Jul 29 19:15:02.651572 ip-10-0-128-101 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 29 19:15:02.651582 ip-10-0-128-101 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 29 19:15:02.651592 ip-10-0-128-101 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 29 19:15:02.651891 ip-10-0-128-101 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 29 19:15:12.780798 ip-10-0-128-101 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 29 19:15:12.780821 ip-10-0-128-101 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 338650f9dbf64f2ebcef440e5d1baf12 --
Jul 29 19:17:11.289484 ip-10-0-128-101 systemd[1]: Starting Kubernetes Kubelet...
Jul 29 19:17:11.700190 ip-10-0-128-101 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 19:17:11.700190 ip-10-0-128-101 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 29 19:17:11.700190 ip-10-0-128-101 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 19:17:11.700190 ip-10-0-128-101 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 29 19:17:11.700190 ip-10-0-128-101 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 19:17:11.701142 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.701000    2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 29 19:17:11.704775 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704755    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 19:17:11.704775 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704774    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 19:17:11.704843 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704781    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 19:17:11.704843 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704785    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 19:17:11.704843 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704789    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 19:17:11.704843 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704791    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 19:17:11.704843 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704799    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 19:17:11.704843 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704802    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 19:17:11.704843 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704805    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 19:17:11.704843 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704808    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 19:17:11.704843 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704811    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 19:17:11.704843 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704814    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 19:17:11.704843 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704816    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 19:17:11.704843 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704819    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 19:17:11.704843 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704821    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 19:17:11.704843 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704824    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 19:17:11.704843 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704826    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 19:17:11.704843 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704829    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 19:17:11.704843 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704832    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 19:17:11.704843 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704834    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 19:17:11.704843 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704837    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 19:17:11.704843 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704839    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 19:17:11.705375 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704842    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 19:17:11.705375 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704845    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 19:17:11.705375 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704848    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 19:17:11.705375 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704851    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 19:17:11.705375 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704854    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 19:17:11.705375 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704857    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 19:17:11.705375 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704860    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 19:17:11.705375 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704862    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 19:17:11.705375 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704865    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 19:17:11.705375 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704868    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 19:17:11.705375 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704870    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 19:17:11.705375 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704873    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 19:17:11.705375 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704875    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 19:17:11.705375 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704878    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 19:17:11.705375 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704881    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 19:17:11.705375 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704883    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 19:17:11.705375 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704886    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 19:17:11.705375 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704890    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 19:17:11.705375 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704893    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 19:17:11.705375 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704897    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 19:17:11.705881 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704900    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 19:17:11.705881 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704903    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 19:17:11.705881 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704905    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 19:17:11.705881 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704908    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 19:17:11.705881 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704910    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 19:17:11.705881 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704912    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 19:17:11.705881 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704915    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 19:17:11.705881 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704917    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 19:17:11.705881 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704920    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 19:17:11.705881 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704922    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 19:17:11.705881 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704925    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 19:17:11.705881 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704927    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 19:17:11.705881 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704932    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 19:17:11.705881 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704936    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 19:17:11.705881 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704939    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 19:17:11.705881 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704942    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 19:17:11.705881 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704945    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 19:17:11.705881 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704947    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 19:17:11.705881 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704950    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 19:17:11.706359 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704953    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 19:17:11.706359 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704956    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 19:17:11.706359 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704958    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 19:17:11.706359 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704961    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 19:17:11.706359 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704963    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 19:17:11.706359 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704966    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 19:17:11.706359 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704968    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 19:17:11.706359 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704971    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 19:17:11.706359 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704974    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 19:17:11.706359 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704976    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 19:17:11.706359 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704981    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 19:17:11.706359 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704984    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 19:17:11.706359 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704987    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 19:17:11.706359 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704990    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 19:17:11.706359 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704993    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 19:17:11.706359 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704996    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 19:17:11.706359 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.704998    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 19:17:11.706359 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705001    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 19:17:11.706359 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705004    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 19:17:11.706359 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705006    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 19:17:11.706847 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705009    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 19:17:11.706847 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705011    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 19:17:11.706847 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705014    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 19:17:11.706847 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705017    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 19:17:11.706847 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705020    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 19:17:11.706847 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705022    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 19:17:11.706847 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705025    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 19:17:11.706847 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705027    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 19:17:11.706847 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705030    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 19:17:11.706847 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705032    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 19:17:11.706847 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705035    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 19:17:11.706847 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705038    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 19:17:11.706847 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705040    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 19:17:11.706847 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705042    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 19:17:11.706847 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705045    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 19:17:11.706847 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705048    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 19:17:11.706847 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705050    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 19:17:11.706847 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705053    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 19:17:11.706847 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705829    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 19:17:11.706847 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705836    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 19:17:11.707353 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705839    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 19:17:11.707353 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705842    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 19:17:11.707353 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705845    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 19:17:11.707353 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705849    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 19:17:11.707353 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705851    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 19:17:11.707353 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705854    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 19:17:11.707353 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705857    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 19:17:11.707353 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705860    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 19:17:11.707353 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705863    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 19:17:11.707353 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705865    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 19:17:11.707353 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705868    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 19:17:11.707353 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705871    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 19:17:11.707353 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705873    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 19:17:11.707353 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705876    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 19:17:11.707353 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705878    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 19:17:11.707353 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705881    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 19:17:11.707353 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705884    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 19:17:11.707353 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705887    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 19:17:11.707353 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705890    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 19:17:11.707353 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705892    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 19:17:11.707883 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705895    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 19:17:11.707883 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705898    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 19:17:11.707883 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705901    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 19:17:11.707883 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705903    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 19:17:11.707883 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705906    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 19:17:11.707883 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705908    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 19:17:11.707883 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705911    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 19:17:11.707883 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705913    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 19:17:11.707883 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705916    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 19:17:11.707883 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705918    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 19:17:11.707883 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705921    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 19:17:11.707883 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705923    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 19:17:11.707883 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705926    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 19:17:11.707883 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705928    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 19:17:11.707883 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705931    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 19:17:11.707883 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705933    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 19:17:11.707883 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705936    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 19:17:11.707883 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705939    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 19:17:11.707883 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705942    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 19:17:11.707883 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705945    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 19:17:11.708384 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705947    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 19:17:11.708384 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705950    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 19:17:11.708384 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705952    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 19:17:11.708384 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705955    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 19:17:11.708384 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705957    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 19:17:11.708384 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705960    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 19:17:11.708384 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705962    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 19:17:11.708384 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705965    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 19:17:11.708384 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705967    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 19:17:11.708384 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705970    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 19:17:11.708384 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705973    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 19:17:11.708384 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705975    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 19:17:11.708384 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705978    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 19:17:11.708384 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705981    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 19:17:11.708384 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705983    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 19:17:11.708384 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705986    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 19:17:11.708384 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705988    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 19:17:11.708384 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705990    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 19:17:11.708384 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705993    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 19:17:11.708384 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705996    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 19:17:11.708883 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.705998    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 19:17:11.708883 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706001    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 19:17:11.708883 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706004    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 19:17:11.708883 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706008    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 19:17:11.708883 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706011    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 19:17:11.708883 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706014    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 19:17:11.708883 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706016    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 19:17:11.708883 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706018    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 19:17:11.708883 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706021    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 19:17:11.708883 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706024    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 19:17:11.708883 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706026    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 19:17:11.708883 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706029    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 19:17:11.708883 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706032    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 19:17:11.708883 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706034    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 19:17:11.708883 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706037    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 19:17:11.708883 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706039    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 19:17:11.708883 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706041    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 19:17:11.708883 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706044    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 19:17:11.708883 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706046    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 19:17:11.708883 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706049    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 19:17:11.709389 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706052    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 19:17:11.709389 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706056    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 19:17:11.709389 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706059    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 19:17:11.709389 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706061    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 19:17:11.709389 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706064    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 19:17:11.709389 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706066    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 19:17:11.709389 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706069    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 19:17:11.709389 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706071    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 19:17:11.709389 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706074    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 19:17:11.709389 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706076    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 19:17:11.709389 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706079    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 19:17:11.709389 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706081    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 19:17:11.709389 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706085    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 19:17:11.709389 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706088    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 19:17:11.709389 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706092    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 19:17:11.709389 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706096    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 19:17:11.709389 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706099    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 19:17:11.709389 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706512    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 19:17:11.709389 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706517    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 19:17:11.709869 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706520    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 19:17:11.709869 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706523    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 19:17:11.709869 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706526    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 19:17:11.709869 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706531    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 19:17:11.709869 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706534    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 19:17:11.709869 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706537    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 19:17:11.709869 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706539    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 19:17:11.709869 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706542    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 19:17:11.709869 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706545    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 19:17:11.709869 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706547    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 19:17:11.709869 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706550    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 19:17:11.709869 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706553    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 19:17:11.709869 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706555    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 19:17:11.709869 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706558    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 19:17:11.709869 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706561    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 19:17:11.709869 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706563    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 19:17:11.709869 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706566    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 19:17:11.709869 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706569    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 19:17:11.709869 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706572    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 19:17:11.709869 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706574    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 19:17:11.710375 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706577    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 19:17:11.710375 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706579    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 19:17:11.710375 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706582    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 19:17:11.710375 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706585    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 19:17:11.710375 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706587    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 19:17:11.710375 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706590    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 19:17:11.710375 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706593    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 19:17:11.710375 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706595    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 19:17:11.710375 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706598    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 19:17:11.710375 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706600    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 19:17:11.710375 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706603    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 19:17:11.710375 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706606    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 19:17:11.710375 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706608    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 19:17:11.710375 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706611    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 19:17:11.710375 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706615    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 19:17:11.710375 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706620    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 19:17:11.710375 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706622    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 19:17:11.710375 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706625    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 19:17:11.710375 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706628    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 19:17:11.710841 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706630    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 19:17:11.710841 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706633    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 19:17:11.710841 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706635    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 19:17:11.710841 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706638    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 19:17:11.710841 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706640    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 19:17:11.710841 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706643    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 19:17:11.710841 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706646    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 19:17:11.710841 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706648    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 19:17:11.710841 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706651    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 19:17:11.710841 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706654    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 19:17:11.710841 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706657    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 19:17:11.710841 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706659    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 19:17:11.710841 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706662    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 19:17:11.710841 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706665    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 19:17:11.710841 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706668    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 19:17:11.710841 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706670    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 19:17:11.710841 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706673    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 19:17:11.710841 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706675    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 19:17:11.710841 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706678    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 19:17:11.710841 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706680    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 19:17:11.711350 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706683    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 19:17:11.711350 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706685    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 19:17:11.711350 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706688    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 19:17:11.711350 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706690    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 19:17:11.711350 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706692    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 19:17:11.711350 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706695    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 19:17:11.711350 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706698    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 19:17:11.711350 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706700    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 19:17:11.711350 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706704    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 19:17:11.711350 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706707    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 19:17:11.711350 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706710    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 19:17:11.711350 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706712    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 19:17:11.711350 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706714    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 19:17:11.711350 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706717    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 19:17:11.711350 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706720    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 19:17:11.711350 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706723    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 19:17:11.711350 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706725    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 19:17:11.711350 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706728    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 19:17:11.711350 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706731    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 19:17:11.711350 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706734    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 19:17:11.711855 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706736    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 19:17:11.711855 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706739    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 19:17:11.711855 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706741    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 19:17:11.711855 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706744    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 19:17:11.711855 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706748    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 19:17:11.711855 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706752    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 19:17:11.711855 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706755    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 19:17:11.711855 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706758    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 19:17:11.711855 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706761    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 19:17:11.711855 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706764    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 19:17:11.711855 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706767    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 19:17:11.711855 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706770    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 19:17:11.711855 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706772    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 19:17:11.711855 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706775    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 19:17:11.711855 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706777    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 19:17:11.711855 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706780    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 19:17:11.711855 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706782    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 19:17:11.711855 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.706785    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 19:17:11.711855 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.706854    2583 flags.go:64] FLAG: --address="0.0.0.0"
Jul 29 19:17:11.712338 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.706862    2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 29 19:17:11.712338 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.706870    2583 flags.go:64] FLAG: --anonymous-auth="true"
Jul 29 19:17:11.712338 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.706874    2583 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 29 19:17:11.712338 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.706879    2583 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 29 19:17:11.712338 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.706883    2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 29 19:17:11.712338 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.706887    2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 29 19:17:11.712338 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.706892    2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 29 19:17:11.712338 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.706895    2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 29 19:17:11.712338 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.706898    2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 29 19:17:11.712338 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.706902    2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 29 19:17:11.712338 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.706905    2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 29 19:17:11.712338 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.706908    2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 29 19:17:11.712338 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.706912    2583 flags.go:64] FLAG: --cgroup-root=""
Jul 29 19:17:11.712338 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.706915    2583 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 29 19:17:11.712338 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.706918    2583 flags.go:64] FLAG: --client-ca-file=""
Jul 29 19:17:11.712338 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.706921    2583 flags.go:64] FLAG: --cloud-provider="external"
Jul 29 19:17:11.712338 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.706924    2583 flags.go:64] FLAG: --cluster-dns="[]"
Jul 29 19:17:11.712338 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.706929    2583 flags.go:64] FLAG: --cluster-domain=""
Jul 29 19:17:11.712338 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.706932    2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 29 19:17:11.712338 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.706935    2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 29 19:17:11.712338 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.706938    2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 29 19:17:11.712338 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.706942    2583 flags.go:64] FLAG: --container-log-max-files="5"
Jul 29 19:17:11.712338 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.706946    2583 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 29 19:17:11.712338 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.706949    2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 29 19:17:11.712913 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.706952    2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 29 19:17:11.712913 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.706955    2583 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 29 19:17:11.712913 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.706958    2583 flags.go:64] FLAG: --contention-profiling="false"
Jul 29 19:17:11.712913 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.706961    2583 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 29 19:17:11.712913 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.706964    2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 29 19:17:11.712913 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.706968    2583 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 29 19:17:11.712913 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.706971    2583 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 29 19:17:11.712913 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.706975    2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 29 19:17:11.712913 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.706978    2583 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 29 19:17:11.712913 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.706982    2583 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 29 19:17:11.712913 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.706985    2583 flags.go:64] FLAG: --enable-load-reader="false"
Jul 29 19:17:11.712913 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.706989    2583 flags.go:64] FLAG: --enable-server="true"
Jul 29 19:17:11.712913 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.706993    2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 29 19:17:11.712913 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.706997    2583 flags.go:64] FLAG: --event-burst="100"
Jul 29 19:17:11.712913 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707000    2583 flags.go:64] FLAG: --event-qps="50"
Jul 29 19:17:11.712913 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707003    2583 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 29 19:17:11.712913 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707006    2583 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 29 19:17:11.712913 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707009    2583 flags.go:64] FLAG: --eviction-hard=""
Jul 29 19:17:11.712913 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707013    2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 29 19:17:11.712913 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707016    2583 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 29 19:17:11.712913 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707019    2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 29 19:17:11.712913 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707023    2583 flags.go:64] FLAG: --eviction-soft=""
Jul 29 19:17:11.712913 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707026    2583 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 29 19:17:11.712913 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707029    2583 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 29 19:17:11.712913 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707032    2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 29 19:17:11.713630 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707035    2583 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 29 19:17:11.713630 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707038    2583 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 29 19:17:11.713630 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707041    2583 flags.go:64] FLAG: --fail-swap-on="true"
Jul 29 19:17:11.713630 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707044    2583 flags.go:64] FLAG: --feature-gates=""
Jul 29 19:17:11.713630 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707048    2583 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 29 19:17:11.713630 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707051    2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 29 19:17:11.713630 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707054    2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 29 19:17:11.713630 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707057    2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 29 19:17:11.713630 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707060    2583 flags.go:64] FLAG: --healthz-port="10248"
Jul 29 19:17:11.713630 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707063    2583 flags.go:64] FLAG: --help="false"
Jul 29 19:17:11.713630 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707066    2583 flags.go:64] FLAG: --hostname-override="ip-10-0-128-101.ec2.internal"
Jul 29 19:17:11.713630 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707069    2583 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 29 19:17:11.713630 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707072    2583 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 29 19:17:11.713630 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707075    2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 29 19:17:11.713630 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707078    2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 29 19:17:11.713630 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707082    2583 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 29 19:17:11.713630 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707086    2583 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 29 19:17:11.713630 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707089    2583 flags.go:64] FLAG: --image-service-endpoint=""
Jul 29 19:17:11.713630 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707092    2583 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 29 19:17:11.713630 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707095    2583 flags.go:64] FLAG: --kube-api-burst="100"
Jul 29 19:17:11.713630 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707099    2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 29 19:17:11.713630 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707103    2583 flags.go:64] FLAG: --kube-api-qps="50"
Jul 29 19:17:11.713630 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707105    2583 flags.go:64] FLAG: --kube-reserved=""
Jul 29 19:17:11.713630 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707108    2583 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 29 19:17:11.714241 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707111    2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 29 19:17:11.714241 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707114    2583 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 29 19:17:11.714241 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707130    2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 29 19:17:11.714241 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707134    2583 flags.go:64] FLAG: --lock-file=""
Jul 29 19:17:11.714241 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707137    2583 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 29 19:17:11.714241 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707140    2583 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 29 19:17:11.714241 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707143    2583 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 29 19:17:11.714241 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707149    2583 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 29 19:17:11.714241 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707152    2583 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 29 19:17:11.714241 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707155    2583 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 29 19:17:11.714241 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707158    2583 flags.go:64] FLAG: --logging-format="text"
Jul 29 19:17:11.714241 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707161    2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 29 19:17:11.714241 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707164    2583 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 29 19:17:11.714241 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707167    2583 flags.go:64] FLAG: --manifest-url=""
Jul 29 19:17:11.714241 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707170    2583 flags.go:64] FLAG: --manifest-url-header=""
Jul 29 19:17:11.714241 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707174    2583 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 29 19:17:11.714241 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707178    2583 flags.go:64] FLAG: --max-open-files="1000000"
Jul 29 19:17:11.714241 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707183    2583 flags.go:64] FLAG: --max-pods="110"
Jul 29 19:17:11.714241 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707186    2583 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 29 19:17:11.714241 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707189    2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 29 19:17:11.714241 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707192    2583 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 29 19:17:11.714241 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707195    2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 29 19:17:11.714241 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707198    2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 29 19:17:11.714241 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707201    2583 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 29 19:17:11.714241 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707206    2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 29 19:17:11.714855 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707213    2583 flags.go:64] FLAG: --node-status-max-images="50"
Jul 29 19:17:11.714855 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707216    2583 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 29 19:17:11.714855 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707219    2583 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 29 19:17:11.714855 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707223    2583 flags.go:64] FLAG: --pod-cidr=""
Jul 29 19:17:11.714855 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707232    2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 29 19:17:11.714855 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707238    2583 flags.go:64] FLAG: --pod-manifest-path=""
Jul 29 19:17:11.714855 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707241    2583 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 29 19:17:11.714855 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707244    2583 flags.go:64] FLAG: --pods-per-core="0"
Jul 29 19:17:11.714855 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707247    2583 flags.go:64] FLAG: --port="10250"
Jul 29 19:17:11.714855 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707251    2583 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 29 19:17:11.714855 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707254    2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09a544a78b4634248"
Jul 29 19:17:11.714855 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707257    2583 flags.go:64] FLAG: --qos-reserved=""
Jul 29 19:17:11.714855 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707260    2583 flags.go:64] FLAG: --read-only-port="10255"
Jul 29 19:17:11.714855 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707263    2583 flags.go:64] FLAG: --register-node="true"
Jul 29 19:17:11.714855 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707266    2583 flags.go:64] FLAG: --register-with-taints=""
Jul 29 19:17:11.714855 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707275    2583 flags.go:64] FLAG: --registry-burst="10"
Jul 29 19:17:11.714855 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707278    2583 flags.go:64] FLAG: --registry-qps="5"
Jul 29 19:17:11.714855 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707280    2583 flags.go:64] FLAG: --reserved-cpus=""
Jul 29 19:17:11.714855 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707284    2583 flags.go:64] FLAG: --reserved-memory=""
Jul 29 19:17:11.714855 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707287    2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 29 19:17:11.714855 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707290    2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 29 19:17:11.714855 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707293    2583 flags.go:64] FLAG: --rotate-certificates="false"
Jul 29 19:17:11.714855 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707296    2583 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 29 19:17:11.714855 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707299    2583 flags.go:64] FLAG: --runonce="false"
Jul 29 19:17:11.714855 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707302    2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 29 19:17:11.715528 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707305    2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 29 19:17:11.715528 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707308    2583 flags.go:64] FLAG: --seccomp-default="false"
Jul 29 19:17:11.715528 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707311    2583 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 29 19:17:11.715528 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707314    2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 29 19:17:11.715528 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707317    2583 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 29 19:17:11.715528 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707321    2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 29 19:17:11.715528 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707324    2583 flags.go:64] FLAG: --storage-driver-password="root"
Jul 29 19:17:11.715528 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707329    2583 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 29 19:17:11.715528 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707332    2583 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 29 19:17:11.715528 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707334    2583 flags.go:64] FLAG: --storage-driver-user="root"
Jul 29 19:17:11.715528 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707338    2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 29 19:17:11.715528 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707341    2583 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 29 19:17:11.715528 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707345    2583 flags.go:64] FLAG: --system-cgroups=""
Jul 29 19:17:11.715528 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707348    2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 29 19:17:11.715528 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707353    2583 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 29 19:17:11.715528 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707357    2583 flags.go:64] FLAG: --tls-cert-file=""
Jul 29 19:17:11.715528 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707361    2583 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 29 19:17:11.715528 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707366    2583 flags.go:64] FLAG: --tls-min-version=""
Jul 29 19:17:11.715528 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707369    2583 flags.go:64] FLAG: --tls-private-key-file=""
Jul 29 19:17:11.715528 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707372    2583 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 29 19:17:11.715528 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707376    2583 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 29 19:17:11.715528 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707379    2583 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 29 19:17:11.715528 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707381    2583 flags.go:64] FLAG: --v="2"
Jul 29 19:17:11.715528 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707385    2583 flags.go:64] FLAG: --version="false"
Jul 29 19:17:11.715528 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707389    2583 flags.go:64] FLAG: --vmodule=""
Jul 29 19:17:11.716153 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707393    2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 29 19:17:11.716153 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707397    2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 29 19:17:11.716153 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707515    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 19:17:11.716153 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707519    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 19:17:11.716153 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707524    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 19:17:11.716153 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707528    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 19:17:11.716153 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707533    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 19:17:11.716153 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707536    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 19:17:11.716153 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707540    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 19:17:11.716153 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707543    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 19:17:11.716153 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707546    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 19:17:11.716153 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707549    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 19:17:11.716153 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707551    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 19:17:11.716153 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707554    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 19:17:11.716153 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707559    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 19:17:11.716153 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707562    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 19:17:11.716153 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707565    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 19:17:11.716153 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707568    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 19:17:11.716153 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707570    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 19:17:11.716623 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707573    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 19:17:11.716623 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707576    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 19:17:11.716623 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707579    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 19:17:11.716623 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707582    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 19:17:11.716623 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707586    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 19:17:11.716623 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707589    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 19:17:11.716623 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707591    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 19:17:11.716623 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707594    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 19:17:11.716623 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707597    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 19:17:11.716623 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707599    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 19:17:11.716623 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707602    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 19:17:11.716623 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707605    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 19:17:11.716623 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707607    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 19:17:11.716623 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707610    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 19:17:11.716623 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707612    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 19:17:11.716623 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707615    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 19:17:11.716623 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707618    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 19:17:11.716623 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707620    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 19:17:11.716623 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707623    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 19:17:11.716623 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707626    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 19:17:11.717154 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707628    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 19:17:11.717154 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707631    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 19:17:11.717154 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707634    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 19:17:11.717154 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707637    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 19:17:11.717154 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707639    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 19:17:11.717154 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707642    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 19:17:11.717154 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707644    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 19:17:11.717154 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707648    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 19:17:11.717154 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707651    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 19:17:11.717154 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707653    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 19:17:11.717154 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707656    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 19:17:11.717154 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707659    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 19:17:11.717154 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707661    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 19:17:11.717154 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707664    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 19:17:11.717154 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707667    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 19:17:11.717154 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707670    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 19:17:11.717154 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707674    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 19:17:11.717154 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707677    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 19:17:11.717154 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707679    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 19:17:11.717631 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707682    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 19:17:11.717631 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707685    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 19:17:11.717631 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707687    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 19:17:11.717631 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707690    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 19:17:11.717631 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707692    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 19:17:11.717631 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707695    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 19:17:11.717631 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707698    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 19:17:11.717631 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707700    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 19:17:11.717631 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707703    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 19:17:11.717631 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707705    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 19:17:11.717631 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707708    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 19:17:11.717631 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707711    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 19:17:11.717631 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707713    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 19:17:11.717631 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707716    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 19:17:11.717631 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707719    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 19:17:11.717631 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707721    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 19:17:11.717631 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707724    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 19:17:11.717631 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707726    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 19:17:11.717631 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707729    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 19:17:11.718094 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707731    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 19:17:11.718094 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707735    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 19:17:11.718094 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707738    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 19:17:11.718094 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707740    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 19:17:11.718094 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707743    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 19:17:11.718094 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707746    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 19:17:11.718094 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707748    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 19:17:11.718094 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707750    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 19:17:11.718094 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707753    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 19:17:11.718094 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707756    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 19:17:11.718094 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707761    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 19:17:11.718094 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707765    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 19:17:11.718094 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707768    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 19:17:11.718094 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707771    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 19:17:11.718094 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707774    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 19:17:11.718094 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707777    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 19:17:11.718094 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707780    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 19:17:11.718094 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707783    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 19:17:11.718094 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707785    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 19:17:11.718094 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707788    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 19:17:11.718595 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707790    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 19:17:11.718595 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707793    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 19:17:11.718595 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707795    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 19:17:11.718595 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.707798    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 19:17:11.718595 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.707804    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 19:17:11.718595 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.715158    2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 29 19:17:11.718595 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.715177    2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 29 19:17:11.718595 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.715195    2583 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 29 19:17:11.718595 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.715201    2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 29 19:17:11.718595 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715266    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 19:17:11.718595 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715271    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 19:17:11.718595 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715274    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 19:17:11.718595 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715277    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 19:17:11.718595 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715280    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 19:17:11.718595 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715284    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 19:17:11.718967 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715287    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 19:17:11.718967 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715289    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 19:17:11.718967 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715292    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 19:17:11.718967 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715295    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 19:17:11.718967 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715298    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 19:17:11.718967 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715301    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 19:17:11.718967 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715303    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 19:17:11.718967 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715306    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 19:17:11.718967 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715309    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 19:17:11.718967 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715312    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 19:17:11.718967 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715315    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 19:17:11.718967 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715318    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 19:17:11.718967 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715320    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 19:17:11.718967 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715323    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 19:17:11.718967 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715325    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 19:17:11.718967 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715328    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 19:17:11.718967 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715331    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 19:17:11.718967 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715333    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 19:17:11.718967 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715336    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 19:17:11.718967 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715339    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 19:17:11.719539 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715341    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 19:17:11.719539 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715343    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 19:17:11.719539 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715346    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 19:17:11.719539 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715355    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 19:17:11.719539 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715363    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 19:17:11.719539 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715366    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 19:17:11.719539 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715368    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 19:17:11.719539 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715371    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 19:17:11.719539 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715374    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 19:17:11.719539 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715376    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 19:17:11.719539 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715379    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 19:17:11.719539 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715382    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 19:17:11.719539 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715384    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 19:17:11.719539 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715387    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 19:17:11.719539 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715389    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 19:17:11.719539 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715392    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 19:17:11.719539 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715395    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 19:17:11.719539 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715397    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 19:17:11.719539 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715400    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 19:17:11.719539 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715404    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 19:17:11.720023 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715408    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 19:17:11.720023 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715410    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 19:17:11.720023 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715413    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 19:17:11.720023 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715416    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 19:17:11.720023 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715419    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 19:17:11.720023 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715421    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 19:17:11.720023 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715424    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 19:17:11.720023 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715426    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 19:17:11.720023 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715429    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 19:17:11.720023 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715435    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 19:17:11.720023 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715438    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 19:17:11.720023 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715440    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 19:17:11.720023 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715443    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 19:17:11.720023 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715445    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 19:17:11.720023 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715448    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 19:17:11.720023 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715451    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 19:17:11.720023 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715459    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 19:17:11.720023 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715462    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 19:17:11.720023 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715464    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 19:17:11.720023 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715467    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 19:17:11.720563 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715470    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 19:17:11.720563 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715472    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 19:17:11.720563 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715475    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 19:17:11.720563 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715477    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 19:17:11.720563 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715480    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 19:17:11.720563 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715482    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 19:17:11.720563 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715485    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 19:17:11.720563 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715488    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 19:17:11.720563 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715490    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 19:17:11.720563 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715493    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 19:17:11.720563 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715495    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 19:17:11.720563 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715498    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 19:17:11.720563 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715502    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 19:17:11.720563 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715506    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 19:17:11.720563 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715508    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 19:17:11.720563 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715511    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 19:17:11.720563 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715514    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 19:17:11.720563 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715517    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 19:17:11.720563 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715519    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 19:17:11.721024 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715522    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 19:17:11.721024 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715525    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 19:17:11.721024 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715528    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 19:17:11.721024 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715531    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 19:17:11.721024 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715535    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 19:17:11.721024 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715538    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 19:17:11.721024 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715541    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 19:17:11.721024 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715544    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 19:17:11.721024 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715547    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 19:17:11.721024 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715550    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 19:17:11.721024 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715558    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 19:17:11.721024 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715561    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 19:17:11.721024 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715564    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 19:17:11.721024 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715566    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 19:17:11.721379 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.715571    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 19:17:11.721379 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715718    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 19:17:11.721379 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715726    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 19:17:11.721379 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715729    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 19:17:11.721379 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715732    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 19:17:11.721379 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715735    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 19:17:11.721379 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715738    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 19:17:11.721379 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715741    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 19:17:11.721379 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715743    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 19:17:11.721379 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715746    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 19:17:11.721379 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715749    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 19:17:11.721379 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715751    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 19:17:11.721379 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715754    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 19:17:11.721379 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715756    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 19:17:11.721379 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715759    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 19:17:11.721752 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715761    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 19:17:11.721752 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715764    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 19:17:11.721752 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715766    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 19:17:11.721752 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715769    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 19:17:11.721752 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715771    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 19:17:11.721752 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715774    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 19:17:11.721752 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715776    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 19:17:11.721752 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715779    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 19:17:11.721752 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715782    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 19:17:11.721752 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715784    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 19:17:11.721752 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715787    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 19:17:11.721752 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715789    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 19:17:11.721752 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715792    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 19:17:11.721752 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715794    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 19:17:11.721752 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715803    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 19:17:11.721752 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715805    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 19:17:11.721752 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715808    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 19:17:11.721752 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715810    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 19:17:11.721752 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715813    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 19:17:11.721752 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715817    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 19:17:11.722264 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715821    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 19:17:11.722264 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715823    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 19:17:11.722264 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715826    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 19:17:11.722264 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715829    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 19:17:11.722264 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715832    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 19:17:11.722264 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715835    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 19:17:11.722264 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715838    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 19:17:11.722264 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715841    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 19:17:11.722264 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715843    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 19:17:11.722264 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715846    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 19:17:11.722264 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715848    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 19:17:11.722264 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715851    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 19:17:11.722264 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715853    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 19:17:11.722264 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715856    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 19:17:11.722264 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715859    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 19:17:11.722264 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715861    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 19:17:11.722264 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715864    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 19:17:11.722264 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715866    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 19:17:11.722264 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715869    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 19:17:11.722721 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715871    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 19:17:11.722721 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715876    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 19:17:11.722721 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715878    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 19:17:11.722721 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715881    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 19:17:11.722721 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715883    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 19:17:11.722721 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715886    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 19:17:11.722721 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715889    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 19:17:11.722721 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715891    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 19:17:11.722721 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715901    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 19:17:11.722721 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715903    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 19:17:11.722721 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715906    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 19:17:11.722721 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715910    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 19:17:11.722721 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715913    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 19:17:11.722721 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715916    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 19:17:11.722721 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715918    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 19:17:11.722721 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715921    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 19:17:11.722721 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715924    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 19:17:11.722721 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715927    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 19:17:11.722721 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715930    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 19:17:11.723240 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715933    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 19:17:11.723240 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715937    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 19:17:11.723240 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715940    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 19:17:11.723240 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715942    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 19:17:11.723240 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715945    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 19:17:11.723240 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715948    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 19:17:11.723240 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715950    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 19:17:11.723240 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715952    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 19:17:11.723240 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715955    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 19:17:11.723240 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715958    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 19:17:11.723240 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715960    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 19:17:11.723240 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715963    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 19:17:11.723240 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715965    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 19:17:11.723240 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715968    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 19:17:11.723240 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715972    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 19:17:11.723240 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715974    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 19:17:11.723240 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715977    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 19:17:11.723240 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715979    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 19:17:11.723240 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715982    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 19:17:11.723240 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715984    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 19:17:11.723716 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715987    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 19:17:11.723716 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715989    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 19:17:11.723716 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.715997    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 19:17:11.723716 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.716000    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 19:17:11.723716 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.716002    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 19:17:11.723716 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.716005    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 19:17:11.723716 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:11.716008    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 19:17:11.723716 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.716013    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 19:17:11.723716 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.716157    2583 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 29 19:17:11.723716 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.720028    2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 29 19:17:11.723716 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.721326    2583 server.go:1019] "Starting client certificate rotation"
Jul 29 19:17:11.723716 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.721464    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 19:17:11.723716 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.721499    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 19:17:11.727320 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.727290    2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 19:17:11.730168 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.730147    2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 19:17:11.735986 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.735969    2583 log.go:25] "Validated CRI v1 runtime API"
Jul 29 19:17:11.738014 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.737998    2583 log.go:25] "Validated CRI v1 image API"
Jul 29 19:17:11.740026 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.740010    2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 29 19:17:11.742580 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.742557    2583 fs.go:135] Filesystem UUIDs: map[0473f894-9214-4135-a0db-064d8fcbbf05:/dev/nvme0n1p4 2bfa7320-3be6-4924-a609-495c2df4b055:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 29 19:17:11.742668 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.742578    2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 29 19:17:11.747673 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.747547    2583 manager.go:216] Machine: {Timestamp:2026-07-29 19:17:11.746357154 +0000 UTC m=+0.334425692 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100724 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22fd624377a21ffb097adcb64cda1c SystemUUID:ec22fd62-4377-a21f-fb09-7adcb64cda1c BootID:338650f9-dbf6-4f2e-bcef-440e5d1baf12 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:eb:bd:43:d2:5b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:eb:bd:43:d2:5b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:72:53:4a:b8:ac:22 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 29 19:17:11.747673 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.747660    2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 29 19:17:11.747844 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.747776    2583 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 29 19:17:11.750723 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.750689    2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 29 19:17:11.750904 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.750726    2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-101.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 29 19:17:11.751421 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.751408    2583 topology_manager.go:138] "Creating topology manager with none policy"
Jul 29 19:17:11.751476 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.751425    2583 container_manager_linux.go:309] "Creating device plugin manager"
Jul 29 19:17:11.751476 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.751443    2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 19:17:11.752054 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.752042    2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 19:17:11.752857 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.752844    2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 29 19:17:11.754959 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.754944    2583 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 19:17:11.755028 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.754997    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 19:17:11.755142 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.755114    2583 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 29 19:17:11.755206 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.755196    2583 kubelet.go:486] "Attempting to sync node with API server"
Jul 29 19:17:11.755253 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.755222    2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 29 19:17:11.755253 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.755243    2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 29 19:17:11.755342 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.755257    2583 kubelet.go:398] "Adding apiserver pod source"
Jul 29 19:17:11.755342 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.755289    2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 29 19:17:11.755732 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.755717    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 19:17:11.755800 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.755740    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 19:17:11.758806 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.758703    2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 29 19:17:11.760683 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.760664    2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 29 19:17:11.760763 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.760697    2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 29 19:17:11.762221 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.762208    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 29 19:17:11.762272 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.762234    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 29 19:17:11.762272 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.762243    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 29 19:17:11.762272 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.762249    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 29 19:17:11.762272 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.762258    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 29 19:17:11.762272 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.762264    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 29 19:17:11.762272 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.762272    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 29 19:17:11.762421 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.762277    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 29 19:17:11.762421 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.762284    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 29 19:17:11.762421 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.762290    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 29 19:17:11.762421 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.762299    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 29 19:17:11.762421 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.762304    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 29 19:17:11.763855 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.763840    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 29 19:17:11.763855 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.763853    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 29 19:17:11.764993 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:11.764967    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-101.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 29 19:17:11.765065 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:11.765018    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 29 19:17:11.767218 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.767204    2583 server.go:1268] "Started kubelet"
Jul 29 19:17:11.767344 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.767293    2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 29 19:17:11.767396 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.767351    2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 29 19:17:11.767396 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.767370    2583 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 29 19:17:11.768061 ip-10-0-128-101 systemd[1]: Started Kubernetes Kubelet.
Jul 29 19:17:11.769454 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.769433    2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 29 19:17:11.769543 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.769526    2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-101.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 29 19:17:11.770436 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.770417    2583 server.go:310] "Adding debug handlers to kubelet server"
Jul 29 19:17:11.774212 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:11.774193    2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 29 19:17:11.777937 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.777923    2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 29 19:17:11.778003 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.777933    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 29 19:17:11.778071 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:11.778027    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-101.ec2.internal\" not found"
Jul 29 19:17:11.778791 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.778776    2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 29 19:17:11.778899 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.778779    2583 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 29 19:17:11.778899 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.778852    2583 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 29 19:17:11.779044 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.778961    2583 reconstruct.go:97] "Volume reconstruction finished"
Jul 29 19:17:11.779044 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.779037    2583 factory.go:55] Registering systemd factory
Jul 29 19:17:11.779044 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.779043    2583 reconciler.go:29] "Reconciler: start to sync state"
Jul 29 19:17:11.779170 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.779051    2583 factory.go:223] Registration of the systemd container factory successfully
Jul 29 19:17:11.779269 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.779256    2583 factory.go:153] Registering CRI-O factory
Jul 29 19:17:11.779300 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.779271    2583 factory.go:223] Registration of the crio container factory successfully
Jul 29 19:17:11.779327 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.779315    2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 29 19:17:11.779356 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.779334    2583 factory.go:103] Registering Raw factory
Jul 29 19:17:11.779356 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.779344    2583 manager.go:1196] Started watching for new ooms in manager
Jul 29 19:17:11.779959 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:11.779939    2583 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-101.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 29 19:17:11.780178 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:11.780159    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 29 19:17:11.780290 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.780276    2583 manager.go:349] Starting recovery of all containers
Jul 29 19:17:11.780769 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:11.779822    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-101.ec2.internal.18c6d8c2674a946b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-101.ec2.internal,UID:ip-10-0-128-101.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-101.ec2.internal,},FirstTimestamp:2026-07-29 19:17:11.767176299 +0000 UTC m=+0.355244836,LastTimestamp:2026-07-29 19:17:11.767176299 +0000 UTC m=+0.355244836,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-101.ec2.internal,}"
Jul 29 19:17:11.780975 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.780957    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pxkxn"
Jul 29 19:17:11.787835 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.787804    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pxkxn"
Jul 29 19:17:11.791772 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.791747    2583 manager.go:354] Recovery completed
Jul 29 19:17:11.793689 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:11.793666    2583 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 29 19:17:11.796829 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.796815    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 19:17:11.798603 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.798586    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-101.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 19:17:11.798663 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.798617    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-101.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 19:17:11.798663 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.798633    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-101.ec2.internal" event="NodeHasSufficientPID"
Jul 29 19:17:11.799208 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.799191    2583 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 29 19:17:11.799208 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.799208    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 19:17:11.799298 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.799208    2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 29 19:17:11.799332 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.799305    2583 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 19:17:11.799803 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.799790    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-101.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 19:17:11.799845 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.799815    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-101.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 19:17:11.799845 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.799825    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-101.ec2.internal" event="NodeHasSufficientPID"
Jul 29 19:17:11.800923 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.800912    2583 policy_none.go:49] "None policy: Start"
Jul 29 19:17:11.800972 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.800927    2583 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 29 19:17:11.800972 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.800939    2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 29 19:17:11.801886 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.801876    2583 policy_none.go:47] "Start"
Jul 29 19:17:11.839991 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.839781    2583 manager.go:337] "Starting Device Plugin manager"
Jul 29 19:17:11.840144 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:11.840015    2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 29 19:17:11.840144 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.840027    2583 server.go:87] "Starting device plugin registration server"
Jul 29 19:17:11.840315 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.840305    2583 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 29 19:17:11.840349 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.840319    2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 29 19:17:11.840450 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.840435    2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 29 19:17:11.840562 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.840526    2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 29 19:17:11.840562 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.840543    2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 29 19:17:11.841074 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:11.841055    2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 29 19:17:11.841160 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:11.841115    2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-101.ec2.internal\" not found"
Jul 29 19:17:11.941367 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.941338    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 19:17:11.942390 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.942316    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-101.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 19:17:11.942390 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.942347    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-101.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 19:17:11.942390 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.942357    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-101.ec2.internal" event="NodeHasSufficientPID"
Jul 29 19:17:11.942390 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.942386    2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-101.ec2.internal"
Jul 29 19:17:11.952378 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.952318    2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-101.ec2.internal"
Jul 29 19:17:11.952378 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:11.952344    2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-101.ec2.internal\": node \"ip-10-0-128-101.ec2.internal\" not found"
Jul 29 19:17:11.973305 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.973259    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 29 19:17:11.974543 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.974522    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 29 19:17:11.974639 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.974558    2583 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 29 19:17:11.974639 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.974582    2583 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 29 19:17:11.974639 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:11.974615    2583 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 29 19:17:11.975007 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:11.974989    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-101.ec2.internal\" not found"
Jul 29 19:17:11.977162 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:11.977146    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 19:17:12.075211 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:12.075172    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-101.ec2.internal\" not found"
Jul 29 19:17:12.075211 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.075155    2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-101.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-101.ec2.internal"]
Jul 29 19:17:12.075424 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.075290    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 19:17:12.076826 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.076811    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-101.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 19:17:12.076903 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.076844    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-101.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 19:17:12.076903 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.076854    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-101.ec2.internal" event="NodeHasSufficientPID"
Jul 29 19:17:12.078021 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.078007    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 19:17:12.078114 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.078100    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-101.ec2.internal"
Jul 29 19:17:12.078177 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.078142    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 19:17:12.078685 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.078669    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-101.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 19:17:12.078732 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.078699    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-101.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 19:17:12.078732 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.078713    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-101.ec2.internal" event="NodeHasSufficientPID"
Jul 29 19:17:12.078802 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.078755    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-101.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 19:17:12.078802 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.078781    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-101.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 19:17:12.078802 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.078793    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-101.ec2.internal" event="NodeHasSufficientPID"
Jul 29 19:17:12.079904 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.079891    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-101.ec2.internal"
Jul 29 19:17:12.079953 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.079915    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 19:17:12.080423 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.080390    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b19b16543121e99642de34dc778bf29b-config\") pod \"kube-apiserver-proxy-ip-10-0-128-101.ec2.internal\" (UID: \"b19b16543121e99642de34dc778bf29b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-101.ec2.internal"
Jul 29 19:17:12.080525 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.080423    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e3140649e42883a9530def746eafe57a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-101.ec2.internal\" (UID: \"e3140649e42883a9530def746eafe57a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-101.ec2.internal"
Jul 29 19:17:12.080525 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.080446    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3140649e42883a9530def746eafe57a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-101.ec2.internal\" (UID: \"e3140649e42883a9530def746eafe57a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-101.ec2.internal"
Jul 29 19:17:12.080605 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.080541    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-101.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 19:17:12.080605 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.080564    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-101.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 19:17:12.080605 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.080574    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-101.ec2.internal" event="NodeHasSufficientPID"
Jul 29 19:17:12.107998 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:12.107970    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-101.ec2.internal\" not found" node="ip-10-0-128-101.ec2.internal"
Jul 29 19:17:12.112513 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:12.112492    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-101.ec2.internal\" not found" node="ip-10-0-128-101.ec2.internal"
Jul 29 19:17:12.175331 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:12.175298    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-101.ec2.internal\" not found"
Jul 29 19:17:12.181286 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.181255    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e3140649e42883a9530def746eafe57a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-101.ec2.internal\" (UID: \"e3140649e42883a9530def746eafe57a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-101.ec2.internal"
Jul 29 19:17:12.181286 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.181289    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3140649e42883a9530def746eafe57a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-101.ec2.internal\" (UID: \"e3140649e42883a9530def746eafe57a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-101.ec2.internal"
Jul 29 19:17:12.181484 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.181307    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b19b16543121e99642de34dc778bf29b-config\") pod \"kube-apiserver-proxy-ip-10-0-128-101.ec2.internal\" (UID: \"b19b16543121e99642de34dc778bf29b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-101.ec2.internal"
Jul 29 19:17:12.181484 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.181343    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b19b16543121e99642de34dc778bf29b-config\") pod \"kube-apiserver-proxy-ip-10-0-128-101.ec2.internal\" (UID: \"b19b16543121e99642de34dc778bf29b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-101.ec2.internal"
Jul 29 19:17:12.181484 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.181364    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3140649e42883a9530def746eafe57a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-101.ec2.internal\" (UID: \"e3140649e42883a9530def746eafe57a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-101.ec2.internal"
Jul 29 19:17:12.181484 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.181368    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e3140649e42883a9530def746eafe57a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-101.ec2.internal\" (UID: \"e3140649e42883a9530def746eafe57a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-101.ec2.internal"
Jul 29 19:17:12.276198 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:12.276170    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-101.ec2.internal\" not found"
Jul 29 19:17:12.376778 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:12.376728    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-101.ec2.internal\" not found"
Jul 29 19:17:12.410235 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.410211    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-101.ec2.internal"
Jul 29 19:17:12.415349 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.415331    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-101.ec2.internal"
Jul 29 19:17:12.477429 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:12.477400    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-101.ec2.internal\" not found"
Jul 29 19:17:12.577971 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:12.577893    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-101.ec2.internal\" not found"
Jul 29 19:17:12.600207 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.600184    2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 19:17:12.678634 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.678600    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-101.ec2.internal"
Jul 29 19:17:12.690880 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.690861    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 19:17:12.692513 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.692496    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-101.ec2.internal"
Jul 29 19:17:12.702268 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.702249    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 19:17:12.721402 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.721377    2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 29 19:17:12.721561 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.721542    2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 29 19:17:12.721621 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.721548    2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 29 19:17:12.755749 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.755699    2583 apiserver.go:52] "Watching apiserver"
Jul 29 19:17:12.767360 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.767332    2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 29 19:17:12.767709 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.767682    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vw8v","openshift-dns/node-resolver-mlk48","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-101.ec2.internal","openshift-multus/network-metrics-daemon-4wlg9","openshift-network-diagnostics/network-check-target-cmjv9","openshift-cluster-node-tuning-operator/tuned-nz9jf","openshift-image-registry/node-ca-98vz6","openshift-multus/multus-additional-cni-plugins-tpwtr","openshift-multus/multus-gdg8t","openshift-network-operator/iptables-alerter-n6l8q","openshift-ovn-kubernetes/ovnkube-node-jpgfd","kube-system/global-pull-secret-syncer-pp24q","kube-system/konnectivity-agent-m8f8f","kube-system/kube-apiserver-proxy-ip-10-0-128-101.ec2.internal"]
Jul 29 19:17:12.769294 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.769262    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-98vz6"
Jul 29 19:17:12.770442 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.770424    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m8f8f"
Jul 29 19:17:12.771932 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.771902    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-t2h9c\""
Jul 29 19:17:12.772298 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.772174    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 29 19:17:12.772298 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.772219    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 29 19:17:12.772298 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.772313    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 29 19:17:12.773885 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.772479    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-j9lch\""
Jul 29 19:17:12.773885 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.772578    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 29 19:17:12.773885 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.772672    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 29 19:17:12.773885 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.772897    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tpwtr"
Jul 29 19:17:12.774256 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.774187    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wlg9"
Jul 29 19:17:12.774356 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:12.774253    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wlg9" podUID="d9ab35af-cc45-4dc0-ac22-cc434c73ee9e"
Jul 29 19:17:12.774707 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.774686    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 29 19:17:12.774997 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.774984    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 29 19:17:12.775260 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.775244    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 29 19:17:12.775371 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.775353    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cmjv9"
Jul 29 19:17:12.775442 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.775370    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 29 19:17:12.775442 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:12.775417    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cmjv9" podUID="0a4b19f9-f7aa-4860-b947-e21d47ed4b21"
Jul 29 19:17:12.775442 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.775353    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 29 19:17:12.775725 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.775705    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nbnlk\""
Jul 29 19:17:12.776812 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.776795    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nz9jf"
Jul 29 19:17:12.777908 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.777892    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.778078 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.778057    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 29 19:17:12.778609 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.778595    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 29 19:17:12.779142 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.779109    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vw8v"
Jul 29 19:17:12.779221 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.779204    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 29 19:17:12.779278 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.779256    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gvfqr\""
Jul 29 19:17:12.779617 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.779594    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4c5xf\""
Jul 29 19:17:12.779686 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.779630    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 29 19:17:12.780446 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.780430    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mlk48"
Jul 29 19:17:12.780812 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.780795    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 29 19:17:12.780886 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.780849    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mnnpm\""
Jul 29 19:17:12.781116 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.781100    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 29 19:17:12.781305 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.781290    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 29 19:17:12.781898 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.781882    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-n6l8q"
Jul 29 19:17:12.782178 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.782143    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 29 19:17:12.782262 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.782217    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 29 19:17:12.782362 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.782347    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-dnhfq\""
Jul 29 19:17:12.782970 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.782956    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.783693 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.783676    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 29 19:17:12.783786 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.783717    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nlf74\""
Jul 29 19:17:12.783786 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.783717    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 29 19:17:12.783786 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.783719    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 29 19:17:12.784217 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.784203    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp24q"
Jul 29 19:17:12.784271 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:12.784255    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp24q" podUID="2485c438-0e6d-4aae-bab6-8dcda418e188"
Jul 29 19:17:12.784405 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.784384    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1d4c55d2-4264-40c8-9dff-1bcda7ec672d-device-dir\") pod \"aws-ebs-csi-driver-node-6vw8v\" (UID: \"1d4c55d2-4264-40c8-9dff-1bcda7ec672d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vw8v"
Jul 29 19:17:12.784531 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.784409    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zhd7\" (UniqueName: \"kubernetes.io/projected/1d4c55d2-4264-40c8-9dff-1bcda7ec672d-kube-api-access-9zhd7\") pod \"aws-ebs-csi-driver-node-6vw8v\" (UID: \"1d4c55d2-4264-40c8-9dff-1bcda7ec672d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vw8v"
Jul 29 19:17:12.784531 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.784426    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c57e5316-49cb-4992-904a-4b14b8c881e2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tpwtr\" (UID: \"c57e5316-49cb-4992-904a-4b14b8c881e2\") " pod="openshift-multus/multus-additional-cni-plugins-tpwtr"
Jul 29 19:17:12.784531 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.784449    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c57e5316-49cb-4992-904a-4b14b8c881e2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tpwtr\" (UID: \"c57e5316-49cb-4992-904a-4b14b8c881e2\") " pod="openshift-multus/multus-additional-cni-plugins-tpwtr"
Jul 29 19:17:12.784531 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.784468    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p562c\" (UniqueName: \"kubernetes.io/projected/c57e5316-49cb-4992-904a-4b14b8c881e2-kube-api-access-p562c\") pod \"multus-additional-cni-plugins-tpwtr\" (UID: \"c57e5316-49cb-4992-904a-4b14b8c881e2\") " pod="openshift-multus/multus-additional-cni-plugins-tpwtr"
Jul 29 19:17:12.784531 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.784487    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9ab35af-cc45-4dc0-ac22-cc434c73ee9e-metrics-certs\") pod \"network-metrics-daemon-4wlg9\" (UID: \"d9ab35af-cc45-4dc0-ac22-cc434c73ee9e\") " pod="openshift-multus/network-metrics-daemon-4wlg9"
Jul 29 19:17:12.784531 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.784508    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f2d56b2-16f1-460a-b498-1bad91e77dd1-host\") pod \"node-ca-98vz6\" (UID: \"4f2d56b2-16f1-460a-b498-1bad91e77dd1\") " pod="openshift-image-registry/node-ca-98vz6"
Jul 29 19:17:12.784939 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.784570    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/02b321ee-f4cc-4042-b292-926a6165fd9b-cnibin\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.784939 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.784607    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/02b321ee-f4cc-4042-b292-926a6165fd9b-os-release\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.784939 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.784637    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c57e5316-49cb-4992-904a-4b14b8c881e2-system-cni-dir\") pod \"multus-additional-cni-plugins-tpwtr\" (UID: \"c57e5316-49cb-4992-904a-4b14b8c881e2\") " pod="openshift-multus/multus-additional-cni-plugins-tpwtr"
Jul 29 19:17:12.784939 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.784669    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c57e5316-49cb-4992-904a-4b14b8c881e2-os-release\") pod \"multus-additional-cni-plugins-tpwtr\" (UID: \"c57e5316-49cb-4992-904a-4b14b8c881e2\") " pod="openshift-multus/multus-additional-cni-plugins-tpwtr"
Jul 29 19:17:12.784939 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.784710    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55857bde-1ffc-488e-9c18-dac2e7736f26-var-lib-kubelet\") pod \"tuned-nz9jf\" (UID: \"55857bde-1ffc-488e-9c18-dac2e7736f26\") " pod="openshift-cluster-node-tuning-operator/tuned-nz9jf"
Jul 29 19:17:12.784939 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.784737    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/55857bde-1ffc-488e-9c18-dac2e7736f26-host\") pod \"tuned-nz9jf\" (UID: \"55857bde-1ffc-488e-9c18-dac2e7736f26\") " pod="openshift-cluster-node-tuning-operator/tuned-nz9jf"
Jul 29 19:17:12.784939 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.784769    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/55857bde-1ffc-488e-9c18-dac2e7736f26-etc-tuned\") pod \"tuned-nz9jf\" (UID: \"55857bde-1ffc-488e-9c18-dac2e7736f26\") " pod="openshift-cluster-node-tuning-operator/tuned-nz9jf"
Jul 29 19:17:12.784939 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.784784    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 29 19:17:12.784939 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.784793    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1d4c55d2-4264-40c8-9dff-1bcda7ec672d-socket-dir\") pod \"aws-ebs-csi-driver-node-6vw8v\" (UID: \"1d4c55d2-4264-40c8-9dff-1bcda7ec672d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vw8v"
Jul 29 19:17:12.784939 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.784817    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1d4c55d2-4264-40c8-9dff-1bcda7ec672d-registration-dir\") pod \"aws-ebs-csi-driver-node-6vw8v\" (UID: \"1d4c55d2-4264-40c8-9dff-1bcda7ec672d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vw8v"
Jul 29 19:17:12.784939 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.784850    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02b321ee-f4cc-4042-b292-926a6165fd9b-multus-cni-dir\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.784939 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.784872    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/02b321ee-f4cc-4042-b292-926a6165fd9b-multus-socket-dir-parent\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.784939 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.784902    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/02b321ee-f4cc-4042-b292-926a6165fd9b-host-run-k8s-cni-cncf-io\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.786092 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.784945    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/02b321ee-f4cc-4042-b292-926a6165fd9b-host-var-lib-cni-multus\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.786092 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.784973    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/02b321ee-f4cc-4042-b292-926a6165fd9b-host-run-multus-certs\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.786092 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.784996    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/02b321ee-f4cc-4042-b292-926a6165fd9b-host-run-netns\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.786092 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.785003    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 29 19:17:12.786092 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.785033    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/02b321ee-f4cc-4042-b292-926a6165fd9b-hostroot\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.786092 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.785060    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsc2t\" (UniqueName: \"kubernetes.io/projected/02b321ee-f4cc-4042-b292-926a6165fd9b-kube-api-access-zsc2t\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.786092 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.785081    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1d4c55d2-4264-40c8-9dff-1bcda7ec672d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6vw8v\" (UID: \"1d4c55d2-4264-40c8-9dff-1bcda7ec672d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vw8v"
Jul 29 19:17:12.786092 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.785115    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-g6hf5\""
Jul 29 19:17:12.786092 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.785114    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1d4c55d2-4264-40c8-9dff-1bcda7ec672d-sys-fs\") pod \"aws-ebs-csi-driver-node-6vw8v\" (UID: \"1d4c55d2-4264-40c8-9dff-1bcda7ec672d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vw8v"
Jul 29 19:17:12.786092 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.785161    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c57e5316-49cb-4992-904a-4b14b8c881e2-cni-binary-copy\") pod \"multus-additional-cni-plugins-tpwtr\" (UID: \"c57e5316-49cb-4992-904a-4b14b8c881e2\") " pod="openshift-multus/multus-additional-cni-plugins-tpwtr"
Jul 29 19:17:12.786092 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.785183    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8vf2\" (UniqueName: \"kubernetes.io/projected/d9ab35af-cc45-4dc0-ac22-cc434c73ee9e-kube-api-access-m8vf2\") pod \"network-metrics-daemon-4wlg9\" (UID: \"d9ab35af-cc45-4dc0-ac22-cc434c73ee9e\") " pod="openshift-multus/network-metrics-daemon-4wlg9"
Jul 29 19:17:12.786092 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.785205    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/55857bde-1ffc-488e-9c18-dac2e7736f26-etc-sysctl-d\") pod \"tuned-nz9jf\" (UID: \"55857bde-1ffc-488e-9c18-dac2e7736f26\") " pod="openshift-cluster-node-tuning-operator/tuned-nz9jf"
Jul 29 19:17:12.786092 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.785216    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 29 19:17:12.786092 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.785246    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfpkf\" (UniqueName: \"kubernetes.io/projected/55857bde-1ffc-488e-9c18-dac2e7736f26-kube-api-access-qfpkf\") pod \"tuned-nz9jf\" (UID: \"55857bde-1ffc-488e-9c18-dac2e7736f26\") " pod="openshift-cluster-node-tuning-operator/tuned-nz9jf"
Jul 29 19:17:12.786092 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.785289    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4f2d56b2-16f1-460a-b498-1bad91e77dd1-serviceca\") pod \"node-ca-98vz6\" (UID: \"4f2d56b2-16f1-460a-b498-1bad91e77dd1\") " pod="openshift-image-registry/node-ca-98vz6"
Jul 29 19:17:12.786092 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.785312    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 29 19:17:12.786092 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.785311    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1d4c55d2-4264-40c8-9dff-1bcda7ec672d-etc-selinux\") pod \"aws-ebs-csi-driver-node-6vw8v\" (UID: \"1d4c55d2-4264-40c8-9dff-1bcda7ec672d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vw8v"
Jul 29 19:17:12.786092 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.785415    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55857bde-1ffc-488e-9c18-dac2e7736f26-tmp\") pod \"tuned-nz9jf\" (UID: \"55857bde-1ffc-488e-9c18-dac2e7736f26\") " pod="openshift-cluster-node-tuning-operator/tuned-nz9jf"
Jul 29 19:17:12.786092 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.785440    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02b321ee-f4cc-4042-b292-926a6165fd9b-host-var-lib-kubelet\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.787166 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.785464    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/55857bde-1ffc-488e-9c18-dac2e7736f26-etc-systemd\") pod \"tuned-nz9jf\" (UID: \"55857bde-1ffc-488e-9c18-dac2e7736f26\") " pod="openshift-cluster-node-tuning-operator/tuned-nz9jf"
Jul 29 19:17:12.787166 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.785500    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/55857bde-1ffc-488e-9c18-dac2e7736f26-sys\") pod \"tuned-nz9jf\" (UID: \"55857bde-1ffc-488e-9c18-dac2e7736f26\") " pod="openshift-cluster-node-tuning-operator/tuned-nz9jf"
Jul 29 19:17:12.787166 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.785529    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/02b321ee-f4cc-4042-b292-926a6165fd9b-multus-daemon-config\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.787166 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.785569    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 29 19:17:12.787166 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.785551    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02b321ee-f4cc-4042-b292-926a6165fd9b-etc-kubernetes\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.787166 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.785607    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55857bde-1ffc-488e-9c18-dac2e7736f26-etc-kubernetes\") pod \"tuned-nz9jf\" (UID: \"55857bde-1ffc-488e-9c18-dac2e7736f26\") " pod="openshift-cluster-node-tuning-operator/tuned-nz9jf"
Jul 29 19:17:12.787166 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.785629    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/efd6982e-9638-4562-9ff3-9aab71caae71-agent-certs\") pod \"konnectivity-agent-m8f8f\" (UID: \"efd6982e-9638-4562-9ff3-9aab71caae71\") " pod="kube-system/konnectivity-agent-m8f8f"
Jul 29 19:17:12.787166 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.785670    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c57e5316-49cb-4992-904a-4b14b8c881e2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tpwtr\" (UID: \"c57e5316-49cb-4992-904a-4b14b8c881e2\") " pod="openshift-multus/multus-additional-cni-plugins-tpwtr"
Jul 29 19:17:12.787166 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.785693    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/55857bde-1ffc-488e-9c18-dac2e7736f26-etc-sysconfig\") pod \"tuned-nz9jf\" (UID: \"55857bde-1ffc-488e-9c18-dac2e7736f26\") " pod="openshift-cluster-node-tuning-operator/tuned-nz9jf"
Jul 29 19:17:12.787166 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.785712    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/55857bde-1ffc-488e-9c18-dac2e7736f26-lib-modules\") pod \"tuned-nz9jf\" (UID: \"55857bde-1ffc-488e-9c18-dac2e7736f26\") " pod="openshift-cluster-node-tuning-operator/tuned-nz9jf"
Jul 29 19:17:12.787166 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.785736    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 29 19:17:12.787166 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.785754    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkb6j\" (UniqueName: \"kubernetes.io/projected/4f2d56b2-16f1-460a-b498-1bad91e77dd1-kube-api-access-tkb6j\") pod \"node-ca-98vz6\" (UID: \"4f2d56b2-16f1-460a-b498-1bad91e77dd1\") " pod="openshift-image-registry/node-ca-98vz6"
Jul 29 19:17:12.787166 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.785782    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/02b321ee-f4cc-4042-b292-926a6165fd9b-host-var-lib-cni-bin\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.787166 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.785806    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/55857bde-1ffc-488e-9c18-dac2e7736f26-etc-modprobe-d\") pod \"tuned-nz9jf\" (UID: \"55857bde-1ffc-488e-9c18-dac2e7736f26\") " pod="openshift-cluster-node-tuning-operator/tuned-nz9jf"
Jul 29 19:17:12.787166 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.785827    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/55857bde-1ffc-488e-9c18-dac2e7736f26-etc-sysctl-conf\") pod \"tuned-nz9jf\" (UID: \"55857bde-1ffc-488e-9c18-dac2e7736f26\") " pod="openshift-cluster-node-tuning-operator/tuned-nz9jf"
Jul 29 19:17:12.787166 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.785848    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/55857bde-1ffc-488e-9c18-dac2e7736f26-run\") pod \"tuned-nz9jf\" (UID: \"55857bde-1ffc-488e-9c18-dac2e7736f26\") " pod="openshift-cluster-node-tuning-operator/tuned-nz9jf"
Jul 29 19:17:12.787166 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.785869    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/02b321ee-f4cc-4042-b292-926a6165fd9b-cni-binary-copy\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.787166 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.785892    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/02b321ee-f4cc-4042-b292-926a6165fd9b-multus-conf-dir\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.787656 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.785915    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/efd6982e-9638-4562-9ff3-9aab71caae71-konnectivity-ca\") pod \"konnectivity-agent-m8f8f\" (UID: \"efd6982e-9638-4562-9ff3-9aab71caae71\") " pod="kube-system/konnectivity-agent-m8f8f"
Jul 29 19:17:12.787656 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.785938    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c57e5316-49cb-4992-904a-4b14b8c881e2-cnibin\") pod \"multus-additional-cni-plugins-tpwtr\" (UID: \"c57e5316-49cb-4992-904a-4b14b8c881e2\") " pod="openshift-multus/multus-additional-cni-plugins-tpwtr"
Jul 29 19:17:12.787656 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.786004    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwjg2\" (UniqueName: \"kubernetes.io/projected/0a4b19f9-f7aa-4860-b947-e21d47ed4b21-kube-api-access-gwjg2\") pod \"network-check-target-cmjv9\" (UID: \"0a4b19f9-f7aa-4860-b947-e21d47ed4b21\") " pod="openshift-network-diagnostics/network-check-target-cmjv9"
Jul 29 19:17:12.787656 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.786032    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02b321ee-f4cc-4042-b292-926a6165fd9b-system-cni-dir\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.791072 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.791054    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 19:17:12.792092 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.792062    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 19:12:13 +0000 UTC" deadline="2028-03-06 02:19:55.41870292 +0000 UTC"
Jul 29 19:17:12.792171 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.792093    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14047h2m42.62661404s"
Jul 29 19:17:12.815864 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.815844    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2ffph"
Jul 29 19:17:12.823851 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.823829    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2ffph"
Jul 29 19:17:12.879781 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.879754    2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 29 19:17:12.886840 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.886810    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4f2d56b2-16f1-460a-b498-1bad91e77dd1-serviceca\") pod \"node-ca-98vz6\" (UID: \"4f2d56b2-16f1-460a-b498-1bad91e77dd1\") " pod="openshift-image-registry/node-ca-98vz6"
Jul 29 19:17:12.886979 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.886848    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1d4c55d2-4264-40c8-9dff-1bcda7ec672d-etc-selinux\") pod \"aws-ebs-csi-driver-node-6vw8v\" (UID: \"1d4c55d2-4264-40c8-9dff-1bcda7ec672d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vw8v"
Jul 29 19:17:12.886979 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.886869    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55857bde-1ffc-488e-9c18-dac2e7736f26-tmp\") pod \"tuned-nz9jf\" (UID: \"55857bde-1ffc-488e-9c18-dac2e7736f26\") " pod="openshift-cluster-node-tuning-operator/tuned-nz9jf"
Jul 29 19:17:12.886979 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.886892    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02b321ee-f4cc-4042-b292-926a6165fd9b-host-var-lib-kubelet\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.886979 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.886917    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-env-overrides\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.886979 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.886939    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/55857bde-1ffc-488e-9c18-dac2e7736f26-etc-systemd\") pod \"tuned-nz9jf\" (UID: \"55857bde-1ffc-488e-9c18-dac2e7736f26\") " pod="openshift-cluster-node-tuning-operator/tuned-nz9jf"
Jul 29 19:17:12.886979 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.886961    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/55857bde-1ffc-488e-9c18-dac2e7736f26-sys\") pod \"tuned-nz9jf\" (UID: \"55857bde-1ffc-488e-9c18-dac2e7736f26\") " pod="openshift-cluster-node-tuning-operator/tuned-nz9jf"
Jul 29 19:17:12.887288 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.886983    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/02b321ee-f4cc-4042-b292-926a6165fd9b-multus-daemon-config\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.887288 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.886994    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1d4c55d2-4264-40c8-9dff-1bcda7ec672d-etc-selinux\") pod \"aws-ebs-csi-driver-node-6vw8v\" (UID: \"1d4c55d2-4264-40c8-9dff-1bcda7ec672d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vw8v"
Jul 29 19:17:12.887288 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.887006    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02b321ee-f4cc-4042-b292-926a6165fd9b-etc-kubernetes\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.887288 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.887028    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-ovnkube-config\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.887288 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.887054    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55857bde-1ffc-488e-9c18-dac2e7736f26-etc-kubernetes\") pod \"tuned-nz9jf\" (UID: \"55857bde-1ffc-488e-9c18-dac2e7736f26\") " pod="openshift-cluster-node-tuning-operator/tuned-nz9jf"
Jul 29 19:17:12.887288 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.887078    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/efd6982e-9638-4562-9ff3-9aab71caae71-agent-certs\") pod \"konnectivity-agent-m8f8f\" (UID: \"efd6982e-9638-4562-9ff3-9aab71caae71\") " pod="kube-system/konnectivity-agent-m8f8f"
Jul 29 19:17:12.887288 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.887079    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/55857bde-1ffc-488e-9c18-dac2e7736f26-etc-systemd\") pod \"tuned-nz9jf\" (UID: \"55857bde-1ffc-488e-9c18-dac2e7736f26\") " pod="openshift-cluster-node-tuning-operator/tuned-nz9jf"
Jul 29 19:17:12.887288 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.887103    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-etc-openvswitch\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.887288 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.887157    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-run-ovn\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.887288 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.887182    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-node-log\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.887288 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.887203    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55857bde-1ffc-488e-9c18-dac2e7736f26-etc-kubernetes\") pod \"tuned-nz9jf\" (UID: \"55857bde-1ffc-488e-9c18-dac2e7736f26\") " pod="openshift-cluster-node-tuning-operator/tuned-nz9jf"
Jul 29 19:17:12.887288 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.887209    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c57e5316-49cb-4992-904a-4b14b8c881e2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tpwtr\" (UID: \"c57e5316-49cb-4992-904a-4b14b8c881e2\") " pod="openshift-multus/multus-additional-cni-plugins-tpwtr"
Jul 29 19:17:12.887288 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.887239    2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 29 19:17:12.887288 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.887249    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02b321ee-f4cc-4042-b292-926a6165fd9b-etc-kubernetes\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.887959 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.887296    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/55857bde-1ffc-488e-9c18-dac2e7736f26-etc-sysconfig\") pod \"tuned-nz9jf\" (UID: \"55857bde-1ffc-488e-9c18-dac2e7736f26\") " pod="openshift-cluster-node-tuning-operator/tuned-nz9jf"
Jul 29 19:17:12.887959 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.887348    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4f2d56b2-16f1-460a-b498-1bad91e77dd1-serviceca\") pod \"node-ca-98vz6\" (UID: \"4f2d56b2-16f1-460a-b498-1bad91e77dd1\") " pod="openshift-image-registry/node-ca-98vz6"
Jul 29 19:17:12.887959 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.887354    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/55857bde-1ffc-488e-9c18-dac2e7736f26-lib-modules\") pod \"tuned-nz9jf\" (UID: \"55857bde-1ffc-488e-9c18-dac2e7736f26\") " pod="openshift-cluster-node-tuning-operator/tuned-nz9jf"
Jul 29 19:17:12.887959 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.887425    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkb6j\" (UniqueName: \"kubernetes.io/projected/4f2d56b2-16f1-460a-b498-1bad91e77dd1-kube-api-access-tkb6j\") pod \"node-ca-98vz6\" (UID: \"4f2d56b2-16f1-460a-b498-1bad91e77dd1\") " pod="openshift-image-registry/node-ca-98vz6"
Jul 29 19:17:12.887959 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.887452    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/02b321ee-f4cc-4042-b292-926a6165fd9b-host-var-lib-cni-bin\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.887959 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.887456    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/55857bde-1ffc-488e-9c18-dac2e7736f26-lib-modules\") pod \"tuned-nz9jf\" (UID: \"55857bde-1ffc-488e-9c18-dac2e7736f26\") " pod="openshift-cluster-node-tuning-operator/tuned-nz9jf"
Jul 29 19:17:12.887959 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.887482    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-var-lib-openvswitch\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.887959 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.887508    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.887959 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.887535    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/558f3f9a-7e78-4810-ba73-42063864201e-tmp-dir\") pod \"node-resolver-mlk48\" (UID: \"558f3f9a-7e78-4810-ba73-42063864201e\") " pod="openshift-dns/node-resolver-mlk48"
Jul 29 19:17:12.887959 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.887537    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/55857bde-1ffc-488e-9c18-dac2e7736f26-etc-sysconfig\") pod \"tuned-nz9jf\" (UID: \"55857bde-1ffc-488e-9c18-dac2e7736f26\") " pod="openshift-cluster-node-tuning-operator/tuned-nz9jf"
Jul 29 19:17:12.887959 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.887581    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/55857bde-1ffc-488e-9c18-dac2e7736f26-etc-modprobe-d\") pod \"tuned-nz9jf\" (UID: \"55857bde-1ffc-488e-9c18-dac2e7736f26\") " pod="openshift-cluster-node-tuning-operator/tuned-nz9jf"
Jul 29 19:17:12.887959 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.887607    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02b321ee-f4cc-4042-b292-926a6165fd9b-host-var-lib-kubelet\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.887959 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.887608    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/55857bde-1ffc-488e-9c18-dac2e7736f26-sys\") pod \"tuned-nz9jf\" (UID: \"55857bde-1ffc-488e-9c18-dac2e7736f26\") " pod="openshift-cluster-node-tuning-operator/tuned-nz9jf"
Jul 29 19:17:12.887959 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.887629    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/02b321ee-f4cc-4042-b292-926a6165fd9b-host-var-lib-cni-bin\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.887959 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.887657    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/55857bde-1ffc-488e-9c18-dac2e7736f26-etc-sysctl-conf\") pod \"tuned-nz9jf\" (UID: \"55857bde-1ffc-488e-9c18-dac2e7736f26\") " pod="openshift-cluster-node-tuning-operator/tuned-nz9jf"
Jul 29 19:17:12.887959 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.887682    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/55857bde-1ffc-488e-9c18-dac2e7736f26-run\") pod \"tuned-nz9jf\" (UID: \"55857bde-1ffc-488e-9c18-dac2e7736f26\") " pod="openshift-cluster-node-tuning-operator/tuned-nz9jf"
Jul 29 19:17:12.887959 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.887723    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/02b321ee-f4cc-4042-b292-926a6165fd9b-multus-daemon-config\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.887959 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.887758    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/55857bde-1ffc-488e-9c18-dac2e7736f26-etc-modprobe-d\") pod \"tuned-nz9jf\" (UID: \"55857bde-1ffc-488e-9c18-dac2e7736f26\") " pod="openshift-cluster-node-tuning-operator/tuned-nz9jf"
Jul 29 19:17:12.889300 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.887768    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c57e5316-49cb-4992-904a-4b14b8c881e2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tpwtr\" (UID: \"c57e5316-49cb-4992-904a-4b14b8c881e2\") " pod="openshift-multus/multus-additional-cni-plugins-tpwtr"
Jul 29 19:17:12.889300 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.887797    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/02b321ee-f4cc-4042-b292-926a6165fd9b-cni-binary-copy\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.889300 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.887819    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/02b321ee-f4cc-4042-b292-926a6165fd9b-multus-conf-dir\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.889300 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.887871    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/55857bde-1ffc-488e-9c18-dac2e7736f26-run\") pod \"tuned-nz9jf\" (UID: \"55857bde-1ffc-488e-9c18-dac2e7736f26\") " pod="openshift-cluster-node-tuning-operator/tuned-nz9jf"
Jul 29 19:17:12.889300 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.887891    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/efd6982e-9638-4562-9ff3-9aab71caae71-konnectivity-ca\") pod \"konnectivity-agent-m8f8f\" (UID: \"efd6982e-9638-4562-9ff3-9aab71caae71\") " pod="kube-system/konnectivity-agent-m8f8f"
Jul 29 19:17:12.889300 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.887918    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/55857bde-1ffc-488e-9c18-dac2e7736f26-etc-sysctl-conf\") pod \"tuned-nz9jf\" (UID: \"55857bde-1ffc-488e-9c18-dac2e7736f26\") " pod="openshift-cluster-node-tuning-operator/tuned-nz9jf"
Jul 29 19:17:12.889300 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.887908    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/02b321ee-f4cc-4042-b292-926a6165fd9b-multus-conf-dir\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.889300 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.887936    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-run-systemd\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.889300 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.887962    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6zv2\" (UniqueName: \"kubernetes.io/projected/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-kube-api-access-l6zv2\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.889300 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.887987    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2485c438-0e6d-4aae-bab6-8dcda418e188-kubelet-config\") pod \"global-pull-secret-syncer-pp24q\" (UID: \"2485c438-0e6d-4aae-bab6-8dcda418e188\") " pod="kube-system/global-pull-secret-syncer-pp24q"
Jul 29 19:17:12.889300 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888013    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c57e5316-49cb-4992-904a-4b14b8c881e2-cnibin\") pod \"multus-additional-cni-plugins-tpwtr\" (UID: \"c57e5316-49cb-4992-904a-4b14b8c881e2\") " pod="openshift-multus/multus-additional-cni-plugins-tpwtr"
Jul 29 19:17:12.889300 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888042    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwjg2\" (UniqueName: \"kubernetes.io/projected/0a4b19f9-f7aa-4860-b947-e21d47ed4b21-kube-api-access-gwjg2\") pod \"network-check-target-cmjv9\" (UID: \"0a4b19f9-f7aa-4860-b947-e21d47ed4b21\") " pod="openshift-network-diagnostics/network-check-target-cmjv9"
Jul 29 19:17:12.889300 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888105    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c57e5316-49cb-4992-904a-4b14b8c881e2-cnibin\") pod \"multus-additional-cni-plugins-tpwtr\" (UID: \"c57e5316-49cb-4992-904a-4b14b8c881e2\") " pod="openshift-multus/multus-additional-cni-plugins-tpwtr"
Jul 29 19:17:12.889300 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888142    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02b321ee-f4cc-4042-b292-926a6165fd9b-system-cni-dir\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.889300 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888159    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1d4c55d2-4264-40c8-9dff-1bcda7ec672d-device-dir\") pod \"aws-ebs-csi-driver-node-6vw8v\" (UID: \"1d4c55d2-4264-40c8-9dff-1bcda7ec672d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vw8v"
Jul 29 19:17:12.889300 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888176    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zhd7\" (UniqueName: \"kubernetes.io/projected/1d4c55d2-4264-40c8-9dff-1bcda7ec672d-kube-api-access-9zhd7\") pod \"aws-ebs-csi-driver-node-6vw8v\" (UID: \"1d4c55d2-4264-40c8-9dff-1bcda7ec672d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vw8v"
Jul 29 19:17:12.889300 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888194    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-host-kubelet\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.890211 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888214    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-systemd-units\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.890211 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888231    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1d4c55d2-4264-40c8-9dff-1bcda7ec672d-device-dir\") pod \"aws-ebs-csi-driver-node-6vw8v\" (UID: \"1d4c55d2-4264-40c8-9dff-1bcda7ec672d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vw8v"
Jul 29 19:17:12.890211 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888191    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02b321ee-f4cc-4042-b292-926a6165fd9b-system-cni-dir\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.890211 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888233    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-host-run-ovn-kubernetes\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.890211 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888267    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-ovn-node-metrics-cert\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.890211 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888286    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c57e5316-49cb-4992-904a-4b14b8c881e2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tpwtr\" (UID: \"c57e5316-49cb-4992-904a-4b14b8c881e2\") " pod="openshift-multus/multus-additional-cni-plugins-tpwtr"
Jul 29 19:17:12.890211 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888307    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c57e5316-49cb-4992-904a-4b14b8c881e2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tpwtr\" (UID: \"c57e5316-49cb-4992-904a-4b14b8c881e2\") " pod="openshift-multus/multus-additional-cni-plugins-tpwtr"
Jul 29 19:17:12.890211 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888339    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p562c\" (UniqueName: \"kubernetes.io/projected/c57e5316-49cb-4992-904a-4b14b8c881e2-kube-api-access-p562c\") pod \"multus-additional-cni-plugins-tpwtr\" (UID: \"c57e5316-49cb-4992-904a-4b14b8c881e2\") " pod="openshift-multus/multus-additional-cni-plugins-tpwtr"
Jul 29 19:17:12.890211 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888346    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/efd6982e-9638-4562-9ff3-9aab71caae71-konnectivity-ca\") pod \"konnectivity-agent-m8f8f\" (UID: \"efd6982e-9638-4562-9ff3-9aab71caae71\") " pod="kube-system/konnectivity-agent-m8f8f"
Jul 29 19:17:12.890211 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888376    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-run-openvswitch\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.890211 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888340    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/02b321ee-f4cc-4042-b292-926a6165fd9b-cni-binary-copy\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.890211 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888467    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9ab35af-cc45-4dc0-ac22-cc434c73ee9e-metrics-certs\") pod \"network-metrics-daemon-4wlg9\" (UID: \"d9ab35af-cc45-4dc0-ac22-cc434c73ee9e\") " pod="openshift-multus/network-metrics-daemon-4wlg9"
Jul 29 19:17:12.890211 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888488    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f2d56b2-16f1-460a-b498-1bad91e77dd1-host\") pod \"node-ca-98vz6\" (UID: \"4f2d56b2-16f1-460a-b498-1bad91e77dd1\") " pod="openshift-image-registry/node-ca-98vz6"
Jul 29 19:17:12.890211 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888524    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/02b321ee-f4cc-4042-b292-926a6165fd9b-cnibin\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.890211 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888547    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/02b321ee-f4cc-4042-b292-926a6165fd9b-os-release\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.890211 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888571    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c57e5316-49cb-4992-904a-4b14b8c881e2-system-cni-dir\") pod \"multus-additional-cni-plugins-tpwtr\" (UID: \"c57e5316-49cb-4992-904a-4b14b8c881e2\") " pod="openshift-multus/multus-additional-cni-plugins-tpwtr"
Jul 29 19:17:12.890211 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888589    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f2d56b2-16f1-460a-b498-1bad91e77dd1-host\") pod \"node-ca-98vz6\" (UID: \"4f2d56b2-16f1-460a-b498-1bad91e77dd1\") " pod="openshift-image-registry/node-ca-98vz6"
Jul 29 19:17:12.890960 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888595    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c57e5316-49cb-4992-904a-4b14b8c881e2-os-release\") pod \"multus-additional-cni-plugins-tpwtr\" (UID: \"c57e5316-49cb-4992-904a-4b14b8c881e2\") " pod="openshift-multus/multus-additional-cni-plugins-tpwtr"
Jul 29 19:17:12.890960 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888608    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c57e5316-49cb-4992-904a-4b14b8c881e2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tpwtr\" (UID: \"c57e5316-49cb-4992-904a-4b14b8c881e2\") " pod="openshift-multus/multus-additional-cni-plugins-tpwtr"
Jul 29 19:17:12.890960 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888617    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55857bde-1ffc-488e-9c18-dac2e7736f26-var-lib-kubelet\") pod \"tuned-nz9jf\" (UID: \"55857bde-1ffc-488e-9c18-dac2e7736f26\") " pod="openshift-cluster-node-tuning-operator/tuned-nz9jf"
Jul 29 19:17:12.890960 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888648    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/55857bde-1ffc-488e-9c18-dac2e7736f26-host\") pod \"tuned-nz9jf\" (UID: \"55857bde-1ffc-488e-9c18-dac2e7736f26\") " pod="openshift-cluster-node-tuning-operator/tuned-nz9jf"
Jul 29 19:17:12.890960 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:12.888658    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:17:12.890960 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888665    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55857bde-1ffc-488e-9c18-dac2e7736f26-var-lib-kubelet\") pod \"tuned-nz9jf\" (UID: \"55857bde-1ffc-488e-9c18-dac2e7736f26\") " pod="openshift-cluster-node-tuning-operator/tuned-nz9jf"
Jul 29 19:17:12.890960 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888674    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/55857bde-1ffc-488e-9c18-dac2e7736f26-etc-tuned\") pod \"tuned-nz9jf\" (UID: \"55857bde-1ffc-488e-9c18-dac2e7736f26\") " pod="openshift-cluster-node-tuning-operator/tuned-nz9jf"
Jul 29 19:17:12.890960 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888709    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/02b321ee-f4cc-4042-b292-926a6165fd9b-cnibin\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.890960 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:12.888719    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9ab35af-cc45-4dc0-ac22-cc434c73ee9e-metrics-certs podName:d9ab35af-cc45-4dc0-ac22-cc434c73ee9e nodeName:}" failed. No retries permitted until 2026-07-29 19:17:13.388690714 +0000 UTC m=+1.976759254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9ab35af-cc45-4dc0-ac22-cc434c73ee9e-metrics-certs") pod "network-metrics-daemon-4wlg9" (UID: "d9ab35af-cc45-4dc0-ac22-cc434c73ee9e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:17:12.890960 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888735    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1d4c55d2-4264-40c8-9dff-1bcda7ec672d-socket-dir\") pod \"aws-ebs-csi-driver-node-6vw8v\" (UID: \"1d4c55d2-4264-40c8-9dff-1bcda7ec672d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vw8v"
Jul 29 19:17:12.890960 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888753    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1d4c55d2-4264-40c8-9dff-1bcda7ec672d-registration-dir\") pod \"aws-ebs-csi-driver-node-6vw8v\" (UID: \"1d4c55d2-4264-40c8-9dff-1bcda7ec672d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vw8v"
Jul 29 19:17:12.890960 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888767    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/02b321ee-f4cc-4042-b292-926a6165fd9b-os-release\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.890960 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888775    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-host-run-netns\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.890960 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888794    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-host-slash\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.890960 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888809    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-log-socket\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.890960 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888823    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-host-cni-netd\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.890960 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888825    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c57e5316-49cb-4992-904a-4b14b8c881e2-os-release\") pod \"multus-additional-cni-plugins-tpwtr\" (UID: \"c57e5316-49cb-4992-904a-4b14b8c881e2\") " pod="openshift-multus/multus-additional-cni-plugins-tpwtr"
Jul 29 19:17:12.891463 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888838    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2485c438-0e6d-4aae-bab6-8dcda418e188-dbus\") pod \"global-pull-secret-syncer-pp24q\" (UID: \"2485c438-0e6d-4aae-bab6-8dcda418e188\") " pod="kube-system/global-pull-secret-syncer-pp24q"
Jul 29 19:17:12.891463 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888878    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/55857bde-1ffc-488e-9c18-dac2e7736f26-host\") pod \"tuned-nz9jf\" (UID: \"55857bde-1ffc-488e-9c18-dac2e7736f26\") " pod="openshift-cluster-node-tuning-operator/tuned-nz9jf"
Jul 29 19:17:12.891463 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888907    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/558f3f9a-7e78-4810-ba73-42063864201e-hosts-file\") pod \"node-resolver-mlk48\" (UID: \"558f3f9a-7e78-4810-ba73-42063864201e\") " pod="openshift-dns/node-resolver-mlk48"
Jul 29 19:17:12.891463 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888932    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0504596f-5a2f-4305-a319-ce48cbd6d199-iptables-alerter-script\") pod \"iptables-alerter-n6l8q\" (UID: \"0504596f-5a2f-4305-a319-ce48cbd6d199\") " pod="openshift-network-operator/iptables-alerter-n6l8q"
Jul 29 19:17:12.891463 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888949    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwgm6\" (UniqueName: \"kubernetes.io/projected/0504596f-5a2f-4305-a319-ce48cbd6d199-kube-api-access-bwgm6\") pod \"iptables-alerter-n6l8q\" (UID: \"0504596f-5a2f-4305-a319-ce48cbd6d199\") " pod="openshift-network-operator/iptables-alerter-n6l8q"
Jul 29 19:17:12.891463 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888965    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02b321ee-f4cc-4042-b292-926a6165fd9b-multus-cni-dir\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.891463 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888982    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/02b321ee-f4cc-4042-b292-926a6165fd9b-multus-socket-dir-parent\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.891463 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.888999    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-host-cni-bin\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.891463 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.889016    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w48kw\" (UniqueName: \"kubernetes.io/projected/558f3f9a-7e78-4810-ba73-42063864201e-kube-api-access-w48kw\") pod \"node-resolver-mlk48\" (UID: \"558f3f9a-7e78-4810-ba73-42063864201e\") " pod="openshift-dns/node-resolver-mlk48"
Jul 29 19:17:12.891463 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.889053    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/02b321ee-f4cc-4042-b292-926a6165fd9b-host-run-k8s-cni-cncf-io\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.891463 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.889074    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/02b321ee-f4cc-4042-b292-926a6165fd9b-host-var-lib-cni-multus\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.891463 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.889100    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/02b321ee-f4cc-4042-b292-926a6165fd9b-host-run-multus-certs\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.891463 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.889146    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-ovnkube-script-lib\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.891463 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.889146    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1d4c55d2-4264-40c8-9dff-1bcda7ec672d-socket-dir\") pod \"aws-ebs-csi-driver-node-6vw8v\" (UID: \"1d4c55d2-4264-40c8-9dff-1bcda7ec672d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vw8v"
Jul 29 19:17:12.891463 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.889180    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/02b321ee-f4cc-4042-b292-926a6165fd9b-host-run-netns\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.891463 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.889209    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/02b321ee-f4cc-4042-b292-926a6165fd9b-hostroot\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.891463 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.889237    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsc2t\" (UniqueName: \"kubernetes.io/projected/02b321ee-f4cc-4042-b292-926a6165fd9b-kube-api-access-zsc2t\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.891962 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.889261    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1d4c55d2-4264-40c8-9dff-1bcda7ec672d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6vw8v\" (UID: \"1d4c55d2-4264-40c8-9dff-1bcda7ec672d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vw8v"
Jul 29 19:17:12.891962 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.889287    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1d4c55d2-4264-40c8-9dff-1bcda7ec672d-sys-fs\") pod \"aws-ebs-csi-driver-node-6vw8v\" (UID: \"1d4c55d2-4264-40c8-9dff-1bcda7ec672d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vw8v"
Jul 29 19:17:12.891962 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.889289    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02b321ee-f4cc-4042-b292-926a6165fd9b-multus-cni-dir\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.891962 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.889316    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2485c438-0e6d-4aae-bab6-8dcda418e188-original-pull-secret\") pod \"global-pull-secret-syncer-pp24q\" (UID: \"2485c438-0e6d-4aae-bab6-8dcda418e188\") " pod="kube-system/global-pull-secret-syncer-pp24q"
Jul 29 19:17:12.891962 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.889342    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0504596f-5a2f-4305-a319-ce48cbd6d199-host-slash\") pod \"iptables-alerter-n6l8q\" (UID: \"0504596f-5a2f-4305-a319-ce48cbd6d199\") " pod="openshift-network-operator/iptables-alerter-n6l8q"
Jul 29 19:17:12.891962 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.889346    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/02b321ee-f4cc-4042-b292-926a6165fd9b-multus-socket-dir-parent\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.891962 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.889357    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c57e5316-49cb-4992-904a-4b14b8c881e2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tpwtr\" (UID: \"c57e5316-49cb-4992-904a-4b14b8c881e2\") " pod="openshift-multus/multus-additional-cni-plugins-tpwtr"
Jul 29 19:17:12.891962 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.889375    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c57e5316-49cb-4992-904a-4b14b8c881e2-cni-binary-copy\") pod \"multus-additional-cni-plugins-tpwtr\" (UID: \"c57e5316-49cb-4992-904a-4b14b8c881e2\") " pod="openshift-multus/multus-additional-cni-plugins-tpwtr"
Jul 29 19:17:12.891962 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.889399    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1d4c55d2-4264-40c8-9dff-1bcda7ec672d-registration-dir\") pod \"aws-ebs-csi-driver-node-6vw8v\" (UID: \"1d4c55d2-4264-40c8-9dff-1bcda7ec672d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vw8v"
Jul 29 19:17:12.891962 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.889402    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8vf2\" (UniqueName: \"kubernetes.io/projected/d9ab35af-cc45-4dc0-ac22-cc434c73ee9e-kube-api-access-m8vf2\") pod \"network-metrics-daemon-4wlg9\" (UID: \"d9ab35af-cc45-4dc0-ac22-cc434c73ee9e\") " pod="openshift-multus/network-metrics-daemon-4wlg9"
Jul 29 19:17:12.891962 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.889408    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/02b321ee-f4cc-4042-b292-926a6165fd9b-host-run-netns\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.891962 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.889442    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/55857bde-1ffc-488e-9c18-dac2e7736f26-etc-sysctl-d\") pod \"tuned-nz9jf\" (UID: \"55857bde-1ffc-488e-9c18-dac2e7736f26\") " pod="openshift-cluster-node-tuning-operator/tuned-nz9jf"
Jul 29 19:17:12.891962 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.889468    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c57e5316-49cb-4992-904a-4b14b8c881e2-system-cni-dir\") pod \"multus-additional-cni-plugins-tpwtr\" (UID: \"c57e5316-49cb-4992-904a-4b14b8c881e2\") " pod="openshift-multus/multus-additional-cni-plugins-tpwtr"
Jul 29 19:17:12.891962 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.889487    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfpkf\" (UniqueName: \"kubernetes.io/projected/55857bde-1ffc-488e-9c18-dac2e7736f26-kube-api-access-qfpkf\") pod \"tuned-nz9jf\" (UID: \"55857bde-1ffc-488e-9c18-dac2e7736f26\") " pod="openshift-cluster-node-tuning-operator/tuned-nz9jf"
Jul 29 19:17:12.891962 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.889532    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/02b321ee-f4cc-4042-b292-926a6165fd9b-host-var-lib-cni-multus\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.891962 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.889600    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/02b321ee-f4cc-4042-b292-926a6165fd9b-host-run-k8s-cni-cncf-io\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.891962 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.889631    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/02b321ee-f4cc-4042-b292-926a6165fd9b-host-run-multus-certs\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.892453 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.889654    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/02b321ee-f4cc-4042-b292-926a6165fd9b-hostroot\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.892453 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.889674    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1d4c55d2-4264-40c8-9dff-1bcda7ec672d-sys-fs\") pod \"aws-ebs-csi-driver-node-6vw8v\" (UID: \"1d4c55d2-4264-40c8-9dff-1bcda7ec672d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vw8v"
Jul 29 19:17:12.892453 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.889752    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/55857bde-1ffc-488e-9c18-dac2e7736f26-etc-sysctl-d\") pod \"tuned-nz9jf\" (UID: \"55857bde-1ffc-488e-9c18-dac2e7736f26\") " pod="openshift-cluster-node-tuning-operator/tuned-nz9jf"
Jul 29 19:17:12.892453 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.889791    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1d4c55d2-4264-40c8-9dff-1bcda7ec672d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6vw8v\" (UID: \"1d4c55d2-4264-40c8-9dff-1bcda7ec672d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vw8v"
Jul 29 19:17:12.892453 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.889992    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c57e5316-49cb-4992-904a-4b14b8c881e2-cni-binary-copy\") pod \"multus-additional-cni-plugins-tpwtr\" (UID: \"c57e5316-49cb-4992-904a-4b14b8c881e2\") " pod="openshift-multus/multus-additional-cni-plugins-tpwtr"
Jul 29 19:17:12.892453 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.890796    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55857bde-1ffc-488e-9c18-dac2e7736f26-tmp\") pod \"tuned-nz9jf\" (UID: \"55857bde-1ffc-488e-9c18-dac2e7736f26\") " pod="openshift-cluster-node-tuning-operator/tuned-nz9jf"
Jul 29 19:17:12.892453 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.890900    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/efd6982e-9638-4562-9ff3-9aab71caae71-agent-certs\") pod \"konnectivity-agent-m8f8f\" (UID: \"efd6982e-9638-4562-9ff3-9aab71caae71\") " pod="kube-system/konnectivity-agent-m8f8f"
Jul 29 19:17:12.892453 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.891005    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/55857bde-1ffc-488e-9c18-dac2e7736f26-etc-tuned\") pod \"tuned-nz9jf\" (UID: \"55857bde-1ffc-488e-9c18-dac2e7736f26\") " pod="openshift-cluster-node-tuning-operator/tuned-nz9jf"
Jul 29 19:17:12.897104 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:12.897083    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 19:17:12.897104 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:12.897104    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 19:17:12.897314 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:12.897129    2583 projected.go:196] Error preparing data for projected volume kube-api-access-gwjg2 for pod openshift-network-diagnostics/network-check-target-cmjv9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:17:12.897314 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:12.897193    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a4b19f9-f7aa-4860-b947-e21d47ed4b21-kube-api-access-gwjg2 podName:0a4b19f9-f7aa-4860-b947-e21d47ed4b21 nodeName:}" failed. No retries permitted until 2026-07-29 19:17:13.397179613 +0000 UTC m=+1.985248142 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gwjg2" (UniqueName: "kubernetes.io/projected/0a4b19f9-f7aa-4860-b947-e21d47ed4b21-kube-api-access-gwjg2") pod "network-check-target-cmjv9" (UID: "0a4b19f9-f7aa-4860-b947-e21d47ed4b21") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:17:12.897581 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.897559    2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 19:17:12.897581 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.897573    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkb6j\" (UniqueName: \"kubernetes.io/projected/4f2d56b2-16f1-460a-b498-1bad91e77dd1-kube-api-access-tkb6j\") pod \"node-ca-98vz6\" (UID: \"4f2d56b2-16f1-460a-b498-1bad91e77dd1\") " pod="openshift-image-registry/node-ca-98vz6"
Jul 29 19:17:12.899459 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.899433    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zhd7\" (UniqueName: \"kubernetes.io/projected/1d4c55d2-4264-40c8-9dff-1bcda7ec672d-kube-api-access-9zhd7\") pod \"aws-ebs-csi-driver-node-6vw8v\" (UID: \"1d4c55d2-4264-40c8-9dff-1bcda7ec672d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vw8v"
Jul 29 19:17:12.903545 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.903522    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8vf2\" (UniqueName: \"kubernetes.io/projected/d9ab35af-cc45-4dc0-ac22-cc434c73ee9e-kube-api-access-m8vf2\") pod \"network-metrics-daemon-4wlg9\" (UID: \"d9ab35af-cc45-4dc0-ac22-cc434c73ee9e\") " pod="openshift-multus/network-metrics-daemon-4wlg9"
Jul 29 19:17:12.903927 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.903905    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsc2t\" (UniqueName: \"kubernetes.io/projected/02b321ee-f4cc-4042-b292-926a6165fd9b-kube-api-access-zsc2t\") pod \"multus-gdg8t\" (UID: \"02b321ee-f4cc-4042-b292-926a6165fd9b\") " pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:12.905285 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.905262    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfpkf\" (UniqueName: \"kubernetes.io/projected/55857bde-1ffc-488e-9c18-dac2e7736f26-kube-api-access-qfpkf\") pod \"tuned-nz9jf\" (UID: \"55857bde-1ffc-488e-9c18-dac2e7736f26\") " pod="openshift-cluster-node-tuning-operator/tuned-nz9jf"
Jul 29 19:17:12.905892 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.905875    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p562c\" (UniqueName: \"kubernetes.io/projected/c57e5316-49cb-4992-904a-4b14b8c881e2-kube-api-access-p562c\") pod \"multus-additional-cni-plugins-tpwtr\" (UID: \"c57e5316-49cb-4992-904a-4b14b8c881e2\") " pod="openshift-multus/multus-additional-cni-plugins-tpwtr"
Jul 29 19:17:12.963297 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:12.963262    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3140649e42883a9530def746eafe57a.slice/crio-82b6268c2d1fafbbc6e647ba75a3671470ed685956d907a399b13481c0d2d406 WatchSource:0}: Error finding container 82b6268c2d1fafbbc6e647ba75a3671470ed685956d907a399b13481c0d2d406: Status 404 returned error can't find the container with id 82b6268c2d1fafbbc6e647ba75a3671470ed685956d907a399b13481c0d2d406
Jul 29 19:17:12.963730 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:12.963704    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb19b16543121e99642de34dc778bf29b.slice/crio-85e16835774c35e6974e035c65b0578e84d6932bd2d71910f93dc59d47b48572 WatchSource:0}: Error finding container 85e16835774c35e6974e035c65b0578e84d6932bd2d71910f93dc59d47b48572: Status 404 returned error can't find the container with id 85e16835774c35e6974e035c65b0578e84d6932bd2d71910f93dc59d47b48572
Jul 29 19:17:12.968825 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.968809    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 19:17:12.977652 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.977611    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-101.ec2.internal" event={"ID":"e3140649e42883a9530def746eafe57a","Type":"ContainerStarted","Data":"82b6268c2d1fafbbc6e647ba75a3671470ed685956d907a399b13481c0d2d406"}
Jul 29 19:17:12.978676 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.978656    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-101.ec2.internal" event={"ID":"b19b16543121e99642de34dc778bf29b","Type":"ContainerStarted","Data":"85e16835774c35e6974e035c65b0578e84d6932bd2d71910f93dc59d47b48572"}
Jul 29 19:17:12.990658 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.990632    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-host-run-ovn-kubernetes\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.990812 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.990666    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-ovn-node-metrics-cert\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.990812 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.990689    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-run-openvswitch\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.990812 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.990734    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-host-run-ovn-kubernetes\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.990812 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.990729    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-host-run-netns\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.990812 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.990778    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-run-openvswitch\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.990812 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.990784    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-host-slash\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.990812 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.990802    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-host-run-netns\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.990812 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.990808    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-log-socket\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.991219 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.990826    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-host-slash\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.991219 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.990830    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-host-cni-netd\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.991219 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.990853    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-host-cni-netd\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.991219 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.990860    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2485c438-0e6d-4aae-bab6-8dcda418e188-dbus\") pod \"global-pull-secret-syncer-pp24q\" (UID: \"2485c438-0e6d-4aae-bab6-8dcda418e188\") " pod="kube-system/global-pull-secret-syncer-pp24q"
Jul 29 19:17:12.991219 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.990867    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-log-socket\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.991219 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.990883    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/558f3f9a-7e78-4810-ba73-42063864201e-hosts-file\") pod \"node-resolver-mlk48\" (UID: \"558f3f9a-7e78-4810-ba73-42063864201e\") " pod="openshift-dns/node-resolver-mlk48"
Jul 29 19:17:12.991219 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.990908    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0504596f-5a2f-4305-a319-ce48cbd6d199-iptables-alerter-script\") pod \"iptables-alerter-n6l8q\" (UID: \"0504596f-5a2f-4305-a319-ce48cbd6d199\") " pod="openshift-network-operator/iptables-alerter-n6l8q"
Jul 29 19:17:12.991219 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.990924    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2485c438-0e6d-4aae-bab6-8dcda418e188-dbus\") pod \"global-pull-secret-syncer-pp24q\" (UID: \"2485c438-0e6d-4aae-bab6-8dcda418e188\") " pod="kube-system/global-pull-secret-syncer-pp24q"
Jul 29 19:17:12.991219 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.990933    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwgm6\" (UniqueName: \"kubernetes.io/projected/0504596f-5a2f-4305-a319-ce48cbd6d199-kube-api-access-bwgm6\") pod \"iptables-alerter-n6l8q\" (UID: \"0504596f-5a2f-4305-a319-ce48cbd6d199\") " pod="openshift-network-operator/iptables-alerter-n6l8q"
Jul 29 19:17:12.991219 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.990945    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/558f3f9a-7e78-4810-ba73-42063864201e-hosts-file\") pod \"node-resolver-mlk48\" (UID: \"558f3f9a-7e78-4810-ba73-42063864201e\") " pod="openshift-dns/node-resolver-mlk48"
Jul 29 19:17:12.991219 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.990959    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-host-cni-bin\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.991219 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.990983    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w48kw\" (UniqueName: \"kubernetes.io/projected/558f3f9a-7e78-4810-ba73-42063864201e-kube-api-access-w48kw\") pod \"node-resolver-mlk48\" (UID: \"558f3f9a-7e78-4810-ba73-42063864201e\") " pod="openshift-dns/node-resolver-mlk48"
Jul 29 19:17:12.991219 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.991037    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-host-cni-bin\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.991219 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.991076    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-ovnkube-script-lib\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.991219 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.991103    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2485c438-0e6d-4aae-bab6-8dcda418e188-original-pull-secret\") pod \"global-pull-secret-syncer-pp24q\" (UID: \"2485c438-0e6d-4aae-bab6-8dcda418e188\") " pod="kube-system/global-pull-secret-syncer-pp24q"
Jul 29 19:17:12.991219 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.991133    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0504596f-5a2f-4305-a319-ce48cbd6d199-host-slash\") pod \"iptables-alerter-n6l8q\" (UID: \"0504596f-5a2f-4305-a319-ce48cbd6d199\") " pod="openshift-network-operator/iptables-alerter-n6l8q"
Jul 29 19:17:12.991219 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.991184    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0504596f-5a2f-4305-a319-ce48cbd6d199-host-slash\") pod \"iptables-alerter-n6l8q\" (UID: \"0504596f-5a2f-4305-a319-ce48cbd6d199\") " pod="openshift-network-operator/iptables-alerter-n6l8q"
Jul 29 19:17:12.992073 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.991185    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-env-overrides\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.992073 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.991225    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-ovnkube-config\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.992073 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.991242    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-etc-openvswitch\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.992073 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.991270    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-run-ovn\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.992073 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.991294    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-node-log\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.992073 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.991298    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-etc-openvswitch\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.992073 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.991325    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-var-lib-openvswitch\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.992073 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.991339    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-run-ovn\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.992073 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.991352    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.992073 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.991376    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-node-log\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.992073 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.991378    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/558f3f9a-7e78-4810-ba73-42063864201e-tmp-dir\") pod \"node-resolver-mlk48\" (UID: \"558f3f9a-7e78-4810-ba73-42063864201e\") " pod="openshift-dns/node-resolver-mlk48"
Jul 29 19:17:12.992073 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.991410    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-var-lib-openvswitch\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.992073 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.991417    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-run-systemd\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.992073 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.991443    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6zv2\" (UniqueName: \"kubernetes.io/projected/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-kube-api-access-l6zv2\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.992073 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.991467    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2485c438-0e6d-4aae-bab6-8dcda418e188-kubelet-config\") pod \"global-pull-secret-syncer-pp24q\" (UID: \"2485c438-0e6d-4aae-bab6-8dcda418e188\") " pod="kube-system/global-pull-secret-syncer-pp24q"
Jul 29 19:17:12.992073 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.991505    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-host-kubelet\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.992073 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.991514    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0504596f-5a2f-4305-a319-ce48cbd6d199-iptables-alerter-script\") pod \"iptables-alerter-n6l8q\" (UID: \"0504596f-5a2f-4305-a319-ce48cbd6d199\") " pod="openshift-network-operator/iptables-alerter-n6l8q"
Jul 29 19:17:12.992590 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.991526    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-systemd-units\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.992590 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.991596    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-systemd-units\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.992590 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.991622    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-env-overrides\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.992590 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.991632    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.992590 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.991623    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-ovnkube-script-lib\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.992590 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.991651    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2485c438-0e6d-4aae-bab6-8dcda418e188-kubelet-config\") pod \"global-pull-secret-syncer-pp24q\" (UID: \"2485c438-0e6d-4aae-bab6-8dcda418e188\") " pod="kube-system/global-pull-secret-syncer-pp24q"
Jul 29 19:17:12.992590 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.991659    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-host-kubelet\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.992590 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.991683    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/558f3f9a-7e78-4810-ba73-42063864201e-tmp-dir\") pod \"node-resolver-mlk48\" (UID: \"558f3f9a-7e78-4810-ba73-42063864201e\") " pod="openshift-dns/node-resolver-mlk48"
Jul 29 19:17:12.992590 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:12.991698    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 19:17:12.992590 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:12.991771    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2485c438-0e6d-4aae-bab6-8dcda418e188-original-pull-secret podName:2485c438-0e6d-4aae-bab6-8dcda418e188 nodeName:}" failed. No retries permitted until 2026-07-29 19:17:13.491757892 +0000 UTC m=+2.079826420 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2485c438-0e6d-4aae-bab6-8dcda418e188-original-pull-secret") pod "global-pull-secret-syncer-pp24q" (UID: "2485c438-0e6d-4aae-bab6-8dcda418e188") : object "kube-system"/"original-pull-secret" not registered
Jul 29 19:17:12.992590 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.991833    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-run-systemd\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.992590 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.992067    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-ovnkube-config\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:12.993113 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:12.993096    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-ovn-node-metrics-cert\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:13.001999 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:13.001978    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w48kw\" (UniqueName: \"kubernetes.io/projected/558f3f9a-7e78-4810-ba73-42063864201e-kube-api-access-w48kw\") pod \"node-resolver-mlk48\" (UID: \"558f3f9a-7e78-4810-ba73-42063864201e\") " pod="openshift-dns/node-resolver-mlk48"
Jul 29 19:17:13.002256 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:13.002237    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwgm6\" (UniqueName: \"kubernetes.io/projected/0504596f-5a2f-4305-a319-ce48cbd6d199-kube-api-access-bwgm6\") pod \"iptables-alerter-n6l8q\" (UID: \"0504596f-5a2f-4305-a319-ce48cbd6d199\") " pod="openshift-network-operator/iptables-alerter-n6l8q"
Jul 29 19:17:13.002303 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:13.002241    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6zv2\" (UniqueName: \"kubernetes.io/projected/f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b-kube-api-access-l6zv2\") pod \"ovnkube-node-jpgfd\" (UID: \"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:13.090519 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:13.090445    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-98vz6"
Jul 29 19:17:13.098441 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:13.098415    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f2d56b2_16f1_460a_b498_1bad91e77dd1.slice/crio-033be7f596f6988bf6e03395c3dda36916ca87e8819fb3cc246df18889d2e7b3 WatchSource:0}: Error finding container 033be7f596f6988bf6e03395c3dda36916ca87e8819fb3cc246df18889d2e7b3: Status 404 returned error can't find the container with id 033be7f596f6988bf6e03395c3dda36916ca87e8819fb3cc246df18889d2e7b3
Jul 29 19:17:13.113760 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:13.113738    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m8f8f"
Jul 29 19:17:13.120715 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:13.120692    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tpwtr"
Jul 29 19:17:13.121020 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:13.120995    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefd6982e_9638_4562_9ff3_9aab71caae71.slice/crio-42386823b4db88d5de8642ac92c51e6aa8bf7176790c058971d380d81987d5ae WatchSource:0}: Error finding container 42386823b4db88d5de8642ac92c51e6aa8bf7176790c058971d380d81987d5ae: Status 404 returned error can't find the container with id 42386823b4db88d5de8642ac92c51e6aa8bf7176790c058971d380d81987d5ae
Jul 29 19:17:13.129314 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:13.129291    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc57e5316_49cb_4992_904a_4b14b8c881e2.slice/crio-8992e439c88fb2f6f0562c81b94fa7d9f197c08d988fc0ce6508f288b1890cfe WatchSource:0}: Error finding container 8992e439c88fb2f6f0562c81b94fa7d9f197c08d988fc0ce6508f288b1890cfe: Status 404 returned error can't find the container with id 8992e439c88fb2f6f0562c81b94fa7d9f197c08d988fc0ce6508f288b1890cfe
Jul 29 19:17:13.139815 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:13.139795    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nz9jf"
Jul 29 19:17:13.146731 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:13.146685    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55857bde_1ffc_488e_9c18_dac2e7736f26.slice/crio-d03442dabe14e75c9f829c2afd56dd035a6ce78e1278d30d5f9b855f9629f502 WatchSource:0}: Error finding container d03442dabe14e75c9f829c2afd56dd035a6ce78e1278d30d5f9b855f9629f502: Status 404 returned error can't find the container with id d03442dabe14e75c9f829c2afd56dd035a6ce78e1278d30d5f9b855f9629f502
Jul 29 19:17:13.155603 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:13.155578    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gdg8t"
Jul 29 19:17:13.162784 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:13.162758    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02b321ee_f4cc_4042_b292_926a6165fd9b.slice/crio-bd2e422063c2819e2c5aeb15ed81e69962bb6480b57f9f483a1efcc7d7767acb WatchSource:0}: Error finding container bd2e422063c2819e2c5aeb15ed81e69962bb6480b57f9f483a1efcc7d7767acb: Status 404 returned error can't find the container with id bd2e422063c2819e2c5aeb15ed81e69962bb6480b57f9f483a1efcc7d7767acb
Jul 29 19:17:13.177796 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:13.177777    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vw8v"
Jul 29 19:17:13.185144 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:13.185110    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d4c55d2_4264_40c8_9dff_1bcda7ec672d.slice/crio-af4279a4681163a88d72f7345b929eefbc92f3ffe7888d1602185fbb59192b47 WatchSource:0}: Error finding container af4279a4681163a88d72f7345b929eefbc92f3ffe7888d1602185fbb59192b47: Status 404 returned error can't find the container with id af4279a4681163a88d72f7345b929eefbc92f3ffe7888d1602185fbb59192b47
Jul 29 19:17:13.192868 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:13.192846    2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 19:17:13.197578 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:13.197560    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mlk48"
Jul 29 19:17:13.205759 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:13.205731    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod558f3f9a_7e78_4810_ba73_42063864201e.slice/crio-857402af1fda23be5af1d45024319e3914b323fa134046a132bc3ddea3ff8813 WatchSource:0}: Error finding container 857402af1fda23be5af1d45024319e3914b323fa134046a132bc3ddea3ff8813: Status 404 returned error can't find the container with id 857402af1fda23be5af1d45024319e3914b323fa134046a132bc3ddea3ff8813
Jul 29 19:17:13.214470 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:13.214454    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-n6l8q"
Jul 29 19:17:13.218851 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:13.218834    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:13.221979 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:13.221958    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0504596f_5a2f_4305_a319_ce48cbd6d199.slice/crio-7132fe5505067d3f69932ecbf3f1f4da98ec0bca524e25061c6cff59c65e05a5 WatchSource:0}: Error finding container 7132fe5505067d3f69932ecbf3f1f4da98ec0bca524e25061c6cff59c65e05a5: Status 404 returned error can't find the container with id 7132fe5505067d3f69932ecbf3f1f4da98ec0bca524e25061c6cff59c65e05a5
Jul 29 19:17:13.226521 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:17:13.226500    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf46c5d88_5a0f_4f1c_b71c_d6503ed5ba4b.slice/crio-2ac956517c7c645c1c3d19217f95c4ac053d0393dbaad1465fa903d5d4d7206f WatchSource:0}: Error finding container 2ac956517c7c645c1c3d19217f95c4ac053d0393dbaad1465fa903d5d4d7206f: Status 404 returned error can't find the container with id 2ac956517c7c645c1c3d19217f95c4ac053d0393dbaad1465fa903d5d4d7206f
Jul 29 19:17:13.394578 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:13.394494    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9ab35af-cc45-4dc0-ac22-cc434c73ee9e-metrics-certs\") pod \"network-metrics-daemon-4wlg9\" (UID: \"d9ab35af-cc45-4dc0-ac22-cc434c73ee9e\") " pod="openshift-multus/network-metrics-daemon-4wlg9"
Jul 29 19:17:13.394719 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:13.394647    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:17:13.394719 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:13.394704    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9ab35af-cc45-4dc0-ac22-cc434c73ee9e-metrics-certs podName:d9ab35af-cc45-4dc0-ac22-cc434c73ee9e nodeName:}" failed. No retries permitted until 2026-07-29 19:17:14.394686362 +0000 UTC m=+2.982754896 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9ab35af-cc45-4dc0-ac22-cc434c73ee9e-metrics-certs") pod "network-metrics-daemon-4wlg9" (UID: "d9ab35af-cc45-4dc0-ac22-cc434c73ee9e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:17:13.495493 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:13.495464    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwjg2\" (UniqueName: \"kubernetes.io/projected/0a4b19f9-f7aa-4860-b947-e21d47ed4b21-kube-api-access-gwjg2\") pod \"network-check-target-cmjv9\" (UID: \"0a4b19f9-f7aa-4860-b947-e21d47ed4b21\") " pod="openshift-network-diagnostics/network-check-target-cmjv9"
Jul 29 19:17:13.495648 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:13.495539    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2485c438-0e6d-4aae-bab6-8dcda418e188-original-pull-secret\") pod \"global-pull-secret-syncer-pp24q\" (UID: \"2485c438-0e6d-4aae-bab6-8dcda418e188\") " pod="kube-system/global-pull-secret-syncer-pp24q"
Jul 29 19:17:13.495723 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:13.495651    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 19:17:13.495723 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:13.495667    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 19:17:13.495723 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:13.495698    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 19:17:13.495723 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:13.495712    2583 projected.go:196] Error preparing data for projected volume kube-api-access-gwjg2 for pod openshift-network-diagnostics/network-check-target-cmjv9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:17:13.495977 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:13.495743    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2485c438-0e6d-4aae-bab6-8dcda418e188-original-pull-secret podName:2485c438-0e6d-4aae-bab6-8dcda418e188 nodeName:}" failed. No retries permitted until 2026-07-29 19:17:14.495716416 +0000 UTC m=+3.083784965 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2485c438-0e6d-4aae-bab6-8dcda418e188-original-pull-secret") pod "global-pull-secret-syncer-pp24q" (UID: "2485c438-0e6d-4aae-bab6-8dcda418e188") : object "kube-system"/"original-pull-secret" not registered
Jul 29 19:17:13.495977 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:13.495765    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a4b19f9-f7aa-4860-b947-e21d47ed4b21-kube-api-access-gwjg2 podName:0a4b19f9-f7aa-4860-b947-e21d47ed4b21 nodeName:}" failed. No retries permitted until 2026-07-29 19:17:14.495752207 +0000 UTC m=+3.083820747 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gwjg2" (UniqueName: "kubernetes.io/projected/0a4b19f9-f7aa-4860-b947-e21d47ed4b21-kube-api-access-gwjg2") pod "network-check-target-cmjv9" (UID: "0a4b19f9-f7aa-4860-b947-e21d47ed4b21") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:17:15.185430 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:15.185320    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 19:12:14 +0000 UTC" deadline="2028-01-22 04:19:54.277353265 +0000 UTC"
Jul 29 19:17:15.185430 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:15.185381    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12993h2m39.091976234s"
Jul 29 19:17:15.283167 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:15.283136    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 19:17:15.337622 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:15.337587    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cmjv9"
Jul 29 19:17:15.337822 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:15.337715    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cmjv9" podUID="0a4b19f9-f7aa-4860-b947-e21d47ed4b21"
Jul 29 19:17:15.384104 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:15.384021    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gdg8t" event={"ID":"02b321ee-f4cc-4042-b292-926a6165fd9b","Type":"ContainerStarted","Data":"bd2e422063c2819e2c5aeb15ed81e69962bb6480b57f9f483a1efcc7d7767acb"}
Jul 29 19:17:15.407114 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:15.407005    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tpwtr" event={"ID":"c57e5316-49cb-4992-904a-4b14b8c881e2","Type":"ContainerStarted","Data":"8992e439c88fb2f6f0562c81b94fa7d9f197c08d988fc0ce6508f288b1890cfe"}
Jul 29 19:17:15.423120 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:15.423081    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd" event={"ID":"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b","Type":"ContainerStarted","Data":"2ac956517c7c645c1c3d19217f95c4ac053d0393dbaad1465fa903d5d4d7206f"}
Jul 29 19:17:15.443693 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:15.443512    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nz9jf" event={"ID":"55857bde-1ffc-488e-9c18-dac2e7736f26","Type":"ContainerStarted","Data":"d03442dabe14e75c9f829c2afd56dd035a6ce78e1278d30d5f9b855f9629f502"}
Jul 29 19:17:15.446636 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:15.446604    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m8f8f" event={"ID":"efd6982e-9638-4562-9ff3-9aab71caae71","Type":"ContainerStarted","Data":"42386823b4db88d5de8642ac92c51e6aa8bf7176790c058971d380d81987d5ae"}
Jul 29 19:17:15.448343 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:15.448318    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-98vz6" event={"ID":"4f2d56b2-16f1-460a-b498-1bad91e77dd1","Type":"ContainerStarted","Data":"033be7f596f6988bf6e03395c3dda36916ca87e8819fb3cc246df18889d2e7b3"}
Jul 29 19:17:15.451097 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:15.451069    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-n6l8q" event={"ID":"0504596f-5a2f-4305-a319-ce48cbd6d199","Type":"ContainerStarted","Data":"7132fe5505067d3f69932ecbf3f1f4da98ec0bca524e25061c6cff59c65e05a5"}
Jul 29 19:17:15.469291 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:15.469202    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mlk48" event={"ID":"558f3f9a-7e78-4810-ba73-42063864201e","Type":"ContainerStarted","Data":"857402af1fda23be5af1d45024319e3914b323fa134046a132bc3ddea3ff8813"}
Jul 29 19:17:15.484021 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:15.483975    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vw8v" event={"ID":"1d4c55d2-4264-40c8-9dff-1bcda7ec672d","Type":"ContainerStarted","Data":"af4279a4681163a88d72f7345b929eefbc92f3ffe7888d1602185fbb59192b47"}
Jul 29 19:17:15.763440 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:15.763265    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9ab35af-cc45-4dc0-ac22-cc434c73ee9e-metrics-certs\") pod \"network-metrics-daemon-4wlg9\" (UID: \"d9ab35af-cc45-4dc0-ac22-cc434c73ee9e\") " pod="openshift-multus/network-metrics-daemon-4wlg9"
Jul 29 19:17:15.763593 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:15.763468    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:17:15.763593 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:15.763542    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9ab35af-cc45-4dc0-ac22-cc434c73ee9e-metrics-certs podName:d9ab35af-cc45-4dc0-ac22-cc434c73ee9e nodeName:}" failed. No retries permitted until 2026-07-29 19:17:17.763521829 +0000 UTC m=+4.991910945 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9ab35af-cc45-4dc0-ac22-cc434c73ee9e-metrics-certs") pod "network-metrics-daemon-4wlg9" (UID: "d9ab35af-cc45-4dc0-ac22-cc434c73ee9e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:17:15.864609 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:15.864198    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2485c438-0e6d-4aae-bab6-8dcda418e188-original-pull-secret\") pod \"global-pull-secret-syncer-pp24q\" (UID: \"2485c438-0e6d-4aae-bab6-8dcda418e188\") " pod="kube-system/global-pull-secret-syncer-pp24q"
Jul 29 19:17:15.864609 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:15.864271    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwjg2\" (UniqueName: \"kubernetes.io/projected/0a4b19f9-f7aa-4860-b947-e21d47ed4b21-kube-api-access-gwjg2\") pod \"network-check-target-cmjv9\" (UID: \"0a4b19f9-f7aa-4860-b947-e21d47ed4b21\") " pod="openshift-network-diagnostics/network-check-target-cmjv9"
Jul 29 19:17:15.864609 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:15.864424    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 19:17:15.864609 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:15.864447    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 19:17:15.864609 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:15.864459    2583 projected.go:196] Error preparing data for projected volume kube-api-access-gwjg2 for pod openshift-network-diagnostics/network-check-target-cmjv9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:17:15.864609 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:15.864518    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a4b19f9-f7aa-4860-b947-e21d47ed4b21-kube-api-access-gwjg2 podName:0a4b19f9-f7aa-4860-b947-e21d47ed4b21 nodeName:}" failed. No retries permitted until 2026-07-29 19:17:17.86449926 +0000 UTC m=+5.092888367 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gwjg2" (UniqueName: "kubernetes.io/projected/0a4b19f9-f7aa-4860-b947-e21d47ed4b21-kube-api-access-gwjg2") pod "network-check-target-cmjv9" (UID: "0a4b19f9-f7aa-4860-b947-e21d47ed4b21") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:17:15.865127 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:15.864935    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 19:17:15.865127 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:15.864994    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2485c438-0e6d-4aae-bab6-8dcda418e188-original-pull-secret podName:2485c438-0e6d-4aae-bab6-8dcda418e188 nodeName:}" failed. No retries permitted until 2026-07-29 19:17:17.86497992 +0000 UTC m=+5.093369028 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2485c438-0e6d-4aae-bab6-8dcda418e188-original-pull-secret") pod "global-pull-secret-syncer-pp24q" (UID: "2485c438-0e6d-4aae-bab6-8dcda418e188") : object "kube-system"/"original-pull-secret" not registered
Jul 29 19:17:16.186004 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:16.185961    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 19:12:14 +0000 UTC" deadline="2028-01-29 03:16:12.231256585 +0000 UTC"
Jul 29 19:17:16.186004 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:16.186002    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13159h58m56.045259503s"
Jul 29 19:17:16.330618 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:16.330420    2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 19:17:16.334511 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:16.334484    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wlg9"
Jul 29 19:17:16.334648 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:16.334622    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wlg9" podUID="d9ab35af-cc45-4dc0-ac22-cc434c73ee9e"
Jul 29 19:17:16.335041 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:16.335010    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp24q"
Jul 29 19:17:16.335145 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:16.335108    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp24q" podUID="2485c438-0e6d-4aae-bab6-8dcda418e188"
Jul 29 19:17:17.335070 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:17.335037    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cmjv9"
Jul 29 19:17:17.335766 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:17.335171    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cmjv9" podUID="0a4b19f9-f7aa-4860-b947-e21d47ed4b21"
Jul 29 19:17:17.790174 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:17.790124    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9ab35af-cc45-4dc0-ac22-cc434c73ee9e-metrics-certs\") pod \"network-metrics-daemon-4wlg9\" (UID: \"d9ab35af-cc45-4dc0-ac22-cc434c73ee9e\") " pod="openshift-multus/network-metrics-daemon-4wlg9"
Jul 29 19:17:17.790361 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:17.790275    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:17:17.790361 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:17.790341    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9ab35af-cc45-4dc0-ac22-cc434c73ee9e-metrics-certs podName:d9ab35af-cc45-4dc0-ac22-cc434c73ee9e nodeName:}" failed. No retries permitted until 2026-07-29 19:17:21.790322153 +0000 UTC m=+9.018711263 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9ab35af-cc45-4dc0-ac22-cc434c73ee9e-metrics-certs") pod "network-metrics-daemon-4wlg9" (UID: "d9ab35af-cc45-4dc0-ac22-cc434c73ee9e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:17:17.890948 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:17.890887    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2485c438-0e6d-4aae-bab6-8dcda418e188-original-pull-secret\") pod \"global-pull-secret-syncer-pp24q\" (UID: \"2485c438-0e6d-4aae-bab6-8dcda418e188\") " pod="kube-system/global-pull-secret-syncer-pp24q"
Jul 29 19:17:17.891144 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:17.890984    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwjg2\" (UniqueName: \"kubernetes.io/projected/0a4b19f9-f7aa-4860-b947-e21d47ed4b21-kube-api-access-gwjg2\") pod \"network-check-target-cmjv9\" (UID: \"0a4b19f9-f7aa-4860-b947-e21d47ed4b21\") " pod="openshift-network-diagnostics/network-check-target-cmjv9"
Jul 29 19:17:17.891144 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:17.891137    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 19:17:17.891259 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:17.891156    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 19:17:17.891259 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:17.891171    2583 projected.go:196] Error preparing data for projected volume kube-api-access-gwjg2 for pod openshift-network-diagnostics/network-check-target-cmjv9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:17:17.891259 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:17.891230    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a4b19f9-f7aa-4860-b947-e21d47ed4b21-kube-api-access-gwjg2 podName:0a4b19f9-f7aa-4860-b947-e21d47ed4b21 nodeName:}" failed. No retries permitted until 2026-07-29 19:17:21.89121261 +0000 UTC m=+9.119601731 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gwjg2" (UniqueName: "kubernetes.io/projected/0a4b19f9-f7aa-4860-b947-e21d47ed4b21-kube-api-access-gwjg2") pod "network-check-target-cmjv9" (UID: "0a4b19f9-f7aa-4860-b947-e21d47ed4b21") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:17:17.891645 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:17.891625    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 19:17:17.891745 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:17.891682    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2485c438-0e6d-4aae-bab6-8dcda418e188-original-pull-secret podName:2485c438-0e6d-4aae-bab6-8dcda418e188 nodeName:}" failed. No retries permitted until 2026-07-29 19:17:21.891667088 +0000 UTC m=+9.120056197 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2485c438-0e6d-4aae-bab6-8dcda418e188-original-pull-secret") pod "global-pull-secret-syncer-pp24q" (UID: "2485c438-0e6d-4aae-bab6-8dcda418e188") : object "kube-system"/"original-pull-secret" not registered
Jul 29 19:17:18.335395 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:18.335293    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wlg9"
Jul 29 19:17:18.335395 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:18.335315    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp24q"
Jul 29 19:17:18.335936 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:18.335442    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp24q" podUID="2485c438-0e6d-4aae-bab6-8dcda418e188"
Jul 29 19:17:18.335936 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:18.335566    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wlg9" podUID="d9ab35af-cc45-4dc0-ac22-cc434c73ee9e"
Jul 29 19:17:19.343516 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:19.343483    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cmjv9"
Jul 29 19:17:19.344009 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:19.343618    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cmjv9" podUID="0a4b19f9-f7aa-4860-b947-e21d47ed4b21"
Jul 29 19:17:20.335393 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:20.335265    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp24q"
Jul 29 19:17:20.335587 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:20.335416    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp24q" podUID="2485c438-0e6d-4aae-bab6-8dcda418e188"
Jul 29 19:17:20.335990 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:20.335837    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wlg9"
Jul 29 19:17:20.335990 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:20.335947    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wlg9" podUID="d9ab35af-cc45-4dc0-ac22-cc434c73ee9e"
Jul 29 19:17:21.336012 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:21.335494    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cmjv9"
Jul 29 19:17:21.336012 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:21.335627    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cmjv9" podUID="0a4b19f9-f7aa-4860-b947-e21d47ed4b21"
Jul 29 19:17:21.823220 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:21.822624    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9ab35af-cc45-4dc0-ac22-cc434c73ee9e-metrics-certs\") pod \"network-metrics-daemon-4wlg9\" (UID: \"d9ab35af-cc45-4dc0-ac22-cc434c73ee9e\") " pod="openshift-multus/network-metrics-daemon-4wlg9"
Jul 29 19:17:21.823220 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:21.822794    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:17:21.823220 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:21.822874    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9ab35af-cc45-4dc0-ac22-cc434c73ee9e-metrics-certs podName:d9ab35af-cc45-4dc0-ac22-cc434c73ee9e nodeName:}" failed. No retries permitted until 2026-07-29 19:17:29.822854292 +0000 UTC m=+17.051243399 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9ab35af-cc45-4dc0-ac22-cc434c73ee9e-metrics-certs") pod "network-metrics-daemon-4wlg9" (UID: "d9ab35af-cc45-4dc0-ac22-cc434c73ee9e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:17:21.923606 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:21.923258    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2485c438-0e6d-4aae-bab6-8dcda418e188-original-pull-secret\") pod \"global-pull-secret-syncer-pp24q\" (UID: \"2485c438-0e6d-4aae-bab6-8dcda418e188\") " pod="kube-system/global-pull-secret-syncer-pp24q"
Jul 29 19:17:21.923606 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:21.923337    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwjg2\" (UniqueName: \"kubernetes.io/projected/0a4b19f9-f7aa-4860-b947-e21d47ed4b21-kube-api-access-gwjg2\") pod \"network-check-target-cmjv9\" (UID: \"0a4b19f9-f7aa-4860-b947-e21d47ed4b21\") " pod="openshift-network-diagnostics/network-check-target-cmjv9"
Jul 29 19:17:21.923606 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:21.923460    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 19:17:21.923606 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:21.923499    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 19:17:21.923606 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:21.923515    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 19:17:21.923606 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:21.923528    2583 projected.go:196] Error preparing data for projected volume kube-api-access-gwjg2 for pod openshift-network-diagnostics/network-check-target-cmjv9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:17:21.923606 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:21.923535    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2485c438-0e6d-4aae-bab6-8dcda418e188-original-pull-secret podName:2485c438-0e6d-4aae-bab6-8dcda418e188 nodeName:}" failed. No retries permitted until 2026-07-29 19:17:29.923518419 +0000 UTC m=+17.151907523 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2485c438-0e6d-4aae-bab6-8dcda418e188-original-pull-secret") pod "global-pull-secret-syncer-pp24q" (UID: "2485c438-0e6d-4aae-bab6-8dcda418e188") : object "kube-system"/"original-pull-secret" not registered
Jul 29 19:17:21.923606 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:21.923577    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a4b19f9-f7aa-4860-b947-e21d47ed4b21-kube-api-access-gwjg2 podName:0a4b19f9-f7aa-4860-b947-e21d47ed4b21 nodeName:}" failed. No retries permitted until 2026-07-29 19:17:29.923561066 +0000 UTC m=+17.151950209 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gwjg2" (UniqueName: "kubernetes.io/projected/0a4b19f9-f7aa-4860-b947-e21d47ed4b21-kube-api-access-gwjg2") pod "network-check-target-cmjv9" (UID: "0a4b19f9-f7aa-4860-b947-e21d47ed4b21") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:17:22.335677 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:22.335234    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp24q"
Jul 29 19:17:22.335677 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:22.335280    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wlg9"
Jul 29 19:17:22.335677 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:22.335415    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp24q" podUID="2485c438-0e6d-4aae-bab6-8dcda418e188"
Jul 29 19:17:22.335677 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:22.335559    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wlg9" podUID="d9ab35af-cc45-4dc0-ac22-cc434c73ee9e"
Jul 29 19:17:23.336878 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:23.336381    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cmjv9"
Jul 29 19:17:23.336878 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:23.336489    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cmjv9" podUID="0a4b19f9-f7aa-4860-b947-e21d47ed4b21"
Jul 29 19:17:24.334796 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:24.334711    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wlg9"
Jul 29 19:17:24.334971 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:24.334855    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wlg9" podUID="d9ab35af-cc45-4dc0-ac22-cc434c73ee9e"
Jul 29 19:17:24.334971 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:24.334909    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp24q"
Jul 29 19:17:24.335090 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:24.335022    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp24q" podUID="2485c438-0e6d-4aae-bab6-8dcda418e188"
Jul 29 19:17:25.335445 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:25.335410    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cmjv9"
Jul 29 19:17:25.335911 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:25.335533    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cmjv9" podUID="0a4b19f9-f7aa-4860-b947-e21d47ed4b21"
Jul 29 19:17:26.335250 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:26.335213    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wlg9"
Jul 29 19:17:26.335417 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:26.335253    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp24q"
Jul 29 19:17:26.335417 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:26.335366    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wlg9" podUID="d9ab35af-cc45-4dc0-ac22-cc434c73ee9e"
Jul 29 19:17:26.335819 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:26.335488    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp24q" podUID="2485c438-0e6d-4aae-bab6-8dcda418e188"
Jul 29 19:17:27.334814 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:27.334779    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cmjv9"
Jul 29 19:17:27.334976 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:27.334924    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cmjv9" podUID="0a4b19f9-f7aa-4860-b947-e21d47ed4b21"
Jul 29 19:17:28.334605 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:28.334572    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wlg9"
Jul 29 19:17:28.335022 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:28.334680    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wlg9" podUID="d9ab35af-cc45-4dc0-ac22-cc434c73ee9e"
Jul 29 19:17:28.335022 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:28.334708    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp24q"
Jul 29 19:17:28.335022 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:28.334770    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp24q" podUID="2485c438-0e6d-4aae-bab6-8dcda418e188"
Jul 29 19:17:29.334960 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:29.334928    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cmjv9"
Jul 29 19:17:29.335362 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:29.335046    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cmjv9" podUID="0a4b19f9-f7aa-4860-b947-e21d47ed4b21"
Jul 29 19:17:29.887999 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:29.887821    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9ab35af-cc45-4dc0-ac22-cc434c73ee9e-metrics-certs\") pod \"network-metrics-daemon-4wlg9\" (UID: \"d9ab35af-cc45-4dc0-ac22-cc434c73ee9e\") " pod="openshift-multus/network-metrics-daemon-4wlg9"
Jul 29 19:17:29.888154 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:29.887954    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:17:29.888154 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:29.888110    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9ab35af-cc45-4dc0-ac22-cc434c73ee9e-metrics-certs podName:d9ab35af-cc45-4dc0-ac22-cc434c73ee9e nodeName:}" failed. No retries permitted until 2026-07-29 19:17:45.888093198 +0000 UTC m=+33.116482309 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9ab35af-cc45-4dc0-ac22-cc434c73ee9e-metrics-certs") pod "network-metrics-daemon-4wlg9" (UID: "d9ab35af-cc45-4dc0-ac22-cc434c73ee9e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:17:29.988947 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:29.988921    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwjg2\" (UniqueName: \"kubernetes.io/projected/0a4b19f9-f7aa-4860-b947-e21d47ed4b21-kube-api-access-gwjg2\") pod \"network-check-target-cmjv9\" (UID: \"0a4b19f9-f7aa-4860-b947-e21d47ed4b21\") " pod="openshift-network-diagnostics/network-check-target-cmjv9"
Jul 29 19:17:29.989080 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:29.989005    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2485c438-0e6d-4aae-bab6-8dcda418e188-original-pull-secret\") pod \"global-pull-secret-syncer-pp24q\" (UID: \"2485c438-0e6d-4aae-bab6-8dcda418e188\") " pod="kube-system/global-pull-secret-syncer-pp24q"
Jul 29 19:17:29.989120 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:29.989081    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 19:17:29.989120 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:29.989086    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 19:17:29.989120 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:29.989106    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 19:17:29.989120 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:29.989116    2583 projected.go:196] Error preparing data for projected volume kube-api-access-gwjg2 for pod openshift-network-diagnostics/network-check-target-cmjv9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:17:29.989245 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:29.989124    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2485c438-0e6d-4aae-bab6-8dcda418e188-original-pull-secret podName:2485c438-0e6d-4aae-bab6-8dcda418e188 nodeName:}" failed. No retries permitted until 2026-07-29 19:17:45.989111196 +0000 UTC m=+33.217500301 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2485c438-0e6d-4aae-bab6-8dcda418e188-original-pull-secret") pod "global-pull-secret-syncer-pp24q" (UID: "2485c438-0e6d-4aae-bab6-8dcda418e188") : object "kube-system"/"original-pull-secret" not registered
Jul 29 19:17:29.989245 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:29.989163    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a4b19f9-f7aa-4860-b947-e21d47ed4b21-kube-api-access-gwjg2 podName:0a4b19f9-f7aa-4860-b947-e21d47ed4b21 nodeName:}" failed. No retries permitted until 2026-07-29 19:17:45.989152376 +0000 UTC m=+33.217541481 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gwjg2" (UniqueName: "kubernetes.io/projected/0a4b19f9-f7aa-4860-b947-e21d47ed4b21-kube-api-access-gwjg2") pod "network-check-target-cmjv9" (UID: "0a4b19f9-f7aa-4860-b947-e21d47ed4b21") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:17:30.335445 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:30.335382    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wlg9"
Jul 29 19:17:30.335913 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:30.335383    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp24q"
Jul 29 19:17:30.335913 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:30.335488    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wlg9" podUID="d9ab35af-cc45-4dc0-ac22-cc434c73ee9e"
Jul 29 19:17:30.335913 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:30.335585    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp24q" podUID="2485c438-0e6d-4aae-bab6-8dcda418e188"
Jul 29 19:17:31.335056 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:31.335027    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cmjv9"
Jul 29 19:17:31.335224 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:31.335149    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cmjv9" podUID="0a4b19f9-f7aa-4860-b947-e21d47ed4b21"
Jul 29 19:17:32.335201 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:32.335166    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp24q"
Jul 29 19:17:32.335641 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:32.335169    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wlg9"
Jul 29 19:17:32.335641 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:32.335288    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp24q" podUID="2485c438-0e6d-4aae-bab6-8dcda418e188"
Jul 29 19:17:32.335641 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:32.335420    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wlg9" podUID="d9ab35af-cc45-4dc0-ac22-cc434c73ee9e"
Jul 29 19:17:33.335749 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:33.335717    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cmjv9"
Jul 29 19:17:33.336213 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:33.335836    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cmjv9" podUID="0a4b19f9-f7aa-4860-b947-e21d47ed4b21"
Jul 29 19:17:34.334792 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:34.334769    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wlg9"
Jul 29 19:17:34.334896 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:34.334770    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp24q"
Jul 29 19:17:34.334896 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:34.334884    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wlg9" podUID="d9ab35af-cc45-4dc0-ac22-cc434c73ee9e"
Jul 29 19:17:34.334994 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:34.334974    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp24q" podUID="2485c438-0e6d-4aae-bab6-8dcda418e188"
Jul 29 19:17:34.562186 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:34.559778    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-101.ec2.internal" event={"ID":"b19b16543121e99642de34dc778bf29b","Type":"ContainerStarted","Data":"1d9c082289c7508fd9da7ef693e7882fb0bb5eef577aa40022017548d752c7f8"}
Jul 29 19:17:34.563284 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:34.563259    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gdg8t" event={"ID":"02b321ee-f4cc-4042-b292-926a6165fd9b","Type":"ContainerStarted","Data":"c1a30e5ab98b757c1941bc7bb1c001349b6696cd91781dfdc36543c954f16ee5"}
Jul 29 19:17:34.565482 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:34.565464    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd" event={"ID":"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b","Type":"ContainerStarted","Data":"106eb7ec14502520dd9f045cd423badcb2b1379576b70400d7e5d2fcd6b2ef32"}
Jul 29 19:17:34.565584 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:34.565531    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd" event={"ID":"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b","Type":"ContainerStarted","Data":"4a64ddaed59d20f6f9e204dd73ba5c3236f8bebfd1c4aba5c371626591f4f410"}
Jul 29 19:17:34.566701 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:34.566682    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nz9jf" event={"ID":"55857bde-1ffc-488e-9c18-dac2e7736f26","Type":"ContainerStarted","Data":"bb69b68ffeb51082b9f011f5e4b9245793fc0b97b22f484c0ba2adc7d1df596a"}
Jul 29 19:17:34.577429 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:34.577384    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-101.ec2.internal" podStartSLOduration=20.577373743 podStartE2EDuration="20.577373743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:17:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:17:34.576617954 +0000 UTC m=+21.805007078" watchObservedRunningTime="2026-07-29 19:17:34.577373743 +0000 UTC m=+21.805762869"
Jul 29 19:17:34.597471 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:34.597416    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nz9jf" podStartSLOduration=2.120928551 podStartE2EDuration="21.597397748s" totalImagesPullingTime="19.476469197s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:17:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:17:34.596474591 +0000 UTC m=+21.824863718" watchObservedRunningTime="2026-07-29 19:17:34.597397748 +0000 UTC m=+21.825786878"
Jul 29 19:17:34.619648 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:34.619604    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-gdg8t" podStartSLOduration=2.126328528 podStartE2EDuration="21.619588221s" totalImagesPullingTime="19.493259693s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:17:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:17:34.619140429 +0000 UTC m=+21.847529557" watchObservedRunningTime="2026-07-29 19:17:34.619588221 +0000 UTC m=+21.847977346"
Jul 29 19:17:35.334928 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:35.334907    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cmjv9"
Jul 29 19:17:35.335037 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:35.335020    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cmjv9" podUID="0a4b19f9-f7aa-4860-b947-e21d47ed4b21"
Jul 29 19:17:35.569455 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:35.569419    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-n6l8q" event={"ID":"0504596f-5a2f-4305-a319-ce48cbd6d199","Type":"ContainerStarted","Data":"674c3597440f541da5435152c63d1377ed72865795ee4b675be53b044e33b166"}
Jul 29 19:17:35.570748 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:35.570725    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mlk48" event={"ID":"558f3f9a-7e78-4810-ba73-42063864201e","Type":"ContainerStarted","Data":"301c90db2c9d4528ec91281182b410aa6ae7e4e42f3ea0e8c2077be79194cb22"}
Jul 29 19:17:35.572098 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:35.572067    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vw8v" event={"ID":"1d4c55d2-4264-40c8-9dff-1bcda7ec672d","Type":"ContainerStarted","Data":"368fd0dd3cc26e81e4d62e8265a0f68c44fdc2940698f887357dcfc7e4703bd9"}
Jul 29 19:17:35.573414 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:35.573391    2583 generic.go:356] "Generic (PLEG): container finished" podID="c57e5316-49cb-4992-904a-4b14b8c881e2" containerID="a519f5bf426fd42d20618268afd391571e8bd37c2171a8c7f07a1b2d03d1c471" exitCode=0
Jul 29 19:17:35.573507 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:35.573470    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tpwtr" event={"ID":"c57e5316-49cb-4992-904a-4b14b8c881e2","Type":"ContainerDied","Data":"a519f5bf426fd42d20618268afd391571e8bd37c2171a8c7f07a1b2d03d1c471"}
Jul 29 19:17:35.574953 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:35.574933    2583 generic.go:356] "Generic (PLEG): container finished" podID="e3140649e42883a9530def746eafe57a" containerID="cfb9fca27f2bed59b3010fc2f9c041bbc3310ff4ff4c87cdccc72c9a72111aad" exitCode=0
Jul 29 19:17:35.575017 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:35.575003    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-101.ec2.internal" event={"ID":"e3140649e42883a9530def746eafe57a","Type":"ContainerDied","Data":"cfb9fca27f2bed59b3010fc2f9c041bbc3310ff4ff4c87cdccc72c9a72111aad"}
Jul 29 19:17:35.578076 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:35.578032    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd" event={"ID":"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b","Type":"ContainerStarted","Data":"5f69bcd2552428dfa948a25a8335a04062aee0fabc26b568e030d3034b0a5456"}
Jul 29 19:17:35.578144 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:35.578094    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd" event={"ID":"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b","Type":"ContainerStarted","Data":"2eca2ffb568b75d797d6037e71dbb09315ec93a4f90f3254a35b3c38a05c241b"}
Jul 29 19:17:35.578144 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:35.578124    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd" event={"ID":"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b","Type":"ContainerStarted","Data":"505b0b07b10e5a136f727fe0c0b37595ada4491c226288255f10930be0968028"}
Jul 29 19:17:35.578226 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:35.578158    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd" event={"ID":"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b","Type":"ContainerStarted","Data":"01189795bb70f8ef24a08b64273f1f70f375511963d6c8896bb8da42eb6401cc"}
Jul 29 19:17:35.579287 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:35.579257    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m8f8f" event={"ID":"efd6982e-9638-4562-9ff3-9aab71caae71","Type":"ContainerStarted","Data":"aee30adb093f279682dde412b41740da251920646dcb3b108c0f95ea1a736c53"}
Jul 29 19:17:35.580615 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:35.580599    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-98vz6" event={"ID":"4f2d56b2-16f1-460a-b498-1bad91e77dd1","Type":"ContainerStarted","Data":"94db1eadca0b11b68faa26df10f48ddcbdc500442bee8a77788dae706f24a3a7"}
Jul 29 19:17:35.583712 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:35.583676    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-n6l8q" podStartSLOduration=3.183661344 podStartE2EDuration="22.583664351s" totalImagesPullingTime="19.400003007s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:17:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:17:35.583444141 +0000 UTC m=+22.811833267" watchObservedRunningTime="2026-07-29 19:17:35.583664351 +0000 UTC m=+22.812053477"
Jul 29 19:17:35.618069 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:35.618034    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-m8f8f" podStartSLOduration=3.11712792 podStartE2EDuration="22.618022334s" totalImagesPullingTime="19.500894414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:17:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:17:35.597454545 +0000 UTC m=+22.825843671" watchObservedRunningTime="2026-07-29 19:17:35.618022334 +0000 UTC m=+22.846411462"
Jul 29 19:17:35.633956 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:35.633739    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mlk48" podStartSLOduration=3.217930761 podStartE2EDuration="22.633727139s" totalImagesPullingTime="19.415796378s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:17:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:17:35.633271243 +0000 UTC m=+22.861660360" watchObservedRunningTime="2026-07-29 19:17:35.633727139 +0000 UTC m=+22.862116266"
Jul 29 19:17:35.662213 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:35.662180    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-98vz6" podStartSLOduration=7.342250273 podStartE2EDuration="22.662169311s" totalImagesPullingTime="15.319919038s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:17:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:17:35.661940289 +0000 UTC m=+22.890329415" watchObservedRunningTime="2026-07-29 19:17:35.662169311 +0000 UTC m=+22.890558436"
Jul 29 19:17:36.335570 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:36.335529    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wlg9"
Jul 29 19:17:36.335736 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:36.335529    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp24q"
Jul 29 19:17:36.335736 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:36.335653    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wlg9" podUID="d9ab35af-cc45-4dc0-ac22-cc434c73ee9e"
Jul 29 19:17:36.335736 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:36.335692    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp24q" podUID="2485c438-0e6d-4aae-bab6-8dcda418e188"
Jul 29 19:17:36.368665 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:36.368638    2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 29 19:17:36.585465 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:36.585381    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vw8v" event={"ID":"1d4c55d2-4264-40c8-9dff-1bcda7ec672d","Type":"ContainerStarted","Data":"bc077ead7d9b4848b5bd3a7ea5caddeba3df83d410d2cd1070e0c855db7e26c9"}
Jul 29 19:17:36.587351 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:36.587315    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-101.ec2.internal" event={"ID":"e3140649e42883a9530def746eafe57a","Type":"ContainerStarted","Data":"8fc45080ec3375f15b50fd82be4334b6bb34dc0c7226f56f9d42282547ac5738"}
Jul 29 19:17:36.605300 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:36.605252    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-101.ec2.internal" podStartSLOduration=22.605233692 podStartE2EDuration="22.605233692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:17:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:17:36.604573073 +0000 UTC m=+23.832962212" watchObservedRunningTime="2026-07-29 19:17:36.605233692 +0000 UTC m=+23.833622819"
Jul 29 19:17:37.216609 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:37.216493    2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T19:17:36.368659596Z","UUID":"b526ae77-9427-42b1-acc2-2c17ded903e8","Handler":null,"Name":"","Endpoint":""}
Jul 29 19:17:37.218702 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:37.218668    2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 29 19:17:37.218855 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:37.218713    2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 29 19:17:37.335560 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:37.335532    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cmjv9"
Jul 29 19:17:37.335669 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:37.335651    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cmjv9" podUID="0a4b19f9-f7aa-4860-b947-e21d47ed4b21"
Jul 29 19:17:37.592791 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:37.592756    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd" event={"ID":"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b","Type":"ContainerStarted","Data":"d9bec8de0bac0ee6402d6d3f51d0d86566b1ca1eba0975f85ee57f97e0b23f67"}
Jul 29 19:17:37.597835 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:37.596522    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vw8v" event={"ID":"1d4c55d2-4264-40c8-9dff-1bcda7ec672d","Type":"ContainerStarted","Data":"b8d1d07cf4ca1cb8b96d7341546a8c670c618bc53e84c75cf9f682d745a630c0"}
Jul 29 19:17:37.613703 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:37.613659    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vw8v" podStartSLOduration=2.452808863 podStartE2EDuration="24.613647473s" totalImagesPullingTime="22.16083861s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:17:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:17:37.613516478 +0000 UTC m=+24.841905605" watchObservedRunningTime="2026-07-29 19:17:37.613647473 +0000 UTC m=+24.842036598"
Jul 29 19:17:38.335194 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:38.335161    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wlg9"
Jul 29 19:17:38.335194 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:38.335187    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp24q"
Jul 29 19:17:38.335454 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:38.335280    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wlg9" podUID="d9ab35af-cc45-4dc0-ac22-cc434c73ee9e"
Jul 29 19:17:38.335454 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:38.335426    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp24q" podUID="2485c438-0e6d-4aae-bab6-8dcda418e188"
Jul 29 19:17:39.335577 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:39.335540    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cmjv9"
Jul 29 19:17:39.336090 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:39.335673    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cmjv9" podUID="0a4b19f9-f7aa-4860-b947-e21d47ed4b21"
Jul 29 19:17:40.040009 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:40.039978    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-m8f8f"
Jul 29 19:17:40.041059 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:40.041033    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-m8f8f"
Jul 29 19:17:40.334918 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:40.334835    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wlg9"
Jul 29 19:17:40.334918 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:40.334865    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp24q"
Jul 29 19:17:40.335090 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:40.334950    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wlg9" podUID="d9ab35af-cc45-4dc0-ac22-cc434c73ee9e"
Jul 29 19:17:40.335090 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:40.335018    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp24q" podUID="2485c438-0e6d-4aae-bab6-8dcda418e188"
Jul 29 19:17:40.607100 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:40.606777    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd" event={"ID":"f46c5d88-5a0f-4f1c-b71c-d6503ed5ba4b","Type":"ContainerStarted","Data":"48d51c6796a1fbb1f8acc96e8d4483935aab3b00ce1916e5525547ddf1ff2344"}
Jul 29 19:17:40.634701 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:40.634634    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd" podStartSLOduration=7.872680773 podStartE2EDuration="27.634610868s" totalImagesPullingTime="19.761930095s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:17:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:17:40.633967643 +0000 UTC m=+27.862356768" watchObservedRunningTime="2026-07-29 19:17:40.634610868 +0000 UTC m=+27.862999997"
Jul 29 19:17:41.335402 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:41.335364    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cmjv9"
Jul 29 19:17:41.335731 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:41.335485    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cmjv9" podUID="0a4b19f9-f7aa-4860-b947-e21d47ed4b21"
Jul 29 19:17:41.610410 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:41.610313    2583 generic.go:356] "Generic (PLEG): container finished" podID="c57e5316-49cb-4992-904a-4b14b8c881e2" containerID="fb0fb7382b0d9537e900f6f7a4d6c3dbc70156a60c20932fbfed46a6a590ecec" exitCode=0
Jul 29 19:17:41.611259 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:41.610403    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tpwtr" event={"ID":"c57e5316-49cb-4992-904a-4b14b8c881e2","Type":"ContainerDied","Data":"fb0fb7382b0d9537e900f6f7a4d6c3dbc70156a60c20932fbfed46a6a590ecec"}
Jul 29 19:17:41.611259 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:41.611046    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:41.611259 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:41.611130    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:41.611259 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:41.611180    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:41.627422 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:41.627392    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:41.627564 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:41.627498    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:17:42.116860 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:42.116825    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-m8f8f"
Jul 29 19:17:42.117016 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:42.116987    2583 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 29 19:17:42.117562 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:42.117528    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-m8f8f"
Jul 29 19:17:42.335599 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:42.335384    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wlg9"
Jul 29 19:17:42.335761 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:42.335384    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp24q"
Jul 29 19:17:42.335761 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:42.335747    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wlg9" podUID="d9ab35af-cc45-4dc0-ac22-cc434c73ee9e"
Jul 29 19:17:42.335898 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:42.335879    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp24q" podUID="2485c438-0e6d-4aae-bab6-8dcda418e188"
Jul 29 19:17:42.428757 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:42.428658    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cmjv9"]
Jul 29 19:17:42.428964 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:42.428780    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cmjv9"
Jul 29 19:17:42.428964 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:42.428912    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cmjv9" podUID="0a4b19f9-f7aa-4860-b947-e21d47ed4b21"
Jul 29 19:17:42.431907 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:42.431874    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pp24q"]
Jul 29 19:17:42.432441 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:42.432415    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4wlg9"]
Jul 29 19:17:42.615488 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:42.615465    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wlg9"
Jul 29 19:17:42.615488 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:42.615487    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp24q"
Jul 29 19:17:42.616099 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:42.615504    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tpwtr" event={"ID":"c57e5316-49cb-4992-904a-4b14b8c881e2","Type":"ContainerStarted","Data":"bdc9544aa797d9e9aaec7d339c326816e4cb25284bb325851d41b8f2244f2def"}
Jul 29 19:17:42.616099 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:42.615589    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wlg9" podUID="d9ab35af-cc45-4dc0-ac22-cc434c73ee9e"
Jul 29 19:17:42.616099 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:42.615703    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp24q" podUID="2485c438-0e6d-4aae-bab6-8dcda418e188"
Jul 29 19:17:43.619766 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:43.619728    2583 generic.go:356] "Generic (PLEG): container finished" podID="c57e5316-49cb-4992-904a-4b14b8c881e2" containerID="bdc9544aa797d9e9aaec7d339c326816e4cb25284bb325851d41b8f2244f2def" exitCode=0
Jul 29 19:17:43.620285 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:43.619830    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tpwtr" event={"ID":"c57e5316-49cb-4992-904a-4b14b8c881e2","Type":"ContainerDied","Data":"bdc9544aa797d9e9aaec7d339c326816e4cb25284bb325851d41b8f2244f2def"}
Jul 29 19:17:44.335312 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:44.335232    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wlg9"
Jul 29 19:17:44.335312 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:44.335275    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp24q"
Jul 29 19:17:44.335312 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:44.335275    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cmjv9"
Jul 29 19:17:44.335507 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:44.335349    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wlg9" podUID="d9ab35af-cc45-4dc0-ac22-cc434c73ee9e"
Jul 29 19:17:44.335507 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:44.335442    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp24q" podUID="2485c438-0e6d-4aae-bab6-8dcda418e188"
Jul 29 19:17:44.335572 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:44.335513    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cmjv9" podUID="0a4b19f9-f7aa-4860-b947-e21d47ed4b21"
Jul 29 19:17:44.628286 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:44.628251    2583 generic.go:356] "Generic (PLEG): container finished" podID="c57e5316-49cb-4992-904a-4b14b8c881e2" containerID="8a77aff173db9c522360c3a8f608fd36736e6c8a6df6817e67c4a6e74fb73162" exitCode=0
Jul 29 19:17:44.628692 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:44.628314    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tpwtr" event={"ID":"c57e5316-49cb-4992-904a-4b14b8c881e2","Type":"ContainerDied","Data":"8a77aff173db9c522360c3a8f608fd36736e6c8a6df6817e67c4a6e74fb73162"}
Jul 29 19:17:45.906709 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:45.906667    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9ab35af-cc45-4dc0-ac22-cc434c73ee9e-metrics-certs\") pod \"network-metrics-daemon-4wlg9\" (UID: \"d9ab35af-cc45-4dc0-ac22-cc434c73ee9e\") " pod="openshift-multus/network-metrics-daemon-4wlg9"
Jul 29 19:17:45.907188 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:45.906816    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:17:45.907188 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:45.906884    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9ab35af-cc45-4dc0-ac22-cc434c73ee9e-metrics-certs podName:d9ab35af-cc45-4dc0-ac22-cc434c73ee9e nodeName:}" failed. No retries permitted until 2026-07-29 19:18:17.906866748 +0000 UTC m=+65.135255853 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9ab35af-cc45-4dc0-ac22-cc434c73ee9e-metrics-certs") pod "network-metrics-daemon-4wlg9" (UID: "d9ab35af-cc45-4dc0-ac22-cc434c73ee9e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:17:46.007869 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:46.007828    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwjg2\" (UniqueName: \"kubernetes.io/projected/0a4b19f9-f7aa-4860-b947-e21d47ed4b21-kube-api-access-gwjg2\") pod \"network-check-target-cmjv9\" (UID: \"0a4b19f9-f7aa-4860-b947-e21d47ed4b21\") " pod="openshift-network-diagnostics/network-check-target-cmjv9"
Jul 29 19:17:46.008085 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:46.007918    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2485c438-0e6d-4aae-bab6-8dcda418e188-original-pull-secret\") pod \"global-pull-secret-syncer-pp24q\" (UID: \"2485c438-0e6d-4aae-bab6-8dcda418e188\") " pod="kube-system/global-pull-secret-syncer-pp24q"
Jul 29 19:17:46.008085 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:46.008019    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 19:17:46.008085 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:46.008037    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 19:17:46.008085 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:46.008053    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 19:17:46.008085 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:46.008068    2583 projected.go:196] Error preparing data for projected volume kube-api-access-gwjg2 for pod openshift-network-diagnostics/network-check-target-cmjv9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:17:46.008357 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:46.008127    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2485c438-0e6d-4aae-bab6-8dcda418e188-original-pull-secret podName:2485c438-0e6d-4aae-bab6-8dcda418e188 nodeName:}" failed. No retries permitted until 2026-07-29 19:18:18.008086264 +0000 UTC m=+65.236475369 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2485c438-0e6d-4aae-bab6-8dcda418e188-original-pull-secret") pod "global-pull-secret-syncer-pp24q" (UID: "2485c438-0e6d-4aae-bab6-8dcda418e188") : object "kube-system"/"original-pull-secret" not registered
Jul 29 19:17:46.008357 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:46.008148    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a4b19f9-f7aa-4860-b947-e21d47ed4b21-kube-api-access-gwjg2 podName:0a4b19f9-f7aa-4860-b947-e21d47ed4b21 nodeName:}" failed. No retries permitted until 2026-07-29 19:18:18.008138128 +0000 UTC m=+65.236527241 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gwjg2" (UniqueName: "kubernetes.io/projected/0a4b19f9-f7aa-4860-b947-e21d47ed4b21-kube-api-access-gwjg2") pod "network-check-target-cmjv9" (UID: "0a4b19f9-f7aa-4860-b947-e21d47ed4b21") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:17:46.335008 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:46.334918    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cmjv9"
Jul 29 19:17:46.335008 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:46.334959    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp24q"
Jul 29 19:17:46.335008 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:46.334959    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wlg9"
Jul 29 19:17:46.335292 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:46.335047    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cmjv9" podUID="0a4b19f9-f7aa-4860-b947-e21d47ed4b21"
Jul 29 19:17:46.335292 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:46.335157    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp24q" podUID="2485c438-0e6d-4aae-bab6-8dcda418e188"
Jul 29 19:17:46.335292 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:17:46.335258    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wlg9" podUID="d9ab35af-cc45-4dc0-ac22-cc434c73ee9e"
Jul 29 19:17:47.061712 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:47.061681    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mlk48_558f3f9a-7e78-4810-ba73-42063864201e/dns-node-resolver/0.log"
Jul 29 19:17:47.834580 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:47.834387    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-98vz6_4f2d56b2-16f1-460a-b498-1bad91e77dd1/node-ca/0.log"
Jul 29 19:17:48.081010 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:48.080932    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-101.ec2.internal" event="NodeReady"
Jul 29 19:17:48.081389 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:48.081102    2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 29 19:17:48.334675 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:48.334598    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cmjv9"
Jul 29 19:17:48.334855 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:48.334704    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp24q"
Jul 29 19:17:48.334927 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:48.334897    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wlg9"
Jul 29 19:17:48.336930 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:48.336903    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qxkk6\""
Jul 29 19:17:48.337450 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:48.337429    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 19:17:48.337450 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:48.337450    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 19:17:48.337622 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:48.337606    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 19:17:48.337708 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:48.337687    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 19:17:48.337902 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:48.337814    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-b2qz7\""
Jul 29 19:17:49.643617 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:49.643585    2583 generic.go:356] "Generic (PLEG): container finished" podID="c57e5316-49cb-4992-904a-4b14b8c881e2" containerID="c98e1ff5abbedc1e7f7b60ad6be8b029baf7d93afc2ef469994f6304a4c76be3" exitCode=0
Jul 29 19:17:49.644004 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:49.643639    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tpwtr" event={"ID":"c57e5316-49cb-4992-904a-4b14b8c881e2","Type":"ContainerDied","Data":"c98e1ff5abbedc1e7f7b60ad6be8b029baf7d93afc2ef469994f6304a4c76be3"}
Jul 29 19:17:50.648853 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:50.648817    2583 generic.go:356] "Generic (PLEG): container finished" podID="c57e5316-49cb-4992-904a-4b14b8c881e2" containerID="b8651b84561b2f9ef2e130a6d67883fd3558e3daefc999a5633b421283107eee" exitCode=0
Jul 29 19:17:50.649469 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:50.648877    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tpwtr" event={"ID":"c57e5316-49cb-4992-904a-4b14b8c881e2","Type":"ContainerDied","Data":"b8651b84561b2f9ef2e130a6d67883fd3558e3daefc999a5633b421283107eee"}
Jul 29 19:17:51.653971 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:51.653934    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tpwtr" event={"ID":"c57e5316-49cb-4992-904a-4b14b8c881e2","Type":"ContainerStarted","Data":"eaef65602c35155e95e030bc4e09b0842604d5494c5b06f25d094d4d420a654f"}
Jul 29 19:17:51.677646 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:17:51.677596    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-tpwtr" podStartSLOduration=7.914780268 podStartE2EDuration="38.67758188s" totalImagesPullingTime="30.762801612s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:17:13 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:17:51.67713314 +0000 UTC m=+38.905522265" watchObservedRunningTime="2026-07-29 19:17:51.67758188 +0000 UTC m=+38.905971005"
Jul 29 19:18:08.983272 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:08.983232    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-sz59c"]
Jul 29 19:18:08.985268 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:08.985243    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-sz59c"
Jul 29 19:18:08.987814 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:08.987777    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 29 19:18:08.987950 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:08.987930    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 29 19:18:08.988654 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:08.988634    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-lnqm7\""
Jul 29 19:18:09.001959 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.001940    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-sz59c"]
Jul 29 19:18:09.059324 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.059304    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/78662159-1110-4413-ab45-219ed9d4ede9-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-sz59c\" (UID: \"78662159-1110-4413-ab45-219ed9d4ede9\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sz59c"
Jul 29 19:18:09.059430 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.059349    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/78662159-1110-4413-ab45-219ed9d4ede9-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-sz59c\" (UID: \"78662159-1110-4413-ab45-219ed9d4ede9\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sz59c"
Jul 29 19:18:09.124451 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.124424    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-64ddfd448b-wp5n4"]
Jul 29 19:18:09.126392 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.126369    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hr2f6"]
Jul 29 19:18:09.126510 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.126491    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64ddfd448b-wp5n4"
Jul 29 19:18:09.127916 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.127899    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-r9gkq"]
Jul 29 19:18:09.128164 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.128023    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hr2f6"
Jul 29 19:18:09.132318 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.131420    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c797f987d-zw59v"]
Jul 29 19:18:09.132420 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.132291    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r9gkq"
Jul 29 19:18:09.133286 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.133264    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 29 19:18:09.133398 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.133381    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 29 19:18:09.133487 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.133469    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2xxt4\""
Jul 29 19:18:09.133539 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.133477    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 29 19:18:09.133590 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.133553    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 29 19:18:09.133590 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.133565    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 29 19:18:09.133672 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.133627    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 29 19:18:09.133883 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.133869    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d7cfb778-rzp77"]
Jul 29 19:18:09.133983 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.133968    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c797f987d-zw59v"
Jul 29 19:18:09.135761 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.135742    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7fbf44d9c7-zvzrx"]
Jul 29 19:18:09.135985 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.135962    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d7cfb778-rzp77"
Jul 29 19:18:09.137514 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.137498    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fbf44d9c7-zvzrx"
Jul 29 19:18:09.138693 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.138677    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 29 19:18:09.140215 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.140201    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 29 19:18:09.141282 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.141262    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 29 19:18:09.141377 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.141266    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 29 19:18:09.142304 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.142285    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xs7xs\""
Jul 29 19:18:09.142489 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.142474    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 29 19:18:09.142568 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.142540    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 29 19:18:09.142765 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.142620    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 29 19:18:09.142765 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.142576    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 29 19:18:09.143072 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.143050    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 29 19:18:09.143181 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.143163    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hhdrq\""
Jul 29 19:18:09.143309 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.143292    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 29 19:18:09.144184 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.144171    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-85kx2\""
Jul 29 19:18:09.144342 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.144332    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 29 19:18:09.149333 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.149287    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-64ddfd448b-wp5n4"]
Jul 29 19:18:09.149980 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.149962    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r9gkq"]
Jul 29 19:18:09.150684 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.150658    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c797f987d-zw59v"]
Jul 29 19:18:09.152910 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.152892    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hr2f6"]
Jul 29 19:18:09.153865 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.153846    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7fbf44d9c7-zvzrx"]
Jul 29 19:18:09.159079 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.159062    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 29 19:18:09.160514 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.160499    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d7da528d-c57f-471d-92a3-821b1045f55f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-c797f987d-zw59v\" (UID: \"d7da528d-c57f-471d-92a3-821b1045f55f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c797f987d-zw59v"
Jul 29 19:18:09.160575 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.160521    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be1b9e57-7c37-40bf-90db-0699e42f5c68-registry-tls\") pod \"image-registry-7fbf44d9c7-zvzrx\" (UID: \"be1b9e57-7c37-40bf-90db-0699e42f5c68\") " pod="openshift-image-registry/image-registry-7fbf44d9c7-zvzrx"
Jul 29 19:18:09.160575 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.160538    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slq4q\" (UniqueName: \"kubernetes.io/projected/38b8311f-6cea-4246-ac0a-619be43452e0-kube-api-access-slq4q\") pod \"ingress-canary-hr2f6\" (UID: \"38b8311f-6cea-4246-ac0a-619be43452e0\") " pod="openshift-ingress-canary/ingress-canary-hr2f6"
Jul 29 19:18:09.160575 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.160552    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/93412f51-94fe-40a4-b1bb-10f5bb4ee2a9-config-volume\") pod \"dns-default-r9gkq\" (UID: \"93412f51-94fe-40a4-b1bb-10f5bb4ee2a9\") " pod="openshift-dns/dns-default-r9gkq"
Jul 29 19:18:09.160575 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.160567    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/402395a1-90da-48c5-b125-8ebb36f3e199-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-58d7cfb778-rzp77\" (UID: \"402395a1-90da-48c5-b125-8ebb36f3e199\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d7cfb778-rzp77"
Jul 29 19:18:09.160768 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.160591    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/78662159-1110-4413-ab45-219ed9d4ede9-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-sz59c\" (UID: \"78662159-1110-4413-ab45-219ed9d4ede9\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sz59c"
Jul 29 19:18:09.160768 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.160640    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/402395a1-90da-48c5-b125-8ebb36f3e199-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-58d7cfb778-rzp77\" (UID: \"402395a1-90da-48c5-b125-8ebb36f3e199\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d7cfb778-rzp77"
Jul 29 19:18:09.160768 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.160658    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/402395a1-90da-48c5-b125-8ebb36f3e199-ca\") pod \"cluster-proxy-proxy-agent-58d7cfb778-rzp77\" (UID: \"402395a1-90da-48c5-b125-8ebb36f3e199\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d7cfb778-rzp77"
Jul 29 19:18:09.160768 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.160690    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bad3b996-769f-49cb-8edc-096993a68736-tmp\") pod \"klusterlet-addon-workmgr-64ddfd448b-wp5n4\" (UID: \"bad3b996-769f-49cb-8edc-096993a68736\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64ddfd448b-wp5n4"
Jul 29 19:18:09.160768 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.160704    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be1b9e57-7c37-40bf-90db-0699e42f5c68-registry-certificates\") pod \"image-registry-7fbf44d9c7-zvzrx\" (UID: \"be1b9e57-7c37-40bf-90db-0699e42f5c68\") " pod="openshift-image-registry/image-registry-7fbf44d9c7-zvzrx"
Jul 29 19:18:09.160768 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.160731    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-475wf\" (UniqueName: \"kubernetes.io/projected/93412f51-94fe-40a4-b1bb-10f5bb4ee2a9-kube-api-access-475wf\") pod \"dns-default-r9gkq\" (UID: \"93412f51-94fe-40a4-b1bb-10f5bb4ee2a9\") " pod="openshift-dns/dns-default-r9gkq"
Jul 29 19:18:09.160768 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.160749    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be1b9e57-7c37-40bf-90db-0699e42f5c68-bound-sa-token\") pod \"image-registry-7fbf44d9c7-zvzrx\" (UID: \"be1b9e57-7c37-40bf-90db-0699e42f5c68\") " pod="openshift-image-registry/image-registry-7fbf44d9c7-zvzrx"
Jul 29 19:18:09.160768 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.160762    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/402395a1-90da-48c5-b125-8ebb36f3e199-hub\") pod \"cluster-proxy-proxy-agent-58d7cfb778-rzp77\" (UID: \"402395a1-90da-48c5-b125-8ebb36f3e199\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d7cfb778-rzp77"
Jul 29 19:18:09.161189 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.160784    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dblpj\" (UniqueName: \"kubernetes.io/projected/d7da528d-c57f-471d-92a3-821b1045f55f-kube-api-access-dblpj\") pod \"managed-serviceaccount-addon-agent-c797f987d-zw59v\" (UID: \"d7da528d-c57f-471d-92a3-821b1045f55f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c797f987d-zw59v"
Jul 29 19:18:09.161189 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.160835    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be1b9e57-7c37-40bf-90db-0699e42f5c68-trusted-ca\") pod \"image-registry-7fbf44d9c7-zvzrx\" (UID: \"be1b9e57-7c37-40bf-90db-0699e42f5c68\") " pod="openshift-image-registry/image-registry-7fbf44d9c7-zvzrx"
Jul 29 19:18:09.161189 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.160853    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be1b9e57-7c37-40bf-90db-0699e42f5c68-installation-pull-secrets\") pod \"image-registry-7fbf44d9c7-zvzrx\" (UID: \"be1b9e57-7c37-40bf-90db-0699e42f5c68\") " pod="openshift-image-registry/image-registry-7fbf44d9c7-zvzrx"
Jul 29 19:18:09.161189 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.160871    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/93412f51-94fe-40a4-b1bb-10f5bb4ee2a9-metrics-tls\") pod \"dns-default-r9gkq\" (UID: \"93412f51-94fe-40a4-b1bb-10f5bb4ee2a9\") " pod="openshift-dns/dns-default-r9gkq"
Jul 29 19:18:09.161189 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.160886    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2vlq\" (UniqueName: \"kubernetes.io/projected/402395a1-90da-48c5-b125-8ebb36f3e199-kube-api-access-k2vlq\") pod \"cluster-proxy-proxy-agent-58d7cfb778-rzp77\" (UID: \"402395a1-90da-48c5-b125-8ebb36f3e199\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d7cfb778-rzp77"
Jul 29 19:18:09.161189 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.160914    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be1b9e57-7c37-40bf-90db-0699e42f5c68-image-registry-private-configuration\") pod \"image-registry-7fbf44d9c7-zvzrx\" (UID: \"be1b9e57-7c37-40bf-90db-0699e42f5c68\") " pod="openshift-image-registry/image-registry-7fbf44d9c7-zvzrx"
Jul 29 19:18:09.161189 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.160942    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkbtj\" (UniqueName: \"kubernetes.io/projected/bad3b996-769f-49cb-8edc-096993a68736-kube-api-access-pkbtj\") pod \"klusterlet-addon-workmgr-64ddfd448b-wp5n4\" (UID: \"bad3b996-769f-49cb-8edc-096993a68736\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64ddfd448b-wp5n4"
Jul 29 19:18:09.161189 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.160960    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93412f51-94fe-40a4-b1bb-10f5bb4ee2a9-tmp-dir\") pod \"dns-default-r9gkq\" (UID: \"93412f51-94fe-40a4-b1bb-10f5bb4ee2a9\") " pod="openshift-dns/dns-default-r9gkq"
Jul 29 19:18:09.161189 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.160991    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bad3b996-769f-49cb-8edc-096993a68736-klusterlet-config\") pod \"klusterlet-addon-workmgr-64ddfd448b-wp5n4\" (UID: \"bad3b996-769f-49cb-8edc-096993a68736\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64ddfd448b-wp5n4"
Jul 29 19:18:09.161189 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.161016    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be1b9e57-7c37-40bf-90db-0699e42f5c68-ca-trust-extracted\") pod \"image-registry-7fbf44d9c7-zvzrx\" (UID: \"be1b9e57-7c37-40bf-90db-0699e42f5c68\") " pod="openshift-image-registry/image-registry-7fbf44d9c7-zvzrx"
Jul 29 19:18:09.161189 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.161060    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/78662159-1110-4413-ab45-219ed9d4ede9-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-sz59c\" (UID: \"78662159-1110-4413-ab45-219ed9d4ede9\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sz59c"
Jul 29 19:18:09.161189 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.161076    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2845\" (UniqueName: \"kubernetes.io/projected/be1b9e57-7c37-40bf-90db-0699e42f5c68-kube-api-access-v2845\") pod \"image-registry-7fbf44d9c7-zvzrx\" (UID: \"be1b9e57-7c37-40bf-90db-0699e42f5c68\") " pod="openshift-image-registry/image-registry-7fbf44d9c7-zvzrx"
Jul 29 19:18:09.161189 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.161098    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/38b8311f-6cea-4246-ac0a-619be43452e0-cert\") pod \"ingress-canary-hr2f6\" (UID: \"38b8311f-6cea-4246-ac0a-619be43452e0\") " pod="openshift-ingress-canary/ingress-canary-hr2f6"
Jul 29 19:18:09.161189 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.161120    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/402395a1-90da-48c5-b125-8ebb36f3e199-additional-service-ca\") pod \"cluster-proxy-proxy-agent-58d7cfb778-rzp77\" (UID: \"402395a1-90da-48c5-b125-8ebb36f3e199\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d7cfb778-rzp77"
Jul 29 19:18:09.161667 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.161344    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/78662159-1110-4413-ab45-219ed9d4ede9-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-sz59c\" (UID: \"78662159-1110-4413-ab45-219ed9d4ede9\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sz59c"
Jul 29 19:18:09.164881 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.164863    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/78662159-1110-4413-ab45-219ed9d4ede9-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-sz59c\" (UID: \"78662159-1110-4413-ab45-219ed9d4ede9\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sz59c"
Jul 29 19:18:09.179959 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.179939    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d7cfb778-rzp77"]
Jul 29 19:18:09.190187 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.190168    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nzzqf"]
Jul 29 19:18:09.192373 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.192358    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7fbf44d9c7-zvzrx"]
Jul 29 19:18:09.192472 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.192459    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nzzqf"
Jul 29 19:18:09.192518 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:18:09.192485    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-v2845 registry-certificates registry-tls trusted-ca], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-7fbf44d9c7-zvzrx" podUID="be1b9e57-7c37-40bf-90db-0699e42f5c68"
Jul 29 19:18:09.194487 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.194465    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 29 19:18:09.194695 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.194675    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 29 19:18:09.195112 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.195088    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nb5jk\""
Jul 29 19:18:09.195112 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.195089    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 29 19:18:09.195262 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.195136    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 29 19:18:09.206280 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.206256    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nzzqf"]
Jul 29 19:18:09.262101 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.262044    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7bafa333-3be5-44ae-b6a2-255fd658a919-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nzzqf\" (UID: \"7bafa333-3be5-44ae-b6a2-255fd658a919\") " pod="openshift-insights/insights-runtime-extractor-nzzqf"
Jul 29 19:18:09.262101 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.262073    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7bafa333-3be5-44ae-b6a2-255fd658a919-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nzzqf\" (UID: \"7bafa333-3be5-44ae-b6a2-255fd658a919\") " pod="openshift-insights/insights-runtime-extractor-nzzqf"
Jul 29 19:18:09.262263 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.262111    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/402395a1-90da-48c5-b125-8ebb36f3e199-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-58d7cfb778-rzp77\" (UID: \"402395a1-90da-48c5-b125-8ebb36f3e199\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d7cfb778-rzp77"
Jul 29 19:18:09.262263 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.262150    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/402395a1-90da-48c5-b125-8ebb36f3e199-ca\") pod \"cluster-proxy-proxy-agent-58d7cfb778-rzp77\" (UID: \"402395a1-90da-48c5-b125-8ebb36f3e199\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d7cfb778-rzp77"
Jul 29 19:18:09.262263 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.262177    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bad3b996-769f-49cb-8edc-096993a68736-tmp\") pod \"klusterlet-addon-workmgr-64ddfd448b-wp5n4\" (UID: \"bad3b996-769f-49cb-8edc-096993a68736\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64ddfd448b-wp5n4"
Jul 29 19:18:09.262263 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.262200    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be1b9e57-7c37-40bf-90db-0699e42f5c68-registry-certificates\") pod \"image-registry-7fbf44d9c7-zvzrx\" (UID: \"be1b9e57-7c37-40bf-90db-0699e42f5c68\") " pod="openshift-image-registry/image-registry-7fbf44d9c7-zvzrx"
Jul 29 19:18:09.262263 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.262229    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-475wf\" (UniqueName: \"kubernetes.io/projected/93412f51-94fe-40a4-b1bb-10f5bb4ee2a9-kube-api-access-475wf\") pod \"dns-default-r9gkq\" (UID: \"93412f51-94fe-40a4-b1bb-10f5bb4ee2a9\") " pod="openshift-dns/dns-default-r9gkq"
Jul 29 19:18:09.262263 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.262259    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be1b9e57-7c37-40bf-90db-0699e42f5c68-bound-sa-token\") pod \"image-registry-7fbf44d9c7-zvzrx\" (UID: \"be1b9e57-7c37-40bf-90db-0699e42f5c68\") " pod="openshift-image-registry/image-registry-7fbf44d9c7-zvzrx"
Jul 29 19:18:09.262546 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.262280    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/402395a1-90da-48c5-b125-8ebb36f3e199-hub\") pod \"cluster-proxy-proxy-agent-58d7cfb778-rzp77\" (UID: \"402395a1-90da-48c5-b125-8ebb36f3e199\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d7cfb778-rzp77"
Jul 29 19:18:09.262546 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.262307    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dblpj\" (UniqueName: \"kubernetes.io/projected/d7da528d-c57f-471d-92a3-821b1045f55f-kube-api-access-dblpj\") pod \"managed-serviceaccount-addon-agent-c797f987d-zw59v\" (UID: \"d7da528d-c57f-471d-92a3-821b1045f55f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c797f987d-zw59v"
Jul 29 19:18:09.262546 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.262481    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be1b9e57-7c37-40bf-90db-0699e42f5c68-trusted-ca\") pod \"image-registry-7fbf44d9c7-zvzrx\" (UID: \"be1b9e57-7c37-40bf-90db-0699e42f5c68\") " pod="openshift-image-registry/image-registry-7fbf44d9c7-zvzrx"
Jul 29 19:18:09.262546 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.262519    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be1b9e57-7c37-40bf-90db-0699e42f5c68-installation-pull-secrets\") pod \"image-registry-7fbf44d9c7-zvzrx\" (UID: \"be1b9e57-7c37-40bf-90db-0699e42f5c68\") " pod="openshift-image-registry/image-registry-7fbf44d9c7-zvzrx"
Jul 29 19:18:09.262739 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.262556    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/93412f51-94fe-40a4-b1bb-10f5bb4ee2a9-metrics-tls\") pod \"dns-default-r9gkq\" (UID: \"93412f51-94fe-40a4-b1bb-10f5bb4ee2a9\") " pod="openshift-dns/dns-default-r9gkq"
Jul 29 19:18:09.262739 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.262563    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bad3b996-769f-49cb-8edc-096993a68736-tmp\") pod \"klusterlet-addon-workmgr-64ddfd448b-wp5n4\" (UID: \"bad3b996-769f-49cb-8edc-096993a68736\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64ddfd448b-wp5n4"
Jul 29 19:18:09.262739 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.262583    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2vlq\" (UniqueName: \"kubernetes.io/projected/402395a1-90da-48c5-b125-8ebb36f3e199-kube-api-access-k2vlq\") pod \"cluster-proxy-proxy-agent-58d7cfb778-rzp77\" (UID: \"402395a1-90da-48c5-b125-8ebb36f3e199\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d7cfb778-rzp77"
Jul 29 19:18:09.262739 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.262622    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be1b9e57-7c37-40bf-90db-0699e42f5c68-image-registry-private-configuration\") pod \"image-registry-7fbf44d9c7-zvzrx\" (UID: \"be1b9e57-7c37-40bf-90db-0699e42f5c68\") " pod="openshift-image-registry/image-registry-7fbf44d9c7-zvzrx"
Jul 29 19:18:09.262739 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.262656    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkbtj\" (UniqueName: \"kubernetes.io/projected/bad3b996-769f-49cb-8edc-096993a68736-kube-api-access-pkbtj\") pod \"klusterlet-addon-workmgr-64ddfd448b-wp5n4\" (UID: \"bad3b996-769f-49cb-8edc-096993a68736\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64ddfd448b-wp5n4"
Jul 29 19:18:09.262739 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.262685    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7bafa333-3be5-44ae-b6a2-255fd658a919-crio-socket\") pod \"insights-runtime-extractor-nzzqf\" (UID: \"7bafa333-3be5-44ae-b6a2-255fd658a919\") " pod="openshift-insights/insights-runtime-extractor-nzzqf"
Jul 29 19:18:09.263058 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.262711    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7bafa333-3be5-44ae-b6a2-255fd658a919-data-volume\") pod \"insights-runtime-extractor-nzzqf\" (UID: \"7bafa333-3be5-44ae-b6a2-255fd658a919\") " pod="openshift-insights/insights-runtime-extractor-nzzqf"
Jul 29 19:18:09.263058 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.262777    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93412f51-94fe-40a4-b1bb-10f5bb4ee2a9-tmp-dir\") pod \"dns-default-r9gkq\" (UID: \"93412f51-94fe-40a4-b1bb-10f5bb4ee2a9\") " pod="openshift-dns/dns-default-r9gkq"
Jul 29 19:18:09.263058 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.262830    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6p5f\" (UniqueName: \"kubernetes.io/projected/7bafa333-3be5-44ae-b6a2-255fd658a919-kube-api-access-g6p5f\") pod \"insights-runtime-extractor-nzzqf\" (UID: \"7bafa333-3be5-44ae-b6a2-255fd658a919\") " pod="openshift-insights/insights-runtime-extractor-nzzqf"
Jul 29 19:18:09.263058 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.262863    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bad3b996-769f-49cb-8edc-096993a68736-klusterlet-config\") pod \"klusterlet-addon-workmgr-64ddfd448b-wp5n4\" (UID: \"bad3b996-769f-49cb-8edc-096993a68736\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64ddfd448b-wp5n4"
Jul 29 19:18:09.263058 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.262888    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be1b9e57-7c37-40bf-90db-0699e42f5c68-ca-trust-extracted\") pod \"image-registry-7fbf44d9c7-zvzrx\" (UID: \"be1b9e57-7c37-40bf-90db-0699e42f5c68\") " pod="openshift-image-registry/image-registry-7fbf44d9c7-zvzrx"
Jul 29 19:18:09.263058 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.262921    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2845\" (UniqueName: \"kubernetes.io/projected/be1b9e57-7c37-40bf-90db-0699e42f5c68-kube-api-access-v2845\") pod \"image-registry-7fbf44d9c7-zvzrx\" (UID: \"be1b9e57-7c37-40bf-90db-0699e42f5c68\") " pod="openshift-image-registry/image-registry-7fbf44d9c7-zvzrx"
Jul 29 19:18:09.263058 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.262961    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/38b8311f-6cea-4246-ac0a-619be43452e0-cert\") pod \"ingress-canary-hr2f6\" (UID: \"38b8311f-6cea-4246-ac0a-619be43452e0\") " pod="openshift-ingress-canary/ingress-canary-hr2f6"
Jul 29 19:18:09.263058 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.262989    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/402395a1-90da-48c5-b125-8ebb36f3e199-additional-service-ca\") pod \"cluster-proxy-proxy-agent-58d7cfb778-rzp77\" (UID: \"402395a1-90da-48c5-b125-8ebb36f3e199\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d7cfb778-rzp77"
Jul 29 19:18:09.263058 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.263026    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d7da528d-c57f-471d-92a3-821b1045f55f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-c797f987d-zw59v\" (UID: \"d7da528d-c57f-471d-92a3-821b1045f55f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c797f987d-zw59v"
Jul 29 19:18:09.263058 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.263052    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be1b9e57-7c37-40bf-90db-0699e42f5c68-registry-tls\") pod \"image-registry-7fbf44d9c7-zvzrx\" (UID: \"be1b9e57-7c37-40bf-90db-0699e42f5c68\") " pod="openshift-image-registry/image-registry-7fbf44d9c7-zvzrx"
Jul 29 19:18:09.263058 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.263055    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be1b9e57-7c37-40bf-90db-0699e42f5c68-registry-certificates\") pod \"image-registry-7fbf44d9c7-zvzrx\" (UID: \"be1b9e57-7c37-40bf-90db-0699e42f5c68\") " pod="openshift-image-registry/image-registry-7fbf44d9c7-zvzrx"
Jul 29 19:18:09.263652 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.263081    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slq4q\" (UniqueName: \"kubernetes.io/projected/38b8311f-6cea-4246-ac0a-619be43452e0-kube-api-access-slq4q\") pod \"ingress-canary-hr2f6\" (UID: \"38b8311f-6cea-4246-ac0a-619be43452e0\") " pod="openshift-ingress-canary/ingress-canary-hr2f6"
Jul 29 19:18:09.263652 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.263104    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/93412f51-94fe-40a4-b1bb-10f5bb4ee2a9-config-volume\") pod \"dns-default-r9gkq\" (UID: \"93412f51-94fe-40a4-b1bb-10f5bb4ee2a9\") " pod="openshift-dns/dns-default-r9gkq"
Jul 29 19:18:09.263652 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.263127    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/402395a1-90da-48c5-b125-8ebb36f3e199-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-58d7cfb778-rzp77\" (UID: \"402395a1-90da-48c5-b125-8ebb36f3e199\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d7cfb778-rzp77"
Jul 29 19:18:09.263652 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.263484    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be1b9e57-7c37-40bf-90db-0699e42f5c68-ca-trust-extracted\") pod \"image-registry-7fbf44d9c7-zvzrx\" (UID: \"be1b9e57-7c37-40bf-90db-0699e42f5c68\") " pod="openshift-image-registry/image-registry-7fbf44d9c7-zvzrx"
Jul 29 19:18:09.264446 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.264068    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be1b9e57-7c37-40bf-90db-0699e42f5c68-trusted-ca\") pod \"image-registry-7fbf44d9c7-zvzrx\" (UID: \"be1b9e57-7c37-40bf-90db-0699e42f5c68\") " pod="openshift-image-registry/image-registry-7fbf44d9c7-zvzrx"
Jul 29 19:18:09.264446 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.264224    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93412f51-94fe-40a4-b1bb-10f5bb4ee2a9-tmp-dir\") pod \"dns-default-r9gkq\" (UID: \"93412f51-94fe-40a4-b1bb-10f5bb4ee2a9\") " pod="openshift-dns/dns-default-r9gkq"
Jul 29 19:18:09.265303 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.265282    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/402395a1-90da-48c5-b125-8ebb36f3e199-hub\") pod \"cluster-proxy-proxy-agent-58d7cfb778-rzp77\" (UID: \"402395a1-90da-48c5-b125-8ebb36f3e199\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d7cfb778-rzp77"
Jul 29 19:18:09.266113 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.265624    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/93412f51-94fe-40a4-b1bb-10f5bb4ee2a9-metrics-tls\") pod \"dns-default-r9gkq\" (UID: \"93412f51-94fe-40a4-b1bb-10f5bb4ee2a9\") " pod="openshift-dns/dns-default-r9gkq"
Jul 29 19:18:09.266113 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.265969    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/93412f51-94fe-40a4-b1bb-10f5bb4ee2a9-config-volume\") pod \"dns-default-r9gkq\" (UID: \"93412f51-94fe-40a4-b1bb-10f5bb4ee2a9\") " pod="openshift-dns/dns-default-r9gkq"
Jul 29 19:18:09.266113 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.266075    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/402395a1-90da-48c5-b125-8ebb36f3e199-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-58d7cfb778-rzp77\" (UID: \"402395a1-90da-48c5-b125-8ebb36f3e199\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d7cfb778-rzp77"
Jul 29 19:18:09.266113 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.266081    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/402395a1-90da-48c5-b125-8ebb36f3e199-additional-service-ca\") pod \"cluster-proxy-proxy-agent-58d7cfb778-rzp77\" (UID: \"402395a1-90da-48c5-b125-8ebb36f3e199\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d7cfb778-rzp77"
Jul 29 19:18:09.266412 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.266360    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/402395a1-90da-48c5-b125-8ebb36f3e199-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-58d7cfb778-rzp77\" (UID: \"402395a1-90da-48c5-b125-8ebb36f3e199\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d7cfb778-rzp77"
Jul 29 19:18:09.266412 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.266389    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be1b9e57-7c37-40bf-90db-0699e42f5c68-installation-pull-secrets\") pod \"image-registry-7fbf44d9c7-zvzrx\" (UID: \"be1b9e57-7c37-40bf-90db-0699e42f5c68\") " pod="openshift-image-registry/image-registry-7fbf44d9c7-zvzrx"
Jul 29 19:18:09.267005 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.266964    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/402395a1-90da-48c5-b125-8ebb36f3e199-ca\") pod \"cluster-proxy-proxy-agent-58d7cfb778-rzp77\" (UID: \"402395a1-90da-48c5-b125-8ebb36f3e199\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d7cfb778-rzp77"
Jul 29 19:18:09.267094 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.267020    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be1b9e57-7c37-40bf-90db-0699e42f5c68-image-registry-private-configuration\") pod \"image-registry-7fbf44d9c7-zvzrx\" (UID: \"be1b9e57-7c37-40bf-90db-0699e42f5c68\") " pod="openshift-image-registry/image-registry-7fbf44d9c7-zvzrx"
Jul 29 19:18:09.267674 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.267652    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d7da528d-c57f-471d-92a3-821b1045f55f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-c797f987d-zw59v\" (UID: \"d7da528d-c57f-471d-92a3-821b1045f55f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c797f987d-zw59v"
Jul 29 19:18:09.267793 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.267760    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be1b9e57-7c37-40bf-90db-0699e42f5c68-registry-tls\") pod \"image-registry-7fbf44d9c7-zvzrx\" (UID: \"be1b9e57-7c37-40bf-90db-0699e42f5c68\") " pod="openshift-image-registry/image-registry-7fbf44d9c7-zvzrx"
Jul 29 19:18:09.268022 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.268007    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bad3b996-769f-49cb-8edc-096993a68736-klusterlet-config\") pod \"klusterlet-addon-workmgr-64ddfd448b-wp5n4\" (UID: \"bad3b996-769f-49cb-8edc-096993a68736\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64ddfd448b-wp5n4"
Jul 29 19:18:09.268554 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.268539    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/38b8311f-6cea-4246-ac0a-619be43452e0-cert\") pod \"ingress-canary-hr2f6\" (UID: \"38b8311f-6cea-4246-ac0a-619be43452e0\") " pod="openshift-ingress-canary/ingress-canary-hr2f6"
Jul 29 19:18:09.281056 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.281034    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slq4q\" (UniqueName: \"kubernetes.io/projected/38b8311f-6cea-4246-ac0a-619be43452e0-kube-api-access-slq4q\") pod \"ingress-canary-hr2f6\" (UID: \"38b8311f-6cea-4246-ac0a-619be43452e0\") " pod="openshift-ingress-canary/ingress-canary-hr2f6"
Jul 29 19:18:09.281316 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.281292    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dblpj\" (UniqueName: \"kubernetes.io/projected/d7da528d-c57f-471d-92a3-821b1045f55f-kube-api-access-dblpj\") pod \"managed-serviceaccount-addon-agent-c797f987d-zw59v\" (UID: \"d7da528d-c57f-471d-92a3-821b1045f55f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c797f987d-zw59v"
Jul 29 19:18:09.281413 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.281292    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2vlq\" (UniqueName: \"kubernetes.io/projected/402395a1-90da-48c5-b125-8ebb36f3e199-kube-api-access-k2vlq\") pod \"cluster-proxy-proxy-agent-58d7cfb778-rzp77\" (UID: \"402395a1-90da-48c5-b125-8ebb36f3e199\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d7cfb778-rzp77"
Jul 29 19:18:09.281413 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.281353    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-475wf\" (UniqueName: \"kubernetes.io/projected/93412f51-94fe-40a4-b1bb-10f5bb4ee2a9-kube-api-access-475wf\") pod \"dns-default-r9gkq\" (UID: \"93412f51-94fe-40a4-b1bb-10f5bb4ee2a9\") " pod="openshift-dns/dns-default-r9gkq"
Jul 29 19:18:09.282615 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.282600    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be1b9e57-7c37-40bf-90db-0699e42f5c68-bound-sa-token\") pod \"image-registry-7fbf44d9c7-zvzrx\" (UID: \"be1b9e57-7c37-40bf-90db-0699e42f5c68\") " pod="openshift-image-registry/image-registry-7fbf44d9c7-zvzrx"
Jul 29 19:18:09.284690 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.284669    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkbtj\" (UniqueName: \"kubernetes.io/projected/bad3b996-769f-49cb-8edc-096993a68736-kube-api-access-pkbtj\") pod \"klusterlet-addon-workmgr-64ddfd448b-wp5n4\" (UID: \"bad3b996-769f-49cb-8edc-096993a68736\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64ddfd448b-wp5n4"
Jul 29 19:18:09.284867 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.284853    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2845\" (UniqueName: \"kubernetes.io/projected/be1b9e57-7c37-40bf-90db-0699e42f5c68-kube-api-access-v2845\") pod \"image-registry-7fbf44d9c7-zvzrx\" (UID: \"be1b9e57-7c37-40bf-90db-0699e42f5c68\") " pod="openshift-image-registry/image-registry-7fbf44d9c7-zvzrx"
Jul 29 19:18:09.293591 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.293573    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-sz59c"
Jul 29 19:18:09.364590 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.364525    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7bafa333-3be5-44ae-b6a2-255fd658a919-crio-socket\") pod \"insights-runtime-extractor-nzzqf\" (UID: \"7bafa333-3be5-44ae-b6a2-255fd658a919\") " pod="openshift-insights/insights-runtime-extractor-nzzqf"
Jul 29 19:18:09.364590 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.364566    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7bafa333-3be5-44ae-b6a2-255fd658a919-data-volume\") pod \"insights-runtime-extractor-nzzqf\" (UID: \"7bafa333-3be5-44ae-b6a2-255fd658a919\") " pod="openshift-insights/insights-runtime-extractor-nzzqf"
Jul 29 19:18:09.364776 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.364603    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6p5f\" (UniqueName: \"kubernetes.io/projected/7bafa333-3be5-44ae-b6a2-255fd658a919-kube-api-access-g6p5f\") pod \"insights-runtime-extractor-nzzqf\" (UID: \"7bafa333-3be5-44ae-b6a2-255fd658a919\") " pod="openshift-insights/insights-runtime-extractor-nzzqf"
Jul 29 19:18:09.364776 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.364668    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7bafa333-3be5-44ae-b6a2-255fd658a919-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nzzqf\" (UID: \"7bafa333-3be5-44ae-b6a2-255fd658a919\") " pod="openshift-insights/insights-runtime-extractor-nzzqf"
Jul 29 19:18:09.364776 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.364695    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7bafa333-3be5-44ae-b6a2-255fd658a919-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nzzqf\" (UID: \"7bafa333-3be5-44ae-b6a2-255fd658a919\") " pod="openshift-insights/insights-runtime-extractor-nzzqf"
Jul 29 19:18:09.364964 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.364934    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7bafa333-3be5-44ae-b6a2-255fd658a919-crio-socket\") pod \"insights-runtime-extractor-nzzqf\" (UID: \"7bafa333-3be5-44ae-b6a2-255fd658a919\") " pod="openshift-insights/insights-runtime-extractor-nzzqf"
Jul 29 19:18:09.365226 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.365204    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7bafa333-3be5-44ae-b6a2-255fd658a919-data-volume\") pod \"insights-runtime-extractor-nzzqf\" (UID: \"7bafa333-3be5-44ae-b6a2-255fd658a919\") " pod="openshift-insights/insights-runtime-extractor-nzzqf"
Jul 29 19:18:09.365562 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.365538    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7bafa333-3be5-44ae-b6a2-255fd658a919-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nzzqf\" (UID: \"7bafa333-3be5-44ae-b6a2-255fd658a919\") " pod="openshift-insights/insights-runtime-extractor-nzzqf"
Jul 29 19:18:09.367207 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.367184    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7bafa333-3be5-44ae-b6a2-255fd658a919-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nzzqf\" (UID: \"7bafa333-3be5-44ae-b6a2-255fd658a919\") " pod="openshift-insights/insights-runtime-extractor-nzzqf"
Jul 29 19:18:09.375928 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.375908    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6p5f\" (UniqueName: \"kubernetes.io/projected/7bafa333-3be5-44ae-b6a2-255fd658a919-kube-api-access-g6p5f\") pod \"insights-runtime-extractor-nzzqf\" (UID: \"7bafa333-3be5-44ae-b6a2-255fd658a919\") " pod="openshift-insights/insights-runtime-extractor-nzzqf"
Jul 29 19:18:09.419424 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.419393    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-sz59c"]
Jul 29 19:18:09.422026 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:18:09.421997    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78662159_1110_4413_ab45_219ed9d4ede9.slice/crio-16cc451b88257793a15cbda10c98db79a4422d1a398106d74eb7f39772402ead WatchSource:0}: Error finding container 16cc451b88257793a15cbda10c98db79a4422d1a398106d74eb7f39772402ead: Status 404 returned error can't find the container with id 16cc451b88257793a15cbda10c98db79a4422d1a398106d74eb7f39772402ead
Jul 29 19:18:09.439218 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.439184    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64ddfd448b-wp5n4"
Jul 29 19:18:09.445548 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.445528    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hr2f6"
Jul 29 19:18:09.453450 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.453434    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r9gkq"
Jul 29 19:18:09.468340 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.468315    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c797f987d-zw59v"
Jul 29 19:18:09.477547 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.477376    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d7cfb778-rzp77"
Jul 29 19:18:09.510187 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.509958    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nzzqf"
Jul 29 19:18:09.628669 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.628585    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-64ddfd448b-wp5n4"]
Jul 29 19:18:09.640591 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:18:09.640561    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbad3b996_769f_49cb_8edc_096993a68736.slice/crio-8b5d756bc8f0ae63c7c295f18c69ad32b2184967b9432ae244e2dcb3d6686a9a WatchSource:0}: Error finding container 8b5d756bc8f0ae63c7c295f18c69ad32b2184967b9432ae244e2dcb3d6686a9a: Status 404 returned error can't find the container with id 8b5d756bc8f0ae63c7c295f18c69ad32b2184967b9432ae244e2dcb3d6686a9a
Jul 29 19:18:09.690510 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.690477    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c797f987d-zw59v"]
Jul 29 19:18:09.693163 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:18:09.693137    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7da528d_c57f_471d_92a3_821b1045f55f.slice/crio-00167715ab0836fbe5666dd6002713a29d0e73f2df3fb17e8da52d2d2e7feeef WatchSource:0}: Error finding container 00167715ab0836fbe5666dd6002713a29d0e73f2df3fb17e8da52d2d2e7feeef: Status 404 returned error can't find the container with id 00167715ab0836fbe5666dd6002713a29d0e73f2df3fb17e8da52d2d2e7feeef
Jul 29 19:18:09.698355 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.698285    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c797f987d-zw59v" event={"ID":"d7da528d-c57f-471d-92a3-821b1045f55f","Type":"ContainerStarted","Data":"00167715ab0836fbe5666dd6002713a29d0e73f2df3fb17e8da52d2d2e7feeef"}
Jul 29 19:18:09.699412 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.699389    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64ddfd448b-wp5n4" event={"ID":"bad3b996-769f-49cb-8edc-096993a68736","Type":"ContainerStarted","Data":"8b5d756bc8f0ae63c7c295f18c69ad32b2184967b9432ae244e2dcb3d6686a9a"}
Jul 29 19:18:09.700922 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.700904    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fbf44d9c7-zvzrx"
Jul 29 19:18:09.700992 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.700932    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-sz59c" event={"ID":"78662159-1110-4413-ab45-219ed9d4ede9","Type":"ContainerStarted","Data":"16cc451b88257793a15cbda10c98db79a4422d1a398106d74eb7f39772402ead"}
Jul 29 19:18:09.707115 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.707098    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fbf44d9c7-zvzrx"
Jul 29 19:18:09.714861 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.714843    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nzzqf"]
Jul 29 19:18:09.718494 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:18:09.718472    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bafa333_3be5_44ae_b6a2_255fd658a919.slice/crio-c07f9eb5f9567c0cd2e2e899651ee5541aa4a7836ebbf8638ec222a8d902b67c WatchSource:0}: Error finding container c07f9eb5f9567c0cd2e2e899651ee5541aa4a7836ebbf8638ec222a8d902b67c: Status 404 returned error can't find the container with id c07f9eb5f9567c0cd2e2e899651ee5541aa4a7836ebbf8638ec222a8d902b67c
Jul 29 19:18:09.767861 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.767790    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be1b9e57-7c37-40bf-90db-0699e42f5c68-installation-pull-secrets\") pod \"be1b9e57-7c37-40bf-90db-0699e42f5c68\" (UID: \"be1b9e57-7c37-40bf-90db-0699e42f5c68\") "
Jul 29 19:18:09.767861 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.767846    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be1b9e57-7c37-40bf-90db-0699e42f5c68-ca-trust-extracted\") pod \"be1b9e57-7c37-40bf-90db-0699e42f5c68\" (UID: \"be1b9e57-7c37-40bf-90db-0699e42f5c68\") "
Jul 29 19:18:09.768007 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.767955    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be1b9e57-7c37-40bf-90db-0699e42f5c68-image-registry-private-configuration\") pod \"be1b9e57-7c37-40bf-90db-0699e42f5c68\" (UID: \"be1b9e57-7c37-40bf-90db-0699e42f5c68\") "
Jul 29 19:18:09.768007 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.767991    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be1b9e57-7c37-40bf-90db-0699e42f5c68-registry-tls\") pod \"be1b9e57-7c37-40bf-90db-0699e42f5c68\" (UID: \"be1b9e57-7c37-40bf-90db-0699e42f5c68\") "
Jul 29 19:18:09.768114 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.768026    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be1b9e57-7c37-40bf-90db-0699e42f5c68-trusted-ca\") pod \"be1b9e57-7c37-40bf-90db-0699e42f5c68\" (UID: \"be1b9e57-7c37-40bf-90db-0699e42f5c68\") "
Jul 29 19:18:09.768114 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.768050    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be1b9e57-7c37-40bf-90db-0699e42f5c68-bound-sa-token\") pod \"be1b9e57-7c37-40bf-90db-0699e42f5c68\" (UID: \"be1b9e57-7c37-40bf-90db-0699e42f5c68\") "
Jul 29 19:18:09.768114 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.768074    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2845\" (UniqueName: \"kubernetes.io/projected/be1b9e57-7c37-40bf-90db-0699e42f5c68-kube-api-access-v2845\") pod \"be1b9e57-7c37-40bf-90db-0699e42f5c68\" (UID: \"be1b9e57-7c37-40bf-90db-0699e42f5c68\") "
Jul 29 19:18:09.768114 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.768081    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be1b9e57-7c37-40bf-90db-0699e42f5c68-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "be1b9e57-7c37-40bf-90db-0699e42f5c68" (UID: "be1b9e57-7c37-40bf-90db-0699e42f5c68"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:18:09.768114 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.768106    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be1b9e57-7c37-40bf-90db-0699e42f5c68-registry-certificates\") pod \"be1b9e57-7c37-40bf-90db-0699e42f5c68\" (UID: \"be1b9e57-7c37-40bf-90db-0699e42f5c68\") "
Jul 29 19:18:09.768367 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.768350    2583 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be1b9e57-7c37-40bf-90db-0699e42f5c68-ca-trust-extracted\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:18:09.768695 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.768671    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be1b9e57-7c37-40bf-90db-0699e42f5c68-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "be1b9e57-7c37-40bf-90db-0699e42f5c68" (UID: "be1b9e57-7c37-40bf-90db-0699e42f5c68"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:18:09.768768 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.768725    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be1b9e57-7c37-40bf-90db-0699e42f5c68-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "be1b9e57-7c37-40bf-90db-0699e42f5c68" (UID: "be1b9e57-7c37-40bf-90db-0699e42f5c68"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:18:09.769925 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.769902    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be1b9e57-7c37-40bf-90db-0699e42f5c68-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "be1b9e57-7c37-40bf-90db-0699e42f5c68" (UID: "be1b9e57-7c37-40bf-90db-0699e42f5c68"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:18:09.770074 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.770053    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be1b9e57-7c37-40bf-90db-0699e42f5c68-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "be1b9e57-7c37-40bf-90db-0699e42f5c68" (UID: "be1b9e57-7c37-40bf-90db-0699e42f5c68"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:18:09.770166 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.770082    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be1b9e57-7c37-40bf-90db-0699e42f5c68-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "be1b9e57-7c37-40bf-90db-0699e42f5c68" (UID: "be1b9e57-7c37-40bf-90db-0699e42f5c68"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:18:09.770438 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.770419    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be1b9e57-7c37-40bf-90db-0699e42f5c68-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "be1b9e57-7c37-40bf-90db-0699e42f5c68" (UID: "be1b9e57-7c37-40bf-90db-0699e42f5c68"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:18:09.770497 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.770433    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be1b9e57-7c37-40bf-90db-0699e42f5c68-kube-api-access-v2845" (OuterVolumeSpecName: "kube-api-access-v2845") pod "be1b9e57-7c37-40bf-90db-0699e42f5c68" (UID: "be1b9e57-7c37-40bf-90db-0699e42f5c68"). InnerVolumeSpecName "kube-api-access-v2845". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:18:09.853873 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.853774    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r9gkq"]
Jul 29 19:18:09.856536 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.856515    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hr2f6"]
Jul 29 19:18:09.857835 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:18:09.857794    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93412f51_94fe_40a4_b1bb_10f5bb4ee2a9.slice/crio-9cb89232f8bba241aa57f56b0a637a9e21d15ec0e9ee289ecb6c37da019641e6 WatchSource:0}: Error finding container 9cb89232f8bba241aa57f56b0a637a9e21d15ec0e9ee289ecb6c37da019641e6: Status 404 returned error can't find the container with id 9cb89232f8bba241aa57f56b0a637a9e21d15ec0e9ee289ecb6c37da019641e6
Jul 29 19:18:09.859258 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:18:09.859236    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38b8311f_6cea_4246_ac0a_619be43452e0.slice/crio-965f6755dd63966c199c6c5c94e84043c5a3df588bd23ebad89e5b49e3596625 WatchSource:0}: Error finding container 965f6755dd63966c199c6c5c94e84043c5a3df588bd23ebad89e5b49e3596625: Status 404 returned error can't find the container with id 965f6755dd63966c199c6c5c94e84043c5a3df588bd23ebad89e5b49e3596625
Jul 29 19:18:09.868634 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.868614    2583 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be1b9e57-7c37-40bf-90db-0699e42f5c68-image-registry-private-configuration\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:18:09.868698 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.868636    2583 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be1b9e57-7c37-40bf-90db-0699e42f5c68-registry-tls\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:18:09.868698 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.868645    2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be1b9e57-7c37-40bf-90db-0699e42f5c68-trusted-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:18:09.868698 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.868654    2583 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be1b9e57-7c37-40bf-90db-0699e42f5c68-bound-sa-token\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:18:09.868698 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.868662    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2845\" (UniqueName: \"kubernetes.io/projected/be1b9e57-7c37-40bf-90db-0699e42f5c68-kube-api-access-v2845\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:18:09.868698 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.868670    2583 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be1b9e57-7c37-40bf-90db-0699e42f5c68-registry-certificates\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:18:09.868698 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.868677    2583 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be1b9e57-7c37-40bf-90db-0699e42f5c68-installation-pull-secrets\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:18:09.890309 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:09.890280    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d7cfb778-rzp77"]
Jul 29 19:18:09.892959 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:18:09.892934    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod402395a1_90da_48c5_b125_8ebb36f3e199.slice/crio-a697d142c18fd4e3683367b1c0d4a9e2c27e3d1f1fa2a7a11524f5c96f1ccb51 WatchSource:0}: Error finding container a697d142c18fd4e3683367b1c0d4a9e2c27e3d1f1fa2a7a11524f5c96f1ccb51: Status 404 returned error can't find the container with id a697d142c18fd4e3683367b1c0d4a9e2c27e3d1f1fa2a7a11524f5c96f1ccb51
Jul 29 19:18:10.708441 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:10.707690    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r9gkq" event={"ID":"93412f51-94fe-40a4-b1bb-10f5bb4ee2a9","Type":"ContainerStarted","Data":"9cb89232f8bba241aa57f56b0a637a9e21d15ec0e9ee289ecb6c37da019641e6"}
Jul 29 19:18:10.712113 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:10.712034    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d7cfb778-rzp77" event={"ID":"402395a1-90da-48c5-b125-8ebb36f3e199","Type":"ContainerStarted","Data":"a697d142c18fd4e3683367b1c0d4a9e2c27e3d1f1fa2a7a11524f5c96f1ccb51"}
Jul 29 19:18:10.713545 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:10.713507    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hr2f6" event={"ID":"38b8311f-6cea-4246-ac0a-619be43452e0","Type":"ContainerStarted","Data":"965f6755dd63966c199c6c5c94e84043c5a3df588bd23ebad89e5b49e3596625"}
Jul 29 19:18:10.719481 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:10.717397    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fbf44d9c7-zvzrx"
Jul 29 19:18:10.719481 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:10.717958    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nzzqf" event={"ID":"7bafa333-3be5-44ae-b6a2-255fd658a919","Type":"ContainerStarted","Data":"c07f9eb5f9567c0cd2e2e899651ee5541aa4a7836ebbf8638ec222a8d902b67c"}
Jul 29 19:18:10.769026 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:10.768996    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7fbf44d9c7-zvzrx"]
Jul 29 19:18:10.797695 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:10.797609    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7fbf44d9c7-zvzrx"]
Jul 29 19:18:11.341953 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:11.341719    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be1b9e57-7c37-40bf-90db-0699e42f5c68" path="/var/lib/kubelet/pods/be1b9e57-7c37-40bf-90db-0699e42f5c68/volumes"
Jul 29 19:18:11.731621 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:11.730878    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-sz59c" event={"ID":"78662159-1110-4413-ab45-219ed9d4ede9","Type":"ContainerStarted","Data":"dec1041fe0bd99f351e1271537610a20710c9b4d1bb2d85b0c9e5f49853c00d8"}
Jul 29 19:18:13.639364 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:13.639334    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jpgfd"
Jul 29 19:18:13.703074 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:13.703015    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-sz59c" podStartSLOduration=4.016409848 podStartE2EDuration="5.702990911s" totalImagesPullingTime="1.686581063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:18:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:18:11.747614376 +0000 UTC m=+58.976003503" watchObservedRunningTime="2026-07-29 19:18:13.702990911 +0000 UTC m=+60.931380040"
Jul 29 19:18:17.935173 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:17.935128    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9ab35af-cc45-4dc0-ac22-cc434c73ee9e-metrics-certs\") pod \"network-metrics-daemon-4wlg9\" (UID: \"d9ab35af-cc45-4dc0-ac22-cc434c73ee9e\") " pod="openshift-multus/network-metrics-daemon-4wlg9"
Jul 29 19:18:17.937881 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:17.937856    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 19:18:17.948380 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:17.948348    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9ab35af-cc45-4dc0-ac22-cc434c73ee9e-metrics-certs\") pod \"network-metrics-daemon-4wlg9\" (UID: \"d9ab35af-cc45-4dc0-ac22-cc434c73ee9e\") " pod="openshift-multus/network-metrics-daemon-4wlg9"
Jul 29 19:18:18.035791 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:18.035749    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwjg2\" (UniqueName: \"kubernetes.io/projected/0a4b19f9-f7aa-4860-b947-e21d47ed4b21-kube-api-access-gwjg2\") pod \"network-check-target-cmjv9\" (UID: \"0a4b19f9-f7aa-4860-b947-e21d47ed4b21\") " pod="openshift-network-diagnostics/network-check-target-cmjv9"
Jul 29 19:18:18.035984 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:18.035824    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2485c438-0e6d-4aae-bab6-8dcda418e188-original-pull-secret\") pod \"global-pull-secret-syncer-pp24q\" (UID: \"2485c438-0e6d-4aae-bab6-8dcda418e188\") " pod="kube-system/global-pull-secret-syncer-pp24q"
Jul 29 19:18:18.038628 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:18.038601    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 19:18:18.038779 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:18.038760    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 19:18:18.048661 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:18.048634    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2485c438-0e6d-4aae-bab6-8dcda418e188-original-pull-secret\") pod \"global-pull-secret-syncer-pp24q\" (UID: \"2485c438-0e6d-4aae-bab6-8dcda418e188\") " pod="kube-system/global-pull-secret-syncer-pp24q"
Jul 29 19:18:18.048661 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:18.048652    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 19:18:18.053628 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:18.053609    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp24q"
Jul 29 19:18:18.059730 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:18.059709    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwjg2\" (UniqueName: \"kubernetes.io/projected/0a4b19f9-f7aa-4860-b947-e21d47ed4b21-kube-api-access-gwjg2\") pod \"network-check-target-cmjv9\" (UID: \"0a4b19f9-f7aa-4860-b947-e21d47ed4b21\") " pod="openshift-network-diagnostics/network-check-target-cmjv9"
Jul 29 19:18:18.067968 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:18.067947    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qxkk6\""
Jul 29 19:18:18.073045 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:18.073025    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wlg9"
Jul 29 19:18:18.349214 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:18.349138    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-b2qz7\""
Jul 29 19:18:18.357229 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:18.357201    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cmjv9"
Jul 29 19:18:21.286974 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:21.286947    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4wlg9"]
Jul 29 19:18:21.529623 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:21.529494    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pp24q"]
Jul 29 19:18:21.530522 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:21.530462    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cmjv9"]
Jul 29 19:18:21.532296 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:18:21.532249    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a4b19f9_f7aa_4860_b947_e21d47ed4b21.slice/crio-07cba0f1be98a56be45a0368965ae02997318c37209b40508c97b02c1f3b6013 WatchSource:0}: Error finding container 07cba0f1be98a56be45a0368965ae02997318c37209b40508c97b02c1f3b6013: Status 404 returned error can't find the container with id 07cba0f1be98a56be45a0368965ae02997318c37209b40508c97b02c1f3b6013
Jul 29 19:18:21.533434 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:18:21.533414    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2485c438_0e6d_4aae_bab6_8dcda418e188.slice/crio-7d9fe9987a0623d685d97dbd6d279f92d76aa7322b876c78e7e1a63b151f739d WatchSource:0}: Error finding container 7d9fe9987a0623d685d97dbd6d279f92d76aa7322b876c78e7e1a63b151f739d: Status 404 returned error can't find the container with id 7d9fe9987a0623d685d97dbd6d279f92d76aa7322b876c78e7e1a63b151f739d
Jul 29 19:18:21.761419 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:21.761326    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c797f987d-zw59v" event={"ID":"d7da528d-c57f-471d-92a3-821b1045f55f","Type":"ContainerStarted","Data":"70b63575187fb095e9cc3d2aef686731c238aa89cbaa43af55fcf87e3eefc2cc"}
Jul 29 19:18:21.762767 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:21.762742    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nzzqf" event={"ID":"7bafa333-3be5-44ae-b6a2-255fd658a919","Type":"ContainerStarted","Data":"8b01de4855f89034cbc73cf6145c26ff4a344d737489b9539f74dafd92cb816a"}
Jul 29 19:18:21.763891 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:21.763870    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pp24q" event={"ID":"2485c438-0e6d-4aae-bab6-8dcda418e188","Type":"ContainerStarted","Data":"7d9fe9987a0623d685d97dbd6d279f92d76aa7322b876c78e7e1a63b151f739d"}
Jul 29 19:18:21.764985 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:21.764963    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cmjv9" event={"ID":"0a4b19f9-f7aa-4860-b947-e21d47ed4b21","Type":"ContainerStarted","Data":"07cba0f1be98a56be45a0368965ae02997318c37209b40508c97b02c1f3b6013"}
Jul 29 19:18:21.766376 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:21.766350    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64ddfd448b-wp5n4" event={"ID":"bad3b996-769f-49cb-8edc-096993a68736","Type":"ContainerStarted","Data":"a7883b5302d882a5a469809107be0afa53cb2031e8b143a16332c91235a5b489"}
Jul 29 19:18:21.766509 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:21.766491    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64ddfd448b-wp5n4"
Jul 29 19:18:21.768216 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:21.768197    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64ddfd448b-wp5n4"
Jul 29 19:18:21.768664 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:21.768648    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r9gkq" event={"ID":"93412f51-94fe-40a4-b1bb-10f5bb4ee2a9","Type":"ContainerStarted","Data":"b068b6fd13e2fd34ededae7cd3ad1e09f2925402da07512844391031062b487b"}
Jul 29 19:18:21.768725 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:21.768680    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r9gkq" event={"ID":"93412f51-94fe-40a4-b1bb-10f5bb4ee2a9","Type":"ContainerStarted","Data":"a866590784929896369974d03d36814094ec134733b55025f85af29f986e820e"}
Jul 29 19:18:21.768725 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:21.768710    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-r9gkq"
Jul 29 19:18:21.773308 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:21.773287    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d7cfb778-rzp77" event={"ID":"402395a1-90da-48c5-b125-8ebb36f3e199","Type":"ContainerStarted","Data":"10b5bd0c173a97b08bd6898501d4d7e99e7d8d0cc55a6ea39d8e063a85dc4c1c"}
Jul 29 19:18:21.773405 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:21.773317    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d7cfb778-rzp77" event={"ID":"402395a1-90da-48c5-b125-8ebb36f3e199","Type":"ContainerStarted","Data":"dcc5c4313b2f9fa960d8182d8a7964779b0c778b56773f9104d36d0553b88b36"}
Jul 29 19:18:21.773405 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:21.773330    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d7cfb778-rzp77" event={"ID":"402395a1-90da-48c5-b125-8ebb36f3e199","Type":"ContainerStarted","Data":"3635bbd9b91967620c13bbd9dd11d9a00cba2e218802952708cf4afcd5911e8c"}
Jul 29 19:18:21.775003 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:21.774983    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hr2f6" event={"ID":"38b8311f-6cea-4246-ac0a-619be43452e0","Type":"ContainerStarted","Data":"144b0978385d1bbdbe69423f44cb2c5f3d123e54a8627f1a265c43af20b6e970"}
Jul 29 19:18:21.776296 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:21.776277    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4wlg9" event={"ID":"d9ab35af-cc45-4dc0-ac22-cc434c73ee9e","Type":"ContainerStarted","Data":"2268c60af57c469284459631cb1990c4b05bc0c33f4cbb1e0025f05ece673b5c"}
Jul 29 19:18:21.787934 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:21.787885    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c797f987d-zw59v" podStartSLOduration=1.346945996 podStartE2EDuration="12.787871575s" totalImagesPullingTime="11.440925579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:18:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:18:21.786781661 +0000 UTC m=+69.015170788" watchObservedRunningTime="2026-07-29 19:18:21.787871575 +0000 UTC m=+69.016260679"
Jul 29 19:18:21.812522 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:21.812372    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hr2f6" podStartSLOduration=1.541132376 podStartE2EDuration="12.812360231s" totalImagesPullingTime="11.271227855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:18:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:18:21.81118719 +0000 UTC m=+69.039576316" watchObservedRunningTime="2026-07-29 19:18:21.812360231 +0000 UTC m=+69.040749357"
Jul 29 19:18:21.847173 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:21.847082    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64ddfd448b-wp5n4" podStartSLOduration=1.354369816 podStartE2EDuration="12.847061208s" totalImagesPullingTime="11.492691392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:18:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:18:21.846000599 +0000 UTC m=+69.074389726" watchObservedRunningTime="2026-07-29 19:18:21.847061208 +0000 UTC m=+69.075450334"
Jul 29 19:18:21.876325 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:21.876270    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-r9gkq" podStartSLOduration=1.606394743 podStartE2EDuration="12.876249941s" totalImagesPullingTime="11.269855198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:18:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:18:21.875070079 +0000 UTC m=+69.103459206" watchObservedRunningTime="2026-07-29 19:18:21.876249941 +0000 UTC m=+69.104639068"
Jul 29 19:18:21.900685 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:21.900637    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d7cfb778-rzp77" podStartSLOduration=1.659266928 podStartE2EDuration="12.900621001s" totalImagesPullingTime="11.241354073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:18:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:18:21.8997952 +0000 UTC m=+69.128184327" watchObservedRunningTime="2026-07-29 19:18:21.900621001 +0000 UTC m=+69.129010126"
Jul 29 19:18:27.105518 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:27.105483    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-d5m5t"]
Jul 29 19:18:27.109883 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:27.109860    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-d5m5t"
Jul 29 19:18:27.113133 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:27.113066    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 29 19:18:27.113255 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:27.113155    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bcfpk\""
Jul 29 19:18:27.113255 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:27.113155    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 29 19:18:27.113757 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:27.113741    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 29 19:18:27.114237 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:27.114216    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 29 19:18:27.114357 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:27.114217    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 29 19:18:27.114536 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:27.114514    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 29 19:18:27.214279 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:27.214245    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b6c4faa3-5b24-4394-a73d-915b1b9911f5-node-exporter-textfile\") pod \"node-exporter-d5m5t\" (UID: \"b6c4faa3-5b24-4394-a73d-915b1b9911f5\") " pod="openshift-monitoring/node-exporter-d5m5t"
Jul 29 19:18:27.214279 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:27.214285    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6c4faa3-5b24-4394-a73d-915b1b9911f5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d5m5t\" (UID: \"b6c4faa3-5b24-4394-a73d-915b1b9911f5\") " pod="openshift-monitoring/node-exporter-d5m5t"
Jul 29 19:18:27.214516 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:27.214312    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b6c4faa3-5b24-4394-a73d-915b1b9911f5-node-exporter-tls\") pod \"node-exporter-d5m5t\" (UID: \"b6c4faa3-5b24-4394-a73d-915b1b9911f5\") " pod="openshift-monitoring/node-exporter-d5m5t"
Jul 29 19:18:27.214516 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:27.214337    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b6c4faa3-5b24-4394-a73d-915b1b9911f5-node-exporter-accelerators-collector-config\") pod \"node-exporter-d5m5t\" (UID: \"b6c4faa3-5b24-4394-a73d-915b1b9911f5\") " pod="openshift-monitoring/node-exporter-d5m5t"
Jul 29 19:18:27.214516 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:27.214363    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6c4faa3-5b24-4394-a73d-915b1b9911f5-sys\") pod \"node-exporter-d5m5t\" (UID: \"b6c4faa3-5b24-4394-a73d-915b1b9911f5\") " pod="openshift-monitoring/node-exporter-d5m5t"
Jul 29 19:18:27.214516 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:27.214383    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rldk2\" (UniqueName: \"kubernetes.io/projected/b6c4faa3-5b24-4394-a73d-915b1b9911f5-kube-api-access-rldk2\") pod \"node-exporter-d5m5t\" (UID: \"b6c4faa3-5b24-4394-a73d-915b1b9911f5\") " pod="openshift-monitoring/node-exporter-d5m5t"
Jul 29 19:18:27.214516 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:27.214409    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b6c4faa3-5b24-4394-a73d-915b1b9911f5-node-exporter-wtmp\") pod \"node-exporter-d5m5t\" (UID: \"b6c4faa3-5b24-4394-a73d-915b1b9911f5\") " pod="openshift-monitoring/node-exporter-d5m5t"
Jul 29 19:18:27.214704 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:27.214515    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b6c4faa3-5b24-4394-a73d-915b1b9911f5-root\") pod \"node-exporter-d5m5t\" (UID: \"b6c4faa3-5b24-4394-a73d-915b1b9911f5\") " pod="openshift-monitoring/node-exporter-d5m5t"
Jul 29 19:18:27.214704 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:27.214551    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6c4faa3-5b24-4394-a73d-915b1b9911f5-metrics-client-ca\") pod \"node-exporter-d5m5t\" (UID: \"b6c4faa3-5b24-4394-a73d-915b1b9911f5\") " pod="openshift-monitoring/node-exporter-d5m5t"
Jul 29 19:18:27.315689 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:27.315653    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b6c4faa3-5b24-4394-a73d-915b1b9911f5-root\") pod \"node-exporter-d5m5t\" (UID: \"b6c4faa3-5b24-4394-a73d-915b1b9911f5\") " pod="openshift-monitoring/node-exporter-d5m5t"
Jul 29 19:18:27.315900 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:27.315710    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6c4faa3-5b24-4394-a73d-915b1b9911f5-metrics-client-ca\") pod \"node-exporter-d5m5t\" (UID: \"b6c4faa3-5b24-4394-a73d-915b1b9911f5\") " pod="openshift-monitoring/node-exporter-d5m5t"
Jul 29 19:18:27.315900 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:27.315763    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b6c4faa3-5b24-4394-a73d-915b1b9911f5-node-exporter-textfile\") pod \"node-exporter-d5m5t\" (UID: \"b6c4faa3-5b24-4394-a73d-915b1b9911f5\") " pod="openshift-monitoring/node-exporter-d5m5t"
Jul 29 19:18:27.315900 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:27.315779    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6c4faa3-5b24-4394-a73d-915b1b9911f5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d5m5t\" (UID: \"b6c4faa3-5b24-4394-a73d-915b1b9911f5\") " pod="openshift-monitoring/node-exporter-d5m5t"
Jul 29 19:18:27.315900 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:27.315791    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b6c4faa3-5b24-4394-a73d-915b1b9911f5-root\") pod \"node-exporter-d5m5t\" (UID: \"b6c4faa3-5b24-4394-a73d-915b1b9911f5\") " pod="openshift-monitoring/node-exporter-d5m5t"
Jul 29 19:18:27.315900 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:27.315794    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b6c4faa3-5b24-4394-a73d-915b1b9911f5-node-exporter-tls\") pod \"node-exporter-d5m5t\" (UID: \"b6c4faa3-5b24-4394-a73d-915b1b9911f5\") " pod="openshift-monitoring/node-exporter-d5m5t"
Jul 29 19:18:27.315900 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:27.315882    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b6c4faa3-5b24-4394-a73d-915b1b9911f5-node-exporter-accelerators-collector-config\") pod \"node-exporter-d5m5t\" (UID: \"b6c4faa3-5b24-4394-a73d-915b1b9911f5\") " pod="openshift-monitoring/node-exporter-d5m5t"
Jul 29 19:18:27.316275 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:27.315927    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6c4faa3-5b24-4394-a73d-915b1b9911f5-sys\") pod \"node-exporter-d5m5t\" (UID: \"b6c4faa3-5b24-4394-a73d-915b1b9911f5\") " pod="openshift-monitoring/node-exporter-d5m5t"
Jul 29 19:18:27.316275 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:27.315963    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rldk2\" (UniqueName: \"kubernetes.io/projected/b6c4faa3-5b24-4394-a73d-915b1b9911f5-kube-api-access-rldk2\") pod \"node-exporter-d5m5t\" (UID: \"b6c4faa3-5b24-4394-a73d-915b1b9911f5\") " pod="openshift-monitoring/node-exporter-d5m5t"
Jul 29 19:18:27.316275 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:27.315991    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b6c4faa3-5b24-4394-a73d-915b1b9911f5-node-exporter-wtmp\") pod \"node-exporter-d5m5t\" (UID: \"b6c4faa3-5b24-4394-a73d-915b1b9911f5\") " pod="openshift-monitoring/node-exporter-d5m5t"
Jul 29 19:18:27.316275 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:27.316024    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6c4faa3-5b24-4394-a73d-915b1b9911f5-sys\") pod \"node-exporter-d5m5t\" (UID: \"b6c4faa3-5b24-4394-a73d-915b1b9911f5\") " pod="openshift-monitoring/node-exporter-d5m5t"
Jul 29 19:18:27.316275 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:27.316219    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b6c4faa3-5b24-4394-a73d-915b1b9911f5-node-exporter-textfile\") pod \"node-exporter-d5m5t\" (UID: \"b6c4faa3-5b24-4394-a73d-915b1b9911f5\") " pod="openshift-monitoring/node-exporter-d5m5t"
Jul 29 19:18:27.316521 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:27.316436    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6c4faa3-5b24-4394-a73d-915b1b9911f5-metrics-client-ca\") pod \"node-exporter-d5m5t\" (UID: \"b6c4faa3-5b24-4394-a73d-915b1b9911f5\") " pod="openshift-monitoring/node-exporter-d5m5t"
Jul 29 19:18:27.316521 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:27.316449    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b6c4faa3-5b24-4394-a73d-915b1b9911f5-node-exporter-wtmp\") pod \"node-exporter-d5m5t\" (UID: \"b6c4faa3-5b24-4394-a73d-915b1b9911f5\") " pod="openshift-monitoring/node-exporter-d5m5t"
Jul 29 19:18:27.316521 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:27.316493    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b6c4faa3-5b24-4394-a73d-915b1b9911f5-node-exporter-accelerators-collector-config\") pod \"node-exporter-d5m5t\" (UID: \"b6c4faa3-5b24-4394-a73d-915b1b9911f5\") " pod="openshift-monitoring/node-exporter-d5m5t"
Jul 29 19:18:27.318827 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:27.318777    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6c4faa3-5b24-4394-a73d-915b1b9911f5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d5m5t\" (UID: \"b6c4faa3-5b24-4394-a73d-915b1b9911f5\") " pod="openshift-monitoring/node-exporter-d5m5t"
Jul 29 19:18:27.318943 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:27.318907    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b6c4faa3-5b24-4394-a73d-915b1b9911f5-node-exporter-tls\") pod \"node-exporter-d5m5t\" (UID: \"b6c4faa3-5b24-4394-a73d-915b1b9911f5\") " pod="openshift-monitoring/node-exporter-d5m5t"
Jul 29 19:18:27.327309 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:27.327288    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rldk2\" (UniqueName: \"kubernetes.io/projected/b6c4faa3-5b24-4394-a73d-915b1b9911f5-kube-api-access-rldk2\") pod \"node-exporter-d5m5t\" (UID: \"b6c4faa3-5b24-4394-a73d-915b1b9911f5\") " pod="openshift-monitoring/node-exporter-d5m5t"
Jul 29 19:18:27.420980 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:27.420944    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-d5m5t"
Jul 29 19:18:27.932974 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:18:27.932936    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6c4faa3_5b24_4394_a73d_915b1b9911f5.slice/crio-451bda598107fdec89a5f25e44e6c282fd69a1ebffdf876d2ccf40aead9e6e5e WatchSource:0}: Error finding container 451bda598107fdec89a5f25e44e6c282fd69a1ebffdf876d2ccf40aead9e6e5e: Status 404 returned error can't find the container with id 451bda598107fdec89a5f25e44e6c282fd69a1ebffdf876d2ccf40aead9e6e5e
Jul 29 19:18:28.806236 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:28.806194    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nzzqf" event={"ID":"7bafa333-3be5-44ae-b6a2-255fd658a919","Type":"ContainerStarted","Data":"7ccdbf34673fb37dfb58e286f9801d60a1f906be5136cd45cda7e58005673db8"}
Jul 29 19:18:28.807836 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:28.807789    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pp24q" event={"ID":"2485c438-0e6d-4aae-bab6-8dcda418e188","Type":"ContainerStarted","Data":"6d7f6b48032ef698210a645711222b8f276918e641d9cbd96b8572b8987621c0"}
Jul 29 19:18:28.809501 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:28.809469    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cmjv9" event={"ID":"0a4b19f9-f7aa-4860-b947-e21d47ed4b21","Type":"ContainerStarted","Data":"b2f0de6d88c109411565fdb4a2764117936de92811f6d1e6c6585b8c158c1658"}
Jul 29 19:18:28.809623 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:28.809592    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cmjv9"
Jul 29 19:18:28.810839 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:28.810818    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d5m5t" event={"ID":"b6c4faa3-5b24-4394-a73d-915b1b9911f5","Type":"ContainerStarted","Data":"451bda598107fdec89a5f25e44e6c282fd69a1ebffdf876d2ccf40aead9e6e5e"}
Jul 29 19:18:28.812642 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:28.812618    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4wlg9" event={"ID":"d9ab35af-cc45-4dc0-ac22-cc434c73ee9e","Type":"ContainerStarted","Data":"3cd2aeef268014dd522c940cc02decb9585665825c8794bde118a2e5d86beb2b"}
Jul 29 19:18:28.812741 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:28.812662    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4wlg9" event={"ID":"d9ab35af-cc45-4dc0-ac22-cc434c73ee9e","Type":"ContainerStarted","Data":"185152ac94b209d688f1b2adb94d4826443419b460e3e8ee32f9361eb4cca506"}
Jul 29 19:18:28.824358 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:28.824301    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pp24q" podStartSLOduration=69.431078122 podStartE2EDuration="1m15.824280185s" totalImagesPullingTime="6.393202063s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:17:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:18:28.823592697 +0000 UTC m=+76.051981843" watchObservedRunningTime="2026-07-29 19:18:28.824280185 +0000 UTC m=+76.052669312"
Jul 29 19:18:28.842554 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:28.842494    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-cmjv9" podStartSLOduration=69.448536825 podStartE2EDuration="1m15.842454099s" totalImagesPullingTime="6.393917274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:17:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:18:28.841515403 +0000 UTC m=+76.069904529" watchObservedRunningTime="2026-07-29 19:18:28.842454099 +0000 UTC m=+76.070843216"
Jul 29 19:18:28.858787 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:28.858721    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4wlg9" podStartSLOduration=69.239158803 podStartE2EDuration="1m15.858699296s" totalImagesPullingTime="6.619540493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:17:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:18:28.857845388 +0000 UTC m=+76.086234518" watchObservedRunningTime="2026-07-29 19:18:28.858699296 +0000 UTC m=+76.087088425"
Jul 29 19:18:29.818225 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:29.818134    2583 generic.go:356] "Generic (PLEG): container finished" podID="b6c4faa3-5b24-4394-a73d-915b1b9911f5" containerID="e4c2bbf619b1994ef76c2b2e02a0bf19e9d85c0f7129a0d9f6dbc94ab397b317" exitCode=0
Jul 29 19:18:29.818706 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:29.818236    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d5m5t" event={"ID":"b6c4faa3-5b24-4394-a73d-915b1b9911f5","Type":"ContainerDied","Data":"e4c2bbf619b1994ef76c2b2e02a0bf19e9d85c0f7129a0d9f6dbc94ab397b317"}
Jul 29 19:18:31.783588 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:31.783505    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-r9gkq"
Jul 29 19:18:31.828500 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:31.828464    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nzzqf" event={"ID":"7bafa333-3be5-44ae-b6a2-255fd658a919","Type":"ContainerStarted","Data":"ca5542974f04b27e2ff6b831dc4c2460a105c1c76b6e01cf8432f068944a8245"}
Jul 29 19:18:31.830722 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:31.830693    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d5m5t" event={"ID":"b6c4faa3-5b24-4394-a73d-915b1b9911f5","Type":"ContainerStarted","Data":"087e907637398fbd3804d9949112fc64bb3c9c4d9da68c60ff4ff71f9038715e"}
Jul 29 19:18:31.830880 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:31.830743    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d5m5t" event={"ID":"b6c4faa3-5b24-4394-a73d-915b1b9911f5","Type":"ContainerStarted","Data":"ecba407013ef6811cb2bd9a42d21674d9bd3cb5098938857dd38f632c9b9e7b3"}
Jul 29 19:18:31.846618 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:31.846576    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-nzzqf" podStartSLOduration=1.20471808 podStartE2EDuration="22.846562955s" totalImagesPullingTime="21.641844875s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:18:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:18:31.845272719 +0000 UTC m=+79.073661845" watchObservedRunningTime="2026-07-29 19:18:31.846562955 +0000 UTC m=+79.074952081"
Jul 29 19:18:31.869691 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:31.869648    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-d5m5t" podStartSLOduration=3.209186991 podStartE2EDuration="4.869634909s" totalImagesPullingTime="1.586858815s" totalInitContainerRuntime="73.589103ms" isStatefulPod=true podCreationTimestamp="2026-07-29 19:18:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:18:31.869051373 +0000 UTC m=+79.097440498" watchObservedRunningTime="2026-07-29 19:18:31.869634909 +0000 UTC m=+79.098024034"
Jul 29 19:18:59.821082 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:18:59.821052    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-cmjv9"
Jul 29 19:22:13.152675 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:13.152640    2583 kubelet.go:1648] "Image garbage collection succeeded"
Jul 29 19:22:32.352488 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:32.352454    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hlznp"]
Jul 29 19:22:32.354289 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:32.354273    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hlznp"
Jul 29 19:22:32.356489 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:32.356469    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 29 19:22:32.357029 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:32.357011    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-jbtnn\""
Jul 29 19:22:32.357113 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:32.357043    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 29 19:22:32.363579 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:32.363556    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hlznp"]
Jul 29 19:22:32.429452 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:32.429421    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/426278c0-ce22-43df-8830-06789715ae93-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hlznp\" (UID: \"426278c0-ce22-43df-8830-06789715ae93\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hlznp"
Jul 29 19:22:32.429629 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:32.429475    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbjc8\" (UniqueName: \"kubernetes.io/projected/426278c0-ce22-43df-8830-06789715ae93-kube-api-access-nbjc8\") pod \"openshift-lws-operator-fd8ccff4c-hlznp\" (UID: \"426278c0-ce22-43df-8830-06789715ae93\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hlznp"
Jul 29 19:22:32.530343 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:32.530314    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbjc8\" (UniqueName: \"kubernetes.io/projected/426278c0-ce22-43df-8830-06789715ae93-kube-api-access-nbjc8\") pod \"openshift-lws-operator-fd8ccff4c-hlznp\" (UID: \"426278c0-ce22-43df-8830-06789715ae93\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hlznp"
Jul 29 19:22:32.530449 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:32.530361    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/426278c0-ce22-43df-8830-06789715ae93-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hlznp\" (UID: \"426278c0-ce22-43df-8830-06789715ae93\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hlznp"
Jul 29 19:22:32.530684 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:32.530668    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/426278c0-ce22-43df-8830-06789715ae93-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hlznp\" (UID: \"426278c0-ce22-43df-8830-06789715ae93\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hlznp"
Jul 29 19:22:32.538924 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:32.538897    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbjc8\" (UniqueName: \"kubernetes.io/projected/426278c0-ce22-43df-8830-06789715ae93-kube-api-access-nbjc8\") pod \"openshift-lws-operator-fd8ccff4c-hlznp\" (UID: \"426278c0-ce22-43df-8830-06789715ae93\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hlznp"
Jul 29 19:22:32.662542 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:32.662506    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hlznp"
Jul 29 19:22:32.781893 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:32.781862    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hlznp"]
Jul 29 19:22:32.785098 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:22:32.785071    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod426278c0_ce22_43df_8830_06789715ae93.slice/crio-bec824dc8e90c42059b82af4552af3390caa362261ce08c9e6c5e3af91c75939 WatchSource:0}: Error finding container bec824dc8e90c42059b82af4552af3390caa362261ce08c9e6c5e3af91c75939: Status 404 returned error can't find the container with id bec824dc8e90c42059b82af4552af3390caa362261ce08c9e6c5e3af91c75939
Jul 29 19:22:32.786986 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:32.786971    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 19:22:33.593337 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:33.593286    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hlznp" event={"ID":"426278c0-ce22-43df-8830-06789715ae93","Type":"ContainerStarted","Data":"bec824dc8e90c42059b82af4552af3390caa362261ce08c9e6c5e3af91c75939"}
Jul 29 19:22:35.602264 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:35.602174    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hlznp" event={"ID":"426278c0-ce22-43df-8830-06789715ae93","Type":"ContainerStarted","Data":"d65dc94ec66baebe8370c9c9890e356082cd8cbf50a793cbdb8b6d88d1e51b22"}
Jul 29 19:22:35.627311 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:35.627259    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hlznp" podStartSLOduration=1.144872364 podStartE2EDuration="3.627244707s" totalImagesPullingTime="2.482372343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:22:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:22:35.626535962 +0000 UTC m=+322.854925091" watchObservedRunningTime="2026-07-29 19:22:35.627244707 +0000 UTC m=+322.855633832"
Jul 29 19:22:52.032872 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:52.032832    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-667b7bc75f-g6jzm"]
Jul 29 19:22:52.035091 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:52.035069    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-667b7bc75f-g6jzm"
Jul 29 19:22:52.041907 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:52.041885    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-mq2dv\""
Jul 29 19:22:52.042005 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:52.041924    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 29 19:22:52.043075 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:52.043061    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 29 19:22:52.043129 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:52.043065    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 29 19:22:52.043401 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:52.043387    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 29 19:22:52.061755 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:52.061728    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-667b7bc75f-g6jzm"]
Jul 29 19:22:52.172700 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:52.172664    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cae6c80b-633e-43ba-b0e9-0045ebd3fe37-apiservice-cert\") pod \"opendatahub-operator-controller-manager-667b7bc75f-g6jzm\" (UID: \"cae6c80b-633e-43ba-b0e9-0045ebd3fe37\") " pod="opendatahub/opendatahub-operator-controller-manager-667b7bc75f-g6jzm"
Jul 29 19:22:52.172910 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:52.172736    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cae6c80b-633e-43ba-b0e9-0045ebd3fe37-tmp\") pod \"opendatahub-operator-controller-manager-667b7bc75f-g6jzm\" (UID: \"cae6c80b-633e-43ba-b0e9-0045ebd3fe37\") " pod="opendatahub/opendatahub-operator-controller-manager-667b7bc75f-g6jzm"
Jul 29 19:22:52.172910 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:52.172770    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cae6c80b-633e-43ba-b0e9-0045ebd3fe37-webhook-cert\") pod \"opendatahub-operator-controller-manager-667b7bc75f-g6jzm\" (UID: \"cae6c80b-633e-43ba-b0e9-0045ebd3fe37\") " pod="opendatahub/opendatahub-operator-controller-manager-667b7bc75f-g6jzm"
Jul 29 19:22:52.172910 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:52.172833    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cae6c80b-633e-43ba-b0e9-0045ebd3fe37-manifests\") pod \"opendatahub-operator-controller-manager-667b7bc75f-g6jzm\" (UID: \"cae6c80b-633e-43ba-b0e9-0045ebd3fe37\") " pod="opendatahub/opendatahub-operator-controller-manager-667b7bc75f-g6jzm"
Jul 29 19:22:52.172910 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:52.172855    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2h92\" (UniqueName: \"kubernetes.io/projected/cae6c80b-633e-43ba-b0e9-0045ebd3fe37-kube-api-access-t2h92\") pod \"opendatahub-operator-controller-manager-667b7bc75f-g6jzm\" (UID: \"cae6c80b-633e-43ba-b0e9-0045ebd3fe37\") " pod="opendatahub/opendatahub-operator-controller-manager-667b7bc75f-g6jzm"
Jul 29 19:22:52.274066 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:52.273982    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cae6c80b-633e-43ba-b0e9-0045ebd3fe37-tmp\") pod \"opendatahub-operator-controller-manager-667b7bc75f-g6jzm\" (UID: \"cae6c80b-633e-43ba-b0e9-0045ebd3fe37\") " pod="opendatahub/opendatahub-operator-controller-manager-667b7bc75f-g6jzm"
Jul 29 19:22:52.274066 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:52.274023    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cae6c80b-633e-43ba-b0e9-0045ebd3fe37-webhook-cert\") pod \"opendatahub-operator-controller-manager-667b7bc75f-g6jzm\" (UID: \"cae6c80b-633e-43ba-b0e9-0045ebd3fe37\") " pod="opendatahub/opendatahub-operator-controller-manager-667b7bc75f-g6jzm"
Jul 29 19:22:52.274066 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:52.274050    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cae6c80b-633e-43ba-b0e9-0045ebd3fe37-manifests\") pod \"opendatahub-operator-controller-manager-667b7bc75f-g6jzm\" (UID: \"cae6c80b-633e-43ba-b0e9-0045ebd3fe37\") " pod="opendatahub/opendatahub-operator-controller-manager-667b7bc75f-g6jzm"
Jul 29 19:22:52.274410 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:52.274068    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2h92\" (UniqueName: \"kubernetes.io/projected/cae6c80b-633e-43ba-b0e9-0045ebd3fe37-kube-api-access-t2h92\") pod \"opendatahub-operator-controller-manager-667b7bc75f-g6jzm\" (UID: \"cae6c80b-633e-43ba-b0e9-0045ebd3fe37\") " pod="opendatahub/opendatahub-operator-controller-manager-667b7bc75f-g6jzm"
Jul 29 19:22:52.274410 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:52.274224    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cae6c80b-633e-43ba-b0e9-0045ebd3fe37-apiservice-cert\") pod \"opendatahub-operator-controller-manager-667b7bc75f-g6jzm\" (UID: \"cae6c80b-633e-43ba-b0e9-0045ebd3fe37\") " pod="opendatahub/opendatahub-operator-controller-manager-667b7bc75f-g6jzm"
Jul 29 19:22:52.274637 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:52.274612    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cae6c80b-633e-43ba-b0e9-0045ebd3fe37-manifests\") pod \"opendatahub-operator-controller-manager-667b7bc75f-g6jzm\" (UID: \"cae6c80b-633e-43ba-b0e9-0045ebd3fe37\") " pod="opendatahub/opendatahub-operator-controller-manager-667b7bc75f-g6jzm"
Jul 29 19:22:52.276510 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:52.276482    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cae6c80b-633e-43ba-b0e9-0045ebd3fe37-tmp\") pod \"opendatahub-operator-controller-manager-667b7bc75f-g6jzm\" (UID: \"cae6c80b-633e-43ba-b0e9-0045ebd3fe37\") " pod="opendatahub/opendatahub-operator-controller-manager-667b7bc75f-g6jzm"
Jul 29 19:22:52.276714 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:52.276696    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cae6c80b-633e-43ba-b0e9-0045ebd3fe37-webhook-cert\") pod \"opendatahub-operator-controller-manager-667b7bc75f-g6jzm\" (UID: \"cae6c80b-633e-43ba-b0e9-0045ebd3fe37\") " pod="opendatahub/opendatahub-operator-controller-manager-667b7bc75f-g6jzm"
Jul 29 19:22:52.276785 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:52.276768    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cae6c80b-633e-43ba-b0e9-0045ebd3fe37-apiservice-cert\") pod \"opendatahub-operator-controller-manager-667b7bc75f-g6jzm\" (UID: \"cae6c80b-633e-43ba-b0e9-0045ebd3fe37\") " pod="opendatahub/opendatahub-operator-controller-manager-667b7bc75f-g6jzm"
Jul 29 19:22:52.294990 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:52.294963    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2h92\" (UniqueName: \"kubernetes.io/projected/cae6c80b-633e-43ba-b0e9-0045ebd3fe37-kube-api-access-t2h92\") pod \"opendatahub-operator-controller-manager-667b7bc75f-g6jzm\" (UID: \"cae6c80b-633e-43ba-b0e9-0045ebd3fe37\") " pod="opendatahub/opendatahub-operator-controller-manager-667b7bc75f-g6jzm"
Jul 29 19:22:52.344550 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:52.344518    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-667b7bc75f-g6jzm"
Jul 29 19:22:52.485011 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:52.484988    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-667b7bc75f-g6jzm"]
Jul 29 19:22:52.488039 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:22:52.488016    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcae6c80b_633e_43ba_b0e9_0045ebd3fe37.slice/crio-4a8fa0c21c680ec3f918966ee6f632866d2dd78da59832a761d1ef339aa2cf72 WatchSource:0}: Error finding container 4a8fa0c21c680ec3f918966ee6f632866d2dd78da59832a761d1ef339aa2cf72: Status 404 returned error can't find the container with id 4a8fa0c21c680ec3f918966ee6f632866d2dd78da59832a761d1ef339aa2cf72
Jul 29 19:22:52.658601 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:52.658569    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-667b7bc75f-g6jzm" event={"ID":"cae6c80b-633e-43ba-b0e9-0045ebd3fe37","Type":"ContainerStarted","Data":"4a8fa0c21c680ec3f918966ee6f632866d2dd78da59832a761d1ef339aa2cf72"}
Jul 29 19:22:55.672403 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:55.672303    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-667b7bc75f-g6jzm" event={"ID":"cae6c80b-633e-43ba-b0e9-0045ebd3fe37","Type":"ContainerStarted","Data":"76216f770b90a5fd4b7263fec8215ad720312384409ec654cafedb2fdbe2bf8b"}
Jul 29 19:22:56.677627 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:56.677595    2583 generic.go:356] "Generic (PLEG): container finished" podID="cae6c80b-633e-43ba-b0e9-0045ebd3fe37" containerID="76216f770b90a5fd4b7263fec8215ad720312384409ec654cafedb2fdbe2bf8b" exitCode=0
Jul 29 19:22:56.678012 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:56.677648    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-667b7bc75f-g6jzm" event={"ID":"cae6c80b-633e-43ba-b0e9-0045ebd3fe37","Type":"ContainerDied","Data":"76216f770b90a5fd4b7263fec8215ad720312384409ec654cafedb2fdbe2bf8b"}
Jul 29 19:22:58.685907 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:58.685869    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-667b7bc75f-g6jzm" event={"ID":"cae6c80b-633e-43ba-b0e9-0045ebd3fe37","Type":"ContainerStarted","Data":"25f144f3649ed16fe21b3282087cc07909503a726fa3a7222fec7f98d845371c"}
Jul 29 19:22:58.686300 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:22:58.686052    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-667b7bc75f-g6jzm"
Jul 29 19:23:09.692157 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:09.692080    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-667b7bc75f-g6jzm"
Jul 29 19:23:09.716232 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:09.716178    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-667b7bc75f-g6jzm" podStartSLOduration=13.102915719 podStartE2EDuration="18.716161717s" totalImagesPullingTime="4.563785936s" totalInitContainerRuntime="1.049460062s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:22:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:22:58.727702428 +0000 UTC m=+345.956091553" watchObservedRunningTime="2026-07-29 19:23:09.716161717 +0000 UTC m=+356.944550843"
Jul 29 19:23:12.259203 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:12.259167    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-gqj8n"]
Jul 29 19:23:12.267954 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:12.267930    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-gqj8n"
Jul 29 19:23:12.270274 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:12.270251    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-x24j6\""
Jul 29 19:23:12.271552 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:12.271531    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 29 19:23:12.304292 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:12.304259    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-gqj8n"]
Jul 29 19:23:12.320400 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:12.320363    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qstr\" (UniqueName: \"kubernetes.io/projected/9a18c2d9-4c11-42de-bdea-16ff6e458049-kube-api-access-8qstr\") pod \"odh-model-controller-7998c67dc4-gqj8n\" (UID: \"9a18c2d9-4c11-42de-bdea-16ff6e458049\") " pod="opendatahub/odh-model-controller-7998c67dc4-gqj8n"
Jul 29 19:23:12.320559 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:12.320416    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a18c2d9-4c11-42de-bdea-16ff6e458049-cert\") pod \"odh-model-controller-7998c67dc4-gqj8n\" (UID: \"9a18c2d9-4c11-42de-bdea-16ff6e458049\") " pod="opendatahub/odh-model-controller-7998c67dc4-gqj8n"
Jul 29 19:23:12.421638 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:12.421592    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qstr\" (UniqueName: \"kubernetes.io/projected/9a18c2d9-4c11-42de-bdea-16ff6e458049-kube-api-access-8qstr\") pod \"odh-model-controller-7998c67dc4-gqj8n\" (UID: \"9a18c2d9-4c11-42de-bdea-16ff6e458049\") " pod="opendatahub/odh-model-controller-7998c67dc4-gqj8n"
Jul 29 19:23:12.421870 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:12.421651    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a18c2d9-4c11-42de-bdea-16ff6e458049-cert\") pod \"odh-model-controller-7998c67dc4-gqj8n\" (UID: \"9a18c2d9-4c11-42de-bdea-16ff6e458049\") " pod="opendatahub/odh-model-controller-7998c67dc4-gqj8n"
Jul 29 19:23:12.421870 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:23:12.421756    2583 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 29 19:23:12.421870 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:23:12.421839    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a18c2d9-4c11-42de-bdea-16ff6e458049-cert podName:9a18c2d9-4c11-42de-bdea-16ff6e458049 nodeName:}" failed. No retries permitted until 2026-07-29 19:23:12.921822166 +0000 UTC m=+360.150211285 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/9a18c2d9-4c11-42de-bdea-16ff6e458049-cert") pod "odh-model-controller-7998c67dc4-gqj8n" (UID: "9a18c2d9-4c11-42de-bdea-16ff6e458049") : secret "odh-model-controller-webhook-cert" not found
Jul 29 19:23:12.432642 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:12.432611    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qstr\" (UniqueName: \"kubernetes.io/projected/9a18c2d9-4c11-42de-bdea-16ff6e458049-kube-api-access-8qstr\") pod \"odh-model-controller-7998c67dc4-gqj8n\" (UID: \"9a18c2d9-4c11-42de-bdea-16ff6e458049\") " pod="opendatahub/odh-model-controller-7998c67dc4-gqj8n"
Jul 29 19:23:12.927347 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:12.927305    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a18c2d9-4c11-42de-bdea-16ff6e458049-cert\") pod \"odh-model-controller-7998c67dc4-gqj8n\" (UID: \"9a18c2d9-4c11-42de-bdea-16ff6e458049\") " pod="opendatahub/odh-model-controller-7998c67dc4-gqj8n"
Jul 29 19:23:12.929777 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:12.929753    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a18c2d9-4c11-42de-bdea-16ff6e458049-cert\") pod \"odh-model-controller-7998c67dc4-gqj8n\" (UID: \"9a18c2d9-4c11-42de-bdea-16ff6e458049\") " pod="opendatahub/odh-model-controller-7998c67dc4-gqj8n"
Jul 29 19:23:13.057363 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:13.057325    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-dbbkd"]
Jul 29 19:23:13.059318 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:13.059300    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-dbbkd"
Jul 29 19:23:13.064232 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:13.064210    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 29 19:23:13.085944 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:13.085914    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-dbbkd"]
Jul 29 19:23:13.129305 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:13.129270    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhjxm\" (UniqueName: \"kubernetes.io/projected/3915492a-6290-43a6-b7a0-3643b5085183-kube-api-access-fhjxm\") pod \"model-serving-api-54d89bd5fc-dbbkd\" (UID: \"3915492a-6290-43a6-b7a0-3643b5085183\") " pod="opendatahub/model-serving-api-54d89bd5fc-dbbkd"
Jul 29 19:23:13.129473 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:13.129335    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3915492a-6290-43a6-b7a0-3643b5085183-tls-certs\") pod \"model-serving-api-54d89bd5fc-dbbkd\" (UID: \"3915492a-6290-43a6-b7a0-3643b5085183\") " pod="opendatahub/model-serving-api-54d89bd5fc-dbbkd"
Jul 29 19:23:13.181333 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:13.181251    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-x24j6\""
Jul 29 19:23:13.188165 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:13.188145    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-gqj8n"
Jul 29 19:23:13.230589 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:13.230548    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhjxm\" (UniqueName: \"kubernetes.io/projected/3915492a-6290-43a6-b7a0-3643b5085183-kube-api-access-fhjxm\") pod \"model-serving-api-54d89bd5fc-dbbkd\" (UID: \"3915492a-6290-43a6-b7a0-3643b5085183\") " pod="opendatahub/model-serving-api-54d89bd5fc-dbbkd"
Jul 29 19:23:13.230727 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:13.230654    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3915492a-6290-43a6-b7a0-3643b5085183-tls-certs\") pod \"model-serving-api-54d89bd5fc-dbbkd\" (UID: \"3915492a-6290-43a6-b7a0-3643b5085183\") " pod="opendatahub/model-serving-api-54d89bd5fc-dbbkd"
Jul 29 19:23:13.234711 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:13.234671    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 29 19:23:13.243974 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:13.243951    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3915492a-6290-43a6-b7a0-3643b5085183-tls-certs\") pod \"model-serving-api-54d89bd5fc-dbbkd\" (UID: \"3915492a-6290-43a6-b7a0-3643b5085183\") " pod="opendatahub/model-serving-api-54d89bd5fc-dbbkd"
Jul 29 19:23:13.254409 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:13.254383    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhjxm\" (UniqueName: \"kubernetes.io/projected/3915492a-6290-43a6-b7a0-3643b5085183-kube-api-access-fhjxm\") pod \"model-serving-api-54d89bd5fc-dbbkd\" (UID: \"3915492a-6290-43a6-b7a0-3643b5085183\") " pod="opendatahub/model-serving-api-54d89bd5fc-dbbkd"
Jul 29 19:23:13.321473 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:13.321450    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-gqj8n"]
Jul 29 19:23:13.323720 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:23:13.323694    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a18c2d9_4c11_42de_bdea_16ff6e458049.slice/crio-85a65aa2146c72d3b79e6c1c92406f1290bd503cb68da1523ecaffbb304db1eb WatchSource:0}: Error finding container 85a65aa2146c72d3b79e6c1c92406f1290bd503cb68da1523ecaffbb304db1eb: Status 404 returned error can't find the container with id 85a65aa2146c72d3b79e6c1c92406f1290bd503cb68da1523ecaffbb304db1eb
Jul 29 19:23:13.369060 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:13.369030    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-dbbkd"
Jul 29 19:23:13.519511 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:13.519477    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-dbbkd"]
Jul 29 19:23:13.523849 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:23:13.523823    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3915492a_6290_43a6_b7a0_3643b5085183.slice/crio-2cebc096f8cef3de9335f1f14b55222cb68384056dc1ad2b85b79f44f6f2bb9d WatchSource:0}: Error finding container 2cebc096f8cef3de9335f1f14b55222cb68384056dc1ad2b85b79f44f6f2bb9d: Status 404 returned error can't find the container with id 2cebc096f8cef3de9335f1f14b55222cb68384056dc1ad2b85b79f44f6f2bb9d
Jul 29 19:23:13.738524 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:13.738438    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-dbbkd" event={"ID":"3915492a-6290-43a6-b7a0-3643b5085183","Type":"ContainerStarted","Data":"2cebc096f8cef3de9335f1f14b55222cb68384056dc1ad2b85b79f44f6f2bb9d"}
Jul 29 19:23:13.739529 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:13.739508    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-gqj8n" event={"ID":"9a18c2d9-4c11-42de-bdea-16ff6e458049","Type":"ContainerStarted","Data":"85a65aa2146c72d3b79e6c1c92406f1290bd503cb68da1523ecaffbb304db1eb"}
Jul 29 19:23:15.748112 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:15.748077    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-dbbkd" event={"ID":"3915492a-6290-43a6-b7a0-3643b5085183","Type":"ContainerStarted","Data":"1cc1af1a6c5d740ffced5f0fbe57df30c511a32cefa64ca88260a14509024c8b"}
Jul 29 19:23:15.748558 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:15.748214    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-dbbkd"
Jul 29 19:23:15.771785 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:15.771735    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-dbbkd" podStartSLOduration=1.195861181 podStartE2EDuration="2.771718477s" totalImagesPullingTime="1.575857296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:23:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:23:15.77134591 +0000 UTC m=+362.999735037" watchObservedRunningTime="2026-07-29 19:23:15.771718477 +0000 UTC m=+363.000107605"
Jul 29 19:23:18.760789 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:18.760749    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-gqj8n" event={"ID":"9a18c2d9-4c11-42de-bdea-16ff6e458049","Type":"ContainerStarted","Data":"9c8672ef4543cfe00c6c9cbe1b0c15770a1f5140766804a323b5b8a924fdf476"}
Jul 29 19:23:18.761177 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:18.760841    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-gqj8n"
Jul 29 19:23:18.783890 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:18.782083    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-gqj8n" podStartSLOduration=2.098422478 podStartE2EDuration="6.782060822s" totalImagesPullingTime="4.683638344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:23:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:23:18.778875791 +0000 UTC m=+366.007264927" watchObservedRunningTime="2026-07-29 19:23:18.782060822 +0000 UTC m=+366.010449950"
Jul 29 19:23:23.004305 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:23.004262    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7b6ffddf6-tcktk"]
Jul 29 19:23:23.006622 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:23.006600    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7b6ffddf6-tcktk"
Jul 29 19:23:23.011467 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:23.011438    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 29 19:23:23.011467 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:23.011460    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 29 19:23:23.011627 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:23.011605    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 29 19:23:23.011812 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:23.011790    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 29 19:23:23.038200 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:23.038171    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7b6ffddf6-tcktk"]
Jul 29 19:23:23.112900 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:23.112851    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54036b22-87b4-4104-bd40-56da3fb1d01f-tmp\") pod \"kube-auth-proxy-7b6ffddf6-tcktk\" (UID: \"54036b22-87b4-4104-bd40-56da3fb1d01f\") " pod="openshift-ingress/kube-auth-proxy-7b6ffddf6-tcktk"
Jul 29 19:23:23.112900 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:23.112902    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54036b22-87b4-4104-bd40-56da3fb1d01f-tls-certs\") pod \"kube-auth-proxy-7b6ffddf6-tcktk\" (UID: \"54036b22-87b4-4104-bd40-56da3fb1d01f\") " pod="openshift-ingress/kube-auth-proxy-7b6ffddf6-tcktk"
Jul 29 19:23:23.113105 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:23.112999    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nllmp\" (UniqueName: \"kubernetes.io/projected/54036b22-87b4-4104-bd40-56da3fb1d01f-kube-api-access-nllmp\") pod \"kube-auth-proxy-7b6ffddf6-tcktk\" (UID: \"54036b22-87b4-4104-bd40-56da3fb1d01f\") " pod="openshift-ingress/kube-auth-proxy-7b6ffddf6-tcktk"
Jul 29 19:23:23.214309 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:23.214259    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nllmp\" (UniqueName: \"kubernetes.io/projected/54036b22-87b4-4104-bd40-56da3fb1d01f-kube-api-access-nllmp\") pod \"kube-auth-proxy-7b6ffddf6-tcktk\" (UID: \"54036b22-87b4-4104-bd40-56da3fb1d01f\") " pod="openshift-ingress/kube-auth-proxy-7b6ffddf6-tcktk"
Jul 29 19:23:23.214496 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:23.214333    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54036b22-87b4-4104-bd40-56da3fb1d01f-tmp\") pod \"kube-auth-proxy-7b6ffddf6-tcktk\" (UID: \"54036b22-87b4-4104-bd40-56da3fb1d01f\") " pod="openshift-ingress/kube-auth-proxy-7b6ffddf6-tcktk"
Jul 29 19:23:23.214496 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:23.214359    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54036b22-87b4-4104-bd40-56da3fb1d01f-tls-certs\") pod \"kube-auth-proxy-7b6ffddf6-tcktk\" (UID: \"54036b22-87b4-4104-bd40-56da3fb1d01f\") " pod="openshift-ingress/kube-auth-proxy-7b6ffddf6-tcktk"
Jul 29 19:23:23.216723 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:23.216685    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54036b22-87b4-4104-bd40-56da3fb1d01f-tmp\") pod \"kube-auth-proxy-7b6ffddf6-tcktk\" (UID: \"54036b22-87b4-4104-bd40-56da3fb1d01f\") " pod="openshift-ingress/kube-auth-proxy-7b6ffddf6-tcktk"
Jul 29 19:23:23.217035 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:23.217014    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54036b22-87b4-4104-bd40-56da3fb1d01f-tls-certs\") pod \"kube-auth-proxy-7b6ffddf6-tcktk\" (UID: \"54036b22-87b4-4104-bd40-56da3fb1d01f\") " pod="openshift-ingress/kube-auth-proxy-7b6ffddf6-tcktk"
Jul 29 19:23:23.240311 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:23.240279    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nllmp\" (UniqueName: \"kubernetes.io/projected/54036b22-87b4-4104-bd40-56da3fb1d01f-kube-api-access-nllmp\") pod \"kube-auth-proxy-7b6ffddf6-tcktk\" (UID: \"54036b22-87b4-4104-bd40-56da3fb1d01f\") " pod="openshift-ingress/kube-auth-proxy-7b6ffddf6-tcktk"
Jul 29 19:23:23.316314 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:23.316227    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7b6ffddf6-tcktk"
Jul 29 19:23:23.456465 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:23.456432    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7b6ffddf6-tcktk"]
Jul 29 19:23:23.459545 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:23:23.459520    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54036b22_87b4_4104_bd40_56da3fb1d01f.slice/crio-7ecd8374ba6c04df16d11889ead8f507e84e1dd4e6d046dd66dc5c05cfb8bc35 WatchSource:0}: Error finding container 7ecd8374ba6c04df16d11889ead8f507e84e1dd4e6d046dd66dc5c05cfb8bc35: Status 404 returned error can't find the container with id 7ecd8374ba6c04df16d11889ead8f507e84e1dd4e6d046dd66dc5c05cfb8bc35
Jul 29 19:23:23.781459 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:23.781430    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7b6ffddf6-tcktk" event={"ID":"54036b22-87b4-4104-bd40-56da3fb1d01f","Type":"ContainerStarted","Data":"7ecd8374ba6c04df16d11889ead8f507e84e1dd4e6d046dd66dc5c05cfb8bc35"}
Jul 29 19:23:23.792089 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:23.792058    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-q94h6"]
Jul 29 19:23:23.795006 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:23.794989    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-t9vcv"]
Jul 29 19:23:23.795138 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:23.795122    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-q94h6"
Jul 29 19:23:23.796854 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:23.796838    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t9vcv"
Jul 29 19:23:23.798317 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:23.798296    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-2jhb7\""
Jul 29 19:23:23.798401 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:23.798363    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 29 19:23:23.799705 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:23.799686    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-8qfm6\""
Jul 29 19:23:23.799823 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:23.799775    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 29 19:23:23.814437 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:23.814411    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-t9vcv"]
Jul 29 19:23:23.830839 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:23.830790    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-q94h6"]
Jul 29 19:23:23.919256 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:23.919224    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxlzk\" (UniqueName: \"kubernetes.io/projected/7ec5dddd-2a18-46ce-bbd3-ff5f0c447915-kube-api-access-xxlzk\") pod \"kserve-controller-manager-d6f4b65b8-q94h6\" (UID: \"7ec5dddd-2a18-46ce-bbd3-ff5f0c447915\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-q94h6"
Jul 29 19:23:23.919256 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:23.919261    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/538180db-c08e-4b16-b2b8-ded900ea01e1-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-t9vcv\" (UID: \"538180db-c08e-4b16-b2b8-ded900ea01e1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t9vcv"
Jul 29 19:23:23.919471 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:23.919304    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxbdq\" (UniqueName: \"kubernetes.io/projected/538180db-c08e-4b16-b2b8-ded900ea01e1-kube-api-access-fxbdq\") pod \"llmisvc-controller-manager-777c6bd7d5-t9vcv\" (UID: \"538180db-c08e-4b16-b2b8-ded900ea01e1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t9vcv"
Jul 29 19:23:23.919471 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:23.919328    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ec5dddd-2a18-46ce-bbd3-ff5f0c447915-cert\") pod \"kserve-controller-manager-d6f4b65b8-q94h6\" (UID: \"7ec5dddd-2a18-46ce-bbd3-ff5f0c447915\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-q94h6"
Jul 29 19:23:24.020354 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:24.020312    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxlzk\" (UniqueName: \"kubernetes.io/projected/7ec5dddd-2a18-46ce-bbd3-ff5f0c447915-kube-api-access-xxlzk\") pod \"kserve-controller-manager-d6f4b65b8-q94h6\" (UID: \"7ec5dddd-2a18-46ce-bbd3-ff5f0c447915\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-q94h6"
Jul 29 19:23:24.020354 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:24.020357    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/538180db-c08e-4b16-b2b8-ded900ea01e1-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-t9vcv\" (UID: \"538180db-c08e-4b16-b2b8-ded900ea01e1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t9vcv"
Jul 29 19:23:24.020792 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:24.020405    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxbdq\" (UniqueName: \"kubernetes.io/projected/538180db-c08e-4b16-b2b8-ded900ea01e1-kube-api-access-fxbdq\") pod \"llmisvc-controller-manager-777c6bd7d5-t9vcv\" (UID: \"538180db-c08e-4b16-b2b8-ded900ea01e1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t9vcv"
Jul 29 19:23:24.020792 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:24.020433    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ec5dddd-2a18-46ce-bbd3-ff5f0c447915-cert\") pod \"kserve-controller-manager-d6f4b65b8-q94h6\" (UID: \"7ec5dddd-2a18-46ce-bbd3-ff5f0c447915\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-q94h6"
Jul 29 19:23:24.022853 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:24.022833    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ec5dddd-2a18-46ce-bbd3-ff5f0c447915-cert\") pod \"kserve-controller-manager-d6f4b65b8-q94h6\" (UID: \"7ec5dddd-2a18-46ce-bbd3-ff5f0c447915\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-q94h6"
Jul 29 19:23:24.022902 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:24.022856    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/538180db-c08e-4b16-b2b8-ded900ea01e1-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-t9vcv\" (UID: \"538180db-c08e-4b16-b2b8-ded900ea01e1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t9vcv"
Jul 29 19:23:24.030931 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:24.030905    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxlzk\" (UniqueName: \"kubernetes.io/projected/7ec5dddd-2a18-46ce-bbd3-ff5f0c447915-kube-api-access-xxlzk\") pod \"kserve-controller-manager-d6f4b65b8-q94h6\" (UID: \"7ec5dddd-2a18-46ce-bbd3-ff5f0c447915\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-q94h6"
Jul 29 19:23:24.031594 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:24.031546    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxbdq\" (UniqueName: \"kubernetes.io/projected/538180db-c08e-4b16-b2b8-ded900ea01e1-kube-api-access-fxbdq\") pod \"llmisvc-controller-manager-777c6bd7d5-t9vcv\" (UID: \"538180db-c08e-4b16-b2b8-ded900ea01e1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t9vcv"
Jul 29 19:23:24.107678 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:24.107646    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-q94h6"
Jul 29 19:23:24.112707 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:24.112688    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t9vcv"
Jul 29 19:23:24.320369 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:24.320280    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-q94h6"]
Jul 29 19:23:24.323704 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:23:24.323673    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ec5dddd_2a18_46ce_bbd3_ff5f0c447915.slice/crio-bd4895054fb1845eb19dd53b0dad8c2e184b21b15d1ef1d9ccb15ba2c4e2d623 WatchSource:0}: Error finding container bd4895054fb1845eb19dd53b0dad8c2e184b21b15d1ef1d9ccb15ba2c4e2d623: Status 404 returned error can't find the container with id bd4895054fb1845eb19dd53b0dad8c2e184b21b15d1ef1d9ccb15ba2c4e2d623
Jul 29 19:23:24.338352 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:24.338321    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-t9vcv"]
Jul 29 19:23:24.341073 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:23:24.341045    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod538180db_c08e_4b16_b2b8_ded900ea01e1.slice/crio-7e90cf884508779d8e50b82607c20a3e64d68d0a7a88ac67c06f1d9a68a58cbe WatchSource:0}: Error finding container 7e90cf884508779d8e50b82607c20a3e64d68d0a7a88ac67c06f1d9a68a58cbe: Status 404 returned error can't find the container with id 7e90cf884508779d8e50b82607c20a3e64d68d0a7a88ac67c06f1d9a68a58cbe
Jul 29 19:23:24.786879 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:24.786843    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-q94h6" event={"ID":"7ec5dddd-2a18-46ce-bbd3-ff5f0c447915","Type":"ContainerStarted","Data":"bd4895054fb1845eb19dd53b0dad8c2e184b21b15d1ef1d9ccb15ba2c4e2d623"}
Jul 29 19:23:24.787908 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:24.787883    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t9vcv" event={"ID":"538180db-c08e-4b16-b2b8-ded900ea01e1","Type":"ContainerStarted","Data":"7e90cf884508779d8e50b82607c20a3e64d68d0a7a88ac67c06f1d9a68a58cbe"}
Jul 29 19:23:25.668723 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:25.668686    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-srr62"]
Jul 29 19:23:25.671557 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:25.671533    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-srr62"
Jul 29 19:23:25.673792 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:25.673767    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-zz5px\""
Jul 29 19:23:25.674268 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:25.674095    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 29 19:23:25.674268 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:25.674219    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 29 19:23:25.674846 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:25.674651    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 29 19:23:25.674846 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:25.674747    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 29 19:23:25.690900 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:25.690840    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-srr62"]
Jul 29 19:23:25.734028 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:25.733990    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf6pk\" (UniqueName: \"kubernetes.io/projected/6d180d8b-da5e-4cfb-a409-eef0f14ebe9b-kube-api-access-tf6pk\") pod \"istiod-openshift-gateway-5677ccb49b-srr62\" (UID: \"6d180d8b-da5e-4cfb-a409-eef0f14ebe9b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-srr62"
Jul 29 19:23:25.734211 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:25.734072    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6d180d8b-da5e-4cfb-a409-eef0f14ebe9b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-srr62\" (UID: \"6d180d8b-da5e-4cfb-a409-eef0f14ebe9b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-srr62"
Jul 29 19:23:25.734211 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:25.734137    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d180d8b-da5e-4cfb-a409-eef0f14ebe9b-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-srr62\" (UID: \"6d180d8b-da5e-4cfb-a409-eef0f14ebe9b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-srr62"
Jul 29 19:23:25.734330 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:25.734231    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6d180d8b-da5e-4cfb-a409-eef0f14ebe9b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-srr62\" (UID: \"6d180d8b-da5e-4cfb-a409-eef0f14ebe9b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-srr62"
Jul 29 19:23:25.734330 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:25.734280    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6d180d8b-da5e-4cfb-a409-eef0f14ebe9b-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-srr62\" (UID: \"6d180d8b-da5e-4cfb-a409-eef0f14ebe9b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-srr62"
Jul 29 19:23:25.734330 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:25.734311    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d180d8b-da5e-4cfb-a409-eef0f14ebe9b-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-srr62\" (UID: \"6d180d8b-da5e-4cfb-a409-eef0f14ebe9b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-srr62"
Jul 29 19:23:25.734609 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:25.734353    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6d180d8b-da5e-4cfb-a409-eef0f14ebe9b-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-srr62\" (UID: \"6d180d8b-da5e-4cfb-a409-eef0f14ebe9b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-srr62"
Jul 29 19:23:25.835389 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:25.835350    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6d180d8b-da5e-4cfb-a409-eef0f14ebe9b-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-srr62\" (UID: \"6d180d8b-da5e-4cfb-a409-eef0f14ebe9b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-srr62"
Jul 29 19:23:25.835579 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:25.835398    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d180d8b-da5e-4cfb-a409-eef0f14ebe9b-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-srr62\" (UID: \"6d180d8b-da5e-4cfb-a409-eef0f14ebe9b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-srr62"
Jul 29 19:23:25.835579 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:25.835434    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6d180d8b-da5e-4cfb-a409-eef0f14ebe9b-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-srr62\" (UID: \"6d180d8b-da5e-4cfb-a409-eef0f14ebe9b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-srr62"
Jul 29 19:23:25.835579 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:25.835465    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf6pk\" (UniqueName: \"kubernetes.io/projected/6d180d8b-da5e-4cfb-a409-eef0f14ebe9b-kube-api-access-tf6pk\") pod \"istiod-openshift-gateway-5677ccb49b-srr62\" (UID: \"6d180d8b-da5e-4cfb-a409-eef0f14ebe9b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-srr62"
Jul 29 19:23:25.835579 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:25.835510    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6d180d8b-da5e-4cfb-a409-eef0f14ebe9b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-srr62\" (UID: \"6d180d8b-da5e-4cfb-a409-eef0f14ebe9b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-srr62"
Jul 29 19:23:25.835579 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:25.835547    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d180d8b-da5e-4cfb-a409-eef0f14ebe9b-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-srr62\" (UID: \"6d180d8b-da5e-4cfb-a409-eef0f14ebe9b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-srr62"
Jul 29 19:23:25.835873 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:25.835588    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6d180d8b-da5e-4cfb-a409-eef0f14ebe9b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-srr62\" (UID: \"6d180d8b-da5e-4cfb-a409-eef0f14ebe9b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-srr62"
Jul 29 19:23:25.836412 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:25.836385    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6d180d8b-da5e-4cfb-a409-eef0f14ebe9b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-srr62\" (UID: \"6d180d8b-da5e-4cfb-a409-eef0f14ebe9b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-srr62"
Jul 29 19:23:25.838470 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:25.838440    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6d180d8b-da5e-4cfb-a409-eef0f14ebe9b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-srr62\" (UID: \"6d180d8b-da5e-4cfb-a409-eef0f14ebe9b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-srr62"
Jul 29 19:23:25.838936 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:25.838609    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6d180d8b-da5e-4cfb-a409-eef0f14ebe9b-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-srr62\" (UID: \"6d180d8b-da5e-4cfb-a409-eef0f14ebe9b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-srr62"
Jul 29 19:23:25.838936 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:25.838899    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6d180d8b-da5e-4cfb-a409-eef0f14ebe9b-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-srr62\" (UID: \"6d180d8b-da5e-4cfb-a409-eef0f14ebe9b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-srr62"
Jul 29 19:23:25.839234 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:25.839216    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d180d8b-da5e-4cfb-a409-eef0f14ebe9b-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-srr62\" (UID: \"6d180d8b-da5e-4cfb-a409-eef0f14ebe9b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-srr62"
Jul 29 19:23:25.847129 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:25.847080    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d180d8b-da5e-4cfb-a409-eef0f14ebe9b-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-srr62\" (UID: \"6d180d8b-da5e-4cfb-a409-eef0f14ebe9b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-srr62"
Jul 29 19:23:25.847363 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:25.847316    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf6pk\" (UniqueName: \"kubernetes.io/projected/6d180d8b-da5e-4cfb-a409-eef0f14ebe9b-kube-api-access-tf6pk\") pod \"istiod-openshift-gateway-5677ccb49b-srr62\" (UID: \"6d180d8b-da5e-4cfb-a409-eef0f14ebe9b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-srr62"
Jul 29 19:23:25.984693 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:25.984603    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-srr62"
Jul 29 19:23:26.180639 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:26.180546    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-srr62"]
Jul 29 19:23:26.757676 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:26.757647    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-dbbkd"
Jul 29 19:23:26.799041 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:23:26.799011    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d180d8b_da5e_4cfb_a409_eef0f14ebe9b.slice/crio-bbd921504723489ed965c27583b30f5951d354ec67af559e9d16d16e5224910d WatchSource:0}: Error finding container bbd921504723489ed965c27583b30f5951d354ec67af559e9d16d16e5224910d: Status 404 returned error can't find the container with id bbd921504723489ed965c27583b30f5951d354ec67af559e9d16d16e5224910d
Jul 29 19:23:27.806052 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:27.806018    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-srr62" event={"ID":"6d180d8b-da5e-4cfb-a409-eef0f14ebe9b","Type":"ContainerStarted","Data":"bbd921504723489ed965c27583b30f5951d354ec67af559e9d16d16e5224910d"}
Jul 29 19:23:28.812777 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:28.812740    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7b6ffddf6-tcktk" event={"ID":"54036b22-87b4-4104-bd40-56da3fb1d01f","Type":"ContainerStarted","Data":"15b09a8c5ca6001b4c8ca34c4d5fdcf62b5c9a5aafd3a02b174a676dcd28113c"}
Jul 29 19:23:28.814309 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:28.814280    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-q94h6" event={"ID":"7ec5dddd-2a18-46ce-bbd3-ff5f0c447915","Type":"ContainerStarted","Data":"38c762f161fdf9bbd5611cbfa1f3ac80612d725b01b2f2727506fed97271a5d0"}
Jul 29 19:23:28.814426 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:28.814342    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-q94h6"
Jul 29 19:23:28.815704 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:28.815676    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t9vcv" event={"ID":"538180db-c08e-4b16-b2b8-ded900ea01e1","Type":"ContainerStarted","Data":"60ab35f5a732b050389a24b72e53ba16b932ad17016dcece07a3b7d117159b00"}
Jul 29 19:23:28.815848 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:28.815833    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t9vcv"
Jul 29 19:23:28.838630 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:28.838578    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7b6ffddf6-tcktk" podStartSLOduration=2.272302549 podStartE2EDuration="6.838560357s" totalImagesPullingTime="4.566257808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:23:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:23:28.837904931 +0000 UTC m=+376.066294056" watchObservedRunningTime="2026-07-29 19:23:28.838560357 +0000 UTC m=+376.066949482"
Jul 29 19:23:28.860396 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:28.860343    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t9vcv" podStartSLOduration=2.171884818 podStartE2EDuration="5.86032771s" totalImagesPullingTime="3.688442892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:23:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:23:28.8580172 +0000 UTC m=+376.086406327" watchObservedRunningTime="2026-07-29 19:23:28.86032771 +0000 UTC m=+376.088716836"
Jul 29 19:23:28.876214 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:28.876157    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-q94h6" podStartSLOduration=2.17909133 podStartE2EDuration="5.876140023s" totalImagesPullingTime="3.697048693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:23:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:23:28.875908084 +0000 UTC m=+376.104297211" watchObservedRunningTime="2026-07-29 19:23:28.876140023 +0000 UTC m=+376.104529148"
Jul 29 19:23:29.769159 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:29.769126    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-gqj8n"
Jul 29 19:23:30.987029 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:30.986991    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 29 19:23:30.987301 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:30.987079    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 29 19:23:31.833580 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:31.833533    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-srr62" event={"ID":"6d180d8b-da5e-4cfb-a409-eef0f14ebe9b","Type":"ContainerStarted","Data":"655f88e1ad7f1fd2beee9903bb72fd6334f26c024f28383f874a0406845a97fa"}
Jul 29 19:23:31.833794 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:31.833749    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-srr62"
Jul 29 19:23:32.839711 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:32.839680    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-srr62"
Jul 29 19:23:32.927911 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:32.927854    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-srr62" podStartSLOduration=3.742493932 podStartE2EDuration="7.927837984s" totalImagesPullingTime="4.185344052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:23:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:23:31.872660191 +0000 UTC m=+379.101049326" watchObservedRunningTime="2026-07-29 19:23:32.927837984 +0000 UTC m=+380.156227112"
Jul 29 19:23:59.824409 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:59.824381    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t9vcv"
Jul 29 19:23:59.827425 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:23:59.827405    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-q94h6"
Jul 29 19:24:43.562328 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:24:43.562250    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-6g6tj"]
Jul 29 19:24:43.564311 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:24:43.564289    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-6g6tj"
Jul 29 19:24:43.566319 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:24:43.566299    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 29 19:24:43.566573 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:24:43.566559    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-rt87z\""
Jul 29 19:24:43.566624 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:24:43.566595    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 29 19:24:43.578379 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:24:43.578353    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-6g6tj"]
Jul 29 19:24:43.657920 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:24:43.657883    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22wfz\" (UniqueName: \"kubernetes.io/projected/f7cfc0f9-8792-41bb-82a7-fdc902cc5506-kube-api-access-22wfz\") pod \"authorino-operator-75cc8d5cb7-6g6tj\" (UID: \"f7cfc0f9-8792-41bb-82a7-fdc902cc5506\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-6g6tj"
Jul 29 19:24:43.758884 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:24:43.758844    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22wfz\" (UniqueName: \"kubernetes.io/projected/f7cfc0f9-8792-41bb-82a7-fdc902cc5506-kube-api-access-22wfz\") pod \"authorino-operator-75cc8d5cb7-6g6tj\" (UID: \"f7cfc0f9-8792-41bb-82a7-fdc902cc5506\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-6g6tj"
Jul 29 19:24:43.772184 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:24:43.772157    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22wfz\" (UniqueName: \"kubernetes.io/projected/f7cfc0f9-8792-41bb-82a7-fdc902cc5506-kube-api-access-22wfz\") pod \"authorino-operator-75cc8d5cb7-6g6tj\" (UID: \"f7cfc0f9-8792-41bb-82a7-fdc902cc5506\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-6g6tj"
Jul 29 19:24:43.874464 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:24:43.874433    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-6g6tj"
Jul 29 19:24:44.000401 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:24:44.000376    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-6g6tj"]
Jul 29 19:24:44.002774 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:24:44.002741    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7cfc0f9_8792_41bb_82a7_fdc902cc5506.slice/crio-a9ead0691c9bad484e2c29dd4c9dc63220b5ca8d9bbb1bdf45d428def87bb253 WatchSource:0}: Error finding container a9ead0691c9bad484e2c29dd4c9dc63220b5ca8d9bbb1bdf45d428def87bb253: Status 404 returned error can't find the container with id a9ead0691c9bad484e2c29dd4c9dc63220b5ca8d9bbb1bdf45d428def87bb253
Jul 29 19:24:44.125313 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:24:44.125232    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-6g6tj" event={"ID":"f7cfc0f9-8792-41bb-82a7-fdc902cc5506","Type":"ContainerStarted","Data":"a9ead0691c9bad484e2c29dd4c9dc63220b5ca8d9bbb1bdf45d428def87bb253"}
Jul 29 19:24:47.146834 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:24:47.146779    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-6g6tj" event={"ID":"f7cfc0f9-8792-41bb-82a7-fdc902cc5506","Type":"ContainerStarted","Data":"8ee7e2e8f686608d248f465f4e825e90f6a85094449009aa15c0aab9e9a0611b"}
Jul 29 19:24:47.147228 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:24:47.146927    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-6g6tj"
Jul 29 19:24:47.200317 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:24:47.200266    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-6g6tj" podStartSLOduration=2.066969586 podStartE2EDuration="4.200249929s" totalImagesPullingTime="2.133280343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:24:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:24:47.194847342 +0000 UTC m=+454.423236469" watchObservedRunningTime="2026-07-29 19:24:47.200249929 +0000 UTC m=+454.428639055"
Jul 29 19:24:58.154173 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:24:58.154140    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-6g6tj"
Jul 29 19:25:58.929228 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:25:58.929192    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:25:58.936792 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:25:58.936762    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"
Jul 29 19:25:58.938642 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:25:58.938627    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-shqcg\""
Jul 29 19:25:58.938711 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:25:58.938630    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 29 19:25:58.943257 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:25:58.943233    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:25:58.972305 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:25:58.972271    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:25:59.055997 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:25:59.055955    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k796t\" (UniqueName: \"kubernetes.io/projected/382a805c-5973-428e-888c-1c30e0226476-kube-api-access-k796t\") pod \"limitador-limitador-54d6d988d5-sd7wr\" (UID: \"382a805c-5973-428e-888c-1c30e0226476\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"
Jul 29 19:25:59.056178 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:25:59.056018    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/382a805c-5973-428e-888c-1c30e0226476-config-file\") pod \"limitador-limitador-54d6d988d5-sd7wr\" (UID: \"382a805c-5973-428e-888c-1c30e0226476\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"
Jul 29 19:25:59.157116 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:25:59.157081    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/382a805c-5973-428e-888c-1c30e0226476-config-file\") pod \"limitador-limitador-54d6d988d5-sd7wr\" (UID: \"382a805c-5973-428e-888c-1c30e0226476\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"
Jul 29 19:25:59.157283 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:25:59.157138    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k796t\" (UniqueName: \"kubernetes.io/projected/382a805c-5973-428e-888c-1c30e0226476-kube-api-access-k796t\") pod \"limitador-limitador-54d6d988d5-sd7wr\" (UID: \"382a805c-5973-428e-888c-1c30e0226476\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"
Jul 29 19:25:59.157715 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:25:59.157695    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/382a805c-5973-428e-888c-1c30e0226476-config-file\") pod \"limitador-limitador-54d6d988d5-sd7wr\" (UID: \"382a805c-5973-428e-888c-1c30e0226476\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"
Jul 29 19:25:59.164742 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:25:59.164713    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k796t\" (UniqueName: \"kubernetes.io/projected/382a805c-5973-428e-888c-1c30e0226476-kube-api-access-k796t\") pod \"limitador-limitador-54d6d988d5-sd7wr\" (UID: \"382a805c-5973-428e-888c-1c30e0226476\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"
Jul 29 19:25:59.247875 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:25:59.247751    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"
Jul 29 19:25:59.396613 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:25:59.396574    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:25:59.399920 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:25:59.399890    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod382a805c_5973_428e_888c_1c30e0226476.slice/crio-a998fdf8dad44505ab0791a6f5d7251834bc9f642a76e513bcc9bd048482aa87 WatchSource:0}: Error finding container a998fdf8dad44505ab0791a6f5d7251834bc9f642a76e513bcc9bd048482aa87: Status 404 returned error can't find the container with id a998fdf8dad44505ab0791a6f5d7251834bc9f642a76e513bcc9bd048482aa87
Jul 29 19:25:59.441458 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:25:59.441426    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-sd7wr" event={"ID":"382a805c-5973-428e-888c-1c30e0226476","Type":"ContainerStarted","Data":"a998fdf8dad44505ab0791a6f5d7251834bc9f642a76e513bcc9bd048482aa87"}
Jul 29 19:26:02.456844 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:26:02.456787    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-sd7wr" event={"ID":"382a805c-5973-428e-888c-1c30e0226476","Type":"ContainerStarted","Data":"426a362d28c9a03dc18ffd8988fd3d9e45929a51aa4379200b3d494d1fe69967"}
Jul 29 19:26:02.457258 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:26:02.456980    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"
Jul 29 19:26:02.473812 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:26:02.473760    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-sd7wr" podStartSLOduration=2.151540688 podStartE2EDuration="4.473742393s" totalImagesPullingTime="2.322201705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:25:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:26:02.472533073 +0000 UTC m=+529.700922200" watchObservedRunningTime="2026-07-29 19:26:02.473742393 +0000 UTC m=+529.702131519"
Jul 29 19:26:13.462159 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:26:13.462071    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"
Jul 29 19:27:11.480503 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:27:11.480475    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:27:27.341988 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:27:27.341961    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:27:35.894679 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:27:35.894598    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:27:45.403823 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:27:45.403769    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:27:57.448148 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:27:57.448109    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7"]
Jul 29 19:27:57.450498 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:27:57.450472    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7"
Jul 29 19:27:57.452776 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:27:57.452751    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 29 19:27:57.452932 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:27:57.452898    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 29 19:27:57.453522 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:27:57.453505    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 29 19:27:57.453597 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:27:57.453525    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-452bp\""
Jul 29 19:27:57.462328 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:27:57.462301    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7"]
Jul 29 19:27:57.494902 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:27:57.494873    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8defa80b-2b41-49b3-921b-57792d5fb3f2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7\" (UID: \"8defa80b-2b41-49b3-921b-57792d5fb3f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7"
Jul 29 19:27:57.495052 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:27:57.494908    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8defa80b-2b41-49b3-921b-57792d5fb3f2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7\" (UID: \"8defa80b-2b41-49b3-921b-57792d5fb3f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7"
Jul 29 19:27:57.495052 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:27:57.494943    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkpsh\" (UniqueName: \"kubernetes.io/projected/8defa80b-2b41-49b3-921b-57792d5fb3f2-kube-api-access-bkpsh\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7\" (UID: \"8defa80b-2b41-49b3-921b-57792d5fb3f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7"
Jul 29 19:27:57.495052 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:27:57.494961    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8defa80b-2b41-49b3-921b-57792d5fb3f2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7\" (UID: \"8defa80b-2b41-49b3-921b-57792d5fb3f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7"
Jul 29 19:27:57.495052 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:27:57.494976    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8defa80b-2b41-49b3-921b-57792d5fb3f2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7\" (UID: \"8defa80b-2b41-49b3-921b-57792d5fb3f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7"
Jul 29 19:27:57.495052 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:27:57.495016    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8defa80b-2b41-49b3-921b-57792d5fb3f2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7\" (UID: \"8defa80b-2b41-49b3-921b-57792d5fb3f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7"
Jul 29 19:27:57.595361 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:27:57.595329    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8defa80b-2b41-49b3-921b-57792d5fb3f2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7\" (UID: \"8defa80b-2b41-49b3-921b-57792d5fb3f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7"
Jul 29 19:27:57.595546 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:27:57.595380    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8defa80b-2b41-49b3-921b-57792d5fb3f2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7\" (UID: \"8defa80b-2b41-49b3-921b-57792d5fb3f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7"
Jul 29 19:27:57.595546 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:27:57.595410    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8defa80b-2b41-49b3-921b-57792d5fb3f2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7\" (UID: \"8defa80b-2b41-49b3-921b-57792d5fb3f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7"
Jul 29 19:27:57.595546 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:27:57.595455    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkpsh\" (UniqueName: \"kubernetes.io/projected/8defa80b-2b41-49b3-921b-57792d5fb3f2-kube-api-access-bkpsh\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7\" (UID: \"8defa80b-2b41-49b3-921b-57792d5fb3f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7"
Jul 29 19:27:57.595546 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:27:57.595483    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8defa80b-2b41-49b3-921b-57792d5fb3f2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7\" (UID: \"8defa80b-2b41-49b3-921b-57792d5fb3f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7"
Jul 29 19:27:57.595546 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:27:57.595505    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8defa80b-2b41-49b3-921b-57792d5fb3f2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7\" (UID: \"8defa80b-2b41-49b3-921b-57792d5fb3f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7"
Jul 29 19:27:57.595860 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:27:57.595838    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8defa80b-2b41-49b3-921b-57792d5fb3f2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7\" (UID: \"8defa80b-2b41-49b3-921b-57792d5fb3f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7"
Jul 29 19:27:57.595929 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:27:57.595859    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8defa80b-2b41-49b3-921b-57792d5fb3f2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7\" (UID: \"8defa80b-2b41-49b3-921b-57792d5fb3f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7"
Jul 29 19:27:57.595929 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:27:57.595901    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8defa80b-2b41-49b3-921b-57792d5fb3f2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7\" (UID: \"8defa80b-2b41-49b3-921b-57792d5fb3f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7"
Jul 29 19:27:57.597677 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:27:57.597658    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8defa80b-2b41-49b3-921b-57792d5fb3f2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7\" (UID: \"8defa80b-2b41-49b3-921b-57792d5fb3f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7"
Jul 29 19:27:57.598001 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:27:57.597986    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8defa80b-2b41-49b3-921b-57792d5fb3f2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7\" (UID: \"8defa80b-2b41-49b3-921b-57792d5fb3f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7"
Jul 29 19:27:57.604784 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:27:57.604761    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkpsh\" (UniqueName: \"kubernetes.io/projected/8defa80b-2b41-49b3-921b-57792d5fb3f2-kube-api-access-bkpsh\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7\" (UID: \"8defa80b-2b41-49b3-921b-57792d5fb3f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7"
Jul 29 19:27:57.762245 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:27:57.762143    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7"
Jul 29 19:27:57.895465 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:27:57.895430    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7"]
Jul 29 19:27:57.898664 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:27:57.898626    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8defa80b_2b41_49b3_921b_57792d5fb3f2.slice/crio-2ab72e257d9d3a415f716b915588c6765b2a9d57c7c0c633a5bef8ade9be9beb WatchSource:0}: Error finding container 2ab72e257d9d3a415f716b915588c6765b2a9d57c7c0c633a5bef8ade9be9beb: Status 404 returned error can't find the container with id 2ab72e257d9d3a415f716b915588c6765b2a9d57c7c0c633a5bef8ade9be9beb
Jul 29 19:27:57.899998 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:27:57.899980    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 19:27:57.919741 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:27:57.919706    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7" event={"ID":"8defa80b-2b41-49b3-921b-57792d5fb3f2","Type":"ContainerStarted","Data":"2ab72e257d9d3a415f716b915588c6765b2a9d57c7c0c633a5bef8ade9be9beb"}
Jul 29 19:27:59.006833 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:27:59.006779    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:27:59.929349 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:27:59.929315    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7" event={"ID":"8defa80b-2b41-49b3-921b-57792d5fb3f2","Type":"ContainerStarted","Data":"0c9eedb36355693a46158585a38752322f34d747d9497f9eca1c3e43e73eca52"}
Jul 29 19:27:59.946921 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:27:59.946874    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7" podStartSLOduration=1.896516254 podStartE2EDuration="2.946859757s" totalImagesPullingTime="1.050343503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:27:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:27:59.94548957 +0000 UTC m=+647.173878695" watchObservedRunningTime="2026-07-29 19:27:59.946859757 +0000 UTC m=+647.175248934"
Jul 29 19:28:07.762680 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:28:07.762643    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7"
Jul 29 19:28:07.763092 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:28:07.762699    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7"
Jul 29 19:28:07.774993 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:28:07.774968    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7"
Jul 29 19:28:07.973201 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:28:07.973176    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7"
Jul 29 19:28:12.702491 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:28:12.702451    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:28:29.300782 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:28:29.300744    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:30:00.161637 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:30:00.161600    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755890-2wzkf"]
Jul 29 19:30:00.164090 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:30:00.164074    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755890-2wzkf"
Jul 29 19:30:00.166296 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:30:00.166276    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hq9l2\""
Jul 29 19:30:00.173428 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:30:00.173411    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755890-2wzkf"
Jul 29 19:30:00.173815 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:30:00.173781    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755890-2wzkf"]
Jul 29 19:30:00.304176 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:30:00.304129    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755890-2wzkf"]
Jul 29 19:30:00.306421 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:30:00.306397    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3979261_f3fb_47da_ba69_d7ea080f6f48.slice/crio-d5703d02ea16e6e3f26c773133743d4d2c08000ad97506f8badd55ddb27f6ae9 WatchSource:0}: Error finding container d5703d02ea16e6e3f26c773133743d4d2c08000ad97506f8badd55ddb27f6ae9: Status 404 returned error can't find the container with id d5703d02ea16e6e3f26c773133743d4d2c08000ad97506f8badd55ddb27f6ae9
Jul 29 19:30:00.423703 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:30:00.423616    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755890-2wzkf" event={"ID":"c3979261-f3fb-47da-ba69-d7ea080f6f48","Type":"ContainerStarted","Data":"d5703d02ea16e6e3f26c773133743d4d2c08000ad97506f8badd55ddb27f6ae9"}
Jul 29 19:30:03.438199 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:30:03.438161    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755890-2wzkf" event={"ID":"c3979261-f3fb-47da-ba69-d7ea080f6f48","Type":"ContainerStarted","Data":"5baa23ad4dc74db01ef229a7d14a9925158f33db7bf45dd9a08e2fd7bf2e129b"}
Jul 29 19:30:03.457761 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:30:03.457705    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755890-2wzkf" podStartSLOduration=0.710831862 podStartE2EDuration="3.457688537s" totalImagesPullingTime="2.746856675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:30:03.455755792 +0000 UTC m=+770.684144918" watchObservedRunningTime="2026-07-29 19:30:03.457688537 +0000 UTC m=+770.686077640"
Jul 29 19:30:04.443708 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:30:04.443671    2583 generic.go:356] "Generic (PLEG): container finished" podID="c3979261-f3fb-47da-ba69-d7ea080f6f48" containerID="5baa23ad4dc74db01ef229a7d14a9925158f33db7bf45dd9a08e2fd7bf2e129b" exitCode=0
Jul 29 19:30:04.444103 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:30:04.443755    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755890-2wzkf" event={"ID":"c3979261-f3fb-47da-ba69-d7ea080f6f48","Type":"ContainerDied","Data":"5baa23ad4dc74db01ef229a7d14a9925158f33db7bf45dd9a08e2fd7bf2e129b"}
Jul 29 19:30:05.573656 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:30:05.573632    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755890-2wzkf"
Jul 29 19:30:06.453385 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:30:06.453357    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755890-2wzkf"
Jul 29 19:30:06.453385 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:30:06.453361    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755890-2wzkf" event={"ID":"c3979261-f3fb-47da-ba69-d7ea080f6f48","Type":"ContainerDied","Data":"d5703d02ea16e6e3f26c773133743d4d2c08000ad97506f8badd55ddb27f6ae9"}
Jul 29 19:30:06.453640 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:30:06.453410    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5703d02ea16e6e3f26c773133743d4d2c08000ad97506f8badd55ddb27f6ae9"
Jul 29 19:30:30.512965 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:30:30.512926    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:30:41.278485 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:30:41.278410    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:30:51.006522 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:30:51.006483    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:32:35.999484 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:32:35.999345    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:33:13.800589 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:33:13.800553    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:33:23.106457 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:33:23.106419    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:33:33.211031 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:33:33.210987    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:33:40.699742 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:33:40.699651    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:33:52.506495 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:33:52.506462    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:34:03.705705 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:34:03.705671    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:34:14.797023 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:34:14.796985    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:34:24.955246 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:34:24.955212    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:34:47.157482 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:34:47.157444    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:34:56.095244 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:34:56.095208    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:35:22.507386 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:35:22.507307    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:35:35.105401 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:35:35.105363    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:35:44.102110 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:35:44.102073    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:36:02.718983 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:36:02.718951    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:36:12.205665 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:36:12.205630    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:36:21.402282 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:36:21.402250    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:36:31.501486 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:36:31.501453    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:36:40.944340 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:36:40.944265    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:38:27.907827 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:38:27.907732    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:38:38.993953 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:38:38.993918    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:38:54.116280 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:38:54.116246    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:39:04.407927 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:39:04.407890    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:39:16.195393 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:39:16.195358    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:39:24.525719 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:39:24.525682    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:39:34.795542 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:39:34.795462    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:39:45.243783 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:39:45.243749    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:39:55.900091 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:39:55.900053    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:40:05.218494 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:40:05.218462    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:40:40.413840 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:40:40.413791    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:40:50.336483 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:40:50.336452    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:41:05.823373 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:05.823290    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:41:10.961353 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:10.961319    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv"]
Jul 29 19:41:10.961782 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:10.961763    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3979261-f3fb-47da-ba69-d7ea080f6f48" containerName="cleanup"
Jul 29 19:41:10.961853 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:10.961787    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3979261-f3fb-47da-ba69-d7ea080f6f48" containerName="cleanup"
Jul 29 19:41:10.961921 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:10.961909    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3979261-f3fb-47da-ba69-d7ea080f6f48" containerName="cleanup"
Jul 29 19:41:10.963946 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:10.963930    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv"
Jul 29 19:41:10.966922 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:10.966900    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-569c3022-kserve-self-signed-certs\""
Jul 29 19:41:10.975304 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:10.975275    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv"]
Jul 29 19:41:11.059816 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:11.059773    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6361b312-0163-476f-8be4-dbbb73bd41d9-model-cache\") pod \"e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv\" (UID: \"6361b312-0163-476f-8be4-dbbb73bd41d9\") " pod="llm/e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv"
Jul 29 19:41:11.060020 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:11.059868    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzwhr\" (UniqueName: \"kubernetes.io/projected/6361b312-0163-476f-8be4-dbbb73bd41d9-kube-api-access-qzwhr\") pod \"e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv\" (UID: \"6361b312-0163-476f-8be4-dbbb73bd41d9\") " pod="llm/e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv"
Jul 29 19:41:11.060020 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:11.059898    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6361b312-0163-476f-8be4-dbbb73bd41d9-home\") pod \"e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv\" (UID: \"6361b312-0163-476f-8be4-dbbb73bd41d9\") " pod="llm/e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv"
Jul 29 19:41:11.060020 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:11.059933    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6361b312-0163-476f-8be4-dbbb73bd41d9-tls-certs\") pod \"e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv\" (UID: \"6361b312-0163-476f-8be4-dbbb73bd41d9\") " pod="llm/e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv"
Jul 29 19:41:11.060020 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:11.059956    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6361b312-0163-476f-8be4-dbbb73bd41d9-dshm\") pod \"e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv\" (UID: \"6361b312-0163-476f-8be4-dbbb73bd41d9\") " pod="llm/e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv"
Jul 29 19:41:11.060194 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:11.060059    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6361b312-0163-476f-8be4-dbbb73bd41d9-tmp-dir\") pod \"e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv\" (UID: \"6361b312-0163-476f-8be4-dbbb73bd41d9\") " pod="llm/e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv"
Jul 29 19:41:11.161475 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:11.161433    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6361b312-0163-476f-8be4-dbbb73bd41d9-tmp-dir\") pod \"e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv\" (UID: \"6361b312-0163-476f-8be4-dbbb73bd41d9\") " pod="llm/e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv"
Jul 29 19:41:11.161475 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:11.161475    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6361b312-0163-476f-8be4-dbbb73bd41d9-model-cache\") pod \"e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv\" (UID: \"6361b312-0163-476f-8be4-dbbb73bd41d9\") " pod="llm/e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv"
Jul 29 19:41:11.161741 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:11.161503    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzwhr\" (UniqueName: \"kubernetes.io/projected/6361b312-0163-476f-8be4-dbbb73bd41d9-kube-api-access-qzwhr\") pod \"e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv\" (UID: \"6361b312-0163-476f-8be4-dbbb73bd41d9\") " pod="llm/e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv"
Jul 29 19:41:11.161741 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:11.161521    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6361b312-0163-476f-8be4-dbbb73bd41d9-home\") pod \"e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv\" (UID: \"6361b312-0163-476f-8be4-dbbb73bd41d9\") " pod="llm/e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv"
Jul 29 19:41:11.161741 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:11.161543    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6361b312-0163-476f-8be4-dbbb73bd41d9-tls-certs\") pod \"e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv\" (UID: \"6361b312-0163-476f-8be4-dbbb73bd41d9\") " pod="llm/e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv"
Jul 29 19:41:11.161741 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:11.161563    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6361b312-0163-476f-8be4-dbbb73bd41d9-dshm\") pod \"e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv\" (UID: \"6361b312-0163-476f-8be4-dbbb73bd41d9\") " pod="llm/e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv"
Jul 29 19:41:11.162054 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:11.162032    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6361b312-0163-476f-8be4-dbbb73bd41d9-tmp-dir\") pod \"e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv\" (UID: \"6361b312-0163-476f-8be4-dbbb73bd41d9\") " pod="llm/e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv"
Jul 29 19:41:11.162054 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:11.162044    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6361b312-0163-476f-8be4-dbbb73bd41d9-home\") pod \"e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv\" (UID: \"6361b312-0163-476f-8be4-dbbb73bd41d9\") " pod="llm/e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv"
Jul 29 19:41:11.162147 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:11.162097    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6361b312-0163-476f-8be4-dbbb73bd41d9-model-cache\") pod \"e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv\" (UID: \"6361b312-0163-476f-8be4-dbbb73bd41d9\") " pod="llm/e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv"
Jul 29 19:41:11.163897 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:11.163879    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6361b312-0163-476f-8be4-dbbb73bd41d9-dshm\") pod \"e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv\" (UID: \"6361b312-0163-476f-8be4-dbbb73bd41d9\") " pod="llm/e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv"
Jul 29 19:41:11.164070 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:11.164053    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6361b312-0163-476f-8be4-dbbb73bd41d9-tls-certs\") pod \"e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv\" (UID: \"6361b312-0163-476f-8be4-dbbb73bd41d9\") " pod="llm/e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv"
Jul 29 19:41:11.170204 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:11.170182    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzwhr\" (UniqueName: \"kubernetes.io/projected/6361b312-0163-476f-8be4-dbbb73bd41d9-kube-api-access-qzwhr\") pod \"e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv\" (UID: \"6361b312-0163-476f-8be4-dbbb73bd41d9\") " pod="llm/e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv"
Jul 29 19:41:11.273603 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:11.273503    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv"
Jul 29 19:41:11.413333 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:11.413305    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv"]
Jul 29 19:41:11.415296 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:41:11.415267    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6361b312_0163_476f_8be4_dbbb73bd41d9.slice/crio-bd374a50a3b6b5fbed1dd72490f1545d01210b1b945d118b4715cf561a69d7fb WatchSource:0}: Error finding container bd374a50a3b6b5fbed1dd72490f1545d01210b1b945d118b4715cf561a69d7fb: Status 404 returned error can't find the container with id bd374a50a3b6b5fbed1dd72490f1545d01210b1b945d118b4715cf561a69d7fb
Jul 29 19:41:12.226223 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:12.226190    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv" event={"ID":"6361b312-0163-476f-8be4-dbbb73bd41d9","Type":"ContainerStarted","Data":"502e910a2522fb9a920acceef90e9b5e458b81f2f80aa0f93754c0a7bb93ddd3"}
Jul 29 19:41:12.226223 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:12.226229    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv" event={"ID":"6361b312-0163-476f-8be4-dbbb73bd41d9","Type":"ContainerStarted","Data":"bd374a50a3b6b5fbed1dd72490f1545d01210b1b945d118b4715cf561a69d7fb"}
Jul 29 19:41:12.244526 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:12.244479    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv" podStartSLOduration=2.244461323 podStartE2EDuration="2.244461323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:41:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:41:12.243698282 +0000 UTC m=+1439.472087409" watchObservedRunningTime="2026-07-29 19:41:12.244461323 +0000 UTC m=+1439.472850448"
Jul 29 19:41:21.274633 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:21.274578    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv"
Jul 29 19:41:21.274633 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:21.274637    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv"
Jul 29 19:41:21.287133 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:21.287103    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv"
Jul 29 19:41:21.800212 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:21.800176    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:41:22.278487 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:22.278448    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv"
Jul 29 19:41:32.897020 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:32.896988    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv"]
Jul 29 19:41:35.798023 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:35.797988    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:41:47.916120 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:47.916048    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv" podUID="6361b312-0163-476f-8be4-dbbb73bd41d9" containerName="main" containerID="cri-o://502e910a2522fb9a920acceef90e9b5e458b81f2f80aa0f93754c0a7bb93ddd3" gracePeriod=45
Jul 29 19:41:48.165293 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:48.165266    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv"
Jul 29 19:41:48.285979 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:48.285898    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6361b312-0163-476f-8be4-dbbb73bd41d9-model-cache\") pod \"6361b312-0163-476f-8be4-dbbb73bd41d9\" (UID: \"6361b312-0163-476f-8be4-dbbb73bd41d9\") "
Jul 29 19:41:48.285979 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:48.285936    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6361b312-0163-476f-8be4-dbbb73bd41d9-tls-certs\") pod \"6361b312-0163-476f-8be4-dbbb73bd41d9\" (UID: \"6361b312-0163-476f-8be4-dbbb73bd41d9\") "
Jul 29 19:41:48.285979 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:48.285965    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6361b312-0163-476f-8be4-dbbb73bd41d9-home\") pod \"6361b312-0163-476f-8be4-dbbb73bd41d9\" (UID: \"6361b312-0163-476f-8be4-dbbb73bd41d9\") "
Jul 29 19:41:48.286216 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:48.286084    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6361b312-0163-476f-8be4-dbbb73bd41d9-dshm\") pod \"6361b312-0163-476f-8be4-dbbb73bd41d9\" (UID: \"6361b312-0163-476f-8be4-dbbb73bd41d9\") "
Jul 29 19:41:48.286216 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:48.286129    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6361b312-0163-476f-8be4-dbbb73bd41d9-tmp-dir\") pod \"6361b312-0163-476f-8be4-dbbb73bd41d9\" (UID: \"6361b312-0163-476f-8be4-dbbb73bd41d9\") "
Jul 29 19:41:48.286216 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:48.286157    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6361b312-0163-476f-8be4-dbbb73bd41d9-home" (OuterVolumeSpecName: "home") pod "6361b312-0163-476f-8be4-dbbb73bd41d9" (UID: "6361b312-0163-476f-8be4-dbbb73bd41d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:41:48.286216 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:48.286160    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qzwhr\" (UniqueName: \"kubernetes.io/projected/6361b312-0163-476f-8be4-dbbb73bd41d9-kube-api-access-qzwhr\") pod \"6361b312-0163-476f-8be4-dbbb73bd41d9\" (UID: \"6361b312-0163-476f-8be4-dbbb73bd41d9\") "
Jul 29 19:41:48.286216 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:48.286201    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6361b312-0163-476f-8be4-dbbb73bd41d9-model-cache" (OuterVolumeSpecName: "model-cache") pod "6361b312-0163-476f-8be4-dbbb73bd41d9" (UID: "6361b312-0163-476f-8be4-dbbb73bd41d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:41:48.286453 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:48.286375    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6361b312-0163-476f-8be4-dbbb73bd41d9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6361b312-0163-476f-8be4-dbbb73bd41d9" (UID: "6361b312-0163-476f-8be4-dbbb73bd41d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:41:48.286453 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:48.286404    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6361b312-0163-476f-8be4-dbbb73bd41d9-model-cache\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:41:48.286453 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:48.286416    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6361b312-0163-476f-8be4-dbbb73bd41d9-home\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:41:48.288224 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:48.288192    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6361b312-0163-476f-8be4-dbbb73bd41d9-dshm" (OuterVolumeSpecName: "dshm") pod "6361b312-0163-476f-8be4-dbbb73bd41d9" (UID: "6361b312-0163-476f-8be4-dbbb73bd41d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:41:48.288322 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:48.288250    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6361b312-0163-476f-8be4-dbbb73bd41d9-kube-api-access-qzwhr" (OuterVolumeSpecName: "kube-api-access-qzwhr") pod "6361b312-0163-476f-8be4-dbbb73bd41d9" (UID: "6361b312-0163-476f-8be4-dbbb73bd41d9"). InnerVolumeSpecName "kube-api-access-qzwhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:41:48.288322 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:48.288263    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6361b312-0163-476f-8be4-dbbb73bd41d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6361b312-0163-476f-8be4-dbbb73bd41d9" (UID: "6361b312-0163-476f-8be4-dbbb73bd41d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:41:48.377046 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:48.377008    2583 generic.go:356] "Generic (PLEG): container finished" podID="6361b312-0163-476f-8be4-dbbb73bd41d9" containerID="502e910a2522fb9a920acceef90e9b5e458b81f2f80aa0f93754c0a7bb93ddd3" exitCode=0
Jul 29 19:41:48.377046 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:48.377044    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv"
Jul 29 19:41:48.377259 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:48.377092    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv" event={"ID":"6361b312-0163-476f-8be4-dbbb73bd41d9","Type":"ContainerDied","Data":"502e910a2522fb9a920acceef90e9b5e458b81f2f80aa0f93754c0a7bb93ddd3"}
Jul 29 19:41:48.377259 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:48.377131    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv" event={"ID":"6361b312-0163-476f-8be4-dbbb73bd41d9","Type":"ContainerDied","Data":"bd374a50a3b6b5fbed1dd72490f1545d01210b1b945d118b4715cf561a69d7fb"}
Jul 29 19:41:48.377259 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:48.377154    2583 scope.go:117] "RemoveContainer" containerID="502e910a2522fb9a920acceef90e9b5e458b81f2f80aa0f93754c0a7bb93ddd3"
Jul 29 19:41:48.385681 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:48.385659    2583 scope.go:117] "RemoveContainer" containerID="502e910a2522fb9a920acceef90e9b5e458b81f2f80aa0f93754c0a7bb93ddd3"
Jul 29 19:41:48.385992 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:41:48.385974    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"502e910a2522fb9a920acceef90e9b5e458b81f2f80aa0f93754c0a7bb93ddd3\": container with ID starting with 502e910a2522fb9a920acceef90e9b5e458b81f2f80aa0f93754c0a7bb93ddd3 not found: ID does not exist" containerID="502e910a2522fb9a920acceef90e9b5e458b81f2f80aa0f93754c0a7bb93ddd3"
Jul 29 19:41:48.386047 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:48.386003    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"502e910a2522fb9a920acceef90e9b5e458b81f2f80aa0f93754c0a7bb93ddd3"} err="failed to get container status \"502e910a2522fb9a920acceef90e9b5e458b81f2f80aa0f93754c0a7bb93ddd3\": rpc error: code = NotFound desc = could not find container \"502e910a2522fb9a920acceef90e9b5e458b81f2f80aa0f93754c0a7bb93ddd3\": container with ID starting with 502e910a2522fb9a920acceef90e9b5e458b81f2f80aa0f93754c0a7bb93ddd3 not found: ID does not exist"
Jul 29 19:41:48.387350 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:48.387336    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6361b312-0163-476f-8be4-dbbb73bd41d9-tls-certs\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:41:48.387446 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:48.387354    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6361b312-0163-476f-8be4-dbbb73bd41d9-dshm\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:41:48.387446 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:48.387366    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6361b312-0163-476f-8be4-dbbb73bd41d9-tmp-dir\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:41:48.387446 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:48.387378    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzwhr\" (UniqueName: \"kubernetes.io/projected/6361b312-0163-476f-8be4-dbbb73bd41d9-kube-api-access-qzwhr\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:41:48.396904 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:48.396881    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv"]
Jul 29 19:41:48.400769 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:48.400751    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-569c3022-kserve-6dd4dfbdc9-qmtrv"]
Jul 29 19:41:49.098698 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:49.098664    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:41:49.341166 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:49.341135    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6361b312-0163-476f-8be4-dbbb73bd41d9" path="/var/lib/kubelet/pods/6361b312-0163-476f-8be4-dbbb73bd41d9/volumes"
Jul 29 19:41:56.007666 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:41:56.007629    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:42:10.905960 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:42:10.905927    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:42:20.004117 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:42:20.004086    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:42:29.305852 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:42:29.305818    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:42:37.109249 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:42:37.109159    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:42:48.110339 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:42:48.110306    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:42:57.204622 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:42:57.204587    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:43:09.906670 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:43:09.906634    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:43:19.204054 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:43:19.204012    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:43:28.007885 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:43:28.007757    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:43:38.606009 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:43:38.605968    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:43:47.505739 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:43:47.505701    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:43:54.309095 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:43:54.309062    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:44:10.863782 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:44:10.863696    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:44:22.702821 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:44:22.702767    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:44:31.048295 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:44:31.048261    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:44:41.109374 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:44:41.109336    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:44:49.899167 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:44:49.899131    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:45:00.146623 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:00.146586    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755905-69g49"]
Jul 29 19:45:00.147032 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:00.146906    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6361b312-0163-476f-8be4-dbbb73bd41d9" containerName="main"
Jul 29 19:45:00.147032 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:00.146917    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6361b312-0163-476f-8be4-dbbb73bd41d9" containerName="main"
Jul 29 19:45:00.147032 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:00.146973    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6361b312-0163-476f-8be4-dbbb73bd41d9" containerName="main"
Jul 29 19:45:00.149670 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:00.149653    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755905-69g49"
Jul 29 19:45:00.152271 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:00.152250    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hq9l2\""
Jul 29 19:45:00.158768 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:00.158752    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755905-69g49"
Jul 29 19:45:00.175929 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:00.175896    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755905-69g49"]
Jul 29 19:45:00.294653 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:00.294619    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755905-69g49"]
Jul 29 19:45:00.297734 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:45:00.297710    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod269c6be3_441e_43e8_8931_eec7457dfb01.slice/crio-8f78a0fc87d153912a7b23e46f25fb3f2ad98188aaa02bbae7d1ce6fc37bd1d8 WatchSource:0}: Error finding container 8f78a0fc87d153912a7b23e46f25fb3f2ad98188aaa02bbae7d1ce6fc37bd1d8: Status 404 returned error can't find the container with id 8f78a0fc87d153912a7b23e46f25fb3f2ad98188aaa02bbae7d1ce6fc37bd1d8
Jul 29 19:45:01.169496 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:01.169441    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755905-69g49" event={"ID":"269c6be3-441e-43e8-8931-eec7457dfb01","Type":"ContainerStarted","Data":"0eadf8ae3bdbcaf58e938c1256562c304d32f27ddee3f93a3640bb224fdeae41"}
Jul 29 19:45:01.169496 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:01.169496    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755905-69g49" event={"ID":"269c6be3-441e-43e8-8931-eec7457dfb01","Type":"ContainerStarted","Data":"8f78a0fc87d153912a7b23e46f25fb3f2ad98188aaa02bbae7d1ce6fc37bd1d8"}
Jul 29 19:45:01.192627 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:01.192575    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755905-69g49" podStartSLOduration=1.192561438 podStartE2EDuration="1.192561438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:45:01.19048706 +0000 UTC m=+1668.418876186" watchObservedRunningTime="2026-07-29 19:45:01.192561438 +0000 UTC m=+1668.420950564"
Jul 29 19:45:02.175513 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:02.175477    2583 generic.go:356] "Generic (PLEG): container finished" podID="269c6be3-441e-43e8-8931-eec7457dfb01" containerID="0eadf8ae3bdbcaf58e938c1256562c304d32f27ddee3f93a3640bb224fdeae41" exitCode=0
Jul 29 19:45:02.175992 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:02.175568    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755905-69g49" event={"ID":"269c6be3-441e-43e8-8931-eec7457dfb01","Type":"ContainerDied","Data":"0eadf8ae3bdbcaf58e938c1256562c304d32f27ddee3f93a3640bb224fdeae41"}
Jul 29 19:45:03.308266 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:03.308240    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755905-69g49"
Jul 29 19:45:04.185264 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:04.185220    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755905-69g49" event={"ID":"269c6be3-441e-43e8-8931-eec7457dfb01","Type":"ContainerDied","Data":"8f78a0fc87d153912a7b23e46f25fb3f2ad98188aaa02bbae7d1ce6fc37bd1d8"}
Jul 29 19:45:04.185264 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:04.185258    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755905-69g49"
Jul 29 19:45:04.185479 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:04.185318    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f78a0fc87d153912a7b23e46f25fb3f2ad98188aaa02bbae7d1ce6fc37bd1d8"
Jul 29 19:45:11.528204 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:11.528176    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:45:21.326685 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:21.326650    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e9d45d7a-6c8c56fdf4-gjtf5"]
Jul 29 19:45:21.327183 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:21.327160    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="269c6be3-441e-43e8-8931-eec7457dfb01" containerName="cleanup"
Jul 29 19:45:21.327254 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:21.327187    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="269c6be3-441e-43e8-8931-eec7457dfb01" containerName="cleanup"
Jul 29 19:45:21.327303 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:21.327272    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="269c6be3-441e-43e8-8931-eec7457dfb01" containerName="cleanup"
Jul 29 19:45:21.330229 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:21.330208    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9d45d7a-6c8c56fdf4-gjtf5"
Jul 29 19:45:21.332766 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:21.332734    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6wzbz\""
Jul 29 19:45:21.332932 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:21.332897    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 29 19:45:21.333266 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:21.333251    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e9d45d7a-serving-cert\""
Jul 29 19:45:21.333348 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:21.333280    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 19:45:21.349329 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:21.349290    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e9d45d7a-6c8c56fdf4-gjtf5"]
Jul 29 19:45:21.387173 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:21.387141    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q2b9\" (UniqueName: \"kubernetes.io/projected/c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca-kube-api-access-8q2b9\") pod \"maas-api-e2e-ait-e9d45d7a-6c8c56fdf4-gjtf5\" (UID: \"c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9d45d7a-6c8c56fdf4-gjtf5"
Jul 29 19:45:21.387380 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:21.387180    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca-maas-api-tls\") pod \"maas-api-e2e-ait-e9d45d7a-6c8c56fdf4-gjtf5\" (UID: \"c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9d45d7a-6c8c56fdf4-gjtf5"
Jul 29 19:45:21.387448 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:21.387405    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca-openshift-service-ca\") pod \"maas-api-e2e-ait-e9d45d7a-6c8c56fdf4-gjtf5\" (UID: \"c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9d45d7a-6c8c56fdf4-gjtf5"
Jul 29 19:45:21.488496 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:21.488456    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca-openshift-service-ca\") pod \"maas-api-e2e-ait-e9d45d7a-6c8c56fdf4-gjtf5\" (UID: \"c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9d45d7a-6c8c56fdf4-gjtf5"
Jul 29 19:45:21.488677 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:21.488521    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q2b9\" (UniqueName: \"kubernetes.io/projected/c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca-kube-api-access-8q2b9\") pod \"maas-api-e2e-ait-e9d45d7a-6c8c56fdf4-gjtf5\" (UID: \"c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9d45d7a-6c8c56fdf4-gjtf5"
Jul 29 19:45:21.488677 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:21.488562    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca-maas-api-tls\") pod \"maas-api-e2e-ait-e9d45d7a-6c8c56fdf4-gjtf5\" (UID: \"c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9d45d7a-6c8c56fdf4-gjtf5"
Jul 29 19:45:21.489483 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:21.489453    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca-openshift-service-ca\") pod \"maas-api-e2e-ait-e9d45d7a-6c8c56fdf4-gjtf5\" (UID: \"c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9d45d7a-6c8c56fdf4-gjtf5"
Jul 29 19:45:21.491737 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:21.491708    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca-maas-api-tls\") pod \"maas-api-e2e-ait-e9d45d7a-6c8c56fdf4-gjtf5\" (UID: \"c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9d45d7a-6c8c56fdf4-gjtf5"
Jul 29 19:45:21.497178 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:21.497146    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q2b9\" (UniqueName: \"kubernetes.io/projected/c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca-kube-api-access-8q2b9\") pod \"maas-api-e2e-ait-e9d45d7a-6c8c56fdf4-gjtf5\" (UID: \"c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9d45d7a-6c8c56fdf4-gjtf5"
Jul 29 19:45:21.641723 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:21.641685    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9d45d7a-6c8c56fdf4-gjtf5"
Jul 29 19:45:21.781824 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:21.781785    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e9d45d7a-6c8c56fdf4-gjtf5"]
Jul 29 19:45:21.783954 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:45:21.783926    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3c2d6c8_8a12_4a5e_a44b_8d36efa9d9ca.slice/crio-f95228e16d3e516348b3f71b2e67fe21df433a9b2f65409abd5985e8da9f9511 WatchSource:0}: Error finding container f95228e16d3e516348b3f71b2e67fe21df433a9b2f65409abd5985e8da9f9511: Status 404 returned error can't find the container with id f95228e16d3e516348b3f71b2e67fe21df433a9b2f65409abd5985e8da9f9511
Jul 29 19:45:21.785328 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:21.785309    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 19:45:22.264641 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:22.264608    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9d45d7a-6c8c56fdf4-gjtf5" event={"ID":"c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca","Type":"ContainerStarted","Data":"f95228e16d3e516348b3f71b2e67fe21df433a9b2f65409abd5985e8da9f9511"}
Jul 29 19:45:25.279951 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:25.279912    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9d45d7a-6c8c56fdf4-gjtf5" event={"ID":"c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca","Type":"ContainerStarted","Data":"8303dd5c169ca2b248bd60dde20f2da71774afc74c423db1cca141deb0020215"}
Jul 29 19:45:25.280445 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:25.280039    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9d45d7a-6c8c56fdf4-gjtf5"
Jul 29 19:45:25.297096 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:25.297047    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9d45d7a-6c8c56fdf4-gjtf5" podStartSLOduration=1.772932257 podStartE2EDuration="4.297031288s" totalImagesPullingTime="2.524099031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:45:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:45:25.296123955 +0000 UTC m=+1692.524513075" watchObservedRunningTime="2026-07-29 19:45:25.297031288 +0000 UTC m=+1692.525420413"
Jul 29 19:45:27.797825 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:27.797773    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:45:31.289124 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:31.289088    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9d45d7a-6c8c56fdf4-gjtf5"
Jul 29 19:45:46.755594 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:46.755519    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9d45d7a-9vg5v"]
Jul 29 19:45:46.759003 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:46.758982    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9d45d7a-9vg5v"
Jul 29 19:45:46.761039 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:46.761018    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hq9l2\""
Jul 29 19:45:46.765417 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:46.765396    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9d45d7a-9vg5v"]
Jul 29 19:45:46.797404 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:46.797375    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b60758e-84ce-4c7d-a3bc-b41515a5a584-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e9d45d7a-9vg5v\" (UID: \"3b60758e-84ce-4c7d-a3bc-b41515a5a584\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9d45d7a-9vg5v"
Jul 29 19:45:46.898819 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:46.898762    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b60758e-84ce-4c7d-a3bc-b41515a5a584-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e9d45d7a-9vg5v\" (UID: \"3b60758e-84ce-4c7d-a3bc-b41515a5a584\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9d45d7a-9vg5v"
Jul 29 19:45:46.899371 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:46.899351    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b60758e-84ce-4c7d-a3bc-b41515a5a584-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e9d45d7a-9vg5v\" (UID: \"3b60758e-84ce-4c7d-a3bc-b41515a5a584\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9d45d7a-9vg5v"
Jul 29 19:45:47.069952 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:47.069857    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9d45d7a-9vg5v"
Jul 29 19:45:47.201465 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:47.201440    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9d45d7a-9vg5v"]
Jul 29 19:45:47.203545 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:45:47.203520    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b60758e_84ce_4c7d_a3bc_b41515a5a584.slice/crio-64d9b97931cd4b7d60e2318dda363a441dd685938d7dfd37bbb938fbd7942a86 WatchSource:0}: Error finding container 64d9b97931cd4b7d60e2318dda363a441dd685938d7dfd37bbb938fbd7942a86: Status 404 returned error can't find the container with id 64d9b97931cd4b7d60e2318dda363a441dd685938d7dfd37bbb938fbd7942a86
Jul 29 19:45:47.369482 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:47.369400    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9d45d7a-9vg5v" event={"ID":"3b60758e-84ce-4c7d-a3bc-b41515a5a584","Type":"ContainerStarted","Data":"64d9b97931cd4b7d60e2318dda363a441dd685938d7dfd37bbb938fbd7942a86"}
Jul 29 19:45:48.374947 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:48.374915    2583 generic.go:356] "Generic (PLEG): container finished" podID="3b60758e-84ce-4c7d-a3bc-b41515a5a584" containerID="ddb09ee7bc816f420595584c6384d7bfb5147cf5cb31b8f2f11c92e92ab1c812" exitCode=0
Jul 29 19:45:48.375316 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:48.375013    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9d45d7a-9vg5v" event={"ID":"3b60758e-84ce-4c7d-a3bc-b41515a5a584","Type":"ContainerDied","Data":"ddb09ee7bc816f420595584c6384d7bfb5147cf5cb31b8f2f11c92e92ab1c812"}
Jul 29 19:45:49.499145 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:49.499121    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9d45d7a-9vg5v"
Jul 29 19:45:49.620754 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:49.620723    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b60758e-84ce-4c7d-a3bc-b41515a5a584-maas-api-service-ca\") pod \"3b60758e-84ce-4c7d-a3bc-b41515a5a584\" (UID: \"3b60758e-84ce-4c7d-a3bc-b41515a5a584\") "
Jul 29 19:45:49.621084 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:49.621062    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b60758e-84ce-4c7d-a3bc-b41515a5a584-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3b60758e-84ce-4c7d-a3bc-b41515a5a584" (UID: "3b60758e-84ce-4c7d-a3bc-b41515a5a584"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:45:49.721518 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:49.721438    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b60758e-84ce-4c7d-a3bc-b41515a5a584-maas-api-service-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:45:50.383980 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:50.383951    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9d45d7a-9vg5v"
Jul 29 19:45:50.384178 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:50.383949    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9d45d7a-9vg5v" event={"ID":"3b60758e-84ce-4c7d-a3bc-b41515a5a584","Type":"ContainerDied","Data":"64d9b97931cd4b7d60e2318dda363a441dd685938d7dfd37bbb938fbd7942a86"}
Jul 29 19:45:50.384178 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:50.384072    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64d9b97931cd4b7d60e2318dda363a441dd685938d7dfd37bbb938fbd7942a86"
Jul 29 19:45:51.883621 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:51.883590    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e9d45d7a-6c8c56fdf4-gjtf5"]
Jul 29 19:45:51.884063 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:51.883851    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9d45d7a-6c8c56fdf4-gjtf5" podUID="c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca" containerName="maas-api" containerID="cri-o://8303dd5c169ca2b248bd60dde20f2da71774afc74c423db1cca141deb0020215" gracePeriod=30
Jul 29 19:45:52.095180 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:52.095147    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9d45d7a-9vg5v"]
Jul 29 19:45:52.098448 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:52.098427    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9d45d7a-9vg5v"]
Jul 29 19:45:52.124448 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:52.124426    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9d45d7a-6c8c56fdf4-gjtf5"
Jul 29 19:45:52.240844 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:52.240761    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8q2b9\" (UniqueName: \"kubernetes.io/projected/c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca-kube-api-access-8q2b9\") pod \"c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca\" (UID: \"c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca\") "
Jul 29 19:45:52.240985 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:52.240856    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca-maas-api-tls\") pod \"c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca\" (UID: \"c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca\") "
Jul 29 19:45:52.240985 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:52.240888    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca-openshift-service-ca\") pod \"c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca\" (UID: \"c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca\") "
Jul 29 19:45:52.241237 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:52.241211    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca" (UID: "c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:45:52.242939 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:52.242916    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca-kube-api-access-8q2b9" (OuterVolumeSpecName: "kube-api-access-8q2b9") pod "c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca" (UID: "c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca"). InnerVolumeSpecName "kube-api-access-8q2b9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:45:52.243140 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:52.243121    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca" (UID: "c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:45:52.341608 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:52.341577    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q2b9\" (UniqueName: \"kubernetes.io/projected/c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca-kube-api-access-8q2b9\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:45:52.341608 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:52.341604    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca-maas-api-tls\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:45:52.341608 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:52.341613    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca-openshift-service-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:45:52.393682 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:52.393650    2583 generic.go:356] "Generic (PLEG): container finished" podID="c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca" containerID="8303dd5c169ca2b248bd60dde20f2da71774afc74c423db1cca141deb0020215" exitCode=0
Jul 29 19:45:52.393843 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:52.393695    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9d45d7a-6c8c56fdf4-gjtf5"
Jul 29 19:45:52.393843 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:52.393708    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9d45d7a-6c8c56fdf4-gjtf5" event={"ID":"c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca","Type":"ContainerDied","Data":"8303dd5c169ca2b248bd60dde20f2da71774afc74c423db1cca141deb0020215"}
Jul 29 19:45:52.393843 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:52.393748    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9d45d7a-6c8c56fdf4-gjtf5" event={"ID":"c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca","Type":"ContainerDied","Data":"f95228e16d3e516348b3f71b2e67fe21df433a9b2f65409abd5985e8da9f9511"}
Jul 29 19:45:52.393843 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:52.393772    2583 scope.go:117] "RemoveContainer" containerID="8303dd5c169ca2b248bd60dde20f2da71774afc74c423db1cca141deb0020215"
Jul 29 19:45:52.402113 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:52.402070    2583 scope.go:117] "RemoveContainer" containerID="8303dd5c169ca2b248bd60dde20f2da71774afc74c423db1cca141deb0020215"
Jul 29 19:45:52.402433 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:45:52.402404    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8303dd5c169ca2b248bd60dde20f2da71774afc74c423db1cca141deb0020215\": container with ID starting with 8303dd5c169ca2b248bd60dde20f2da71774afc74c423db1cca141deb0020215 not found: ID does not exist" containerID="8303dd5c169ca2b248bd60dde20f2da71774afc74c423db1cca141deb0020215"
Jul 29 19:45:52.402526 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:52.402445    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8303dd5c169ca2b248bd60dde20f2da71774afc74c423db1cca141deb0020215"} err="failed to get container status \"8303dd5c169ca2b248bd60dde20f2da71774afc74c423db1cca141deb0020215\": rpc error: code = NotFound desc = could not find container \"8303dd5c169ca2b248bd60dde20f2da71774afc74c423db1cca141deb0020215\": container with ID starting with 8303dd5c169ca2b248bd60dde20f2da71774afc74c423db1cca141deb0020215 not found: ID does not exist"
Jul 29 19:45:52.415073 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:52.415050    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e9d45d7a-6c8c56fdf4-gjtf5"]
Jul 29 19:45:52.419106 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:52.419087    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e9d45d7a-6c8c56fdf4-gjtf5"]
Jul 29 19:45:53.340686 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:53.340655    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b60758e-84ce-4c7d-a3bc-b41515a5a584" path="/var/lib/kubelet/pods/3b60758e-84ce-4c7d-a3bc-b41515a5a584/volumes"
Jul 29 19:45:53.341221 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:53.341196    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca" path="/var/lib/kubelet/pods/c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca/volumes"
Jul 29 19:45:56.408785 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:45:56.408756    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:46:11.005197 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:11.005163    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:46:23.789224 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:23.789191    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9a86149a-fscq5"]
Jul 29 19:46:23.789709 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:23.789634    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b60758e-84ce-4c7d-a3bc-b41515a5a584" containerName="revoke-keys"
Jul 29 19:46:23.789709 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:23.789650    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b60758e-84ce-4c7d-a3bc-b41515a5a584" containerName="revoke-keys"
Jul 29 19:46:23.789709 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:23.789679    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca" containerName="maas-api"
Jul 29 19:46:23.789709 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:23.789687    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca" containerName="maas-api"
Jul 29 19:46:23.789951 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:23.789757    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b60758e-84ce-4c7d-a3bc-b41515a5a584" containerName="revoke-keys"
Jul 29 19:46:23.789951 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:23.789772    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3c2d6c8-8a12-4a5e-a44b-8d36efa9d9ca" containerName="maas-api"
Jul 29 19:46:23.792625 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:23.792601    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9a86149a-fscq5"
Jul 29 19:46:23.795989 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:23.795962    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hq9l2\""
Jul 29 19:46:23.796125 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:23.796104    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 19:46:23.809755 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:23.809733    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9a86149a-fscq5"]
Jul 29 19:46:23.891671 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:23.891647    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cef13f6-19f3-4041-8372-437a85c4ad3a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9a86149a-fscq5\" (UID: \"4cef13f6-19f3-4041-8372-437a85c4ad3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9a86149a-fscq5"
Jul 29 19:46:23.993047 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:23.992969    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cef13f6-19f3-4041-8372-437a85c4ad3a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9a86149a-fscq5\" (UID: \"4cef13f6-19f3-4041-8372-437a85c4ad3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9a86149a-fscq5"
Jul 29 19:46:23.993553 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:23.993535    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cef13f6-19f3-4041-8372-437a85c4ad3a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9a86149a-fscq5\" (UID: \"4cef13f6-19f3-4041-8372-437a85c4ad3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9a86149a-fscq5"
Jul 29 19:46:24.101978 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:24.101906    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9a86149a-fscq5"
Jul 29 19:46:24.228344 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:24.228287    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9a86149a-fscq5"]
Jul 29 19:46:24.230536 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:46:24.230502    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cef13f6_19f3_4041_8372_437a85c4ad3a.slice/crio-7bd1e5869255481cd449ab7056017a3158b2812e7788e515f2597142eb3a771e WatchSource:0}: Error finding container 7bd1e5869255481cd449ab7056017a3158b2812e7788e515f2597142eb3a771e: Status 404 returned error can't find the container with id 7bd1e5869255481cd449ab7056017a3158b2812e7788e515f2597142eb3a771e
Jul 29 19:46:24.526907 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:24.526878    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9a86149a-fscq5" event={"ID":"4cef13f6-19f3-4041-8372-437a85c4ad3a","Type":"ContainerStarted","Data":"e8a304e5ce0176eb0d0b4b2171817ef3c79b26a6555790797d146c4a33ee49b3"}
Jul 29 19:46:24.527056 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:24.526950    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9a86149a-fscq5" event={"ID":"4cef13f6-19f3-4041-8372-437a85c4ad3a","Type":"ContainerStarted","Data":"7bd1e5869255481cd449ab7056017a3158b2812e7788e515f2597142eb3a771e"}
Jul 29 19:46:24.549449 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:24.549403    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9a86149a-fscq5" podStartSLOduration=1.549389239 podStartE2EDuration="1.549389239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:46:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:46:24.54748545 +0000 UTC m=+1751.775874575" watchObservedRunningTime="2026-07-29 19:46:24.549389239 +0000 UTC m=+1751.777778364"
Jul 29 19:46:25.532813 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:25.532764    2583 generic.go:356] "Generic (PLEG): container finished" podID="4cef13f6-19f3-4041-8372-437a85c4ad3a" containerID="e8a304e5ce0176eb0d0b4b2171817ef3c79b26a6555790797d146c4a33ee49b3" exitCode=0
Jul 29 19:46:25.533228 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:25.532852    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9a86149a-fscq5" event={"ID":"4cef13f6-19f3-4041-8372-437a85c4ad3a","Type":"ContainerDied","Data":"e8a304e5ce0176eb0d0b4b2171817ef3c79b26a6555790797d146c4a33ee49b3"}
Jul 29 19:46:26.658664 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:26.658640    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9a86149a-fscq5"
Jul 29 19:46:26.711960 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:26.711929    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cef13f6-19f3-4041-8372-437a85c4ad3a-maas-api-service-ca\") pod \"4cef13f6-19f3-4041-8372-437a85c4ad3a\" (UID: \"4cef13f6-19f3-4041-8372-437a85c4ad3a\") "
Jul 29 19:46:26.712260 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:26.712242    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cef13f6-19f3-4041-8372-437a85c4ad3a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4cef13f6-19f3-4041-8372-437a85c4ad3a" (UID: "4cef13f6-19f3-4041-8372-437a85c4ad3a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:46:26.812534 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:26.812462    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cef13f6-19f3-4041-8372-437a85c4ad3a-maas-api-service-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:46:27.542411 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:27.542379    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9a86149a-fscq5" event={"ID":"4cef13f6-19f3-4041-8372-437a85c4ad3a","Type":"ContainerDied","Data":"7bd1e5869255481cd449ab7056017a3158b2812e7788e515f2597142eb3a771e"}
Jul 29 19:46:27.542411 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:27.542416    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9a86149a-fscq5"
Jul 29 19:46:27.542596 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:27.542428    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bd1e5869255481cd449ab7056017a3158b2812e7788e515f2597142eb3a771e"
Jul 29 19:46:30.352575 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:30.352544    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9a86149a-fscq5"]
Jul 29 19:46:30.355122 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:30.355098    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9a86149a-fscq5"]
Jul 29 19:46:31.340608 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:31.340578    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cef13f6-19f3-4041-8372-437a85c4ad3a" path="/var/lib/kubelet/pods/4cef13f6-19f3-4041-8372-437a85c4ad3a/volumes"
Jul 29 19:46:33.897664 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:33.897635    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2f25c52f-5b47868696-8dsz5"]
Jul 29 19:46:33.898092 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:33.898076    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cef13f6-19f3-4041-8372-437a85c4ad3a" containerName="revoke-keys"
Jul 29 19:46:33.898130 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:33.898095    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cef13f6-19f3-4041-8372-437a85c4ad3a" containerName="revoke-keys"
Jul 29 19:46:33.898211 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:33.898200    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cef13f6-19f3-4041-8372-437a85c4ad3a" containerName="revoke-keys"
Jul 29 19:46:33.901559 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:33.901533    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f25c52f-5b47868696-8dsz5"
Jul 29 19:46:33.903607 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:33.903585    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-2f25c52f-serving-cert\""
Jul 29 19:46:33.903926 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:33.903903    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 19:46:33.904042 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:33.903937    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 29 19:46:33.904474 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:33.904457    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6wzbz\""
Jul 29 19:46:33.912445 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:33.912426    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2f25c52f-5b47868696-8dsz5"]
Jul 29 19:46:33.961771 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:33.961745    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6zmp\" (UniqueName: \"kubernetes.io/projected/d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a-kube-api-access-w6zmp\") pod \"maas-api-e2e-ait-2f25c52f-5b47868696-8dsz5\" (UID: \"d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f25c52f-5b47868696-8dsz5"
Jul 29 19:46:33.961947 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:33.961844    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a-maas-api-tls\") pod \"maas-api-e2e-ait-2f25c52f-5b47868696-8dsz5\" (UID: \"d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f25c52f-5b47868696-8dsz5"
Jul 29 19:46:33.961947 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:33.961869    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a-openshift-service-ca\") pod \"maas-api-e2e-ait-2f25c52f-5b47868696-8dsz5\" (UID: \"d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f25c52f-5b47868696-8dsz5"
Jul 29 19:46:34.062490 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:34.062459    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6zmp\" (UniqueName: \"kubernetes.io/projected/d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a-kube-api-access-w6zmp\") pod \"maas-api-e2e-ait-2f25c52f-5b47868696-8dsz5\" (UID: \"d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f25c52f-5b47868696-8dsz5"
Jul 29 19:46:34.062626 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:34.062517    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a-maas-api-tls\") pod \"maas-api-e2e-ait-2f25c52f-5b47868696-8dsz5\" (UID: \"d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f25c52f-5b47868696-8dsz5"
Jul 29 19:46:34.062626 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:34.062535    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a-openshift-service-ca\") pod \"maas-api-e2e-ait-2f25c52f-5b47868696-8dsz5\" (UID: \"d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f25c52f-5b47868696-8dsz5"
Jul 29 19:46:34.062698 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:46:34.062663    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-2f25c52f-serving-cert: secret "maas-api-e2e-ait-2f25c52f-serving-cert" not found
Jul 29 19:46:34.062739 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:46:34.062730    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a-maas-api-tls podName:d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a nodeName:}" failed. No retries permitted until 2026-07-29 19:46:34.562714898 +0000 UTC m=+1761.791104002 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a-maas-api-tls") pod "maas-api-e2e-ait-2f25c52f-5b47868696-8dsz5" (UID: "d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a") : secret "maas-api-e2e-ait-2f25c52f-serving-cert" not found
Jul 29 19:46:34.063136 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:34.063121    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a-openshift-service-ca\") pod \"maas-api-e2e-ait-2f25c52f-5b47868696-8dsz5\" (UID: \"d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f25c52f-5b47868696-8dsz5"
Jul 29 19:46:34.085251 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:34.085227    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6zmp\" (UniqueName: \"kubernetes.io/projected/d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a-kube-api-access-w6zmp\") pod \"maas-api-e2e-ait-2f25c52f-5b47868696-8dsz5\" (UID: \"d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f25c52f-5b47868696-8dsz5"
Jul 29 19:46:34.342074 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:34.341985    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:46:34.567950 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:34.567915    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a-maas-api-tls\") pod \"maas-api-e2e-ait-2f25c52f-5b47868696-8dsz5\" (UID: \"d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f25c52f-5b47868696-8dsz5"
Jul 29 19:46:34.568107 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:46:34.568069    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-2f25c52f-serving-cert: secret "maas-api-e2e-ait-2f25c52f-serving-cert" not found
Jul 29 19:46:34.568153 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:46:34.568148    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a-maas-api-tls podName:d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a nodeName:}" failed. No retries permitted until 2026-07-29 19:46:35.568133825 +0000 UTC m=+1762.796522933 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a-maas-api-tls") pod "maas-api-e2e-ait-2f25c52f-5b47868696-8dsz5" (UID: "d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a") : secret "maas-api-e2e-ait-2f25c52f-serving-cert" not found
Jul 29 19:46:35.575001 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:35.574971    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a-maas-api-tls\") pod \"maas-api-e2e-ait-2f25c52f-5b47868696-8dsz5\" (UID: \"d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f25c52f-5b47868696-8dsz5"
Jul 29 19:46:35.577258 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:35.577241    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a-maas-api-tls\") pod \"maas-api-e2e-ait-2f25c52f-5b47868696-8dsz5\" (UID: \"d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f25c52f-5b47868696-8dsz5"
Jul 29 19:46:35.712820 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:35.712773    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f25c52f-5b47868696-8dsz5"
Jul 29 19:46:35.841320 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:35.841299    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2f25c52f-5b47868696-8dsz5"]
Jul 29 19:46:35.843294 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:46:35.843267    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd76b5e3e_eb3d_4189_a41d_c7e6a9fa817a.slice/crio-f66b3c36e37cdc75df6e1d3a2841254b6cdeb8d64c715c18d7afa2ca9bc855fc WatchSource:0}: Error finding container f66b3c36e37cdc75df6e1d3a2841254b6cdeb8d64c715c18d7afa2ca9bc855fc: Status 404 returned error can't find the container with id f66b3c36e37cdc75df6e1d3a2841254b6cdeb8d64c715c18d7afa2ca9bc855fc
Jul 29 19:46:36.580948 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:36.580865    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f25c52f-5b47868696-8dsz5" event={"ID":"d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a","Type":"ContainerStarted","Data":"731387cbe03bdd5e8ac68528039cab66bb09426a40d4b3d7076c1d69a0779fd4"}
Jul 29 19:46:36.580948 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:36.580926    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f25c52f-5b47868696-8dsz5" event={"ID":"d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a","Type":"ContainerStarted","Data":"f66b3c36e37cdc75df6e1d3a2841254b6cdeb8d64c715c18d7afa2ca9bc855fc"}
Jul 29 19:46:36.581311 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:36.580953    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f25c52f-5b47868696-8dsz5"
Jul 29 19:46:36.603152 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:36.603107    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f25c52f-5b47868696-8dsz5" podStartSLOduration=3.171687367 podStartE2EDuration="3.603091656s" totalImagesPullingTime="431.404289ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:46:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:46:36.601093183 +0000 UTC m=+1763.829482310" watchObservedRunningTime="2026-07-29 19:46:36.603091656 +0000 UTC m=+1763.831480781"
Jul 29 19:46:42.589873 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:42.589841    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f25c52f-5b47868696-8dsz5"
Jul 29 19:46:44.916491 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:44.916444    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:46:59.409947 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:59.409907    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f25c52f-l2szm"]
Jul 29 19:46:59.413185 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:59.413169    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f25c52f-l2szm"
Jul 29 19:46:59.415532 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:59.415511    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hq9l2\""
Jul 29 19:46:59.422188 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:59.422161    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f25c52f-l2szm"]
Jul 29 19:46:59.475672 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:59.475632    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d367b285-743e-46d0-9903-5b0f8860b87a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2f25c52f-l2szm\" (UID: \"d367b285-743e-46d0-9903-5b0f8860b87a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f25c52f-l2szm"
Jul 29 19:46:59.577199 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:59.577165    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d367b285-743e-46d0-9903-5b0f8860b87a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2f25c52f-l2szm\" (UID: \"d367b285-743e-46d0-9903-5b0f8860b87a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f25c52f-l2szm"
Jul 29 19:46:59.577849 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:59.577794    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d367b285-743e-46d0-9903-5b0f8860b87a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2f25c52f-l2szm\" (UID: \"d367b285-743e-46d0-9903-5b0f8860b87a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f25c52f-l2szm"
Jul 29 19:46:59.725721 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:59.725591    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f25c52f-l2szm"
Jul 29 19:46:59.883994 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:46:59.883966    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f25c52f-l2szm"]
Jul 29 19:46:59.885151 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:46:59.885120    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd367b285_743e_46d0_9903_5b0f8860b87a.slice/crio-cac066ab43d93b3fefc73dc1074a3d8061dc03d533c3a15e8e6446f081c36ffb WatchSource:0}: Error finding container cac066ab43d93b3fefc73dc1074a3d8061dc03d533c3a15e8e6446f081c36ffb: Status 404 returned error can't find the container with id cac066ab43d93b3fefc73dc1074a3d8061dc03d533c3a15e8e6446f081c36ffb
Jul 29 19:47:00.681465 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:00.681431    2583 generic.go:356] "Generic (PLEG): container finished" podID="d367b285-743e-46d0-9903-5b0f8860b87a" containerID="d366172556c2df39c388e38b9370df6a4ff84a287540cd33c531f2aaeb25198a" exitCode=0
Jul 29 19:47:00.681839 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:00.681501    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f25c52f-l2szm" event={"ID":"d367b285-743e-46d0-9903-5b0f8860b87a","Type":"ContainerDied","Data":"d366172556c2df39c388e38b9370df6a4ff84a287540cd33c531f2aaeb25198a"}
Jul 29 19:47:00.681839 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:00.681572    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f25c52f-l2szm" event={"ID":"d367b285-743e-46d0-9903-5b0f8860b87a","Type":"ContainerStarted","Data":"cac066ab43d93b3fefc73dc1074a3d8061dc03d533c3a15e8e6446f081c36ffb"}
Jul 29 19:47:01.817903 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:01.817874    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f25c52f-l2szm"
Jul 29 19:47:01.899901 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:01.899868    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d367b285-743e-46d0-9903-5b0f8860b87a-maas-api-service-ca\") pod \"d367b285-743e-46d0-9903-5b0f8860b87a\" (UID: \"d367b285-743e-46d0-9903-5b0f8860b87a\") "
Jul 29 19:47:01.900248 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:01.900220    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d367b285-743e-46d0-9903-5b0f8860b87a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d367b285-743e-46d0-9903-5b0f8860b87a" (UID: "d367b285-743e-46d0-9903-5b0f8860b87a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:47:02.000934 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:02.000858    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d367b285-743e-46d0-9903-5b0f8860b87a-maas-api-service-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:47:02.692000 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:02.691967    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f25c52f-l2szm"
Jul 29 19:47:02.692000 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:02.691977    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f25c52f-l2szm" event={"ID":"d367b285-743e-46d0-9903-5b0f8860b87a","Type":"ContainerDied","Data":"cac066ab43d93b3fefc73dc1074a3d8061dc03d533c3a15e8e6446f081c36ffb"}
Jul 29 19:47:02.692213 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:02.692027    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cac066ab43d93b3fefc73dc1074a3d8061dc03d533c3a15e8e6446f081c36ffb"
Jul 29 19:47:04.336873 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:04.336836    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2f25c52f-5b47868696-8dsz5"]
Jul 29 19:47:04.337291 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:04.337036    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f25c52f-5b47868696-8dsz5" podUID="d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a" containerName="maas-api" containerID="cri-o://731387cbe03bdd5e8ac68528039cab66bb09426a40d4b3d7076c1d69a0779fd4" gracePeriod=30
Jul 29 19:47:04.533320 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:04.533286    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f25c52f-l2szm"]
Jul 29 19:47:04.536951 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:04.536918    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f25c52f-l2szm"]
Jul 29 19:47:04.591172 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:04.591091    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f25c52f-5b47868696-8dsz5"
Jul 29 19:47:04.702194 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:04.702161    2583 generic.go:356] "Generic (PLEG): container finished" podID="d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a" containerID="731387cbe03bdd5e8ac68528039cab66bb09426a40d4b3d7076c1d69a0779fd4" exitCode=0
Jul 29 19:47:04.702363 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:04.702218    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f25c52f-5b47868696-8dsz5"
Jul 29 19:47:04.702363 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:04.702248    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f25c52f-5b47868696-8dsz5" event={"ID":"d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a","Type":"ContainerDied","Data":"731387cbe03bdd5e8ac68528039cab66bb09426a40d4b3d7076c1d69a0779fd4"}
Jul 29 19:47:04.702363 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:04.702307    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f25c52f-5b47868696-8dsz5" event={"ID":"d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a","Type":"ContainerDied","Data":"f66b3c36e37cdc75df6e1d3a2841254b6cdeb8d64c715c18d7afa2ca9bc855fc"}
Jul 29 19:47:04.702363 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:04.702336    2583 scope.go:117] "RemoveContainer" containerID="731387cbe03bdd5e8ac68528039cab66bb09426a40d4b3d7076c1d69a0779fd4"
Jul 29 19:47:04.710458 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:04.710437    2583 scope.go:117] "RemoveContainer" containerID="731387cbe03bdd5e8ac68528039cab66bb09426a40d4b3d7076c1d69a0779fd4"
Jul 29 19:47:04.710759 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:47:04.710734    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"731387cbe03bdd5e8ac68528039cab66bb09426a40d4b3d7076c1d69a0779fd4\": container with ID starting with 731387cbe03bdd5e8ac68528039cab66bb09426a40d4b3d7076c1d69a0779fd4 not found: ID does not exist" containerID="731387cbe03bdd5e8ac68528039cab66bb09426a40d4b3d7076c1d69a0779fd4"
Jul 29 19:47:04.710842 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:04.710766    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"731387cbe03bdd5e8ac68528039cab66bb09426a40d4b3d7076c1d69a0779fd4"} err="failed to get container status \"731387cbe03bdd5e8ac68528039cab66bb09426a40d4b3d7076c1d69a0779fd4\": rpc error: code = NotFound desc = could not find container \"731387cbe03bdd5e8ac68528039cab66bb09426a40d4b3d7076c1d69a0779fd4\": container with ID starting with 731387cbe03bdd5e8ac68528039cab66bb09426a40d4b3d7076c1d69a0779fd4 not found: ID does not exist"
Jul 29 19:47:04.724105 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:04.724070    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6zmp\" (UniqueName: \"kubernetes.io/projected/d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a-kube-api-access-w6zmp\") pod \"d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a\" (UID: \"d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a\") "
Jul 29 19:47:04.724253 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:04.724121    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a-openshift-service-ca\") pod \"d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a\" (UID: \"d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a\") "
Jul 29 19:47:04.724253 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:04.724147    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a-maas-api-tls\") pod \"d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a\" (UID: \"d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a\") "
Jul 29 19:47:04.724500 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:04.724477    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a" (UID: "d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:47:04.726262 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:04.726232    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a" (UID: "d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:47:04.726416 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:04.726390    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a-kube-api-access-w6zmp" (OuterVolumeSpecName: "kube-api-access-w6zmp") pod "d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a" (UID: "d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a"). InnerVolumeSpecName "kube-api-access-w6zmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:47:04.824925 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:04.824896    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a-maas-api-tls\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:47:04.824925 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:04.824922    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6zmp\" (UniqueName: \"kubernetes.io/projected/d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a-kube-api-access-w6zmp\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:47:04.824925 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:04.824935    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a-openshift-service-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:47:05.023499 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:05.023463    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2f25c52f-5b47868696-8dsz5"]
Jul 29 19:47:05.026170 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:05.026139    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2f25c52f-5b47868696-8dsz5"]
Jul 29 19:47:05.344583 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:05.341368    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d367b285-743e-46d0-9903-5b0f8860b87a" path="/var/lib/kubelet/pods/d367b285-743e-46d0-9903-5b0f8860b87a/volumes"
Jul 29 19:47:05.344583 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:05.341954    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a" path="/var/lib/kubelet/pods/d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a/volumes"
Jul 29 19:47:08.401104 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:08.401069    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:47:09.222837 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:09.222792    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ac4a75bf-7d9955bfd9-v5kjd"]
Jul 29 19:47:09.223100 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:09.223088    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d367b285-743e-46d0-9903-5b0f8860b87a" containerName="revoke-keys"
Jul 29 19:47:09.223169 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:09.223100    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d367b285-743e-46d0-9903-5b0f8860b87a" containerName="revoke-keys"
Jul 29 19:47:09.223169 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:09.223111    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a" containerName="maas-api"
Jul 29 19:47:09.223169 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:09.223117    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a" containerName="maas-api"
Jul 29 19:47:09.223320 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:09.223179    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d367b285-743e-46d0-9903-5b0f8860b87a" containerName="revoke-keys"
Jul 29 19:47:09.223320 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:09.223187    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d76b5e3e-eb3d-4189-a41d-c7e6a9fa817a" containerName="maas-api"
Jul 29 19:47:09.227401 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:09.227376    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ac4a75bf-7d9955bfd9-v5kjd"
Jul 29 19:47:09.229529 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:09.229506    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-ac4a75bf-serving-cert\""
Jul 29 19:47:09.229632 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:09.229614    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 19:47:09.229680 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:09.229614    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6wzbz\""
Jul 29 19:47:09.230181 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:09.230165    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 29 19:47:09.239065 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:09.239042    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ac4a75bf-7d9955bfd9-v5kjd"]
Jul 29 19:47:09.364625 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:09.364588    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5a5ea75-09d4-4ca9-b7da-a95f936a8642-maas-api-tls\") pod \"maas-api-e2e-derive-ac4a75bf-7d9955bfd9-v5kjd\" (UID: \"a5a5ea75-09d4-4ca9-b7da-a95f936a8642\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ac4a75bf-7d9955bfd9-v5kjd"
Jul 29 19:47:09.364625 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:09.364622    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5a5ea75-09d4-4ca9-b7da-a95f936a8642-openshift-service-ca\") pod \"maas-api-e2e-derive-ac4a75bf-7d9955bfd9-v5kjd\" (UID: \"a5a5ea75-09d4-4ca9-b7da-a95f936a8642\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ac4a75bf-7d9955bfd9-v5kjd"
Jul 29 19:47:09.364892 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:09.364643    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx8ws\" (UniqueName: \"kubernetes.io/projected/a5a5ea75-09d4-4ca9-b7da-a95f936a8642-kube-api-access-qx8ws\") pod \"maas-api-e2e-derive-ac4a75bf-7d9955bfd9-v5kjd\" (UID: \"a5a5ea75-09d4-4ca9-b7da-a95f936a8642\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ac4a75bf-7d9955bfd9-v5kjd"
Jul 29 19:47:09.465653 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:09.465596    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5a5ea75-09d4-4ca9-b7da-a95f936a8642-maas-api-tls\") pod \"maas-api-e2e-derive-ac4a75bf-7d9955bfd9-v5kjd\" (UID: \"a5a5ea75-09d4-4ca9-b7da-a95f936a8642\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ac4a75bf-7d9955bfd9-v5kjd"
Jul 29 19:47:09.465653 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:09.465651    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5a5ea75-09d4-4ca9-b7da-a95f936a8642-openshift-service-ca\") pod \"maas-api-e2e-derive-ac4a75bf-7d9955bfd9-v5kjd\" (UID: \"a5a5ea75-09d4-4ca9-b7da-a95f936a8642\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ac4a75bf-7d9955bfd9-v5kjd"
Jul 29 19:47:09.466156 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:09.465751    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx8ws\" (UniqueName: \"kubernetes.io/projected/a5a5ea75-09d4-4ca9-b7da-a95f936a8642-kube-api-access-qx8ws\") pod \"maas-api-e2e-derive-ac4a75bf-7d9955bfd9-v5kjd\" (UID: \"a5a5ea75-09d4-4ca9-b7da-a95f936a8642\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ac4a75bf-7d9955bfd9-v5kjd"
Jul 29 19:47:09.466414 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:09.466394    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5a5ea75-09d4-4ca9-b7da-a95f936a8642-openshift-service-ca\") pod \"maas-api-e2e-derive-ac4a75bf-7d9955bfd9-v5kjd\" (UID: \"a5a5ea75-09d4-4ca9-b7da-a95f936a8642\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ac4a75bf-7d9955bfd9-v5kjd"
Jul 29 19:47:09.468167 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:09.468135    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5a5ea75-09d4-4ca9-b7da-a95f936a8642-maas-api-tls\") pod \"maas-api-e2e-derive-ac4a75bf-7d9955bfd9-v5kjd\" (UID: \"a5a5ea75-09d4-4ca9-b7da-a95f936a8642\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ac4a75bf-7d9955bfd9-v5kjd"
Jul 29 19:47:09.474836 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:09.474754    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx8ws\" (UniqueName: \"kubernetes.io/projected/a5a5ea75-09d4-4ca9-b7da-a95f936a8642-kube-api-access-qx8ws\") pod \"maas-api-e2e-derive-ac4a75bf-7d9955bfd9-v5kjd\" (UID: \"a5a5ea75-09d4-4ca9-b7da-a95f936a8642\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ac4a75bf-7d9955bfd9-v5kjd"
Jul 29 19:47:09.537249 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:09.537218    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ac4a75bf-7d9955bfd9-v5kjd"
Jul 29 19:47:09.672442 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:09.672415    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ac4a75bf-7d9955bfd9-v5kjd"]
Jul 29 19:47:09.674916 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:47:09.674888    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5a5ea75_09d4_4ca9_b7da_a95f936a8642.slice/crio-a2d9d654ffad53ace461b97ab9c51a514e9fd3a587e6df0a9bc70b52b3d15852 WatchSource:0}: Error finding container a2d9d654ffad53ace461b97ab9c51a514e9fd3a587e6df0a9bc70b52b3d15852: Status 404 returned error can't find the container with id a2d9d654ffad53ace461b97ab9c51a514e9fd3a587e6df0a9bc70b52b3d15852
Jul 29 19:47:09.724647 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:09.724614    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ac4a75bf-7d9955bfd9-v5kjd" event={"ID":"a5a5ea75-09d4-4ca9-b7da-a95f936a8642","Type":"ContainerStarted","Data":"a2d9d654ffad53ace461b97ab9c51a514e9fd3a587e6df0a9bc70b52b3d15852"}
Jul 29 19:47:10.023370 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:10.023339    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ac4a75bf-57657fc88-gt5jn"]
Jul 29 19:47:10.027840 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:10.027814    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ac4a75bf-57657fc88-gt5jn"
Jul 29 19:47:10.030352 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:10.030315    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-ac4a75bf-dockercfg-k8294\""
Jul 29 19:47:10.030352 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:10.030335    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-ac4a75bf\""
Jul 29 19:47:10.036082 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:10.036062    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ac4a75bf-57657fc88-gt5jn"]
Jul 29 19:47:10.171310 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:10.171284    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa15f562-56f7-4d63-aba9-147757cc77cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ac4a75bf-57657fc88-gt5jn\" (UID: \"aa15f562-56f7-4d63-aba9-147757cc77cc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ac4a75bf-57657fc88-gt5jn"
Jul 29 19:47:10.171480 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:10.171335    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crmr9\" (UniqueName: \"kubernetes.io/projected/aa15f562-56f7-4d63-aba9-147757cc77cc-kube-api-access-crmr9\") pod \"payload-pre-processing-e2e-derive-ac4a75bf-57657fc88-gt5jn\" (UID: \"aa15f562-56f7-4d63-aba9-147757cc77cc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ac4a75bf-57657fc88-gt5jn"
Jul 29 19:47:10.271901 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:10.271871    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa15f562-56f7-4d63-aba9-147757cc77cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ac4a75bf-57657fc88-gt5jn\" (UID: \"aa15f562-56f7-4d63-aba9-147757cc77cc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ac4a75bf-57657fc88-gt5jn"
Jul 29 19:47:10.272066 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:10.271933    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crmr9\" (UniqueName: \"kubernetes.io/projected/aa15f562-56f7-4d63-aba9-147757cc77cc-kube-api-access-crmr9\") pod \"payload-pre-processing-e2e-derive-ac4a75bf-57657fc88-gt5jn\" (UID: \"aa15f562-56f7-4d63-aba9-147757cc77cc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ac4a75bf-57657fc88-gt5jn"
Jul 29 19:47:10.272550 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:10.272524    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa15f562-56f7-4d63-aba9-147757cc77cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ac4a75bf-57657fc88-gt5jn\" (UID: \"aa15f562-56f7-4d63-aba9-147757cc77cc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ac4a75bf-57657fc88-gt5jn"
Jul 29 19:47:10.294302 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:10.294247    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crmr9\" (UniqueName: \"kubernetes.io/projected/aa15f562-56f7-4d63-aba9-147757cc77cc-kube-api-access-crmr9\") pod \"payload-pre-processing-e2e-derive-ac4a75bf-57657fc88-gt5jn\" (UID: \"aa15f562-56f7-4d63-aba9-147757cc77cc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ac4a75bf-57657fc88-gt5jn"
Jul 29 19:47:10.339412 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:10.339384    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ac4a75bf-57657fc88-gt5jn"
Jul 29 19:47:10.673880 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:10.673854    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ac4a75bf-57657fc88-gt5jn"]
Jul 29 19:47:10.676231 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:47:10.676206    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa15f562_56f7_4d63_aba9_147757cc77cc.slice/crio-e5674c0e2d6bc052f9282860cf2c92db65f6bff2907afcdb02733ceb8c6ea16d WatchSource:0}: Error finding container e5674c0e2d6bc052f9282860cf2c92db65f6bff2907afcdb02733ceb8c6ea16d: Status 404 returned error can't find the container with id e5674c0e2d6bc052f9282860cf2c92db65f6bff2907afcdb02733ceb8c6ea16d
Jul 29 19:47:10.729644 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:10.729618    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ac4a75bf-57657fc88-gt5jn" event={"ID":"aa15f562-56f7-4d63-aba9-147757cc77cc","Type":"ContainerStarted","Data":"e5674c0e2d6bc052f9282860cf2c92db65f6bff2907afcdb02733ceb8c6ea16d"}
Jul 29 19:47:10.730956 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:10.730933    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ac4a75bf-7d9955bfd9-v5kjd" event={"ID":"a5a5ea75-09d4-4ca9-b7da-a95f936a8642","Type":"ContainerStarted","Data":"708d9560a8ac85653022d8d8ec6b516c92e1d672cb1c58f7de1b58208be480e7"}
Jul 29 19:47:10.731062 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:10.731050    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ac4a75bf-7d9955bfd9-v5kjd"
Jul 29 19:47:13.747519 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:13.747484    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ac4a75bf-57657fc88-gt5jn" event={"ID":"aa15f562-56f7-4d63-aba9-147757cc77cc","Type":"ContainerStarted","Data":"75b2abff17b21b32dadccaa82b0e5233184fdc9d77a44a9db385f2d1633a2df2"}
Jul 29 19:47:13.748015 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:13.747616    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-ac4a75bf-57657fc88-gt5jn"
Jul 29 19:47:13.767398 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:13.767357    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ac4a75bf-7d9955bfd9-v5kjd" podStartSLOduration=4.389387094 podStartE2EDuration="4.767341747s" totalImagesPullingTime="377.954653ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:47:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:47:10.7604802 +0000 UTC m=+1797.988869325" watchObservedRunningTime="2026-07-29 19:47:13.767341747 +0000 UTC m=+1800.995730875"
Jul 29 19:47:13.767543 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:13.767448    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-ac4a75bf-57657fc88-gt5jn" podStartSLOduration=1.7663516609999999 podStartE2EDuration="3.767431341s" totalImagesPullingTime="2.00107968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:47:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:47:13.765249228 +0000 UTC m=+1800.993638367" watchObservedRunningTime="2026-07-29 19:47:13.767431341 +0000 UTC m=+1800.995820466"
Jul 29 19:47:16.741817 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:16.741774    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ac4a75bf-7d9955bfd9-v5kjd"
Jul 29 19:47:19.998632 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:19.998597    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:47:24.753455 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:24.753424    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-ac4a75bf-57657fc88-gt5jn"
Jul 29 19:47:33.954130 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:33.954098    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac4a75bf-wnd5b"]
Jul 29 19:47:33.957467 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:33.957451    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac4a75bf-wnd5b"
Jul 29 19:47:33.959650 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:33.959632    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hq9l2\""
Jul 29 19:47:33.965355 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:33.965327    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac4a75bf-wnd5b"]
Jul 29 19:47:34.073018 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:34.072982    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13b0957-421d-444a-9d0f-e55aee8136b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ac4a75bf-wnd5b\" (UID: \"e13b0957-421d-444a-9d0f-e55aee8136b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac4a75bf-wnd5b"
Jul 29 19:47:34.173713 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:34.173687    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13b0957-421d-444a-9d0f-e55aee8136b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ac4a75bf-wnd5b\" (UID: \"e13b0957-421d-444a-9d0f-e55aee8136b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac4a75bf-wnd5b"
Jul 29 19:47:34.174301 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:34.174283    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13b0957-421d-444a-9d0f-e55aee8136b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ac4a75bf-wnd5b\" (UID: \"e13b0957-421d-444a-9d0f-e55aee8136b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac4a75bf-wnd5b"
Jul 29 19:47:34.268433 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:34.268368    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac4a75bf-wnd5b"
Jul 29 19:47:34.601323 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:34.601301    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac4a75bf-wnd5b"]
Jul 29 19:47:34.603905 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:47:34.603867    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13b0957_421d_444a_9d0f_e55aee8136b4.slice/crio-a22d5dd80d780624c096d2a21b93982858199dd8dc03f5618e03532a3d70095d WatchSource:0}: Error finding container a22d5dd80d780624c096d2a21b93982858199dd8dc03f5618e03532a3d70095d: Status 404 returned error can't find the container with id a22d5dd80d780624c096d2a21b93982858199dd8dc03f5618e03532a3d70095d
Jul 29 19:47:34.836021 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:34.835993    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac4a75bf-wnd5b" event={"ID":"e13b0957-421d-444a-9d0f-e55aee8136b4","Type":"ContainerStarted","Data":"a22d5dd80d780624c096d2a21b93982858199dd8dc03f5618e03532a3d70095d"}
Jul 29 19:47:35.842265 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:35.842227    2583 generic.go:356] "Generic (PLEG): container finished" podID="e13b0957-421d-444a-9d0f-e55aee8136b4" containerID="ab0b5d87725460927fea2dce92fe9cd3eca1994d8d0e6f4c9f9131e449af9526" exitCode=0
Jul 29 19:47:35.842639 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:35.842299    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac4a75bf-wnd5b" event={"ID":"e13b0957-421d-444a-9d0f-e55aee8136b4","Type":"ContainerDied","Data":"ab0b5d87725460927fea2dce92fe9cd3eca1994d8d0e6f4c9f9131e449af9526"}
Jul 29 19:47:36.973626 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:36.973597    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac4a75bf-wnd5b"
Jul 29 19:47:36.995708 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:36.995683    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13b0957-421d-444a-9d0f-e55aee8136b4-maas-api-service-ca\") pod \"e13b0957-421d-444a-9d0f-e55aee8136b4\" (UID: \"e13b0957-421d-444a-9d0f-e55aee8136b4\") "
Jul 29 19:47:36.996083 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:36.996060    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e13b0957-421d-444a-9d0f-e55aee8136b4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e13b0957-421d-444a-9d0f-e55aee8136b4" (UID: "e13b0957-421d-444a-9d0f-e55aee8136b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:47:37.096975 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:37.096905    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13b0957-421d-444a-9d0f-e55aee8136b4-maas-api-service-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:47:37.851447 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:37.851423    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac4a75bf-wnd5b"
Jul 29 19:47:37.851607 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:37.851459    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac4a75bf-wnd5b" event={"ID":"e13b0957-421d-444a-9d0f-e55aee8136b4","Type":"ContainerDied","Data":"a22d5dd80d780624c096d2a21b93982858199dd8dc03f5618e03532a3d70095d"}
Jul 29 19:47:37.851607 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:37.851506    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a22d5dd80d780624c096d2a21b93982858199dd8dc03f5618e03532a3d70095d"
Jul 29 19:47:39.909292 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:39.909257    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ac4a75bf-7d9955bfd9-v5kjd"]
Jul 29 19:47:39.909747 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:39.909503    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ac4a75bf-7d9955bfd9-v5kjd" podUID="a5a5ea75-09d4-4ca9-b7da-a95f936a8642" containerName="maas-api" containerID="cri-o://708d9560a8ac85653022d8d8ec6b516c92e1d672cb1c58f7de1b58208be480e7" gracePeriod=30
Jul 29 19:47:39.969602 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:39.969572    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ac4a75bf-57657fc88-gt5jn"]
Jul 29 19:47:39.970034 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:39.970001    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-ac4a75bf-57657fc88-gt5jn" podUID="aa15f562-56f7-4d63-aba9-147757cc77cc" containerName="payload-pre-processing" containerID="cri-o://75b2abff17b21b32dadccaa82b0e5233184fdc9d77a44a9db385f2d1633a2df2" gracePeriod=30
Jul 29 19:47:40.100298 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:40.100250    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac4a75bf-wnd5b"]
Jul 29 19:47:40.103063 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:40.103039    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac4a75bf-wnd5b"]
Jul 29 19:47:40.243873 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:40.243793    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ac4a75bf-7d9955bfd9-v5kjd"
Jul 29 19:47:40.246548 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:40.246527    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ac4a75bf-57657fc88-gt5jn"
Jul 29 19:47:40.321914 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:40.321879    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5a5ea75-09d4-4ca9-b7da-a95f936a8642-openshift-service-ca\") pod \"a5a5ea75-09d4-4ca9-b7da-a95f936a8642\" (UID: \"a5a5ea75-09d4-4ca9-b7da-a95f936a8642\") "
Jul 29 19:47:40.321914 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:40.321910    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qx8ws\" (UniqueName: \"kubernetes.io/projected/a5a5ea75-09d4-4ca9-b7da-a95f936a8642-kube-api-access-qx8ws\") pod \"a5a5ea75-09d4-4ca9-b7da-a95f936a8642\" (UID: \"a5a5ea75-09d4-4ca9-b7da-a95f936a8642\") "
Jul 29 19:47:40.322137 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:40.321931    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa15f562-56f7-4d63-aba9-147757cc77cc-plugins-config-volume\") pod \"aa15f562-56f7-4d63-aba9-147757cc77cc\" (UID: \"aa15f562-56f7-4d63-aba9-147757cc77cc\") "
Jul 29 19:47:40.322137 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:40.321960    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crmr9\" (UniqueName: \"kubernetes.io/projected/aa15f562-56f7-4d63-aba9-147757cc77cc-kube-api-access-crmr9\") pod \"aa15f562-56f7-4d63-aba9-147757cc77cc\" (UID: \"aa15f562-56f7-4d63-aba9-147757cc77cc\") "
Jul 29 19:47:40.322137 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:40.322011    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5a5ea75-09d4-4ca9-b7da-a95f936a8642-maas-api-tls\") pod \"a5a5ea75-09d4-4ca9-b7da-a95f936a8642\" (UID: \"a5a5ea75-09d4-4ca9-b7da-a95f936a8642\") "
Jul 29 19:47:40.322314 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:40.322288    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5a5ea75-09d4-4ca9-b7da-a95f936a8642-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a5a5ea75-09d4-4ca9-b7da-a95f936a8642" (UID: "a5a5ea75-09d4-4ca9-b7da-a95f936a8642"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:47:40.322367 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:40.322317    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa15f562-56f7-4d63-aba9-147757cc77cc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "aa15f562-56f7-4d63-aba9-147757cc77cc" (UID: "aa15f562-56f7-4d63-aba9-147757cc77cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:47:40.324185 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:40.324139    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5a5ea75-09d4-4ca9-b7da-a95f936a8642-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a5a5ea75-09d4-4ca9-b7da-a95f936a8642" (UID: "a5a5ea75-09d4-4ca9-b7da-a95f936a8642"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:47:40.324314 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:40.324228    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa15f562-56f7-4d63-aba9-147757cc77cc-kube-api-access-crmr9" (OuterVolumeSpecName: "kube-api-access-crmr9") pod "aa15f562-56f7-4d63-aba9-147757cc77cc" (UID: "aa15f562-56f7-4d63-aba9-147757cc77cc"). InnerVolumeSpecName "kube-api-access-crmr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:47:40.324314 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:40.324236    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5a5ea75-09d4-4ca9-b7da-a95f936a8642-kube-api-access-qx8ws" (OuterVolumeSpecName: "kube-api-access-qx8ws") pod "a5a5ea75-09d4-4ca9-b7da-a95f936a8642" (UID: "a5a5ea75-09d4-4ca9-b7da-a95f936a8642"). InnerVolumeSpecName "kube-api-access-qx8ws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:47:40.423473 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:40.423389    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa15f562-56f7-4d63-aba9-147757cc77cc-plugins-config-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:47:40.423473 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:40.423420    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crmr9\" (UniqueName: \"kubernetes.io/projected/aa15f562-56f7-4d63-aba9-147757cc77cc-kube-api-access-crmr9\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:47:40.423473 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:40.423433    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5a5ea75-09d4-4ca9-b7da-a95f936a8642-maas-api-tls\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:47:40.423473 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:40.423445    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5a5ea75-09d4-4ca9-b7da-a95f936a8642-openshift-service-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:47:40.423473 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:40.423457    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx8ws\" (UniqueName: \"kubernetes.io/projected/a5a5ea75-09d4-4ca9-b7da-a95f936a8642-kube-api-access-qx8ws\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:47:40.865767 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:40.865685    2583 generic.go:356] "Generic (PLEG): container finished" podID="aa15f562-56f7-4d63-aba9-147757cc77cc" containerID="75b2abff17b21b32dadccaa82b0e5233184fdc9d77a44a9db385f2d1633a2df2" exitCode=0
Jul 29 19:47:40.865767 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:40.865732    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ac4a75bf-57657fc88-gt5jn"
Jul 29 19:47:40.865989 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:40.865765    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ac4a75bf-57657fc88-gt5jn" event={"ID":"aa15f562-56f7-4d63-aba9-147757cc77cc","Type":"ContainerDied","Data":"75b2abff17b21b32dadccaa82b0e5233184fdc9d77a44a9db385f2d1633a2df2"}
Jul 29 19:47:40.865989 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:40.865843    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ac4a75bf-57657fc88-gt5jn" event={"ID":"aa15f562-56f7-4d63-aba9-147757cc77cc","Type":"ContainerDied","Data":"e5674c0e2d6bc052f9282860cf2c92db65f6bff2907afcdb02733ceb8c6ea16d"}
Jul 29 19:47:40.865989 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:40.865868    2583 scope.go:117] "RemoveContainer" containerID="75b2abff17b21b32dadccaa82b0e5233184fdc9d77a44a9db385f2d1633a2df2"
Jul 29 19:47:40.867140 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:40.867116    2583 generic.go:356] "Generic (PLEG): container finished" podID="a5a5ea75-09d4-4ca9-b7da-a95f936a8642" containerID="708d9560a8ac85653022d8d8ec6b516c92e1d672cb1c58f7de1b58208be480e7" exitCode=0
Jul 29 19:47:40.867223 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:40.867168    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ac4a75bf-7d9955bfd9-v5kjd"
Jul 29 19:47:40.867284 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:40.867171    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ac4a75bf-7d9955bfd9-v5kjd" event={"ID":"a5a5ea75-09d4-4ca9-b7da-a95f936a8642","Type":"ContainerDied","Data":"708d9560a8ac85653022d8d8ec6b516c92e1d672cb1c58f7de1b58208be480e7"}
Jul 29 19:47:40.867284 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:40.867281    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ac4a75bf-7d9955bfd9-v5kjd" event={"ID":"a5a5ea75-09d4-4ca9-b7da-a95f936a8642","Type":"ContainerDied","Data":"a2d9d654ffad53ace461b97ab9c51a514e9fd3a587e6df0a9bc70b52b3d15852"}
Jul 29 19:47:40.876313 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:40.876291    2583 scope.go:117] "RemoveContainer" containerID="75b2abff17b21b32dadccaa82b0e5233184fdc9d77a44a9db385f2d1633a2df2"
Jul 29 19:47:40.876596 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:47:40.876577    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75b2abff17b21b32dadccaa82b0e5233184fdc9d77a44a9db385f2d1633a2df2\": container with ID starting with 75b2abff17b21b32dadccaa82b0e5233184fdc9d77a44a9db385f2d1633a2df2 not found: ID does not exist" containerID="75b2abff17b21b32dadccaa82b0e5233184fdc9d77a44a9db385f2d1633a2df2"
Jul 29 19:47:40.876701 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:40.876606    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75b2abff17b21b32dadccaa82b0e5233184fdc9d77a44a9db385f2d1633a2df2"} err="failed to get container status \"75b2abff17b21b32dadccaa82b0e5233184fdc9d77a44a9db385f2d1633a2df2\": rpc error: code = NotFound desc = could not find container \"75b2abff17b21b32dadccaa82b0e5233184fdc9d77a44a9db385f2d1633a2df2\": container with ID starting with 75b2abff17b21b32dadccaa82b0e5233184fdc9d77a44a9db385f2d1633a2df2 not found: ID does not exist"
Jul 29 19:47:40.876701 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:40.876629    2583 scope.go:117] "RemoveContainer" containerID="708d9560a8ac85653022d8d8ec6b516c92e1d672cb1c58f7de1b58208be480e7"
Jul 29 19:47:40.884977 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:40.884957    2583 scope.go:117] "RemoveContainer" containerID="708d9560a8ac85653022d8d8ec6b516c92e1d672cb1c58f7de1b58208be480e7"
Jul 29 19:47:40.885225 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:47:40.885206    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"708d9560a8ac85653022d8d8ec6b516c92e1d672cb1c58f7de1b58208be480e7\": container with ID starting with 708d9560a8ac85653022d8d8ec6b516c92e1d672cb1c58f7de1b58208be480e7 not found: ID does not exist" containerID="708d9560a8ac85653022d8d8ec6b516c92e1d672cb1c58f7de1b58208be480e7"
Jul 29 19:47:40.885292 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:40.885235    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"708d9560a8ac85653022d8d8ec6b516c92e1d672cb1c58f7de1b58208be480e7"} err="failed to get container status \"708d9560a8ac85653022d8d8ec6b516c92e1d672cb1c58f7de1b58208be480e7\": rpc error: code = NotFound desc = could not find container \"708d9560a8ac85653022d8d8ec6b516c92e1d672cb1c58f7de1b58208be480e7\": container with ID starting with 708d9560a8ac85653022d8d8ec6b516c92e1d672cb1c58f7de1b58208be480e7 not found: ID does not exist"
Jul 29 19:47:40.901273 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:40.901250    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ac4a75bf-57657fc88-gt5jn"]
Jul 29 19:47:40.910556 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:40.910536    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ac4a75bf-57657fc88-gt5jn"]
Jul 29 19:47:40.923533 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:40.923515    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ac4a75bf-7d9955bfd9-v5kjd"]
Jul 29 19:47:40.929681 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:40.929656    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ac4a75bf-7d9955bfd9-v5kjd"]
Jul 29 19:47:41.340084 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:41.340058    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5a5ea75-09d4-4ca9-b7da-a95f936a8642" path="/var/lib/kubelet/pods/a5a5ea75-09d4-4ca9-b7da-a95f936a8642/volumes"
Jul 29 19:47:41.340379 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:41.340366    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa15f562-56f7-4d63-aba9-147757cc77cc" path="/var/lib/kubelet/pods/aa15f562-56f7-4d63-aba9-147757cc77cc/volumes"
Jul 29 19:47:41.340636 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:41.340625    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e13b0957-421d-444a-9d0f-e55aee8136b4" path="/var/lib/kubelet/pods/e13b0957-421d-444a-9d0f-e55aee8136b4/volumes"
Jul 29 19:47:43.506726 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:47:43.506685    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:49:11.503375 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:11.503340    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:49:23.810926 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:23.810885    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:49:34.013848 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:34.013815    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:49:41.689573 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:41.689543    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cd17a221-76dfd4c49-hqnmh"]
Jul 29 19:49:41.689950 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:41.689854    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa15f562-56f7-4d63-aba9-147757cc77cc" containerName="payload-pre-processing"
Jul 29 19:49:41.689950 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:41.689865    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa15f562-56f7-4d63-aba9-147757cc77cc" containerName="payload-pre-processing"
Jul 29 19:49:41.689950 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:41.689876    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5a5ea75-09d4-4ca9-b7da-a95f936a8642" containerName="maas-api"
Jul 29 19:49:41.689950 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:41.689882    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5a5ea75-09d4-4ca9-b7da-a95f936a8642" containerName="maas-api"
Jul 29 19:49:41.689950 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:41.689888    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e13b0957-421d-444a-9d0f-e55aee8136b4" containerName="revoke-keys"
Jul 29 19:49:41.689950 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:41.689893    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e13b0957-421d-444a-9d0f-e55aee8136b4" containerName="revoke-keys"
Jul 29 19:49:41.689950 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:41.689938    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa15f562-56f7-4d63-aba9-147757cc77cc" containerName="payload-pre-processing"
Jul 29 19:49:41.689950 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:41.689946    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5a5ea75-09d4-4ca9-b7da-a95f936a8642" containerName="maas-api"
Jul 29 19:49:41.689950 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:41.689953    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e13b0957-421d-444a-9d0f-e55aee8136b4" containerName="revoke-keys"
Jul 29 19:49:41.692863 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:41.692838    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd17a221-76dfd4c49-hqnmh"
Jul 29 19:49:41.695057 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:41.695029    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-cd17a221-serving-cert\""
Jul 29 19:49:41.695057 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:41.695050    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 29 19:49:41.695462 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:41.695433    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6wzbz\""
Jul 29 19:49:41.695973 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:41.695955    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 19:49:41.702487 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:41.702469    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cd17a221-76dfd4c49-hqnmh"]
Jul 29 19:49:41.802843 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:41.802789    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d418c1a9-99f3-46f6-91ea-668516f69a8a-openshift-service-ca\") pod \"maas-api-e2e-mt-cd17a221-76dfd4c49-hqnmh\" (UID: \"d418c1a9-99f3-46f6-91ea-668516f69a8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd17a221-76dfd4c49-hqnmh"
Jul 29 19:49:41.803048 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:41.802930    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d418c1a9-99f3-46f6-91ea-668516f69a8a-maas-api-tls\") pod \"maas-api-e2e-mt-cd17a221-76dfd4c49-hqnmh\" (UID: \"d418c1a9-99f3-46f6-91ea-668516f69a8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd17a221-76dfd4c49-hqnmh"
Jul 29 19:49:41.803048 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:41.802956    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqwkr\" (UniqueName: \"kubernetes.io/projected/d418c1a9-99f3-46f6-91ea-668516f69a8a-kube-api-access-cqwkr\") pod \"maas-api-e2e-mt-cd17a221-76dfd4c49-hqnmh\" (UID: \"d418c1a9-99f3-46f6-91ea-668516f69a8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd17a221-76dfd4c49-hqnmh"
Jul 29 19:49:41.903445 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:41.903408    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d418c1a9-99f3-46f6-91ea-668516f69a8a-maas-api-tls\") pod \"maas-api-e2e-mt-cd17a221-76dfd4c49-hqnmh\" (UID: \"d418c1a9-99f3-46f6-91ea-668516f69a8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd17a221-76dfd4c49-hqnmh"
Jul 29 19:49:41.903603 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:41.903451    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqwkr\" (UniqueName: \"kubernetes.io/projected/d418c1a9-99f3-46f6-91ea-668516f69a8a-kube-api-access-cqwkr\") pod \"maas-api-e2e-mt-cd17a221-76dfd4c49-hqnmh\" (UID: \"d418c1a9-99f3-46f6-91ea-668516f69a8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd17a221-76dfd4c49-hqnmh"
Jul 29 19:49:41.903603 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:41.903515    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d418c1a9-99f3-46f6-91ea-668516f69a8a-openshift-service-ca\") pod \"maas-api-e2e-mt-cd17a221-76dfd4c49-hqnmh\" (UID: \"d418c1a9-99f3-46f6-91ea-668516f69a8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd17a221-76dfd4c49-hqnmh"
Jul 29 19:49:41.903603 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:49:41.903564    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-cd17a221-serving-cert: secret "maas-api-e2e-mt-cd17a221-serving-cert" not found
Jul 29 19:49:41.903729 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:49:41.903648    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d418c1a9-99f3-46f6-91ea-668516f69a8a-maas-api-tls podName:d418c1a9-99f3-46f6-91ea-668516f69a8a nodeName:}" failed. No retries permitted until 2026-07-29 19:49:42.403628264 +0000 UTC m=+1949.632017371 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d418c1a9-99f3-46f6-91ea-668516f69a8a-maas-api-tls") pod "maas-api-e2e-mt-cd17a221-76dfd4c49-hqnmh" (UID: "d418c1a9-99f3-46f6-91ea-668516f69a8a") : secret "maas-api-e2e-mt-cd17a221-serving-cert" not found
Jul 29 19:49:41.904279 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:41.904257    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d418c1a9-99f3-46f6-91ea-668516f69a8a-openshift-service-ca\") pod \"maas-api-e2e-mt-cd17a221-76dfd4c49-hqnmh\" (UID: \"d418c1a9-99f3-46f6-91ea-668516f69a8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd17a221-76dfd4c49-hqnmh"
Jul 29 19:49:41.916546 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:41.916521    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqwkr\" (UniqueName: \"kubernetes.io/projected/d418c1a9-99f3-46f6-91ea-668516f69a8a-kube-api-access-cqwkr\") pod \"maas-api-e2e-mt-cd17a221-76dfd4c49-hqnmh\" (UID: \"d418c1a9-99f3-46f6-91ea-668516f69a8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd17a221-76dfd4c49-hqnmh"
Jul 29 19:49:42.390521 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:42.390484    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cd17a221-65f777c97b-2mz2l"]
Jul 29 19:49:42.393451 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:42.393433    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cd17a221-65f777c97b-2mz2l"
Jul 29 19:49:42.395739 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:42.395719    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-cd17a221\""
Jul 29 19:49:42.395834 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:42.395722    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-cd17a221-dockercfg-dnj4f\""
Jul 29 19:49:42.407764 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:42.407737    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cd17a221-65f777c97b-2mz2l"]
Jul 29 19:49:42.408064 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:42.408042    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d418c1a9-99f3-46f6-91ea-668516f69a8a-maas-api-tls\") pod \"maas-api-e2e-mt-cd17a221-76dfd4c49-hqnmh\" (UID: \"d418c1a9-99f3-46f6-91ea-668516f69a8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd17a221-76dfd4c49-hqnmh"
Jul 29 19:49:42.410654 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:42.410604    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d418c1a9-99f3-46f6-91ea-668516f69a8a-maas-api-tls\") pod \"maas-api-e2e-mt-cd17a221-76dfd4c49-hqnmh\" (UID: \"d418c1a9-99f3-46f6-91ea-668516f69a8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd17a221-76dfd4c49-hqnmh"
Jul 29 19:49:42.421482 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:42.421459    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cd17a221-8657c965cd-jb7v5"]
Jul 29 19:49:42.424458 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:42.424444    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cd17a221-8657c965cd-jb7v5"
Jul 29 19:49:42.434024 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:42.434003    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cd17a221-8657c965cd-jb7v5"]
Jul 29 19:49:42.508561 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:42.508531    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn4ck\" (UniqueName: \"kubernetes.io/projected/b6465831-3dbd-4ea7-a7dc-8a77dc1e2bc1-kube-api-access-fn4ck\") pod \"payload-pre-processing-e2e-mt-cd17a221-65f777c97b-2mz2l\" (UID: \"b6465831-3dbd-4ea7-a7dc-8a77dc1e2bc1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cd17a221-65f777c97b-2mz2l"
Jul 29 19:49:42.508718 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:42.508567    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6465831-3dbd-4ea7-a7dc-8a77dc1e2bc1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cd17a221-65f777c97b-2mz2l\" (UID: \"b6465831-3dbd-4ea7-a7dc-8a77dc1e2bc1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cd17a221-65f777c97b-2mz2l"
Jul 29 19:49:42.508718 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:42.508700    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzsx5\" (UniqueName: \"kubernetes.io/projected/6e4cdf2e-2bf0-45cd-86f4-e2617f0fcd74-kube-api-access-lzsx5\") pod \"payload-processing-e2e-mt-cd17a221-8657c965cd-jb7v5\" (UID: \"6e4cdf2e-2bf0-45cd-86f4-e2617f0fcd74\") " pod="openshift-ingress/payload-processing-e2e-mt-cd17a221-8657c965cd-jb7v5"
Jul 29 19:49:42.508831 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:42.508729    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e4cdf2e-2bf0-45cd-86f4-e2617f0fcd74-plugins-config-volume\") pod \"payload-processing-e2e-mt-cd17a221-8657c965cd-jb7v5\" (UID: \"6e4cdf2e-2bf0-45cd-86f4-e2617f0fcd74\") " pod="openshift-ingress/payload-processing-e2e-mt-cd17a221-8657c965cd-jb7v5"
Jul 29 19:49:42.604998 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:42.604963    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd17a221-76dfd4c49-hqnmh"
Jul 29 19:49:42.609106 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:42.609086    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn4ck\" (UniqueName: \"kubernetes.io/projected/b6465831-3dbd-4ea7-a7dc-8a77dc1e2bc1-kube-api-access-fn4ck\") pod \"payload-pre-processing-e2e-mt-cd17a221-65f777c97b-2mz2l\" (UID: \"b6465831-3dbd-4ea7-a7dc-8a77dc1e2bc1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cd17a221-65f777c97b-2mz2l"
Jul 29 19:49:42.609185 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:42.609117    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6465831-3dbd-4ea7-a7dc-8a77dc1e2bc1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cd17a221-65f777c97b-2mz2l\" (UID: \"b6465831-3dbd-4ea7-a7dc-8a77dc1e2bc1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cd17a221-65f777c97b-2mz2l"
Jul 29 19:49:42.609185 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:42.609170    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzsx5\" (UniqueName: \"kubernetes.io/projected/6e4cdf2e-2bf0-45cd-86f4-e2617f0fcd74-kube-api-access-lzsx5\") pod \"payload-processing-e2e-mt-cd17a221-8657c965cd-jb7v5\" (UID: \"6e4cdf2e-2bf0-45cd-86f4-e2617f0fcd74\") " pod="openshift-ingress/payload-processing-e2e-mt-cd17a221-8657c965cd-jb7v5"
Jul 29 19:49:42.609259 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:42.609187    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e4cdf2e-2bf0-45cd-86f4-e2617f0fcd74-plugins-config-volume\") pod \"payload-processing-e2e-mt-cd17a221-8657c965cd-jb7v5\" (UID: \"6e4cdf2e-2bf0-45cd-86f4-e2617f0fcd74\") " pod="openshift-ingress/payload-processing-e2e-mt-cd17a221-8657c965cd-jb7v5"
Jul 29 19:49:42.609875 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:42.609857    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6465831-3dbd-4ea7-a7dc-8a77dc1e2bc1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cd17a221-65f777c97b-2mz2l\" (UID: \"b6465831-3dbd-4ea7-a7dc-8a77dc1e2bc1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cd17a221-65f777c97b-2mz2l"
Jul 29 19:49:42.609920 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:42.609889    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e4cdf2e-2bf0-45cd-86f4-e2617f0fcd74-plugins-config-volume\") pod \"payload-processing-e2e-mt-cd17a221-8657c965cd-jb7v5\" (UID: \"6e4cdf2e-2bf0-45cd-86f4-e2617f0fcd74\") " pod="openshift-ingress/payload-processing-e2e-mt-cd17a221-8657c965cd-jb7v5"
Jul 29 19:49:42.617356 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:42.617329    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn4ck\" (UniqueName: \"kubernetes.io/projected/b6465831-3dbd-4ea7-a7dc-8a77dc1e2bc1-kube-api-access-fn4ck\") pod \"payload-pre-processing-e2e-mt-cd17a221-65f777c97b-2mz2l\" (UID: \"b6465831-3dbd-4ea7-a7dc-8a77dc1e2bc1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cd17a221-65f777c97b-2mz2l"
Jul 29 19:49:42.617450 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:42.617332    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzsx5\" (UniqueName: \"kubernetes.io/projected/6e4cdf2e-2bf0-45cd-86f4-e2617f0fcd74-kube-api-access-lzsx5\") pod \"payload-processing-e2e-mt-cd17a221-8657c965cd-jb7v5\" (UID: \"6e4cdf2e-2bf0-45cd-86f4-e2617f0fcd74\") " pod="openshift-ingress/payload-processing-e2e-mt-cd17a221-8657c965cd-jb7v5"
Jul 29 19:49:42.704299 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:42.704263    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cd17a221-65f777c97b-2mz2l"
Jul 29 19:49:42.732511 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:42.732489    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cd17a221-76dfd4c49-hqnmh"]
Jul 29 19:49:42.734994 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:42.734943    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cd17a221-8657c965cd-jb7v5"
Jul 29 19:49:42.735181 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:49:42.735140    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd418c1a9_99f3_46f6_91ea_668516f69a8a.slice/crio-2051f2c0c2d4b2864a94397922a8ea365135390015965c7664ea89c8de34e4b7 WatchSource:0}: Error finding container 2051f2c0c2d4b2864a94397922a8ea365135390015965c7664ea89c8de34e4b7: Status 404 returned error can't find the container with id 2051f2c0c2d4b2864a94397922a8ea365135390015965c7664ea89c8de34e4b7
Jul 29 19:49:42.847598 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:42.847568    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cd17a221-65f777c97b-2mz2l"]
Jul 29 19:49:42.851505 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:49:42.851471    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6465831_3dbd_4ea7_a7dc_8a77dc1e2bc1.slice/crio-8fa4f35a171cedf73e48377cb5ad780c856c0c238d2eaef65c9f82ac989f62a6 WatchSource:0}: Error finding container 8fa4f35a171cedf73e48377cb5ad780c856c0c238d2eaef65c9f82ac989f62a6: Status 404 returned error can't find the container with id 8fa4f35a171cedf73e48377cb5ad780c856c0c238d2eaef65c9f82ac989f62a6
Jul 29 19:49:42.881075 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:42.881051    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cd17a221-8657c965cd-jb7v5"]
Jul 29 19:49:42.884158 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:49:42.884135    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e4cdf2e_2bf0_45cd_86f4_e2617f0fcd74.slice/crio-6979507ad390f1c6453e938574de20a05d65ab4ccba07c0b6c18093b021250e1 WatchSource:0}: Error finding container 6979507ad390f1c6453e938574de20a05d65ab4ccba07c0b6c18093b021250e1: Status 404 returned error can't find the container with id 6979507ad390f1c6453e938574de20a05d65ab4ccba07c0b6c18093b021250e1
Jul 29 19:49:43.409047 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:43.409003    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cd17a221-8657c965cd-jb7v5" event={"ID":"6e4cdf2e-2bf0-45cd-86f4-e2617f0fcd74","Type":"ContainerStarted","Data":"8c5f6aead1346fafac5c46f1eaab2c75a6b20ebb451183299cdf692c605afbd1"}
Jul 29 19:49:43.409227 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:43.409063    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cd17a221-8657c965cd-jb7v5" event={"ID":"6e4cdf2e-2bf0-45cd-86f4-e2617f0fcd74","Type":"ContainerStarted","Data":"6979507ad390f1c6453e938574de20a05d65ab4ccba07c0b6c18093b021250e1"}
Jul 29 19:49:43.409227 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:43.409115    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-cd17a221-8657c965cd-jb7v5"
Jul 29 19:49:43.410726 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:43.410697    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd17a221-76dfd4c49-hqnmh" event={"ID":"d418c1a9-99f3-46f6-91ea-668516f69a8a","Type":"ContainerStarted","Data":"93cf1e128affe8747d57a91cddc29e78571f142648681855351ac148b3b46d38"}
Jul 29 19:49:43.410859 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:43.410751    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd17a221-76dfd4c49-hqnmh" event={"ID":"d418c1a9-99f3-46f6-91ea-668516f69a8a","Type":"ContainerStarted","Data":"2051f2c0c2d4b2864a94397922a8ea365135390015965c7664ea89c8de34e4b7"}
Jul 29 19:49:43.410859 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:43.410779    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd17a221-76dfd4c49-hqnmh"
Jul 29 19:49:43.412395 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:43.412368    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cd17a221-65f777c97b-2mz2l" event={"ID":"b6465831-3dbd-4ea7-a7dc-8a77dc1e2bc1","Type":"ContainerStarted","Data":"53d279fccddac00ca6b0d0f267f80caf2868a77c7f8e15b3e0dbfb2bd779ae86"}
Jul 29 19:49:43.412517 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:43.412416    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cd17a221-65f777c97b-2mz2l" event={"ID":"b6465831-3dbd-4ea7-a7dc-8a77dc1e2bc1","Type":"ContainerStarted","Data":"8fa4f35a171cedf73e48377cb5ad780c856c0c238d2eaef65c9f82ac989f62a6"}
Jul 29 19:49:43.412517 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:43.412472    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-cd17a221-65f777c97b-2mz2l"
Jul 29 19:49:43.428100 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:43.428053    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-cd17a221-8657c965cd-jb7v5" podStartSLOduration=1.428035923 podStartE2EDuration="1.428035923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:49:43.426792343 +0000 UTC m=+1950.655181471" watchObservedRunningTime="2026-07-29 19:49:43.428035923 +0000 UTC m=+1950.656425048"
Jul 29 19:49:43.445242 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:43.445187    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd17a221-76dfd4c49-hqnmh" podStartSLOduration=1.978735056 podStartE2EDuration="2.445167539s" totalImagesPullingTime="466.432483ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:49:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:49:43.442941035 +0000 UTC m=+1950.671330162" watchObservedRunningTime="2026-07-29 19:49:43.445167539 +0000 UTC m=+1950.673556666"
Jul 29 19:49:43.460650 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:43.460556    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-cd17a221-65f777c97b-2mz2l" podStartSLOduration=1.460539998 podStartE2EDuration="1.460539998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:49:43.460309407 +0000 UTC m=+1950.688698534" watchObservedRunningTime="2026-07-29 19:49:43.460539998 +0000 UTC m=+1950.688929160"
Jul 29 19:49:44.103321 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:44.103288    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:49:49.423102 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:49.423075    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd17a221-76dfd4c49-hqnmh"
Jul 29 19:49:54.419345 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:54.419312    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-cd17a221-8657c965cd-jb7v5"
Jul 29 19:49:54.419833 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:49:54.419375    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-cd17a221-65f777c97b-2mz2l"
Jul 29 19:50:13.766795 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:13.766708    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cd17a221-rzqq8"]
Jul 29 19:50:13.771170 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:13.771151    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cd17a221-rzqq8"
Jul 29 19:50:13.774939 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:13.774914    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hq9l2\""
Jul 29 19:50:13.786913 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:13.786885    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cd17a221-rzqq8"]
Jul 29 19:50:13.877016 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:13.876979    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e82bb715-1233-4e0d-ad5f-b8d0c72c6ea2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-cd17a221-rzqq8\" (UID: \"e82bb715-1233-4e0d-ad5f-b8d0c72c6ea2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cd17a221-rzqq8"
Jul 29 19:50:13.978084 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:13.978050    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e82bb715-1233-4e0d-ad5f-b8d0c72c6ea2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-cd17a221-rzqq8\" (UID: \"e82bb715-1233-4e0d-ad5f-b8d0c72c6ea2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cd17a221-rzqq8"
Jul 29 19:50:13.978664 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:13.978643    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e82bb715-1233-4e0d-ad5f-b8d0c72c6ea2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-cd17a221-rzqq8\" (UID: \"e82bb715-1233-4e0d-ad5f-b8d0c72c6ea2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cd17a221-rzqq8"
Jul 29 19:50:14.080855 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:14.080722    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cd17a221-rzqq8"
Jul 29 19:50:14.212618 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:14.212590    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cd17a221-rzqq8"]
Jul 29 19:50:14.215199 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:50:14.215172    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode82bb715_1233_4e0d_ad5f_b8d0c72c6ea2.slice/crio-9bd9d3b298df967e0deacf220a33b4eb00162b643a45ea6dc3d0a0dc9d320e87 WatchSource:0}: Error finding container 9bd9d3b298df967e0deacf220a33b4eb00162b643a45ea6dc3d0a0dc9d320e87: Status 404 returned error can't find the container with id 9bd9d3b298df967e0deacf220a33b4eb00162b643a45ea6dc3d0a0dc9d320e87
Jul 29 19:50:14.552557 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:14.552517    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cd17a221-rzqq8" event={"ID":"e82bb715-1233-4e0d-ad5f-b8d0c72c6ea2","Type":"ContainerStarted","Data":"222da46c390bfbcc282b8ebf53d53cde8ff340bb6f50723c02501a771e9f294f"}
Jul 29 19:50:14.552731 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:14.552573    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cd17a221-rzqq8" event={"ID":"e82bb715-1233-4e0d-ad5f-b8d0c72c6ea2","Type":"ContainerStarted","Data":"9bd9d3b298df967e0deacf220a33b4eb00162b643a45ea6dc3d0a0dc9d320e87"}
Jul 29 19:50:14.572856 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:14.572783    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cd17a221-rzqq8" podStartSLOduration=1.572767542 podStartE2EDuration="1.572767542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:50:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:50:14.571543051 +0000 UTC m=+1981.799932177" watchObservedRunningTime="2026-07-29 19:50:14.572767542 +0000 UTC m=+1981.801156667"
Jul 29 19:50:15.558532 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:15.558502    2583 generic.go:356] "Generic (PLEG): container finished" podID="e82bb715-1233-4e0d-ad5f-b8d0c72c6ea2" containerID="222da46c390bfbcc282b8ebf53d53cde8ff340bb6f50723c02501a771e9f294f" exitCode=0
Jul 29 19:50:15.558936 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:15.558594    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cd17a221-rzqq8" event={"ID":"e82bb715-1233-4e0d-ad5f-b8d0c72c6ea2","Type":"ContainerDied","Data":"222da46c390bfbcc282b8ebf53d53cde8ff340bb6f50723c02501a771e9f294f"}
Jul 29 19:50:16.692184 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:16.692157    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cd17a221-rzqq8"
Jul 29 19:50:16.800753 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:16.800718    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e82bb715-1233-4e0d-ad5f-b8d0c72c6ea2-maas-api-service-ca\") pod \"e82bb715-1233-4e0d-ad5f-b8d0c72c6ea2\" (UID: \"e82bb715-1233-4e0d-ad5f-b8d0c72c6ea2\") "
Jul 29 19:50:16.801109 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:16.801084    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e82bb715-1233-4e0d-ad5f-b8d0c72c6ea2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e82bb715-1233-4e0d-ad5f-b8d0c72c6ea2" (UID: "e82bb715-1233-4e0d-ad5f-b8d0c72c6ea2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:50:16.901311 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:16.901274    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e82bb715-1233-4e0d-ad5f-b8d0c72c6ea2-maas-api-service-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:50:17.569252 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:17.569226    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cd17a221-rzqq8"
Jul 29 19:50:17.569447 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:17.569269    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cd17a221-rzqq8" event={"ID":"e82bb715-1233-4e0d-ad5f-b8d0c72c6ea2","Type":"ContainerDied","Data":"9bd9d3b298df967e0deacf220a33b4eb00162b643a45ea6dc3d0a0dc9d320e87"}
Jul 29 19:50:17.569447 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:17.569322    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bd9d3b298df967e0deacf220a33b4eb00162b643a45ea6dc3d0a0dc9d320e87"
Jul 29 19:50:24.321670 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:24.321634    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cd17a221-76dfd4c49-hqnmh"]
Jul 29 19:50:24.322256 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:24.321858    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd17a221-76dfd4c49-hqnmh" podUID="d418c1a9-99f3-46f6-91ea-668516f69a8a" containerName="maas-api" containerID="cri-o://93cf1e128affe8747d57a91cddc29e78571f142648681855351ac148b3b46d38" gracePeriod=30
Jul 29 19:50:24.379426 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:24.379392    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cd17a221-8657c965cd-jb7v5"]
Jul 29 19:50:24.379688 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:24.379642    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-cd17a221-8657c965cd-jb7v5" podUID="6e4cdf2e-2bf0-45cd-86f4-e2617f0fcd74" containerName="payload-processing" containerID="cri-o://8c5f6aead1346fafac5c46f1eaab2c75a6b20ebb451183299cdf692c605afbd1" gracePeriod=30
Jul 29 19:50:24.389162 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:24.389130    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cd17a221-65f777c97b-2mz2l"]
Jul 29 19:50:24.389400 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:24.389376    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-cd17a221-65f777c97b-2mz2l" podUID="b6465831-3dbd-4ea7-a7dc-8a77dc1e2bc1" containerName="payload-pre-processing" containerID="cri-o://53d279fccddac00ca6b0d0f267f80caf2868a77c7f8e15b3e0dbfb2bd779ae86" gracePeriod=30
Jul 29 19:50:24.417707 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:24.417645    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-cd17a221-8657c965cd-jb7v5" podUID="6e4cdf2e-2bf0-45cd-86f4-e2617f0fcd74" containerName="payload-processing" probeResult="failure" output="dial tcp 10.133.0.38:9004: connect: connection refused"
Jul 29 19:50:24.417896 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:24.417719    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd17a221-76dfd4c49-hqnmh" podUID="d418c1a9-99f3-46f6-91ea-668516f69a8a" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.36:8443/health\": dial tcp 10.133.0.36:8443: connect: connection refused"
Jul 29 19:50:24.417896 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:24.417645    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-cd17a221-65f777c97b-2mz2l" podUID="b6465831-3dbd-4ea7-a7dc-8a77dc1e2bc1" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.37:9004: connect: connection refused"
Jul 29 19:50:24.504500 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:24.504472    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cd17a221-rzqq8"]
Jul 29 19:50:24.507950 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:24.507926    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cd17a221-rzqq8"]
Jul 29 19:50:24.603748 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:24.603706    2583 generic.go:356] "Generic (PLEG): container finished" podID="b6465831-3dbd-4ea7-a7dc-8a77dc1e2bc1" containerID="53d279fccddac00ca6b0d0f267f80caf2868a77c7f8e15b3e0dbfb2bd779ae86" exitCode=0
Jul 29 19:50:24.603931 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:24.603760    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cd17a221-65f777c97b-2mz2l" event={"ID":"b6465831-3dbd-4ea7-a7dc-8a77dc1e2bc1","Type":"ContainerDied","Data":"53d279fccddac00ca6b0d0f267f80caf2868a77c7f8e15b3e0dbfb2bd779ae86"}
Jul 29 19:50:24.605638 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:24.605615    2583 generic.go:356] "Generic (PLEG): container finished" podID="6e4cdf2e-2bf0-45cd-86f4-e2617f0fcd74" containerID="8c5f6aead1346fafac5c46f1eaab2c75a6b20ebb451183299cdf692c605afbd1" exitCode=0
Jul 29 19:50:24.605782 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:24.605700    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cd17a221-8657c965cd-jb7v5" event={"ID":"6e4cdf2e-2bf0-45cd-86f4-e2617f0fcd74","Type":"ContainerDied","Data":"8c5f6aead1346fafac5c46f1eaab2c75a6b20ebb451183299cdf692c605afbd1"}
Jul 29 19:50:24.607501 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:24.607480    2583 generic.go:356] "Generic (PLEG): container finished" podID="d418c1a9-99f3-46f6-91ea-668516f69a8a" containerID="93cf1e128affe8747d57a91cddc29e78571f142648681855351ac148b3b46d38" exitCode=0
Jul 29 19:50:24.607611 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:24.607507    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd17a221-76dfd4c49-hqnmh" event={"ID":"d418c1a9-99f3-46f6-91ea-668516f69a8a","Type":"ContainerDied","Data":"93cf1e128affe8747d57a91cddc29e78571f142648681855351ac148b3b46d38"}
Jul 29 19:50:24.649774 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:24.649745    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd17a221-76dfd4c49-hqnmh"
Jul 29 19:50:24.677985 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:24.677962    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cd17a221-8657c965cd-jb7v5"
Jul 29 19:50:24.683026 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:24.683008    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cd17a221-65f777c97b-2mz2l"
Jul 29 19:50:24.767620 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:24.767586    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d418c1a9-99f3-46f6-91ea-668516f69a8a-openshift-service-ca\") pod \"d418c1a9-99f3-46f6-91ea-668516f69a8a\" (UID: \"d418c1a9-99f3-46f6-91ea-668516f69a8a\") "
Jul 29 19:50:24.767826 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:24.767632    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d418c1a9-99f3-46f6-91ea-668516f69a8a-maas-api-tls\") pod \"d418c1a9-99f3-46f6-91ea-668516f69a8a\" (UID: \"d418c1a9-99f3-46f6-91ea-668516f69a8a\") "
Jul 29 19:50:24.767826 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:24.767675    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqwkr\" (UniqueName: \"kubernetes.io/projected/d418c1a9-99f3-46f6-91ea-668516f69a8a-kube-api-access-cqwkr\") pod \"d418c1a9-99f3-46f6-91ea-668516f69a8a\" (UID: \"d418c1a9-99f3-46f6-91ea-668516f69a8a\") "
Jul 29 19:50:24.768085 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:24.768054    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d418c1a9-99f3-46f6-91ea-668516f69a8a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d418c1a9-99f3-46f6-91ea-668516f69a8a" (UID: "d418c1a9-99f3-46f6-91ea-668516f69a8a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:50:24.769841 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:24.769820    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d418c1a9-99f3-46f6-91ea-668516f69a8a-kube-api-access-cqwkr" (OuterVolumeSpecName: "kube-api-access-cqwkr") pod "d418c1a9-99f3-46f6-91ea-668516f69a8a" (UID: "d418c1a9-99f3-46f6-91ea-668516f69a8a"). InnerVolumeSpecName "kube-api-access-cqwkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:50:24.769904 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:24.769842    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d418c1a9-99f3-46f6-91ea-668516f69a8a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d418c1a9-99f3-46f6-91ea-668516f69a8a" (UID: "d418c1a9-99f3-46f6-91ea-668516f69a8a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:50:24.868760 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:24.868665    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzsx5\" (UniqueName: \"kubernetes.io/projected/6e4cdf2e-2bf0-45cd-86f4-e2617f0fcd74-kube-api-access-lzsx5\") pod \"6e4cdf2e-2bf0-45cd-86f4-e2617f0fcd74\" (UID: \"6e4cdf2e-2bf0-45cd-86f4-e2617f0fcd74\") "
Jul 29 19:50:24.868760 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:24.868726    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e4cdf2e-2bf0-45cd-86f4-e2617f0fcd74-plugins-config-volume\") pod \"6e4cdf2e-2bf0-45cd-86f4-e2617f0fcd74\" (UID: \"6e4cdf2e-2bf0-45cd-86f4-e2617f0fcd74\") "
Jul 29 19:50:24.868760 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:24.868750    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6465831-3dbd-4ea7-a7dc-8a77dc1e2bc1-plugins-config-volume\") pod \"b6465831-3dbd-4ea7-a7dc-8a77dc1e2bc1\" (UID: \"b6465831-3dbd-4ea7-a7dc-8a77dc1e2bc1\") "
Jul 29 19:50:24.869074 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:24.868784    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fn4ck\" (UniqueName: \"kubernetes.io/projected/b6465831-3dbd-4ea7-a7dc-8a77dc1e2bc1-kube-api-access-fn4ck\") pod \"b6465831-3dbd-4ea7-a7dc-8a77dc1e2bc1\" (UID: \"b6465831-3dbd-4ea7-a7dc-8a77dc1e2bc1\") "
Jul 29 19:50:24.869074 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:24.868937    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d418c1a9-99f3-46f6-91ea-668516f69a8a-maas-api-tls\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:50:24.869074 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:24.868949    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqwkr\" (UniqueName: \"kubernetes.io/projected/d418c1a9-99f3-46f6-91ea-668516f69a8a-kube-api-access-cqwkr\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:50:24.869074 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:24.868958    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d418c1a9-99f3-46f6-91ea-668516f69a8a-openshift-service-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:50:24.869249 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:24.869093    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e4cdf2e-2bf0-45cd-86f4-e2617f0fcd74-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6e4cdf2e-2bf0-45cd-86f4-e2617f0fcd74" (UID: "6e4cdf2e-2bf0-45cd-86f4-e2617f0fcd74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:50:24.869249 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:24.869158    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6465831-3dbd-4ea7-a7dc-8a77dc1e2bc1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b6465831-3dbd-4ea7-a7dc-8a77dc1e2bc1" (UID: "b6465831-3dbd-4ea7-a7dc-8a77dc1e2bc1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:50:24.871005 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:24.870983    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6465831-3dbd-4ea7-a7dc-8a77dc1e2bc1-kube-api-access-fn4ck" (OuterVolumeSpecName: "kube-api-access-fn4ck") pod "b6465831-3dbd-4ea7-a7dc-8a77dc1e2bc1" (UID: "b6465831-3dbd-4ea7-a7dc-8a77dc1e2bc1"). InnerVolumeSpecName "kube-api-access-fn4ck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:50:24.871094 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:24.871013    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e4cdf2e-2bf0-45cd-86f4-e2617f0fcd74-kube-api-access-lzsx5" (OuterVolumeSpecName: "kube-api-access-lzsx5") pod "6e4cdf2e-2bf0-45cd-86f4-e2617f0fcd74" (UID: "6e4cdf2e-2bf0-45cd-86f4-e2617f0fcd74"). InnerVolumeSpecName "kube-api-access-lzsx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:50:24.970050 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:24.970015    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e4cdf2e-2bf0-45cd-86f4-e2617f0fcd74-plugins-config-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:50:24.970050 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:24.970045    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6465831-3dbd-4ea7-a7dc-8a77dc1e2bc1-plugins-config-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:50:24.970050 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:24.970054    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn4ck\" (UniqueName: \"kubernetes.io/projected/b6465831-3dbd-4ea7-a7dc-8a77dc1e2bc1-kube-api-access-fn4ck\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:50:24.970276 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:24.970062    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzsx5\" (UniqueName: \"kubernetes.io/projected/6e4cdf2e-2bf0-45cd-86f4-e2617f0fcd74-kube-api-access-lzsx5\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:50:25.341652 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:25.341623    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e82bb715-1233-4e0d-ad5f-b8d0c72c6ea2" path="/var/lib/kubelet/pods/e82bb715-1233-4e0d-ad5f-b8d0c72c6ea2/volumes"
Jul 29 19:50:25.618590 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:25.618507    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cd17a221-8657c965cd-jb7v5"
Jul 29 19:50:25.618590 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:25.618525    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cd17a221-8657c965cd-jb7v5" event={"ID":"6e4cdf2e-2bf0-45cd-86f4-e2617f0fcd74","Type":"ContainerDied","Data":"6979507ad390f1c6453e938574de20a05d65ab4ccba07c0b6c18093b021250e1"}
Jul 29 19:50:25.618828 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:25.618621    2583 scope.go:117] "RemoveContainer" containerID="8c5f6aead1346fafac5c46f1eaab2c75a6b20ebb451183299cdf692c605afbd1"
Jul 29 19:50:25.620476 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:25.620241    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd17a221-76dfd4c49-hqnmh" event={"ID":"d418c1a9-99f3-46f6-91ea-668516f69a8a","Type":"ContainerDied","Data":"2051f2c0c2d4b2864a94397922a8ea365135390015965c7664ea89c8de34e4b7"}
Jul 29 19:50:25.620476 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:25.620259    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd17a221-76dfd4c49-hqnmh"
Jul 29 19:50:25.622130 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:25.622099    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cd17a221-65f777c97b-2mz2l" event={"ID":"b6465831-3dbd-4ea7-a7dc-8a77dc1e2bc1","Type":"ContainerDied","Data":"8fa4f35a171cedf73e48377cb5ad780c856c0c238d2eaef65c9f82ac989f62a6"}
Jul 29 19:50:25.622242 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:25.622103    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cd17a221-65f777c97b-2mz2l"
Jul 29 19:50:25.630029 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:25.629976    2583 scope.go:117] "RemoveContainer" containerID="93cf1e128affe8747d57a91cddc29e78571f142648681855351ac148b3b46d38"
Jul 29 19:50:25.636509 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:25.636485    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cd17a221-8657c965cd-jb7v5"]
Jul 29 19:50:25.638457 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:25.638433    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cd17a221-8657c965cd-jb7v5"]
Jul 29 19:50:25.640518 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:25.640499    2583 scope.go:117] "RemoveContainer" containerID="53d279fccddac00ca6b0d0f267f80caf2868a77c7f8e15b3e0dbfb2bd779ae86"
Jul 29 19:50:25.650053 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:25.650018    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cd17a221-76dfd4c49-hqnmh"]
Jul 29 19:50:25.651620 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:25.651592    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cd17a221-76dfd4c49-hqnmh"]
Jul 29 19:50:25.661110 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:25.661086    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cd17a221-65f777c97b-2mz2l"]
Jul 29 19:50:25.664596 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:25.664571    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cd17a221-65f777c97b-2mz2l"]
Jul 29 19:50:27.201734 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:27.201696    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:50:27.340477 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:27.340449    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e4cdf2e-2bf0-45cd-86f4-e2617f0fcd74" path="/var/lib/kubelet/pods/6e4cdf2e-2bf0-45cd-86f4-e2617f0fcd74/volumes"
Jul 29 19:50:27.340830 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:27.340795    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6465831-3dbd-4ea7-a7dc-8a77dc1e2bc1" path="/var/lib/kubelet/pods/b6465831-3dbd-4ea7-a7dc-8a77dc1e2bc1/volumes"
Jul 29 19:50:27.341154 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:27.341142    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d418c1a9-99f3-46f6-91ea-668516f69a8a" path="/var/lib/kubelet/pods/d418c1a9-99f3-46f6-91ea-668516f69a8a/volumes"
Jul 29 19:50:54.905517 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:54.905481    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6757cc40-6476c9d48f-dstmd"]
Jul 29 19:50:54.906079 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:54.905778    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e4cdf2e-2bf0-45cd-86f4-e2617f0fcd74" containerName="payload-processing"
Jul 29 19:50:54.906079 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:54.905789    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e4cdf2e-2bf0-45cd-86f4-e2617f0fcd74" containerName="payload-processing"
Jul 29 19:50:54.906079 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:54.905831    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e82bb715-1233-4e0d-ad5f-b8d0c72c6ea2" containerName="revoke-keys"
Jul 29 19:50:54.906079 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:54.905838    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e82bb715-1233-4e0d-ad5f-b8d0c72c6ea2" containerName="revoke-keys"
Jul 29 19:50:54.906079 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:54.905847    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d418c1a9-99f3-46f6-91ea-668516f69a8a" containerName="maas-api"
Jul 29 19:50:54.906079 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:54.905852    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d418c1a9-99f3-46f6-91ea-668516f69a8a" containerName="maas-api"
Jul 29 19:50:54.906079 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:54.905866    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6465831-3dbd-4ea7-a7dc-8a77dc1e2bc1" containerName="payload-pre-processing"
Jul 29 19:50:54.906079 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:54.905871    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6465831-3dbd-4ea7-a7dc-8a77dc1e2bc1" containerName="payload-pre-processing"
Jul 29 19:50:54.906079 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:54.905925    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6465831-3dbd-4ea7-a7dc-8a77dc1e2bc1" containerName="payload-pre-processing"
Jul 29 19:50:54.906079 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:54.905932    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d418c1a9-99f3-46f6-91ea-668516f69a8a" containerName="maas-api"
Jul 29 19:50:54.906079 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:54.905938    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e82bb715-1233-4e0d-ad5f-b8d0c72c6ea2" containerName="revoke-keys"
Jul 29 19:50:54.906079 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:54.905946    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e4cdf2e-2bf0-45cd-86f4-e2617f0fcd74" containerName="payload-processing"
Jul 29 19:50:54.908884 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:54.908867    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6757cc40-6476c9d48f-dstmd"
Jul 29 19:50:54.911106 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:54.911081    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-6757cc40-serving-cert\""
Jul 29 19:50:54.911347 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:54.911334    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 19:50:54.911628 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:54.911612    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6wzbz\""
Jul 29 19:50:54.911688 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:54.911627    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 29 19:50:54.918387 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:54.918358    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6757cc40-6476c9d48f-dstmd"]
Jul 29 19:50:55.006565 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:55.006527    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq4x5\" (UniqueName: \"kubernetes.io/projected/1e49b647-67ab-46e8-9bbb-9bbc1b69b844-kube-api-access-tq4x5\") pod \"maas-api-e2e-mt-6757cc40-6476c9d48f-dstmd\" (UID: \"1e49b647-67ab-46e8-9bbb-9bbc1b69b844\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6757cc40-6476c9d48f-dstmd"
Jul 29 19:50:55.006565 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:55.006567    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e49b647-67ab-46e8-9bbb-9bbc1b69b844-openshift-service-ca\") pod \"maas-api-e2e-mt-6757cc40-6476c9d48f-dstmd\" (UID: \"1e49b647-67ab-46e8-9bbb-9bbc1b69b844\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6757cc40-6476c9d48f-dstmd"
Jul 29 19:50:55.006818 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:55.006596    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e49b647-67ab-46e8-9bbb-9bbc1b69b844-maas-api-tls\") pod \"maas-api-e2e-mt-6757cc40-6476c9d48f-dstmd\" (UID: \"1e49b647-67ab-46e8-9bbb-9bbc1b69b844\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6757cc40-6476c9d48f-dstmd"
Jul 29 19:50:55.108032 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:55.107995    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq4x5\" (UniqueName: \"kubernetes.io/projected/1e49b647-67ab-46e8-9bbb-9bbc1b69b844-kube-api-access-tq4x5\") pod \"maas-api-e2e-mt-6757cc40-6476c9d48f-dstmd\" (UID: \"1e49b647-67ab-46e8-9bbb-9bbc1b69b844\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6757cc40-6476c9d48f-dstmd"
Jul 29 19:50:55.108032 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:55.108034    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e49b647-67ab-46e8-9bbb-9bbc1b69b844-openshift-service-ca\") pod \"maas-api-e2e-mt-6757cc40-6476c9d48f-dstmd\" (UID: \"1e49b647-67ab-46e8-9bbb-9bbc1b69b844\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6757cc40-6476c9d48f-dstmd"
Jul 29 19:50:55.108252 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:55.108067    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e49b647-67ab-46e8-9bbb-9bbc1b69b844-maas-api-tls\") pod \"maas-api-e2e-mt-6757cc40-6476c9d48f-dstmd\" (UID: \"1e49b647-67ab-46e8-9bbb-9bbc1b69b844\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6757cc40-6476c9d48f-dstmd"
Jul 29 19:50:55.108252 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:50:55.108206    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-6757cc40-serving-cert: secret "maas-api-e2e-mt-6757cc40-serving-cert" not found
Jul 29 19:50:55.108333 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:50:55.108286    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e49b647-67ab-46e8-9bbb-9bbc1b69b844-maas-api-tls podName:1e49b647-67ab-46e8-9bbb-9bbc1b69b844 nodeName:}" failed. No retries permitted until 2026-07-29 19:50:55.608263407 +0000 UTC m=+2022.836652513 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1e49b647-67ab-46e8-9bbb-9bbc1b69b844-maas-api-tls") pod "maas-api-e2e-mt-6757cc40-6476c9d48f-dstmd" (UID: "1e49b647-67ab-46e8-9bbb-9bbc1b69b844") : secret "maas-api-e2e-mt-6757cc40-serving-cert" not found
Jul 29 19:50:55.108760 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:55.108742    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e49b647-67ab-46e8-9bbb-9bbc1b69b844-openshift-service-ca\") pod \"maas-api-e2e-mt-6757cc40-6476c9d48f-dstmd\" (UID: \"1e49b647-67ab-46e8-9bbb-9bbc1b69b844\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6757cc40-6476c9d48f-dstmd"
Jul 29 19:50:55.118121 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:55.118091    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq4x5\" (UniqueName: \"kubernetes.io/projected/1e49b647-67ab-46e8-9bbb-9bbc1b69b844-kube-api-access-tq4x5\") pod \"maas-api-e2e-mt-6757cc40-6476c9d48f-dstmd\" (UID: \"1e49b647-67ab-46e8-9bbb-9bbc1b69b844\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6757cc40-6476c9d48f-dstmd"
Jul 29 19:50:55.612767 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:55.612733    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e49b647-67ab-46e8-9bbb-9bbc1b69b844-maas-api-tls\") pod \"maas-api-e2e-mt-6757cc40-6476c9d48f-dstmd\" (UID: \"1e49b647-67ab-46e8-9bbb-9bbc1b69b844\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6757cc40-6476c9d48f-dstmd"
Jul 29 19:50:55.615199 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:55.615170    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e49b647-67ab-46e8-9bbb-9bbc1b69b844-maas-api-tls\") pod \"maas-api-e2e-mt-6757cc40-6476c9d48f-dstmd\" (UID: \"1e49b647-67ab-46e8-9bbb-9bbc1b69b844\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6757cc40-6476c9d48f-dstmd"
Jul 29 19:50:55.653179 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:55.653144    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6757cc40-84997b996-tnxp2"]
Jul 29 19:50:55.656220 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:55.656201    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6757cc40-84997b996-tnxp2"
Jul 29 19:50:55.658201 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:55.658180    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-6757cc40\""
Jul 29 19:50:55.668603 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:55.668578    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6757cc40-84997b996-tnxp2"]
Jul 29 19:50:55.705299 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:55.705264    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6757cc40-5475cd9994-qsd62"]
Jul 29 19:50:55.708289 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:55.708272    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6757cc40-5475cd9994-qsd62"
Jul 29 19:50:55.710362 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:55.710338    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-6757cc40-dockercfg-h56tp\""
Jul 29 19:50:55.713358 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:55.713333    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68c8j\" (UniqueName: \"kubernetes.io/projected/f7739089-0b8f-44c2-b4f1-ca0bef75ac40-kube-api-access-68c8j\") pod \"payload-pre-processing-e2e-mt-6757cc40-84997b996-tnxp2\" (UID: \"f7739089-0b8f-44c2-b4f1-ca0bef75ac40\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6757cc40-84997b996-tnxp2"
Jul 29 19:50:55.713485 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:55.713371    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7739089-0b8f-44c2-b4f1-ca0bef75ac40-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6757cc40-84997b996-tnxp2\" (UID: \"f7739089-0b8f-44c2-b4f1-ca0bef75ac40\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6757cc40-84997b996-tnxp2"
Jul 29 19:50:55.718786 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:55.718760    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6757cc40-5475cd9994-qsd62"]
Jul 29 19:50:55.813875 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:55.813834    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5a386be-f1cf-43ed-9909-ffb4837b839f-plugins-config-volume\") pod \"payload-processing-e2e-mt-6757cc40-5475cd9994-qsd62\" (UID: \"c5a386be-f1cf-43ed-9909-ffb4837b839f\") " pod="openshift-ingress/payload-processing-e2e-mt-6757cc40-5475cd9994-qsd62"
Jul 29 19:50:55.814069 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:55.813919    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7739089-0b8f-44c2-b4f1-ca0bef75ac40-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6757cc40-84997b996-tnxp2\" (UID: \"f7739089-0b8f-44c2-b4f1-ca0bef75ac40\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6757cc40-84997b996-tnxp2"
Jul 29 19:50:55.814069 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:55.813977    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh4zm\" (UniqueName: \"kubernetes.io/projected/c5a386be-f1cf-43ed-9909-ffb4837b839f-kube-api-access-kh4zm\") pod \"payload-processing-e2e-mt-6757cc40-5475cd9994-qsd62\" (UID: \"c5a386be-f1cf-43ed-9909-ffb4837b839f\") " pod="openshift-ingress/payload-processing-e2e-mt-6757cc40-5475cd9994-qsd62"
Jul 29 19:50:55.814069 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:55.814055    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68c8j\" (UniqueName: \"kubernetes.io/projected/f7739089-0b8f-44c2-b4f1-ca0bef75ac40-kube-api-access-68c8j\") pod \"payload-pre-processing-e2e-mt-6757cc40-84997b996-tnxp2\" (UID: \"f7739089-0b8f-44c2-b4f1-ca0bef75ac40\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6757cc40-84997b996-tnxp2"
Jul 29 19:50:55.814633 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:55.814608    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7739089-0b8f-44c2-b4f1-ca0bef75ac40-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6757cc40-84997b996-tnxp2\" (UID: \"f7739089-0b8f-44c2-b4f1-ca0bef75ac40\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6757cc40-84997b996-tnxp2"
Jul 29 19:50:55.821713 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:55.821688    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6757cc40-6476c9d48f-dstmd"
Jul 29 19:50:55.822751 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:55.822724    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68c8j\" (UniqueName: \"kubernetes.io/projected/f7739089-0b8f-44c2-b4f1-ca0bef75ac40-kube-api-access-68c8j\") pod \"payload-pre-processing-e2e-mt-6757cc40-84997b996-tnxp2\" (UID: \"f7739089-0b8f-44c2-b4f1-ca0bef75ac40\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6757cc40-84997b996-tnxp2"
Jul 29 19:50:55.915408 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:55.915351    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5a386be-f1cf-43ed-9909-ffb4837b839f-plugins-config-volume\") pod \"payload-processing-e2e-mt-6757cc40-5475cd9994-qsd62\" (UID: \"c5a386be-f1cf-43ed-9909-ffb4837b839f\") " pod="openshift-ingress/payload-processing-e2e-mt-6757cc40-5475cd9994-qsd62"
Jul 29 19:50:55.915408 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:55.915404    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh4zm\" (UniqueName: \"kubernetes.io/projected/c5a386be-f1cf-43ed-9909-ffb4837b839f-kube-api-access-kh4zm\") pod \"payload-processing-e2e-mt-6757cc40-5475cd9994-qsd62\" (UID: \"c5a386be-f1cf-43ed-9909-ffb4837b839f\") " pod="openshift-ingress/payload-processing-e2e-mt-6757cc40-5475cd9994-qsd62"
Jul 29 19:50:55.916371 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:55.916347    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5a386be-f1cf-43ed-9909-ffb4837b839f-plugins-config-volume\") pod \"payload-processing-e2e-mt-6757cc40-5475cd9994-qsd62\" (UID: \"c5a386be-f1cf-43ed-9909-ffb4837b839f\") " pod="openshift-ingress/payload-processing-e2e-mt-6757cc40-5475cd9994-qsd62"
Jul 29 19:50:55.923966 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:55.923938    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh4zm\" (UniqueName: \"kubernetes.io/projected/c5a386be-f1cf-43ed-9909-ffb4837b839f-kube-api-access-kh4zm\") pod \"payload-processing-e2e-mt-6757cc40-5475cd9994-qsd62\" (UID: \"c5a386be-f1cf-43ed-9909-ffb4837b839f\") " pod="openshift-ingress/payload-processing-e2e-mt-6757cc40-5475cd9994-qsd62"
Jul 29 19:50:55.956559 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:55.956530    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6757cc40-6476c9d48f-dstmd"]
Jul 29 19:50:55.959167 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:50:55.959143    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e49b647_67ab_46e8_9bbb_9bbc1b69b844.slice/crio-3d5a4c8631813402676ad8fb534faecf6a74ca8e5c316a4cf3ec38009064d3ef WatchSource:0}: Error finding container 3d5a4c8631813402676ad8fb534faecf6a74ca8e5c316a4cf3ec38009064d3ef: Status 404 returned error can't find the container with id 3d5a4c8631813402676ad8fb534faecf6a74ca8e5c316a4cf3ec38009064d3ef
Jul 29 19:50:55.960522 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:55.960503    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 19:50:55.966928 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:55.966899    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6757cc40-84997b996-tnxp2"
Jul 29 19:50:56.019239 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:56.019206    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6757cc40-5475cd9994-qsd62"
Jul 29 19:50:56.115124 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:56.114899    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6757cc40-84997b996-tnxp2"]
Jul 29 19:50:56.117543 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:50:56.117514    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7739089_0b8f_44c2_b4f1_ca0bef75ac40.slice/crio-53780f9f766bc495e32d90ddbe64996a759b1c9b96ec7c6ce85694da537eb694 WatchSource:0}: Error finding container 53780f9f766bc495e32d90ddbe64996a759b1c9b96ec7c6ce85694da537eb694: Status 404 returned error can't find the container with id 53780f9f766bc495e32d90ddbe64996a759b1c9b96ec7c6ce85694da537eb694
Jul 29 19:50:56.184905 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:56.184877    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6757cc40-5475cd9994-qsd62"]
Jul 29 19:50:56.187480 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:50:56.187437    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5a386be_f1cf_43ed_9909_ffb4837b839f.slice/crio-8064e49b920be7919713c9cb8b88d87c9b16b797ac7bb2a4c640b37efc139ddb WatchSource:0}: Error finding container 8064e49b920be7919713c9cb8b88d87c9b16b797ac7bb2a4c640b37efc139ddb: Status 404 returned error can't find the container with id 8064e49b920be7919713c9cb8b88d87c9b16b797ac7bb2a4c640b37efc139ddb
Jul 29 19:50:56.770481 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:56.770447    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6757cc40-84997b996-tnxp2" event={"ID":"f7739089-0b8f-44c2-b4f1-ca0bef75ac40","Type":"ContainerStarted","Data":"dd218a160c5477a3482539d7bbe9f85bb97b334059a3532aeddec635c15b1059"}
Jul 29 19:50:56.770679 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:56.770505    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6757cc40-84997b996-tnxp2" event={"ID":"f7739089-0b8f-44c2-b4f1-ca0bef75ac40","Type":"ContainerStarted","Data":"53780f9f766bc495e32d90ddbe64996a759b1c9b96ec7c6ce85694da537eb694"}
Jul 29 19:50:56.770679 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:56.770552    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6757cc40-84997b996-tnxp2"
Jul 29 19:50:56.772050 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:56.772022    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6757cc40-6476c9d48f-dstmd" event={"ID":"1e49b647-67ab-46e8-9bbb-9bbc1b69b844","Type":"ContainerStarted","Data":"78f14bccb5fac25ebf34606ab8545d3b45c8faa192866ad687887089c6b9e047"}
Jul 29 19:50:56.772179 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:56.772065    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6757cc40-6476c9d48f-dstmd" event={"ID":"1e49b647-67ab-46e8-9bbb-9bbc1b69b844","Type":"ContainerStarted","Data":"3d5a4c8631813402676ad8fb534faecf6a74ca8e5c316a4cf3ec38009064d3ef"}
Jul 29 19:50:56.772179 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:56.772098    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6757cc40-6476c9d48f-dstmd"
Jul 29 19:50:56.773577 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:56.773556    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6757cc40-5475cd9994-qsd62" event={"ID":"c5a386be-f1cf-43ed-9909-ffb4837b839f","Type":"ContainerStarted","Data":"3c7e15dc36f223f1279b4264a5112b9abfd39b48b41b547bd3a0b24bd10725fd"}
Jul 29 19:50:56.773697 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:56.773590    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6757cc40-5475cd9994-qsd62" event={"ID":"c5a386be-f1cf-43ed-9909-ffb4837b839f","Type":"ContainerStarted","Data":"8064e49b920be7919713c9cb8b88d87c9b16b797ac7bb2a4c640b37efc139ddb"}
Jul 29 19:50:56.773742 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:56.773697    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-6757cc40-5475cd9994-qsd62"
Jul 29 19:50:56.785653 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:56.785540    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-6757cc40-84997b996-tnxp2" podStartSLOduration=1.785520983 podStartE2EDuration="1.785520983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:50:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:50:56.785141048 +0000 UTC m=+2024.013530175" watchObservedRunningTime="2026-07-29 19:50:56.785520983 +0000 UTC m=+2024.013910109"
Jul 29 19:50:56.801680 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:56.801622    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-6757cc40-5475cd9994-qsd62" podStartSLOduration=1.801599606 podStartE2EDuration="1.801599606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:50:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:50:56.799193429 +0000 UTC m=+2024.027582546" watchObservedRunningTime="2026-07-29 19:50:56.801599606 +0000 UTC m=+2024.029988735"
Jul 29 19:50:56.816471 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:56.816413    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6757cc40-6476c9d48f-dstmd" podStartSLOduration=2.24412072 podStartE2EDuration="2.816396532s" totalImagesPullingTime="572.275812ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:50:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:50:56.814392323 +0000 UTC m=+2024.042781448" watchObservedRunningTime="2026-07-29 19:50:56.816396532 +0000 UTC m=+2024.044785657"
Jul 29 19:50:59.524542 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:50:59.524507    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:51:02.783528 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:51:02.783491    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6757cc40-6476c9d48f-dstmd"
Jul 29 19:51:07.780766 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:51:07.780733    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6757cc40-84997b996-tnxp2"
Jul 29 19:51:07.781238 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:51:07.780900    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-6757cc40-5475cd9994-qsd62"
Jul 29 19:51:41.502683 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:51:41.502597    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:51:55.215722 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:51:55.215682    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:51:59.086440 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:51:59.086405    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6757cc40-txgpz"]
Jul 29 19:51:59.089618 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:51:59.089603    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6757cc40-txgpz"
Jul 29 19:51:59.091836 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:51:59.091814    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hq9l2\""
Jul 29 19:51:59.096713 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:51:59.096460    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6757cc40-txgpz"]
Jul 29 19:51:59.166501 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:51:59.166464    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd6ac5a6-434f-4f07-8c54-59ff978b9c9c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6757cc40-txgpz\" (UID: \"cd6ac5a6-434f-4f07-8c54-59ff978b9c9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6757cc40-txgpz"
Jul 29 19:51:59.267148 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:51:59.267113    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd6ac5a6-434f-4f07-8c54-59ff978b9c9c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6757cc40-txgpz\" (UID: \"cd6ac5a6-434f-4f07-8c54-59ff978b9c9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6757cc40-txgpz"
Jul 29 19:51:59.267743 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:51:59.267722    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd6ac5a6-434f-4f07-8c54-59ff978b9c9c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6757cc40-txgpz\" (UID: \"cd6ac5a6-434f-4f07-8c54-59ff978b9c9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6757cc40-txgpz"
Jul 29 19:51:59.401313 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:51:59.401284    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6757cc40-txgpz"
Jul 29 19:51:59.533370 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:51:59.533328    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6757cc40-txgpz"]
Jul 29 19:51:59.536261 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:51:59.536227    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd6ac5a6_434f_4f07_8c54_59ff978b9c9c.slice/crio-5ab406b9801c2131f312d39600530a3e4725c92a6afa3e1521b8bdfd950c4e7a WatchSource:0}: Error finding container 5ab406b9801c2131f312d39600530a3e4725c92a6afa3e1521b8bdfd950c4e7a: Status 404 returned error can't find the container with id 5ab406b9801c2131f312d39600530a3e4725c92a6afa3e1521b8bdfd950c4e7a
Jul 29 19:52:00.058962 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:00.058874    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6757cc40-txgpz" event={"ID":"cd6ac5a6-434f-4f07-8c54-59ff978b9c9c","Type":"ContainerStarted","Data":"2def6bb9a31417c98fd0a98d1e69eaa4e2ac8861f81c77a8a51915e3fd667674"}
Jul 29 19:52:00.058962 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:00.058931    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6757cc40-txgpz" event={"ID":"cd6ac5a6-434f-4f07-8c54-59ff978b9c9c","Type":"ContainerStarted","Data":"5ab406b9801c2131f312d39600530a3e4725c92a6afa3e1521b8bdfd950c4e7a"}
Jul 29 19:52:00.073893 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:00.073838    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6757cc40-txgpz" podStartSLOduration=1.073796728 podStartE2EDuration="1.073796728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:51:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:52:00.073719162 +0000 UTC m=+2087.302108313" watchObservedRunningTime="2026-07-29 19:52:00.073796728 +0000 UTC m=+2087.302185854"
Jul 29 19:52:01.065060 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:01.065020    2583 generic.go:356] "Generic (PLEG): container finished" podID="cd6ac5a6-434f-4f07-8c54-59ff978b9c9c" containerID="2def6bb9a31417c98fd0a98d1e69eaa4e2ac8861f81c77a8a51915e3fd667674" exitCode=0
Jul 29 19:52:01.065442 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:01.065106    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6757cc40-txgpz" event={"ID":"cd6ac5a6-434f-4f07-8c54-59ff978b9c9c","Type":"ContainerDied","Data":"2def6bb9a31417c98fd0a98d1e69eaa4e2ac8861f81c77a8a51915e3fd667674"}
Jul 29 19:52:02.205405 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:02.205376    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6757cc40-txgpz"
Jul 29 19:52:02.295060 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:02.295020    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd6ac5a6-434f-4f07-8c54-59ff978b9c9c-maas-api-service-ca\") pod \"cd6ac5a6-434f-4f07-8c54-59ff978b9c9c\" (UID: \"cd6ac5a6-434f-4f07-8c54-59ff978b9c9c\") "
Jul 29 19:52:02.295431 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:02.295408    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd6ac5a6-434f-4f07-8c54-59ff978b9c9c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cd6ac5a6-434f-4f07-8c54-59ff978b9c9c" (UID: "cd6ac5a6-434f-4f07-8c54-59ff978b9c9c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:52:02.396333 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:02.396297    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd6ac5a6-434f-4f07-8c54-59ff978b9c9c-maas-api-service-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:52:03.075828 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:03.075788    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6757cc40-txgpz"
Jul 29 19:52:03.076001 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:03.075788    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6757cc40-txgpz" event={"ID":"cd6ac5a6-434f-4f07-8c54-59ff978b9c9c","Type":"ContainerDied","Data":"5ab406b9801c2131f312d39600530a3e4725c92a6afa3e1521b8bdfd950c4e7a"}
Jul 29 19:52:03.076001 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:03.075920    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ab406b9801c2131f312d39600530a3e4725c92a6afa3e1521b8bdfd950c4e7a"
Jul 29 19:52:12.797421 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:12.797385    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:52:13.372095 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:13.372063    2583 scope.go:117] "RemoveContainer" containerID="ddb09ee7bc816f420595584c6384d7bfb5147cf5cb31b8f2f11c92e92ab1c812"
Jul 29 19:52:14.455627 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:14.455593    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6757cc40-6476c9d48f-dstmd"]
Jul 29 19:52:14.456093 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:14.455867    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6757cc40-6476c9d48f-dstmd" podUID="1e49b647-67ab-46e8-9bbb-9bbc1b69b844" containerName="maas-api" containerID="cri-o://78f14bccb5fac25ebf34606ab8545d3b45c8faa192866ad687887089c6b9e047" gracePeriod=30
Jul 29 19:52:14.560357 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:14.560327    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6757cc40-5475cd9994-qsd62"]
Jul 29 19:52:14.560554 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:14.560518    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-6757cc40-5475cd9994-qsd62" podUID="c5a386be-f1cf-43ed-9909-ffb4837b839f" containerName="payload-processing" containerID="cri-o://3c7e15dc36f223f1279b4264a5112b9abfd39b48b41b547bd3a0b24bd10725fd" gracePeriod=30
Jul 29 19:52:14.575886 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:14.575780    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6757cc40-84997b996-tnxp2"]
Jul 29 19:52:14.577055 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:14.576129    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-6757cc40-84997b996-tnxp2" podUID="f7739089-0b8f-44c2-b4f1-ca0bef75ac40" containerName="payload-pre-processing" containerID="cri-o://dd218a160c5477a3482539d7bbe9f85bb97b334059a3532aeddec635c15b1059" gracePeriod=30
Jul 29 19:52:14.724900 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:14.724661    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6757cc40-6476c9d48f-dstmd"
Jul 29 19:52:14.748958 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:14.748924    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6757cc40-txgpz"]
Jul 29 19:52:14.751185 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:14.751166    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6757cc40-txgpz"]
Jul 29 19:52:14.843060 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:14.843035    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6757cc40-5475cd9994-qsd62"
Jul 29 19:52:14.846204 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:14.846183    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6757cc40-84997b996-tnxp2"
Jul 29 19:52:14.895870 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:14.895837    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tq4x5\" (UniqueName: \"kubernetes.io/projected/1e49b647-67ab-46e8-9bbb-9bbc1b69b844-kube-api-access-tq4x5\") pod \"1e49b647-67ab-46e8-9bbb-9bbc1b69b844\" (UID: \"1e49b647-67ab-46e8-9bbb-9bbc1b69b844\") "
Jul 29 19:52:14.896048 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:14.895874    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e49b647-67ab-46e8-9bbb-9bbc1b69b844-maas-api-tls\") pod \"1e49b647-67ab-46e8-9bbb-9bbc1b69b844\" (UID: \"1e49b647-67ab-46e8-9bbb-9bbc1b69b844\") "
Jul 29 19:52:14.896048 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:14.895934    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e49b647-67ab-46e8-9bbb-9bbc1b69b844-openshift-service-ca\") pod \"1e49b647-67ab-46e8-9bbb-9bbc1b69b844\" (UID: \"1e49b647-67ab-46e8-9bbb-9bbc1b69b844\") "
Jul 29 19:52:14.896263 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:14.896236    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e49b647-67ab-46e8-9bbb-9bbc1b69b844-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1e49b647-67ab-46e8-9bbb-9bbc1b69b844" (UID: "1e49b647-67ab-46e8-9bbb-9bbc1b69b844"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:52:14.897979 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:14.897955    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e49b647-67ab-46e8-9bbb-9bbc1b69b844-kube-api-access-tq4x5" (OuterVolumeSpecName: "kube-api-access-tq4x5") pod "1e49b647-67ab-46e8-9bbb-9bbc1b69b844" (UID: "1e49b647-67ab-46e8-9bbb-9bbc1b69b844"). InnerVolumeSpecName "kube-api-access-tq4x5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:52:14.898046 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:14.897993    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e49b647-67ab-46e8-9bbb-9bbc1b69b844-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1e49b647-67ab-46e8-9bbb-9bbc1b69b844" (UID: "1e49b647-67ab-46e8-9bbb-9bbc1b69b844"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:52:14.997344 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:14.997249    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kh4zm\" (UniqueName: \"kubernetes.io/projected/c5a386be-f1cf-43ed-9909-ffb4837b839f-kube-api-access-kh4zm\") pod \"c5a386be-f1cf-43ed-9909-ffb4837b839f\" (UID: \"c5a386be-f1cf-43ed-9909-ffb4837b839f\") "
Jul 29 19:52:14.997344 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:14.997307    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68c8j\" (UniqueName: \"kubernetes.io/projected/f7739089-0b8f-44c2-b4f1-ca0bef75ac40-kube-api-access-68c8j\") pod \"f7739089-0b8f-44c2-b4f1-ca0bef75ac40\" (UID: \"f7739089-0b8f-44c2-b4f1-ca0bef75ac40\") "
Jul 29 19:52:14.997344 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:14.997338    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5a386be-f1cf-43ed-9909-ffb4837b839f-plugins-config-volume\") pod \"c5a386be-f1cf-43ed-9909-ffb4837b839f\" (UID: \"c5a386be-f1cf-43ed-9909-ffb4837b839f\") "
Jul 29 19:52:14.997627 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:14.997386    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7739089-0b8f-44c2-b4f1-ca0bef75ac40-plugins-config-volume\") pod \"f7739089-0b8f-44c2-b4f1-ca0bef75ac40\" (UID: \"f7739089-0b8f-44c2-b4f1-ca0bef75ac40\") "
Jul 29 19:52:14.997627 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:14.997604    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq4x5\" (UniqueName: \"kubernetes.io/projected/1e49b647-67ab-46e8-9bbb-9bbc1b69b844-kube-api-access-tq4x5\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:52:14.997627 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:14.997624    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e49b647-67ab-46e8-9bbb-9bbc1b69b844-maas-api-tls\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:52:14.997787 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:14.997637    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e49b647-67ab-46e8-9bbb-9bbc1b69b844-openshift-service-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:52:14.997787 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:14.997709    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a386be-f1cf-43ed-9909-ffb4837b839f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c5a386be-f1cf-43ed-9909-ffb4837b839f" (UID: "c5a386be-f1cf-43ed-9909-ffb4837b839f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:52:14.997897 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:14.997791    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7739089-0b8f-44c2-b4f1-ca0bef75ac40-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f7739089-0b8f-44c2-b4f1-ca0bef75ac40" (UID: "f7739089-0b8f-44c2-b4f1-ca0bef75ac40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:52:14.999453 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:14.999429    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7739089-0b8f-44c2-b4f1-ca0bef75ac40-kube-api-access-68c8j" (OuterVolumeSpecName: "kube-api-access-68c8j") pod "f7739089-0b8f-44c2-b4f1-ca0bef75ac40" (UID: "f7739089-0b8f-44c2-b4f1-ca0bef75ac40"). InnerVolumeSpecName "kube-api-access-68c8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:52:14.999532 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:14.999454    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a386be-f1cf-43ed-9909-ffb4837b839f-kube-api-access-kh4zm" (OuterVolumeSpecName: "kube-api-access-kh4zm") pod "c5a386be-f1cf-43ed-9909-ffb4837b839f" (UID: "c5a386be-f1cf-43ed-9909-ffb4837b839f"). InnerVolumeSpecName "kube-api-access-kh4zm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:52:15.098297 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:15.098256    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7739089-0b8f-44c2-b4f1-ca0bef75ac40-plugins-config-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:52:15.098297 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:15.098287    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh4zm\" (UniqueName: \"kubernetes.io/projected/c5a386be-f1cf-43ed-9909-ffb4837b839f-kube-api-access-kh4zm\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:52:15.098297 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:15.098299    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68c8j\" (UniqueName: \"kubernetes.io/projected/f7739089-0b8f-44c2-b4f1-ca0bef75ac40-kube-api-access-68c8j\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:52:15.098535 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:15.098313    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5a386be-f1cf-43ed-9909-ffb4837b839f-plugins-config-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:52:15.133933 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:15.133900    2583 generic.go:356] "Generic (PLEG): container finished" podID="c5a386be-f1cf-43ed-9909-ffb4837b839f" containerID="3c7e15dc36f223f1279b4264a5112b9abfd39b48b41b547bd3a0b24bd10725fd" exitCode=0
Jul 29 19:52:15.134082 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:15.133946    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6757cc40-5475cd9994-qsd62"
Jul 29 19:52:15.134082 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:15.133990    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6757cc40-5475cd9994-qsd62" event={"ID":"c5a386be-f1cf-43ed-9909-ffb4837b839f","Type":"ContainerDied","Data":"3c7e15dc36f223f1279b4264a5112b9abfd39b48b41b547bd3a0b24bd10725fd"}
Jul 29 19:52:15.134082 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:15.134041    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6757cc40-5475cd9994-qsd62" event={"ID":"c5a386be-f1cf-43ed-9909-ffb4837b839f","Type":"ContainerDied","Data":"8064e49b920be7919713c9cb8b88d87c9b16b797ac7bb2a4c640b37efc139ddb"}
Jul 29 19:52:15.134082 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:15.134069    2583 scope.go:117] "RemoveContainer" containerID="3c7e15dc36f223f1279b4264a5112b9abfd39b48b41b547bd3a0b24bd10725fd"
Jul 29 19:52:15.135505 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:15.135481    2583 generic.go:356] "Generic (PLEG): container finished" podID="f7739089-0b8f-44c2-b4f1-ca0bef75ac40" containerID="dd218a160c5477a3482539d7bbe9f85bb97b334059a3532aeddec635c15b1059" exitCode=0
Jul 29 19:52:15.135596 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:15.135525    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6757cc40-84997b996-tnxp2"
Jul 29 19:52:15.135596 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:15.135563    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6757cc40-84997b996-tnxp2" event={"ID":"f7739089-0b8f-44c2-b4f1-ca0bef75ac40","Type":"ContainerDied","Data":"dd218a160c5477a3482539d7bbe9f85bb97b334059a3532aeddec635c15b1059"}
Jul 29 19:52:15.135709 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:15.135609    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6757cc40-84997b996-tnxp2" event={"ID":"f7739089-0b8f-44c2-b4f1-ca0bef75ac40","Type":"ContainerDied","Data":"53780f9f766bc495e32d90ddbe64996a759b1c9b96ec7c6ce85694da537eb694"}
Jul 29 19:52:15.136940 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:15.136872    2583 generic.go:356] "Generic (PLEG): container finished" podID="1e49b647-67ab-46e8-9bbb-9bbc1b69b844" containerID="78f14bccb5fac25ebf34606ab8545d3b45c8faa192866ad687887089c6b9e047" exitCode=0
Jul 29 19:52:15.136940 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:15.136922    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6757cc40-6476c9d48f-dstmd"
Jul 29 19:52:15.137104 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:15.136911    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6757cc40-6476c9d48f-dstmd" event={"ID":"1e49b647-67ab-46e8-9bbb-9bbc1b69b844","Type":"ContainerDied","Data":"78f14bccb5fac25ebf34606ab8545d3b45c8faa192866ad687887089c6b9e047"}
Jul 29 19:52:15.137104 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:15.137033    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6757cc40-6476c9d48f-dstmd" event={"ID":"1e49b647-67ab-46e8-9bbb-9bbc1b69b844","Type":"ContainerDied","Data":"3d5a4c8631813402676ad8fb534faecf6a74ca8e5c316a4cf3ec38009064d3ef"}
Jul 29 19:52:15.146795 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:15.146499    2583 scope.go:117] "RemoveContainer" containerID="3c7e15dc36f223f1279b4264a5112b9abfd39b48b41b547bd3a0b24bd10725fd"
Jul 29 19:52:15.152438 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:52:15.148155    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c7e15dc36f223f1279b4264a5112b9abfd39b48b41b547bd3a0b24bd10725fd\": container with ID starting with 3c7e15dc36f223f1279b4264a5112b9abfd39b48b41b547bd3a0b24bd10725fd not found: ID does not exist" containerID="3c7e15dc36f223f1279b4264a5112b9abfd39b48b41b547bd3a0b24bd10725fd"
Jul 29 19:52:15.152438 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:15.148197    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c7e15dc36f223f1279b4264a5112b9abfd39b48b41b547bd3a0b24bd10725fd"} err="failed to get container status \"3c7e15dc36f223f1279b4264a5112b9abfd39b48b41b547bd3a0b24bd10725fd\": rpc error: code = NotFound desc = could not find container \"3c7e15dc36f223f1279b4264a5112b9abfd39b48b41b547bd3a0b24bd10725fd\": container with ID starting with 3c7e15dc36f223f1279b4264a5112b9abfd39b48b41b547bd3a0b24bd10725fd not found: ID does not exist"
Jul 29 19:52:15.152438 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:15.148222    2583 scope.go:117] "RemoveContainer" containerID="dd218a160c5477a3482539d7bbe9f85bb97b334059a3532aeddec635c15b1059"
Jul 29 19:52:15.160284 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:15.160264    2583 scope.go:117] "RemoveContainer" containerID="dd218a160c5477a3482539d7bbe9f85bb97b334059a3532aeddec635c15b1059"
Jul 29 19:52:15.160560 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:52:15.160539    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd218a160c5477a3482539d7bbe9f85bb97b334059a3532aeddec635c15b1059\": container with ID starting with dd218a160c5477a3482539d7bbe9f85bb97b334059a3532aeddec635c15b1059 not found: ID does not exist" containerID="dd218a160c5477a3482539d7bbe9f85bb97b334059a3532aeddec635c15b1059"
Jul 29 19:52:15.160623 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:15.160568    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd218a160c5477a3482539d7bbe9f85bb97b334059a3532aeddec635c15b1059"} err="failed to get container status \"dd218a160c5477a3482539d7bbe9f85bb97b334059a3532aeddec635c15b1059\": rpc error: code = NotFound desc = could not find container \"dd218a160c5477a3482539d7bbe9f85bb97b334059a3532aeddec635c15b1059\": container with ID starting with dd218a160c5477a3482539d7bbe9f85bb97b334059a3532aeddec635c15b1059 not found: ID does not exist"
Jul 29 19:52:15.160623 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:15.160588    2583 scope.go:117] "RemoveContainer" containerID="78f14bccb5fac25ebf34606ab8545d3b45c8faa192866ad687887089c6b9e047"
Jul 29 19:52:15.165887 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:15.165864    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6757cc40-5475cd9994-qsd62"]
Jul 29 19:52:15.168913 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:15.168892    2583 scope.go:117] "RemoveContainer" containerID="78f14bccb5fac25ebf34606ab8545d3b45c8faa192866ad687887089c6b9e047"
Jul 29 19:52:15.169195 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:52:15.169178    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78f14bccb5fac25ebf34606ab8545d3b45c8faa192866ad687887089c6b9e047\": container with ID starting with 78f14bccb5fac25ebf34606ab8545d3b45c8faa192866ad687887089c6b9e047 not found: ID does not exist" containerID="78f14bccb5fac25ebf34606ab8545d3b45c8faa192866ad687887089c6b9e047"
Jul 29 19:52:15.169236 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:15.169201    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78f14bccb5fac25ebf34606ab8545d3b45c8faa192866ad687887089c6b9e047"} err="failed to get container status \"78f14bccb5fac25ebf34606ab8545d3b45c8faa192866ad687887089c6b9e047\": rpc error: code = NotFound desc = could not find container \"78f14bccb5fac25ebf34606ab8545d3b45c8faa192866ad687887089c6b9e047\": container with ID starting with 78f14bccb5fac25ebf34606ab8545d3b45c8faa192866ad687887089c6b9e047 not found: ID does not exist"
Jul 29 19:52:15.169862 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:15.169837    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6757cc40-5475cd9994-qsd62"]
Jul 29 19:52:15.179519 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:15.179488    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6757cc40-6476c9d48f-dstmd"]
Jul 29 19:52:15.182953 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:15.182926    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6757cc40-6476c9d48f-dstmd"]
Jul 29 19:52:15.192561 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:15.192533    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6757cc40-84997b996-tnxp2"]
Jul 29 19:52:15.196519 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:15.196495    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6757cc40-84997b996-tnxp2"]
Jul 29 19:52:15.340658 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:15.340582    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e49b647-67ab-46e8-9bbb-9bbc1b69b844" path="/var/lib/kubelet/pods/1e49b647-67ab-46e8-9bbb-9bbc1b69b844/volumes"
Jul 29 19:52:15.340965 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:15.340951    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5a386be-f1cf-43ed-9909-ffb4837b839f" path="/var/lib/kubelet/pods/c5a386be-f1cf-43ed-9909-ffb4837b839f/volumes"
Jul 29 19:52:15.341227 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:15.341216    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd6ac5a6-434f-4f07-8c54-59ff978b9c9c" path="/var/lib/kubelet/pods/cd6ac5a6-434f-4f07-8c54-59ff978b9c9c/volumes"
Jul 29 19:52:15.341521 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:15.341510    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7739089-0b8f-44c2-b4f1-ca0bef75ac40" path="/var/lib/kubelet/pods/f7739089-0b8f-44c2-b4f1-ca0bef75ac40/volumes"
Jul 29 19:52:24.509331 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:52:24.509295    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:53:13.390439 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:53:13.390367    2583 scope.go:117] "RemoveContainer" containerID="d366172556c2df39c388e38b9370df6a4ff84a287540cd33c531f2aaeb25198a"
Jul 29 19:53:13.397900 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:53:13.397882    2583 scope.go:117] "RemoveContainer" containerID="e8a304e5ce0176eb0d0b4b2171817ef3c79b26a6555790797d146c4a33ee49b3"
Jul 29 19:53:25.231949 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:53:25.231913    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:53:35.004447 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:53:35.004413    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:54:13.427398 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:13.427369    2583 scope.go:117] "RemoveContainer" containerID="ab0b5d87725460927fea2dce92fe9cd3eca1994d8d0e6f4c9f9131e449af9526"
Jul 29 19:54:20.900648 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:20.900612    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-59d5db-596bfd5995-hxwbb"]
Jul 29 19:54:20.901228 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:20.900947    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e49b647-67ab-46e8-9bbb-9bbc1b69b844" containerName="maas-api"
Jul 29 19:54:20.901228 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:20.900959    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e49b647-67ab-46e8-9bbb-9bbc1b69b844" containerName="maas-api"
Jul 29 19:54:20.901228 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:20.900971    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd6ac5a6-434f-4f07-8c54-59ff978b9c9c" containerName="revoke-keys"
Jul 29 19:54:20.901228 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:20.900977    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd6ac5a6-434f-4f07-8c54-59ff978b9c9c" containerName="revoke-keys"
Jul 29 19:54:20.901228 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:20.900984    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7739089-0b8f-44c2-b4f1-ca0bef75ac40" containerName="payload-pre-processing"
Jul 29 19:54:20.901228 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:20.900990    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7739089-0b8f-44c2-b4f1-ca0bef75ac40" containerName="payload-pre-processing"
Jul 29 19:54:20.901228 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:20.900996    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5a386be-f1cf-43ed-9909-ffb4837b839f" containerName="payload-processing"
Jul 29 19:54:20.901228 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:20.901001    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5a386be-f1cf-43ed-9909-ffb4837b839f" containerName="payload-processing"
Jul 29 19:54:20.901228 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:20.901048    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7739089-0b8f-44c2-b4f1-ca0bef75ac40" containerName="payload-pre-processing"
Jul 29 19:54:20.901228 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:20.901056    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd6ac5a6-434f-4f07-8c54-59ff978b9c9c" containerName="revoke-keys"
Jul 29 19:54:20.901228 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:20.901063    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e49b647-67ab-46e8-9bbb-9bbc1b69b844" containerName="maas-api"
Jul 29 19:54:20.901228 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:20.901069    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5a386be-f1cf-43ed-9909-ffb4837b839f" containerName="payload-processing"
Jul 29 19:54:20.903766 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:20.903747    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-59d5db-596bfd5995-hxwbb"
Jul 29 19:54:20.907790 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:20.907769    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-59d5db-serving-cert\""
Jul 29 19:54:20.907924 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:20.907863    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 19:54:20.908510 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:20.908495    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 29 19:54:20.908552 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:20.908533    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6wzbz\""
Jul 29 19:54:20.917329 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:20.917299    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-59d5db-596bfd5995-hxwbb"]
Jul 29 19:54:20.977784 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:20.977757    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eec1d824-d889-4bf6-b600-00181be18634-openshift-service-ca\") pod \"maas-api-e2e-inf-a-59d5db-596bfd5995-hxwbb\" (UID: \"eec1d824-d889-4bf6-b600-00181be18634\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-59d5db-596bfd5995-hxwbb"
Jul 29 19:54:20.977933 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:20.977819    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eec1d824-d889-4bf6-b600-00181be18634-maas-api-tls\") pod \"maas-api-e2e-inf-a-59d5db-596bfd5995-hxwbb\" (UID: \"eec1d824-d889-4bf6-b600-00181be18634\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-59d5db-596bfd5995-hxwbb"
Jul 29 19:54:20.977933 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:20.977864    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6b8f\" (UniqueName: \"kubernetes.io/projected/eec1d824-d889-4bf6-b600-00181be18634-kube-api-access-j6b8f\") pod \"maas-api-e2e-inf-a-59d5db-596bfd5995-hxwbb\" (UID: \"eec1d824-d889-4bf6-b600-00181be18634\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-59d5db-596bfd5995-hxwbb"
Jul 29 19:54:21.078553 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:21.078513    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eec1d824-d889-4bf6-b600-00181be18634-openshift-service-ca\") pod \"maas-api-e2e-inf-a-59d5db-596bfd5995-hxwbb\" (UID: \"eec1d824-d889-4bf6-b600-00181be18634\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-59d5db-596bfd5995-hxwbb"
Jul 29 19:54:21.078706 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:21.078580    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eec1d824-d889-4bf6-b600-00181be18634-maas-api-tls\") pod \"maas-api-e2e-inf-a-59d5db-596bfd5995-hxwbb\" (UID: \"eec1d824-d889-4bf6-b600-00181be18634\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-59d5db-596bfd5995-hxwbb"
Jul 29 19:54:21.078706 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:21.078626    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6b8f\" (UniqueName: \"kubernetes.io/projected/eec1d824-d889-4bf6-b600-00181be18634-kube-api-access-j6b8f\") pod \"maas-api-e2e-inf-a-59d5db-596bfd5995-hxwbb\" (UID: \"eec1d824-d889-4bf6-b600-00181be18634\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-59d5db-596bfd5995-hxwbb"
Jul 29 19:54:21.078857 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:54:21.078720    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-59d5db-serving-cert: secret "maas-api-e2e-inf-a-59d5db-serving-cert" not found
Jul 29 19:54:21.078857 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:54:21.078791    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eec1d824-d889-4bf6-b600-00181be18634-maas-api-tls podName:eec1d824-d889-4bf6-b600-00181be18634 nodeName:}" failed. No retries permitted until 2026-07-29 19:54:21.578775139 +0000 UTC m=+2228.807164243 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eec1d824-d889-4bf6-b600-00181be18634-maas-api-tls") pod "maas-api-e2e-inf-a-59d5db-596bfd5995-hxwbb" (UID: "eec1d824-d889-4bf6-b600-00181be18634") : secret "maas-api-e2e-inf-a-59d5db-serving-cert" not found
Jul 29 19:54:21.079158 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:21.079141    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eec1d824-d889-4bf6-b600-00181be18634-openshift-service-ca\") pod \"maas-api-e2e-inf-a-59d5db-596bfd5995-hxwbb\" (UID: \"eec1d824-d889-4bf6-b600-00181be18634\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-59d5db-596bfd5995-hxwbb"
Jul 29 19:54:21.088494 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:21.088465    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6b8f\" (UniqueName: \"kubernetes.io/projected/eec1d824-d889-4bf6-b600-00181be18634-kube-api-access-j6b8f\") pod \"maas-api-e2e-inf-a-59d5db-596bfd5995-hxwbb\" (UID: \"eec1d824-d889-4bf6-b600-00181be18634\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-59d5db-596bfd5995-hxwbb"
Jul 29 19:54:21.506478 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:21.506441    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-59d5db-84d95c66fb-m48mz"]
Jul 29 19:54:21.509724 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:21.509702    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-59d5db-84d95c66fb-m48mz"
Jul 29 19:54:21.512029 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:21.512006    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-59d5db-dockercfg-x9j9n\""
Jul 29 19:54:21.512122 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:21.512047    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-59d5db\""
Jul 29 19:54:21.519297 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:21.519277    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-59d5db-84d95c66fb-m48mz"]
Jul 29 19:54:21.539460 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:21.539442    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-59d5db-65b8f8c948-qbx6l"]
Jul 29 19:54:21.542329 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:21.542316    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-59d5db-65b8f8c948-qbx6l"
Jul 29 19:54:21.551018 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:21.550994    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-59d5db-65b8f8c948-qbx6l"]
Jul 29 19:54:21.583615 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:21.583593    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch7qx\" (UniqueName: \"kubernetes.io/projected/87cb3932-4c61-4596-ac40-c5ab6eb79329-kube-api-access-ch7qx\") pod \"payload-processing-e2e-inf-a-59d5db-65b8f8c948-qbx6l\" (UID: \"87cb3932-4c61-4596-ac40-c5ab6eb79329\") " pod="openshift-ingress/payload-processing-e2e-inf-a-59d5db-65b8f8c948-qbx6l"
Jul 29 19:54:21.583729 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:21.583636    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/214760ad-73c8-4e9e-b58c-7a2c6a72c272-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-59d5db-84d95c66fb-m48mz\" (UID: \"214760ad-73c8-4e9e-b58c-7a2c6a72c272\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-59d5db-84d95c66fb-m48mz"
Jul 29 19:54:21.583729 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:21.583656    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pmgh\" (UniqueName: \"kubernetes.io/projected/214760ad-73c8-4e9e-b58c-7a2c6a72c272-kube-api-access-2pmgh\") pod \"payload-pre-processing-e2e-inf-a-59d5db-84d95c66fb-m48mz\" (UID: \"214760ad-73c8-4e9e-b58c-7a2c6a72c272\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-59d5db-84d95c66fb-m48mz"
Jul 29 19:54:21.583729 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:21.583726    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87cb3932-4c61-4596-ac40-c5ab6eb79329-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-59d5db-65b8f8c948-qbx6l\" (UID: \"87cb3932-4c61-4596-ac40-c5ab6eb79329\") " pod="openshift-ingress/payload-processing-e2e-inf-a-59d5db-65b8f8c948-qbx6l"
Jul 29 19:54:21.583935 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:21.583792    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eec1d824-d889-4bf6-b600-00181be18634-maas-api-tls\") pod \"maas-api-e2e-inf-a-59d5db-596bfd5995-hxwbb\" (UID: \"eec1d824-d889-4bf6-b600-00181be18634\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-59d5db-596bfd5995-hxwbb"
Jul 29 19:54:21.586760 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:21.586724    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eec1d824-d889-4bf6-b600-00181be18634-maas-api-tls\") pod \"maas-api-e2e-inf-a-59d5db-596bfd5995-hxwbb\" (UID: \"eec1d824-d889-4bf6-b600-00181be18634\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-59d5db-596bfd5995-hxwbb"
Jul 29 19:54:21.684484 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:21.684449    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch7qx\" (UniqueName: \"kubernetes.io/projected/87cb3932-4c61-4596-ac40-c5ab6eb79329-kube-api-access-ch7qx\") pod \"payload-processing-e2e-inf-a-59d5db-65b8f8c948-qbx6l\" (UID: \"87cb3932-4c61-4596-ac40-c5ab6eb79329\") " pod="openshift-ingress/payload-processing-e2e-inf-a-59d5db-65b8f8c948-qbx6l"
Jul 29 19:54:21.684668 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:21.684504    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/214760ad-73c8-4e9e-b58c-7a2c6a72c272-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-59d5db-84d95c66fb-m48mz\" (UID: \"214760ad-73c8-4e9e-b58c-7a2c6a72c272\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-59d5db-84d95c66fb-m48mz"
Jul 29 19:54:21.684668 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:21.684533    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pmgh\" (UniqueName: \"kubernetes.io/projected/214760ad-73c8-4e9e-b58c-7a2c6a72c272-kube-api-access-2pmgh\") pod \"payload-pre-processing-e2e-inf-a-59d5db-84d95c66fb-m48mz\" (UID: \"214760ad-73c8-4e9e-b58c-7a2c6a72c272\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-59d5db-84d95c66fb-m48mz"
Jul 29 19:54:21.684668 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:21.684555    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87cb3932-4c61-4596-ac40-c5ab6eb79329-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-59d5db-65b8f8c948-qbx6l\" (UID: \"87cb3932-4c61-4596-ac40-c5ab6eb79329\") " pod="openshift-ingress/payload-processing-e2e-inf-a-59d5db-65b8f8c948-qbx6l"
Jul 29 19:54:21.685489 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:21.685465    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/214760ad-73c8-4e9e-b58c-7a2c6a72c272-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-59d5db-84d95c66fb-m48mz\" (UID: \"214760ad-73c8-4e9e-b58c-7a2c6a72c272\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-59d5db-84d95c66fb-m48mz"
Jul 29 19:54:21.685545 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:21.685466    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87cb3932-4c61-4596-ac40-c5ab6eb79329-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-59d5db-65b8f8c948-qbx6l\" (UID: \"87cb3932-4c61-4596-ac40-c5ab6eb79329\") " pod="openshift-ingress/payload-processing-e2e-inf-a-59d5db-65b8f8c948-qbx6l"
Jul 29 19:54:21.692910 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:21.692884    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch7qx\" (UniqueName: \"kubernetes.io/projected/87cb3932-4c61-4596-ac40-c5ab6eb79329-kube-api-access-ch7qx\") pod \"payload-processing-e2e-inf-a-59d5db-65b8f8c948-qbx6l\" (UID: \"87cb3932-4c61-4596-ac40-c5ab6eb79329\") " pod="openshift-ingress/payload-processing-e2e-inf-a-59d5db-65b8f8c948-qbx6l"
Jul 29 19:54:21.693022 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:21.692918    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pmgh\" (UniqueName: \"kubernetes.io/projected/214760ad-73c8-4e9e-b58c-7a2c6a72c272-kube-api-access-2pmgh\") pod \"payload-pre-processing-e2e-inf-a-59d5db-84d95c66fb-m48mz\" (UID: \"214760ad-73c8-4e9e-b58c-7a2c6a72c272\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-59d5db-84d95c66fb-m48mz"
Jul 29 19:54:21.813249 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:21.813171    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-59d5db-596bfd5995-hxwbb"
Jul 29 19:54:21.821143 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:21.821119    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-59d5db-84d95c66fb-m48mz"
Jul 29 19:54:21.852080 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:21.852054    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-59d5db-65b8f8c948-qbx6l"
Jul 29 19:54:21.967934 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:21.967844    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-59d5db-596bfd5995-hxwbb"]
Jul 29 19:54:21.970942 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:54:21.970858    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeec1d824_d889_4bf6_b600_00181be18634.slice/crio-cf0072864415800bafc35b9c0a138fabb056b6360cdef58c71e6832d95266df2 WatchSource:0}: Error finding container cf0072864415800bafc35b9c0a138fabb056b6360cdef58c71e6832d95266df2: Status 404 returned error can't find the container with id cf0072864415800bafc35b9c0a138fabb056b6360cdef58c71e6832d95266df2
Jul 29 19:54:21.991481 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:21.991341    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-59d5db-84d95c66fb-m48mz"]
Jul 29 19:54:21.993478 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:54:21.993454    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod214760ad_73c8_4e9e_b58c_7a2c6a72c272.slice/crio-9abf97ab368d8cd637888ce305b17faa86fb24662b1e322b70c85f5f82aed4a0 WatchSource:0}: Error finding container 9abf97ab368d8cd637888ce305b17faa86fb24662b1e322b70c85f5f82aed4a0: Status 404 returned error can't find the container with id 9abf97ab368d8cd637888ce305b17faa86fb24662b1e322b70c85f5f82aed4a0
Jul 29 19:54:22.014568 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:22.014541    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-59d5db-65b8f8c948-qbx6l"]
Jul 29 19:54:22.032872 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:54:22.032844    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87cb3932_4c61_4596_ac40_c5ab6eb79329.slice/crio-1a855098fe5a10ec25cfe072f6a2dc77794ee078874a51973bba0cb51c2b13b5 WatchSource:0}: Error finding container 1a855098fe5a10ec25cfe072f6a2dc77794ee078874a51973bba0cb51c2b13b5: Status 404 returned error can't find the container with id 1a855098fe5a10ec25cfe072f6a2dc77794ee078874a51973bba0cb51c2b13b5
Jul 29 19:54:22.698356 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:22.698322    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-59d5db-84d95c66fb-m48mz" event={"ID":"214760ad-73c8-4e9e-b58c-7a2c6a72c272","Type":"ContainerStarted","Data":"25bfe39f9e70c2a1d5f3a78332c89628a375de30dac880ee8cd7ee451a6e7e9e"}
Jul 29 19:54:22.698541 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:22.698375    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-59d5db-84d95c66fb-m48mz" event={"ID":"214760ad-73c8-4e9e-b58c-7a2c6a72c272","Type":"ContainerStarted","Data":"9abf97ab368d8cd637888ce305b17faa86fb24662b1e322b70c85f5f82aed4a0"}
Jul 29 19:54:22.698541 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:22.698411    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-59d5db-84d95c66fb-m48mz"
Jul 29 19:54:22.699827 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:22.699782    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-59d5db-65b8f8c948-qbx6l" event={"ID":"87cb3932-4c61-4596-ac40-c5ab6eb79329","Type":"ContainerStarted","Data":"d7be693b137691a2fe29fc6d48c4075f41c9d9e3531c6092141bc3c4fa03947b"}
Jul 29 19:54:22.699952 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:22.699838    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-59d5db-65b8f8c948-qbx6l"
Jul 29 19:54:22.699952 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:22.699862    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-59d5db-65b8f8c948-qbx6l" event={"ID":"87cb3932-4c61-4596-ac40-c5ab6eb79329","Type":"ContainerStarted","Data":"1a855098fe5a10ec25cfe072f6a2dc77794ee078874a51973bba0cb51c2b13b5"}
Jul 29 19:54:22.701112 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:22.701092    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-59d5db-596bfd5995-hxwbb" event={"ID":"eec1d824-d889-4bf6-b600-00181be18634","Type":"ContainerStarted","Data":"cf0072864415800bafc35b9c0a138fabb056b6360cdef58c71e6832d95266df2"}
Jul 29 19:54:22.715781 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:22.715739    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-59d5db-84d95c66fb-m48mz" podStartSLOduration=1.715725687 podStartE2EDuration="1.715725687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:54:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:54:22.714309747 +0000 UTC m=+2229.942698876" watchObservedRunningTime="2026-07-29 19:54:22.715725687 +0000 UTC m=+2229.944114813"
Jul 29 19:54:22.731753 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:22.731711    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-59d5db-65b8f8c948-qbx6l" podStartSLOduration=1.731698545 podStartE2EDuration="1.731698545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:54:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:54:22.727657006 +0000 UTC m=+2229.956046132" watchObservedRunningTime="2026-07-29 19:54:22.731698545 +0000 UTC m=+2229.960087672"
Jul 29 19:54:23.708492 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:23.708458    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-59d5db-596bfd5995-hxwbb" event={"ID":"eec1d824-d889-4bf6-b600-00181be18634","Type":"ContainerStarted","Data":"703d231f6cf828daa88a62a22ba4d18b3a66aa8d38cb871b38a112ffe6fe9960"}
Jul 29 19:54:23.708978 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:23.708768    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-59d5db-596bfd5995-hxwbb"
Jul 29 19:54:23.728195 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:23.728153    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-59d5db-596bfd5995-hxwbb" podStartSLOduration=2.941462829 podStartE2EDuration="3.728134368s" totalImagesPullingTime="786.671539ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:54:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:54:23.726207844 +0000 UTC m=+2230.954596972" watchObservedRunningTime="2026-07-29 19:54:23.728134368 +0000 UTC m=+2230.956523494"
Jul 29 19:54:25.792809 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:25.792761    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:54:29.719249 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:29.719217    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-59d5db-596bfd5995-hxwbb"
Jul 29 19:54:33.710880 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:33.710852    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-59d5db-84d95c66fb-m48mz"
Jul 29 19:54:33.711248 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:33.710931    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-59d5db-65b8f8c948-qbx6l"
Jul 29 19:54:49.208741 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:49.208709    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-accf7d-86d774c5f7-w5zpd"]
Jul 29 19:54:49.212061 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:49.212042    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-accf7d-86d774c5f7-w5zpd"
Jul 29 19:54:49.213991 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:49.213960    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-accf7d-serving-cert\""
Jul 29 19:54:49.220898 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:49.220878    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-accf7d-86d774c5f7-w5zpd"]
Jul 29 19:54:49.315395 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:49.315368    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f79a0c1-544a-45d3-be04-b3c02746391f-maas-api-tls\") pod \"maas-api-e2e-inf-b-accf7d-86d774c5f7-w5zpd\" (UID: \"4f79a0c1-544a-45d3-be04-b3c02746391f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-accf7d-86d774c5f7-w5zpd"
Jul 29 19:54:49.315518 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:49.315402    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d86nj\" (UniqueName: \"kubernetes.io/projected/4f79a0c1-544a-45d3-be04-b3c02746391f-kube-api-access-d86nj\") pod \"maas-api-e2e-inf-b-accf7d-86d774c5f7-w5zpd\" (UID: \"4f79a0c1-544a-45d3-be04-b3c02746391f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-accf7d-86d774c5f7-w5zpd"
Jul 29 19:54:49.315518 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:49.315497    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f79a0c1-544a-45d3-be04-b3c02746391f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-accf7d-86d774c5f7-w5zpd\" (UID: \"4f79a0c1-544a-45d3-be04-b3c02746391f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-accf7d-86d774c5f7-w5zpd"
Jul 29 19:54:49.416874 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:49.416842    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f79a0c1-544a-45d3-be04-b3c02746391f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-accf7d-86d774c5f7-w5zpd\" (UID: \"4f79a0c1-544a-45d3-be04-b3c02746391f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-accf7d-86d774c5f7-w5zpd"
Jul 29 19:54:49.417046 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:49.416893    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f79a0c1-544a-45d3-be04-b3c02746391f-maas-api-tls\") pod \"maas-api-e2e-inf-b-accf7d-86d774c5f7-w5zpd\" (UID: \"4f79a0c1-544a-45d3-be04-b3c02746391f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-accf7d-86d774c5f7-w5zpd"
Jul 29 19:54:49.417046 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:49.416916    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d86nj\" (UniqueName: \"kubernetes.io/projected/4f79a0c1-544a-45d3-be04-b3c02746391f-kube-api-access-d86nj\") pod \"maas-api-e2e-inf-b-accf7d-86d774c5f7-w5zpd\" (UID: \"4f79a0c1-544a-45d3-be04-b3c02746391f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-accf7d-86d774c5f7-w5zpd"
Jul 29 19:54:49.417520 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:49.417494    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f79a0c1-544a-45d3-be04-b3c02746391f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-accf7d-86d774c5f7-w5zpd\" (UID: \"4f79a0c1-544a-45d3-be04-b3c02746391f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-accf7d-86d774c5f7-w5zpd"
Jul 29 19:54:49.419227 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:49.419209    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f79a0c1-544a-45d3-be04-b3c02746391f-maas-api-tls\") pod \"maas-api-e2e-inf-b-accf7d-86d774c5f7-w5zpd\" (UID: \"4f79a0c1-544a-45d3-be04-b3c02746391f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-accf7d-86d774c5f7-w5zpd"
Jul 29 19:54:49.432678 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:49.432658    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d86nj\" (UniqueName: \"kubernetes.io/projected/4f79a0c1-544a-45d3-be04-b3c02746391f-kube-api-access-d86nj\") pod \"maas-api-e2e-inf-b-accf7d-86d774c5f7-w5zpd\" (UID: \"4f79a0c1-544a-45d3-be04-b3c02746391f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-accf7d-86d774c5f7-w5zpd"
Jul 29 19:54:49.522710 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:49.522652    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-accf7d-86d774c5f7-w5zpd"
Jul 29 19:54:49.651573 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:49.651541    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-accf7d-86d774c5f7-w5zpd"]
Jul 29 19:54:49.653371 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:54:49.653345    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f79a0c1_544a_45d3_be04_b3c02746391f.slice/crio-76094bdcfbfb663d7b1f961d930bfa98505b15f1a8ff575c8f28de88afad9a24 WatchSource:0}: Error finding container 76094bdcfbfb663d7b1f961d930bfa98505b15f1a8ff575c8f28de88afad9a24: Status 404 returned error can't find the container with id 76094bdcfbfb663d7b1f961d930bfa98505b15f1a8ff575c8f28de88afad9a24
Jul 29 19:54:49.825102 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:49.825013    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-accf7d-86d774c5f7-w5zpd" event={"ID":"4f79a0c1-544a-45d3-be04-b3c02746391f","Type":"ContainerStarted","Data":"76094bdcfbfb663d7b1f961d930bfa98505b15f1a8ff575c8f28de88afad9a24"}
Jul 29 19:54:49.996072 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:49.996041    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-accf7d-755887b768-nzs78"]
Jul 29 19:54:50.000369 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:50.000348    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-accf7d-755887b768-nzs78"
Jul 29 19:54:50.002865 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:50.002833    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-accf7d\""
Jul 29 19:54:50.002986 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:50.002969    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-accf7d-dockercfg-ljxch\""
Jul 29 19:54:50.007726 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:50.007700    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-accf7d-755887b768-nzs78"]
Jul 29 19:54:50.123910 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:50.123882    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9b7899d-03b1-4fda-b0ff-900ed7be3b25-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-accf7d-755887b768-nzs78\" (UID: \"b9b7899d-03b1-4fda-b0ff-900ed7be3b25\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-accf7d-755887b768-nzs78"
Jul 29 19:54:50.124010 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:50.123940    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t65h\" (UniqueName: \"kubernetes.io/projected/b9b7899d-03b1-4fda-b0ff-900ed7be3b25-kube-api-access-5t65h\") pod \"payload-pre-processing-e2e-inf-b-accf7d-755887b768-nzs78\" (UID: \"b9b7899d-03b1-4fda-b0ff-900ed7be3b25\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-accf7d-755887b768-nzs78"
Jul 29 19:54:50.224961 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:50.224933    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9b7899d-03b1-4fda-b0ff-900ed7be3b25-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-accf7d-755887b768-nzs78\" (UID: \"b9b7899d-03b1-4fda-b0ff-900ed7be3b25\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-accf7d-755887b768-nzs78"
Jul 29 19:54:50.225306 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:50.224975    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t65h\" (UniqueName: \"kubernetes.io/projected/b9b7899d-03b1-4fda-b0ff-900ed7be3b25-kube-api-access-5t65h\") pod \"payload-pre-processing-e2e-inf-b-accf7d-755887b768-nzs78\" (UID: \"b9b7899d-03b1-4fda-b0ff-900ed7be3b25\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-accf7d-755887b768-nzs78"
Jul 29 19:54:50.225640 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:50.225620    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9b7899d-03b1-4fda-b0ff-900ed7be3b25-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-accf7d-755887b768-nzs78\" (UID: \"b9b7899d-03b1-4fda-b0ff-900ed7be3b25\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-accf7d-755887b768-nzs78"
Jul 29 19:54:50.232541 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:50.232523    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t65h\" (UniqueName: \"kubernetes.io/projected/b9b7899d-03b1-4fda-b0ff-900ed7be3b25-kube-api-access-5t65h\") pod \"payload-pre-processing-e2e-inf-b-accf7d-755887b768-nzs78\" (UID: \"b9b7899d-03b1-4fda-b0ff-900ed7be3b25\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-accf7d-755887b768-nzs78"
Jul 29 19:54:50.311459 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:50.311435    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-accf7d-755887b768-nzs78"
Jul 29 19:54:50.439475 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:50.439451    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-accf7d-755887b768-nzs78"]
Jul 29 19:54:50.441409 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:54:50.441376    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9b7899d_03b1_4fda_b0ff_900ed7be3b25.slice/crio-e2e09f31f56ffda98ce7f70a1100861b9711852e0c4b01429ed4a47ecabff94b WatchSource:0}: Error finding container e2e09f31f56ffda98ce7f70a1100861b9711852e0c4b01429ed4a47ecabff94b: Status 404 returned error can't find the container with id e2e09f31f56ffda98ce7f70a1100861b9711852e0c4b01429ed4a47ecabff94b
Jul 29 19:54:50.830894 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:50.830853    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-accf7d-755887b768-nzs78" event={"ID":"b9b7899d-03b1-4fda-b0ff-900ed7be3b25","Type":"ContainerStarted","Data":"982dad23ac4a904431d7fdb96dafeaf0689cdbbdef5c8e9dcc973b36ec2b3bfa"}
Jul 29 19:54:50.831072 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:50.830923    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-accf7d-755887b768-nzs78"
Jul 29 19:54:50.831072 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:50.830947    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-accf7d-755887b768-nzs78" event={"ID":"b9b7899d-03b1-4fda-b0ff-900ed7be3b25","Type":"ContainerStarted","Data":"e2e09f31f56ffda98ce7f70a1100861b9711852e0c4b01429ed4a47ecabff94b"}
Jul 29 19:54:50.832348 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:50.832323    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-accf7d-86d774c5f7-w5zpd" event={"ID":"4f79a0c1-544a-45d3-be04-b3c02746391f","Type":"ContainerStarted","Data":"77f545bd07ed04f714b4fc1b0319dddde1d04644a81e5ec4a494de82a866caf4"}
Jul 29 19:54:50.832404 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:50.832390    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-accf7d-86d774c5f7-w5zpd"
Jul 29 19:54:50.848537 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:50.848497    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-accf7d-755887b768-nzs78" podStartSLOduration=1.848484888 podStartE2EDuration="1.848484888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:54:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:54:50.847222058 +0000 UTC m=+2258.075611184" watchObservedRunningTime="2026-07-29 19:54:50.848484888 +0000 UTC m=+2258.076874014"
Jul 29 19:54:50.865927 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:50.865893    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-accf7d-86d774c5f7-w5zpd" podStartSLOduration=1.457199509 podStartE2EDuration="1.865881821s" totalImagesPullingTime="408.682312ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:54:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:54:50.864715219 +0000 UTC m=+2258.093104345" watchObservedRunningTime="2026-07-29 19:54:50.865881821 +0000 UTC m=+2258.094270947"
Jul 29 19:54:56.842568 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:56.842535    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-accf7d-86d774c5f7-w5zpd"
Jul 29 19:54:57.299904 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:54:57.299866    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:55:01.838961 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:55:01.838930    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-accf7d-755887b768-nzs78"
Jul 29 19:55:24.174206 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:55:24.174172    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-59d5db/test-model-59d5db-kserve-d8f7b699f-zl48l"]
Jul 29 19:55:24.177385 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:55:24.177368    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-59d5db/test-model-59d5db-kserve-d8f7b699f-zl48l"
Jul 29 19:55:24.180721 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:55:24.180699    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-59d5db\"/\"test-model-59d5db-kserve-self-signed-certs\""
Jul 29 19:55:24.180897 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:55:24.180736    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-59d5db\"/\"openshift-service-ca.crt\""
Jul 29 19:55:24.180897 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:55:24.180699    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-59d5db\"/\"default-dockercfg-tvzwf\""
Jul 29 19:55:24.180897 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:55:24.180842    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-59d5db\"/\"kube-root-ca.crt\""
Jul 29 19:55:24.189304 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:55:24.189267    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-59d5db/test-model-59d5db-kserve-d8f7b699f-zl48l"]
Jul 29 19:55:24.305350 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:55:24.305313    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e72adc2-52c3-4887-b97a-556ea23d030f-home\") pod \"test-model-59d5db-kserve-d8f7b699f-zl48l\" (UID: \"9e72adc2-52c3-4887-b97a-556ea23d030f\") " pod="ai-tenant-e2e-inf-a-59d5db/test-model-59d5db-kserve-d8f7b699f-zl48l"
Jul 29 19:55:24.305350 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:55:24.305352    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e72adc2-52c3-4887-b97a-556ea23d030f-model-cache\") pod \"test-model-59d5db-kserve-d8f7b699f-zl48l\" (UID: \"9e72adc2-52c3-4887-b97a-556ea23d030f\") " pod="ai-tenant-e2e-inf-a-59d5db/test-model-59d5db-kserve-d8f7b699f-zl48l"
Jul 29 19:55:24.305591 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:55:24.305377    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e72adc2-52c3-4887-b97a-556ea23d030f-tmp-dir\") pod \"test-model-59d5db-kserve-d8f7b699f-zl48l\" (UID: \"9e72adc2-52c3-4887-b97a-556ea23d030f\") " pod="ai-tenant-e2e-inf-a-59d5db/test-model-59d5db-kserve-d8f7b699f-zl48l"
Jul 29 19:55:24.305591 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:55:24.305437    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e72adc2-52c3-4887-b97a-556ea23d030f-tls-certs\") pod \"test-model-59d5db-kserve-d8f7b699f-zl48l\" (UID: \"9e72adc2-52c3-4887-b97a-556ea23d030f\") " pod="ai-tenant-e2e-inf-a-59d5db/test-model-59d5db-kserve-d8f7b699f-zl48l"
Jul 29 19:55:24.305591 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:55:24.305500    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e72adc2-52c3-4887-b97a-556ea23d030f-dshm\") pod \"test-model-59d5db-kserve-d8f7b699f-zl48l\" (UID: \"9e72adc2-52c3-4887-b97a-556ea23d030f\") " pod="ai-tenant-e2e-inf-a-59d5db/test-model-59d5db-kserve-d8f7b699f-zl48l"
Jul 29 19:55:24.305591 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:55:24.305531    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh6dj\" (UniqueName: \"kubernetes.io/projected/9e72adc2-52c3-4887-b97a-556ea23d030f-kube-api-access-wh6dj\") pod \"test-model-59d5db-kserve-d8f7b699f-zl48l\" (UID: \"9e72adc2-52c3-4887-b97a-556ea23d030f\") " pod="ai-tenant-e2e-inf-a-59d5db/test-model-59d5db-kserve-d8f7b699f-zl48l"
Jul 29 19:55:24.406894 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:55:24.406853    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e72adc2-52c3-4887-b97a-556ea23d030f-home\") pod \"test-model-59d5db-kserve-d8f7b699f-zl48l\" (UID: \"9e72adc2-52c3-4887-b97a-556ea23d030f\") " pod="ai-tenant-e2e-inf-a-59d5db/test-model-59d5db-kserve-d8f7b699f-zl48l"
Jul 29 19:55:24.406894 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:55:24.406894    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e72adc2-52c3-4887-b97a-556ea23d030f-model-cache\") pod \"test-model-59d5db-kserve-d8f7b699f-zl48l\" (UID: \"9e72adc2-52c3-4887-b97a-556ea23d030f\") " pod="ai-tenant-e2e-inf-a-59d5db/test-model-59d5db-kserve-d8f7b699f-zl48l"
Jul 29 19:55:24.407144 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:55:24.406927    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e72adc2-52c3-4887-b97a-556ea23d030f-tmp-dir\") pod \"test-model-59d5db-kserve-d8f7b699f-zl48l\" (UID: \"9e72adc2-52c3-4887-b97a-556ea23d030f\") " pod="ai-tenant-e2e-inf-a-59d5db/test-model-59d5db-kserve-d8f7b699f-zl48l"
Jul 29 19:55:24.407144 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:55:24.406961    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e72adc2-52c3-4887-b97a-556ea23d030f-tls-certs\") pod \"test-model-59d5db-kserve-d8f7b699f-zl48l\" (UID: \"9e72adc2-52c3-4887-b97a-556ea23d030f\") " pod="ai-tenant-e2e-inf-a-59d5db/test-model-59d5db-kserve-d8f7b699f-zl48l"
Jul 29 19:55:24.407144 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:55:24.406992    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e72adc2-52c3-4887-b97a-556ea23d030f-dshm\") pod \"test-model-59d5db-kserve-d8f7b699f-zl48l\" (UID: \"9e72adc2-52c3-4887-b97a-556ea23d030f\") " pod="ai-tenant-e2e-inf-a-59d5db/test-model-59d5db-kserve-d8f7b699f-zl48l"
Jul 29 19:55:24.407144 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:55:24.407017    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh6dj\" (UniqueName: \"kubernetes.io/projected/9e72adc2-52c3-4887-b97a-556ea23d030f-kube-api-access-wh6dj\") pod \"test-model-59d5db-kserve-d8f7b699f-zl48l\" (UID: \"9e72adc2-52c3-4887-b97a-556ea23d030f\") " pod="ai-tenant-e2e-inf-a-59d5db/test-model-59d5db-kserve-d8f7b699f-zl48l"
Jul 29 19:55:24.407373 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:55:24.407335    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e72adc2-52c3-4887-b97a-556ea23d030f-home\") pod \"test-model-59d5db-kserve-d8f7b699f-zl48l\" (UID: \"9e72adc2-52c3-4887-b97a-556ea23d030f\") " pod="ai-tenant-e2e-inf-a-59d5db/test-model-59d5db-kserve-d8f7b699f-zl48l"
Jul 29 19:55:24.407373 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:55:24.407349    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e72adc2-52c3-4887-b97a-556ea23d030f-tmp-dir\") pod \"test-model-59d5db-kserve-d8f7b699f-zl48l\" (UID: \"9e72adc2-52c3-4887-b97a-556ea23d030f\") " pod="ai-tenant-e2e-inf-a-59d5db/test-model-59d5db-kserve-d8f7b699f-zl48l"
Jul 29 19:55:24.407634 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:55:24.407609    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e72adc2-52c3-4887-b97a-556ea23d030f-model-cache\") pod \"test-model-59d5db-kserve-d8f7b699f-zl48l\" (UID: \"9e72adc2-52c3-4887-b97a-556ea23d030f\") " pod="ai-tenant-e2e-inf-a-59d5db/test-model-59d5db-kserve-d8f7b699f-zl48l"
Jul 29 19:55:24.409216 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:55:24.409198    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e72adc2-52c3-4887-b97a-556ea23d030f-dshm\") pod \"test-model-59d5db-kserve-d8f7b699f-zl48l\" (UID: \"9e72adc2-52c3-4887-b97a-556ea23d030f\") " pod="ai-tenant-e2e-inf-a-59d5db/test-model-59d5db-kserve-d8f7b699f-zl48l"
Jul 29 19:55:24.409608 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:55:24.409588    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e72adc2-52c3-4887-b97a-556ea23d030f-tls-certs\") pod \"test-model-59d5db-kserve-d8f7b699f-zl48l\" (UID: \"9e72adc2-52c3-4887-b97a-556ea23d030f\") " pod="ai-tenant-e2e-inf-a-59d5db/test-model-59d5db-kserve-d8f7b699f-zl48l"
Jul 29 19:55:24.415459 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:55:24.415431    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh6dj\" (UniqueName: \"kubernetes.io/projected/9e72adc2-52c3-4887-b97a-556ea23d030f-kube-api-access-wh6dj\") pod \"test-model-59d5db-kserve-d8f7b699f-zl48l\" (UID: \"9e72adc2-52c3-4887-b97a-556ea23d030f\") " pod="ai-tenant-e2e-inf-a-59d5db/test-model-59d5db-kserve-d8f7b699f-zl48l"
Jul 29 19:55:24.486428 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:55:24.486340    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-59d5db/test-model-59d5db-kserve-d8f7b699f-zl48l"
Jul 29 19:55:24.618170 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:55:24.618141    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-59d5db/test-model-59d5db-kserve-d8f7b699f-zl48l"]
Jul 29 19:55:24.620749 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:55:24.620725    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e72adc2_52c3_4887_b97a_556ea23d030f.slice/crio-d5836e024c71a527f6f69f28cb18e99315459937deafc2797ddba62c79619c1f WatchSource:0}: Error finding container d5836e024c71a527f6f69f28cb18e99315459937deafc2797ddba62c79619c1f: Status 404 returned error can't find the container with id d5836e024c71a527f6f69f28cb18e99315459937deafc2797ddba62c79619c1f
Jul 29 19:55:24.982117 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:55:24.982075    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-59d5db/test-model-59d5db-kserve-d8f7b699f-zl48l" event={"ID":"9e72adc2-52c3-4887-b97a-556ea23d030f","Type":"ContainerStarted","Data":"c80e3102ca1059d14a701255eb28af84de115e37b986ae000708ef58603d991a"}
Jul 29 19:55:24.982289 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:55:24.982126    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-59d5db/test-model-59d5db-kserve-d8f7b699f-zl48l" event={"ID":"9e72adc2-52c3-4887-b97a-556ea23d030f","Type":"ContainerStarted","Data":"d5836e024c71a527f6f69f28cb18e99315459937deafc2797ddba62c79619c1f"}
Jul 29 19:55:24.999250 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:55:24.999195    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-59d5db/test-model-59d5db-kserve-d8f7b699f-zl48l" podStartSLOduration=0.999178148 podStartE2EDuration="999.178148ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:55:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:55:24.998037313 +0000 UTC m=+2292.226426440" watchObservedRunningTime="2026-07-29 19:55:24.999178148 +0000 UTC m=+2292.227567300"
Jul 29 19:55:29.100638 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:55:29.100606    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:55:34.487434 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:55:34.487398    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-59d5db/test-model-59d5db-kserve-d8f7b699f-zl48l"
Jul 29 19:55:34.487434 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:55:34.487439    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-59d5db/test-model-59d5db-kserve-d8f7b699f-zl48l"
Jul 29 19:55:34.499730 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:55:34.499706    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-59d5db/test-model-59d5db-kserve-d8f7b699f-zl48l"
Jul 29 19:55:35.039561 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:55:35.039531    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-59d5db/test-model-59d5db-kserve-d8f7b699f-zl48l"
Jul 29 19:55:55.298698 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:55:55.298665    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:56:11.602501 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:11.602415    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:56:43.609770 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:43.609733    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-59d5db/test-model-59d5db-kserve-d8f7b699f-zl48l"]
Jul 29 19:56:48.195623 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:48.195590    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:56:55.092744 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:55.092710    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-59d5db-596bfd5995-hxwbb"]
Jul 29 19:56:55.093283 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:55.092988    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-59d5db-596bfd5995-hxwbb" podUID="eec1d824-d889-4bf6-b600-00181be18634" containerName="maas-api" containerID="cri-o://703d231f6cf828daa88a62a22ba4d18b3a66aa8d38cb871b38a112ffe6fe9960" gracePeriod=30
Jul 29 19:56:55.147873 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:55.147839    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-59d5db-65b8f8c948-qbx6l"]
Jul 29 19:56:55.148111 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:55.148085    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-59d5db-65b8f8c948-qbx6l" podUID="87cb3932-4c61-4596-ac40-c5ab6eb79329" containerName="payload-processing" containerID="cri-o://d7be693b137691a2fe29fc6d48c4075f41c9d9e3531c6092141bc3c4fa03947b" gracePeriod=30
Jul 29 19:56:55.152065 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:55.152026    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-59d5db-84d95c66fb-m48mz"]
Jul 29 19:56:55.152268 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:55.152241    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-59d5db-84d95c66fb-m48mz" podUID="214760ad-73c8-4e9e-b58c-7a2c6a72c272" containerName="payload-pre-processing" containerID="cri-o://25bfe39f9e70c2a1d5f3a78332c89628a375de30dac880ee8cd7ee451a6e7e9e" gracePeriod=30
Jul 29 19:56:55.394339 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:55.394310    2583 generic.go:356] "Generic (PLEG): container finished" podID="87cb3932-4c61-4596-ac40-c5ab6eb79329" containerID="d7be693b137691a2fe29fc6d48c4075f41c9d9e3531c6092141bc3c4fa03947b" exitCode=0
Jul 29 19:56:55.394467 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:55.394379    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-59d5db-65b8f8c948-qbx6l" event={"ID":"87cb3932-4c61-4596-ac40-c5ab6eb79329","Type":"ContainerDied","Data":"d7be693b137691a2fe29fc6d48c4075f41c9d9e3531c6092141bc3c4fa03947b"}
Jul 29 19:56:55.394467 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:55.394434    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-59d5db-65b8f8c948-qbx6l" event={"ID":"87cb3932-4c61-4596-ac40-c5ab6eb79329","Type":"ContainerDied","Data":"1a855098fe5a10ec25cfe072f6a2dc77794ee078874a51973bba0cb51c2b13b5"}
Jul 29 19:56:55.394467 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:55.394452    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a855098fe5a10ec25cfe072f6a2dc77794ee078874a51973bba0cb51c2b13b5"
Jul 29 19:56:55.396408 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:55.396382    2583 generic.go:356] "Generic (PLEG): container finished" podID="eec1d824-d889-4bf6-b600-00181be18634" containerID="703d231f6cf828daa88a62a22ba4d18b3a66aa8d38cb871b38a112ffe6fe9960" exitCode=0
Jul 29 19:56:55.396503 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:55.396447    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-59d5db-596bfd5995-hxwbb" event={"ID":"eec1d824-d889-4bf6-b600-00181be18634","Type":"ContainerDied","Data":"703d231f6cf828daa88a62a22ba4d18b3a66aa8d38cb871b38a112ffe6fe9960"}
Jul 29 19:56:55.398242 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:55.398221    2583 generic.go:356] "Generic (PLEG): container finished" podID="214760ad-73c8-4e9e-b58c-7a2c6a72c272" containerID="25bfe39f9e70c2a1d5f3a78332c89628a375de30dac880ee8cd7ee451a6e7e9e" exitCode=0
Jul 29 19:56:55.398349 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:55.398311    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-59d5db-84d95c66fb-m48mz" event={"ID":"214760ad-73c8-4e9e-b58c-7a2c6a72c272","Type":"ContainerDied","Data":"25bfe39f9e70c2a1d5f3a78332c89628a375de30dac880ee8cd7ee451a6e7e9e"}
Jul 29 19:56:55.409532 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:55.409513    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-59d5db-65b8f8c948-qbx6l"
Jul 29 19:56:55.447758 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:55.447735    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-59d5db-84d95c66fb-m48mz"
Jul 29 19:56:55.450761 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:55.450735    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-59d5db-596bfd5995-hxwbb"
Jul 29 19:56:55.467033 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:55.467006    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/214760ad-73c8-4e9e-b58c-7a2c6a72c272-plugins-config-volume\") pod \"214760ad-73c8-4e9e-b58c-7a2c6a72c272\" (UID: \"214760ad-73c8-4e9e-b58c-7a2c6a72c272\") "
Jul 29 19:56:55.467146 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:55.467065    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2pmgh\" (UniqueName: \"kubernetes.io/projected/214760ad-73c8-4e9e-b58c-7a2c6a72c272-kube-api-access-2pmgh\") pod \"214760ad-73c8-4e9e-b58c-7a2c6a72c272\" (UID: \"214760ad-73c8-4e9e-b58c-7a2c6a72c272\") "
Jul 29 19:56:55.467146 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:55.467132    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ch7qx\" (UniqueName: \"kubernetes.io/projected/87cb3932-4c61-4596-ac40-c5ab6eb79329-kube-api-access-ch7qx\") pod \"87cb3932-4c61-4596-ac40-c5ab6eb79329\" (UID: \"87cb3932-4c61-4596-ac40-c5ab6eb79329\") "
Jul 29 19:56:55.467253 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:55.467168    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87cb3932-4c61-4596-ac40-c5ab6eb79329-plugins-config-volume\") pod \"87cb3932-4c61-4596-ac40-c5ab6eb79329\" (UID: \"87cb3932-4c61-4596-ac40-c5ab6eb79329\") "
Jul 29 19:56:55.467438 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:55.467411    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/214760ad-73c8-4e9e-b58c-7a2c6a72c272-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "214760ad-73c8-4e9e-b58c-7a2c6a72c272" (UID: "214760ad-73c8-4e9e-b58c-7a2c6a72c272"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:56:55.467665 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:55.467638    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87cb3932-4c61-4596-ac40-c5ab6eb79329-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "87cb3932-4c61-4596-ac40-c5ab6eb79329" (UID: "87cb3932-4c61-4596-ac40-c5ab6eb79329"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:56:55.469271 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:55.469250    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/214760ad-73c8-4e9e-b58c-7a2c6a72c272-kube-api-access-2pmgh" (OuterVolumeSpecName: "kube-api-access-2pmgh") pod "214760ad-73c8-4e9e-b58c-7a2c6a72c272" (UID: "214760ad-73c8-4e9e-b58c-7a2c6a72c272"). InnerVolumeSpecName "kube-api-access-2pmgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:56:55.469602 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:55.469588    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87cb3932-4c61-4596-ac40-c5ab6eb79329-kube-api-access-ch7qx" (OuterVolumeSpecName: "kube-api-access-ch7qx") pod "87cb3932-4c61-4596-ac40-c5ab6eb79329" (UID: "87cb3932-4c61-4596-ac40-c5ab6eb79329"). InnerVolumeSpecName "kube-api-access-ch7qx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:56:55.568496 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:55.568468    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6b8f\" (UniqueName: \"kubernetes.io/projected/eec1d824-d889-4bf6-b600-00181be18634-kube-api-access-j6b8f\") pod \"eec1d824-d889-4bf6-b600-00181be18634\" (UID: \"eec1d824-d889-4bf6-b600-00181be18634\") "
Jul 29 19:56:55.568655 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:55.568509    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eec1d824-d889-4bf6-b600-00181be18634-openshift-service-ca\") pod \"eec1d824-d889-4bf6-b600-00181be18634\" (UID: \"eec1d824-d889-4bf6-b600-00181be18634\") "
Jul 29 19:56:55.568655 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:55.568543    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eec1d824-d889-4bf6-b600-00181be18634-maas-api-tls\") pod \"eec1d824-d889-4bf6-b600-00181be18634\" (UID: \"eec1d824-d889-4bf6-b600-00181be18634\") "
Jul 29 19:56:55.568655 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:55.568652    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pmgh\" (UniqueName: \"kubernetes.io/projected/214760ad-73c8-4e9e-b58c-7a2c6a72c272-kube-api-access-2pmgh\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:56:55.568758 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:55.568663    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch7qx\" (UniqueName: \"kubernetes.io/projected/87cb3932-4c61-4596-ac40-c5ab6eb79329-kube-api-access-ch7qx\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:56:55.568758 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:55.568672    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87cb3932-4c61-4596-ac40-c5ab6eb79329-plugins-config-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:56:55.568758 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:55.568681    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/214760ad-73c8-4e9e-b58c-7a2c6a72c272-plugins-config-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:56:55.568895 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:55.568869    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eec1d824-d889-4bf6-b600-00181be18634-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "eec1d824-d889-4bf6-b600-00181be18634" (UID: "eec1d824-d889-4bf6-b600-00181be18634"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:56:55.570642 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:55.570611    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eec1d824-d889-4bf6-b600-00181be18634-kube-api-access-j6b8f" (OuterVolumeSpecName: "kube-api-access-j6b8f") pod "eec1d824-d889-4bf6-b600-00181be18634" (UID: "eec1d824-d889-4bf6-b600-00181be18634"). InnerVolumeSpecName "kube-api-access-j6b8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:56:55.570731 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:55.570629    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eec1d824-d889-4bf6-b600-00181be18634-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "eec1d824-d889-4bf6-b600-00181be18634" (UID: "eec1d824-d889-4bf6-b600-00181be18634"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:56:55.669637 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:55.669605    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6b8f\" (UniqueName: \"kubernetes.io/projected/eec1d824-d889-4bf6-b600-00181be18634-kube-api-access-j6b8f\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:56:55.669637 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:55.669634    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eec1d824-d889-4bf6-b600-00181be18634-openshift-service-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:56:55.669850 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:55.669645    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eec1d824-d889-4bf6-b600-00181be18634-maas-api-tls\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:56:56.404293 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:56.404269    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-59d5db-84d95c66fb-m48mz"
Jul 29 19:56:56.404740 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:56.404316    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-59d5db-84d95c66fb-m48mz" event={"ID":"214760ad-73c8-4e9e-b58c-7a2c6a72c272","Type":"ContainerDied","Data":"9abf97ab368d8cd637888ce305b17faa86fb24662b1e322b70c85f5f82aed4a0"}
Jul 29 19:56:56.404740 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:56.404389    2583 scope.go:117] "RemoveContainer" containerID="25bfe39f9e70c2a1d5f3a78332c89628a375de30dac880ee8cd7ee451a6e7e9e"
Jul 29 19:56:56.405633 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:56.405610    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-59d5db-596bfd5995-hxwbb"
Jul 29 19:56:56.405713 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:56.405634    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-59d5db-65b8f8c948-qbx6l"
Jul 29 19:56:56.405713 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:56.405652    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-59d5db-596bfd5995-hxwbb" event={"ID":"eec1d824-d889-4bf6-b600-00181be18634","Type":"ContainerDied","Data":"cf0072864415800bafc35b9c0a138fabb056b6360cdef58c71e6832d95266df2"}
Jul 29 19:56:56.414012 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:56.413995    2583 scope.go:117] "RemoveContainer" containerID="703d231f6cf828daa88a62a22ba4d18b3a66aa8d38cb871b38a112ffe6fe9960"
Jul 29 19:56:56.432281 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:56.432256    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-59d5db-84d95c66fb-m48mz"]
Jul 29 19:56:56.435604 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:56.435585    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-59d5db-84d95c66fb-m48mz"]
Jul 29 19:56:56.447309 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:56.447288    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-59d5db-65b8f8c948-qbx6l"]
Jul 29 19:56:56.450471 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:56.450453    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-59d5db-65b8f8c948-qbx6l"]
Jul 29 19:56:56.461046 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:56.461022    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-59d5db-596bfd5995-hxwbb"]
Jul 29 19:56:56.466598 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:56.466572    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-59d5db-596bfd5995-hxwbb"]
Jul 29 19:56:57.340487 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:57.340456    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="214760ad-73c8-4e9e-b58c-7a2c6a72c272" path="/var/lib/kubelet/pods/214760ad-73c8-4e9e-b58c-7a2c6a72c272/volumes"
Jul 29 19:56:57.340839 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:57.340827    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87cb3932-4c61-4596-ac40-c5ab6eb79329" path="/var/lib/kubelet/pods/87cb3932-4c61-4596-ac40-c5ab6eb79329/volumes"
Jul 29 19:56:57.341164 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:57.341152    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eec1d824-d889-4bf6-b600-00181be18634" path="/var/lib/kubelet/pods/eec1d824-d889-4bf6-b600-00181be18634/volumes"
Jul 29 19:56:58.629245 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:58.629207    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-59d5db/test-model-59d5db-kserve-d8f7b699f-zl48l" podUID="9e72adc2-52c3-4887-b97a-556ea23d030f" containerName="main" containerID="cri-o://c80e3102ca1059d14a701255eb28af84de115e37b986ae000708ef58603d991a" gracePeriod=45
Jul 29 19:56:58.881073 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:58.881002    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-59d5db/test-model-59d5db-kserve-d8f7b699f-zl48l"
Jul 29 19:56:58.996203 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:58.996167    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wh6dj\" (UniqueName: \"kubernetes.io/projected/9e72adc2-52c3-4887-b97a-556ea23d030f-kube-api-access-wh6dj\") pod \"9e72adc2-52c3-4887-b97a-556ea23d030f\" (UID: \"9e72adc2-52c3-4887-b97a-556ea23d030f\") "
Jul 29 19:56:58.996386 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:58.996213    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e72adc2-52c3-4887-b97a-556ea23d030f-home\") pod \"9e72adc2-52c3-4887-b97a-556ea23d030f\" (UID: \"9e72adc2-52c3-4887-b97a-556ea23d030f\") "
Jul 29 19:56:58.996386 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:58.996234    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e72adc2-52c3-4887-b97a-556ea23d030f-tls-certs\") pod \"9e72adc2-52c3-4887-b97a-556ea23d030f\" (UID: \"9e72adc2-52c3-4887-b97a-556ea23d030f\") "
Jul 29 19:56:58.996386 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:58.996269    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e72adc2-52c3-4887-b97a-556ea23d030f-tmp-dir\") pod \"9e72adc2-52c3-4887-b97a-556ea23d030f\" (UID: \"9e72adc2-52c3-4887-b97a-556ea23d030f\") "
Jul 29 19:56:58.996386 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:58.996285    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e72adc2-52c3-4887-b97a-556ea23d030f-dshm\") pod \"9e72adc2-52c3-4887-b97a-556ea23d030f\" (UID: \"9e72adc2-52c3-4887-b97a-556ea23d030f\") "
Jul 29 19:56:58.996386 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:58.996333    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e72adc2-52c3-4887-b97a-556ea23d030f-model-cache\") pod \"9e72adc2-52c3-4887-b97a-556ea23d030f\" (UID: \"9e72adc2-52c3-4887-b97a-556ea23d030f\") "
Jul 29 19:56:58.996658 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:58.996522    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e72adc2-52c3-4887-b97a-556ea23d030f-home" (OuterVolumeSpecName: "home") pod "9e72adc2-52c3-4887-b97a-556ea23d030f" (UID: "9e72adc2-52c3-4887-b97a-556ea23d030f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:56:58.996732 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:58.996702    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e72adc2-52c3-4887-b97a-556ea23d030f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9e72adc2-52c3-4887-b97a-556ea23d030f" (UID: "9e72adc2-52c3-4887-b97a-556ea23d030f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:56:58.996906 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:58.996884    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e72adc2-52c3-4887-b97a-556ea23d030f-model-cache" (OuterVolumeSpecName: "model-cache") pod "9e72adc2-52c3-4887-b97a-556ea23d030f" (UID: "9e72adc2-52c3-4887-b97a-556ea23d030f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:56:58.998570 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:58.998545    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e72adc2-52c3-4887-b97a-556ea23d030f-dshm" (OuterVolumeSpecName: "dshm") pod "9e72adc2-52c3-4887-b97a-556ea23d030f" (UID: "9e72adc2-52c3-4887-b97a-556ea23d030f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:56:58.998570 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:58.998557    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e72adc2-52c3-4887-b97a-556ea23d030f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9e72adc2-52c3-4887-b97a-556ea23d030f" (UID: "9e72adc2-52c3-4887-b97a-556ea23d030f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:56:58.998737 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:58.998581    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e72adc2-52c3-4887-b97a-556ea23d030f-kube-api-access-wh6dj" (OuterVolumeSpecName: "kube-api-access-wh6dj") pod "9e72adc2-52c3-4887-b97a-556ea23d030f" (UID: "9e72adc2-52c3-4887-b97a-556ea23d030f"). InnerVolumeSpecName "kube-api-access-wh6dj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:56:59.097451 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:59.097412    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e72adc2-52c3-4887-b97a-556ea23d030f-model-cache\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:56:59.097451 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:59.097445    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh6dj\" (UniqueName: \"kubernetes.io/projected/9e72adc2-52c3-4887-b97a-556ea23d030f-kube-api-access-wh6dj\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:56:59.097451 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:59.097455    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e72adc2-52c3-4887-b97a-556ea23d030f-home\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:56:59.097683 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:59.097464    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e72adc2-52c3-4887-b97a-556ea23d030f-tls-certs\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:56:59.097683 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:59.097473    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e72adc2-52c3-4887-b97a-556ea23d030f-tmp-dir\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:56:59.097683 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:59.097480    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e72adc2-52c3-4887-b97a-556ea23d030f-dshm\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:56:59.427706 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:59.427670    2583 generic.go:356] "Generic (PLEG): container finished" podID="9e72adc2-52c3-4887-b97a-556ea23d030f" containerID="c80e3102ca1059d14a701255eb28af84de115e37b986ae000708ef58603d991a" exitCode=0
Jul 29 19:56:59.427898 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:59.427711    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-59d5db/test-model-59d5db-kserve-d8f7b699f-zl48l" event={"ID":"9e72adc2-52c3-4887-b97a-556ea23d030f","Type":"ContainerDied","Data":"c80e3102ca1059d14a701255eb28af84de115e37b986ae000708ef58603d991a"}
Jul 29 19:56:59.427898 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:59.427721    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-59d5db/test-model-59d5db-kserve-d8f7b699f-zl48l"
Jul 29 19:56:59.427898 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:59.427745    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-59d5db/test-model-59d5db-kserve-d8f7b699f-zl48l" event={"ID":"9e72adc2-52c3-4887-b97a-556ea23d030f","Type":"ContainerDied","Data":"d5836e024c71a527f6f69f28cb18e99315459937deafc2797ddba62c79619c1f"}
Jul 29 19:56:59.427898 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:59.427761    2583 scope.go:117] "RemoveContainer" containerID="c80e3102ca1059d14a701255eb28af84de115e37b986ae000708ef58603d991a"
Jul 29 19:56:59.436150 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:59.436130    2583 scope.go:117] "RemoveContainer" containerID="c80e3102ca1059d14a701255eb28af84de115e37b986ae000708ef58603d991a"
Jul 29 19:56:59.436421 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:56:59.436399    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c80e3102ca1059d14a701255eb28af84de115e37b986ae000708ef58603d991a\": container with ID starting with c80e3102ca1059d14a701255eb28af84de115e37b986ae000708ef58603d991a not found: ID does not exist" containerID="c80e3102ca1059d14a701255eb28af84de115e37b986ae000708ef58603d991a"
Jul 29 19:56:59.436517 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:59.436426    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c80e3102ca1059d14a701255eb28af84de115e37b986ae000708ef58603d991a"} err="failed to get container status \"c80e3102ca1059d14a701255eb28af84de115e37b986ae000708ef58603d991a\": rpc error: code = NotFound desc = could not find container \"c80e3102ca1059d14a701255eb28af84de115e37b986ae000708ef58603d991a\": container with ID starting with c80e3102ca1059d14a701255eb28af84de115e37b986ae000708ef58603d991a not found: ID does not exist"
Jul 29 19:56:59.444219 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:59.444195    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-59d5db/test-model-59d5db-kserve-d8f7b699f-zl48l"]
Jul 29 19:56:59.449290 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:56:59.449267    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-59d5db/test-model-59d5db-kserve-d8f7b699f-zl48l"]
Jul 29 19:57:01.339778 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:01.339737    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e72adc2-52c3-4887-b97a-556ea23d030f" path="/var/lib/kubelet/pods/9e72adc2-52c3-4887-b97a-556ea23d030f/volumes"
Jul 29 19:57:02.476703 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:02.476665    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-accf7d-hl86b"]
Jul 29 19:57:02.477124 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:02.477030    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87cb3932-4c61-4596-ac40-c5ab6eb79329" containerName="payload-processing"
Jul 29 19:57:02.477124 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:02.477042    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="87cb3932-4c61-4596-ac40-c5ab6eb79329" containerName="payload-processing"
Jul 29 19:57:02.477124 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:02.477051    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eec1d824-d889-4bf6-b600-00181be18634" containerName="maas-api"
Jul 29 19:57:02.477124 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:02.477056    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="eec1d824-d889-4bf6-b600-00181be18634" containerName="maas-api"
Jul 29 19:57:02.477124 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:02.477068    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e72adc2-52c3-4887-b97a-556ea23d030f" containerName="main"
Jul 29 19:57:02.477124 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:02.477073    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e72adc2-52c3-4887-b97a-556ea23d030f" containerName="main"
Jul 29 19:57:02.477124 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:02.477089    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="214760ad-73c8-4e9e-b58c-7a2c6a72c272" containerName="payload-pre-processing"
Jul 29 19:57:02.477124 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:02.477095    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="214760ad-73c8-4e9e-b58c-7a2c6a72c272" containerName="payload-pre-processing"
Jul 29 19:57:02.477553 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:02.477140    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="87cb3932-4c61-4596-ac40-c5ab6eb79329" containerName="payload-processing"
Jul 29 19:57:02.477553 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:02.477148    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="214760ad-73c8-4e9e-b58c-7a2c6a72c272" containerName="payload-pre-processing"
Jul 29 19:57:02.477553 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:02.477153    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e72adc2-52c3-4887-b97a-556ea23d030f" containerName="main"
Jul 29 19:57:02.477553 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:02.477161    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="eec1d824-d889-4bf6-b600-00181be18634" containerName="maas-api"
Jul 29 19:57:02.481314 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:02.481296    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-accf7d-hl86b"
Jul 29 19:57:02.483683 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:02.483668    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hq9l2\""
Jul 29 19:57:02.488845 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:02.488783    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-accf7d-hl86b"]
Jul 29 19:57:02.524924 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:02.524885    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9105397d-bbc4-463b-bc39-0a17f3c6a2fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-accf7d-hl86b\" (UID: \"9105397d-bbc4-463b-bc39-0a17f3c6a2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-accf7d-hl86b"
Jul 29 19:57:02.626260 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:02.626230    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9105397d-bbc4-463b-bc39-0a17f3c6a2fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-accf7d-hl86b\" (UID: \"9105397d-bbc4-463b-bc39-0a17f3c6a2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-accf7d-hl86b"
Jul 29 19:57:02.626926 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:02.626905    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9105397d-bbc4-463b-bc39-0a17f3c6a2fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-accf7d-hl86b\" (UID: \"9105397d-bbc4-463b-bc39-0a17f3c6a2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-accf7d-hl86b"
Jul 29 19:57:02.793577 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:02.793480    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-accf7d-hl86b"
Jul 29 19:57:02.928742 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:02.928718    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-accf7d-hl86b"]
Jul 29 19:57:02.931518 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:57:02.931490    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9105397d_bbc4_463b_bc39_0a17f3c6a2fd.slice/crio-dc2fb5ec32454be876318a374c2562ad777167db99458624ba0d383f1b1bfeee WatchSource:0}: Error finding container dc2fb5ec32454be876318a374c2562ad777167db99458624ba0d383f1b1bfeee: Status 404 returned error can't find the container with id dc2fb5ec32454be876318a374c2562ad777167db99458624ba0d383f1b1bfeee
Jul 29 19:57:03.449228 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:03.449191    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-accf7d-hl86b" event={"ID":"9105397d-bbc4-463b-bc39-0a17f3c6a2fd","Type":"ContainerStarted","Data":"762f7c74e9144ea8c4bdd7e18eb014bd64f8e1e31a69ee567fe2902f96de4965"}
Jul 29 19:57:03.449420 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:03.449252    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-accf7d-hl86b" event={"ID":"9105397d-bbc4-463b-bc39-0a17f3c6a2fd","Type":"ContainerStarted","Data":"dc2fb5ec32454be876318a374c2562ad777167db99458624ba0d383f1b1bfeee"}
Jul 29 19:57:03.465531 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:03.465481    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-accf7d-hl86b" podStartSLOduration=1.465465759 podStartE2EDuration="1.465465759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:57:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:57:03.464537296 +0000 UTC m=+2390.692926423" watchObservedRunningTime="2026-07-29 19:57:03.465465759 +0000 UTC m=+2390.693854946"
Jul 29 19:57:04.455083 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:04.455050    2583 generic.go:356] "Generic (PLEG): container finished" podID="9105397d-bbc4-463b-bc39-0a17f3c6a2fd" containerID="762f7c74e9144ea8c4bdd7e18eb014bd64f8e1e31a69ee567fe2902f96de4965" exitCode=0
Jul 29 19:57:04.455467 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:04.455102    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-accf7d-hl86b" event={"ID":"9105397d-bbc4-463b-bc39-0a17f3c6a2fd","Type":"ContainerDied","Data":"762f7c74e9144ea8c4bdd7e18eb014bd64f8e1e31a69ee567fe2902f96de4965"}
Jul 29 19:57:05.098343 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:05.098310    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:57:05.585408 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:05.585385    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-accf7d-hl86b"
Jul 29 19:57:05.652918 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:05.652883    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9105397d-bbc4-463b-bc39-0a17f3c6a2fd-maas-api-service-ca\") pod \"9105397d-bbc4-463b-bc39-0a17f3c6a2fd\" (UID: \"9105397d-bbc4-463b-bc39-0a17f3c6a2fd\") "
Jul 29 19:57:05.653237 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:05.653210    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9105397d-bbc4-463b-bc39-0a17f3c6a2fd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9105397d-bbc4-463b-bc39-0a17f3c6a2fd" (UID: "9105397d-bbc4-463b-bc39-0a17f3c6a2fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:57:05.754310 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:05.754231    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9105397d-bbc4-463b-bc39-0a17f3c6a2fd-maas-api-service-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:57:06.465619 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:06.465591    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-accf7d-hl86b"
Jul 29 19:57:06.465789 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:06.465588    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-accf7d-hl86b" event={"ID":"9105397d-bbc4-463b-bc39-0a17f3c6a2fd","Type":"ContainerDied","Data":"dc2fb5ec32454be876318a374c2562ad777167db99458624ba0d383f1b1bfeee"}
Jul 29 19:57:06.465789 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:06.465723    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc2fb5ec32454be876318a374c2562ad777167db99458624ba0d383f1b1bfeee"
Jul 29 19:57:12.995562 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:12.995530    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-accf7d-86d774c5f7-w5zpd"]
Jul 29 19:57:12.995983 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:12.995738    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-accf7d-86d774c5f7-w5zpd" podUID="4f79a0c1-544a-45d3-be04-b3c02746391f" containerName="maas-api" containerID="cri-o://77f545bd07ed04f714b4fc1b0319dddde1d04644a81e5ec4a494de82a866caf4" gracePeriod=30
Jul 29 19:57:13.053335 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:13.053303    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-accf7d-755887b768-nzs78"]
Jul 29 19:57:13.053567 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:13.053537    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-accf7d-755887b768-nzs78" podUID="b9b7899d-03b1-4fda-b0ff-900ed7be3b25" containerName="payload-pre-processing" containerID="cri-o://982dad23ac4a904431d7fdb96dafeaf0689cdbbdef5c8e9dcc973b36ec2b3bfa" gracePeriod=30
Jul 29 19:57:13.174634 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:13.174602    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-accf7d-hl86b"]
Jul 29 19:57:13.175909 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:13.175882    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-accf7d-hl86b"]
Jul 29 19:57:13.341428 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:13.341389    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9105397d-bbc4-463b-bc39-0a17f3c6a2fd" path="/var/lib/kubelet/pods/9105397d-bbc4-463b-bc39-0a17f3c6a2fd/volumes"
Jul 29 19:57:13.349844 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:13.349826    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-accf7d-755887b768-nzs78"
Jul 29 19:57:13.424072 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:13.424042    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9b7899d-03b1-4fda-b0ff-900ed7be3b25-plugins-config-volume\") pod \"b9b7899d-03b1-4fda-b0ff-900ed7be3b25\" (UID: \"b9b7899d-03b1-4fda-b0ff-900ed7be3b25\") "
Jul 29 19:57:13.424199 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:13.424091    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5t65h\" (UniqueName: \"kubernetes.io/projected/b9b7899d-03b1-4fda-b0ff-900ed7be3b25-kube-api-access-5t65h\") pod \"b9b7899d-03b1-4fda-b0ff-900ed7be3b25\" (UID: \"b9b7899d-03b1-4fda-b0ff-900ed7be3b25\") "
Jul 29 19:57:13.424487 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:13.424455    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9b7899d-03b1-4fda-b0ff-900ed7be3b25-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b9b7899d-03b1-4fda-b0ff-900ed7be3b25" (UID: "b9b7899d-03b1-4fda-b0ff-900ed7be3b25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:57:13.426927 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:13.426903    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9b7899d-03b1-4fda-b0ff-900ed7be3b25-kube-api-access-5t65h" (OuterVolumeSpecName: "kube-api-access-5t65h") pod "b9b7899d-03b1-4fda-b0ff-900ed7be3b25" (UID: "b9b7899d-03b1-4fda-b0ff-900ed7be3b25"). InnerVolumeSpecName "kube-api-access-5t65h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:57:13.456517 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:13.456498    2583 scope.go:117] "RemoveContainer" containerID="222da46c390bfbcc282b8ebf53d53cde8ff340bb6f50723c02501a771e9f294f"
Jul 29 19:57:13.459770 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:13.459749    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-accf7d-86d774c5f7-w5zpd"
Jul 29 19:57:13.499622 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:13.499591    2583 generic.go:356] "Generic (PLEG): container finished" podID="b9b7899d-03b1-4fda-b0ff-900ed7be3b25" containerID="982dad23ac4a904431d7fdb96dafeaf0689cdbbdef5c8e9dcc973b36ec2b3bfa" exitCode=0
Jul 29 19:57:13.499735 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:13.499634    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-accf7d-755887b768-nzs78"
Jul 29 19:57:13.499735 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:13.499671    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-accf7d-755887b768-nzs78" event={"ID":"b9b7899d-03b1-4fda-b0ff-900ed7be3b25","Type":"ContainerDied","Data":"982dad23ac4a904431d7fdb96dafeaf0689cdbbdef5c8e9dcc973b36ec2b3bfa"}
Jul 29 19:57:13.499735 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:13.499724    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-accf7d-755887b768-nzs78" event={"ID":"b9b7899d-03b1-4fda-b0ff-900ed7be3b25","Type":"ContainerDied","Data":"e2e09f31f56ffda98ce7f70a1100861b9711852e0c4b01429ed4a47ecabff94b"}
Jul 29 19:57:13.499899 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:13.499752    2583 scope.go:117] "RemoveContainer" containerID="982dad23ac4a904431d7fdb96dafeaf0689cdbbdef5c8e9dcc973b36ec2b3bfa"
Jul 29 19:57:13.501145 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:13.501026    2583 generic.go:356] "Generic (PLEG): container finished" podID="4f79a0c1-544a-45d3-be04-b3c02746391f" containerID="77f545bd07ed04f714b4fc1b0319dddde1d04644a81e5ec4a494de82a866caf4" exitCode=0
Jul 29 19:57:13.501145 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:13.501066    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-accf7d-86d774c5f7-w5zpd" event={"ID":"4f79a0c1-544a-45d3-be04-b3c02746391f","Type":"ContainerDied","Data":"77f545bd07ed04f714b4fc1b0319dddde1d04644a81e5ec4a494de82a866caf4"}
Jul 29 19:57:13.501145 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:13.501089    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-accf7d-86d774c5f7-w5zpd"
Jul 29 19:57:13.501145 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:13.501113    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-accf7d-86d774c5f7-w5zpd" event={"ID":"4f79a0c1-544a-45d3-be04-b3c02746391f","Type":"ContainerDied","Data":"76094bdcfbfb663d7b1f961d930bfa98505b15f1a8ff575c8f28de88afad9a24"}
Jul 29 19:57:13.508090 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:13.508074    2583 scope.go:117] "RemoveContainer" containerID="982dad23ac4a904431d7fdb96dafeaf0689cdbbdef5c8e9dcc973b36ec2b3bfa"
Jul 29 19:57:13.508357 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:57:13.508337    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"982dad23ac4a904431d7fdb96dafeaf0689cdbbdef5c8e9dcc973b36ec2b3bfa\": container with ID starting with 982dad23ac4a904431d7fdb96dafeaf0689cdbbdef5c8e9dcc973b36ec2b3bfa not found: ID does not exist" containerID="982dad23ac4a904431d7fdb96dafeaf0689cdbbdef5c8e9dcc973b36ec2b3bfa"
Jul 29 19:57:13.508417 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:13.508365    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"982dad23ac4a904431d7fdb96dafeaf0689cdbbdef5c8e9dcc973b36ec2b3bfa"} err="failed to get container status \"982dad23ac4a904431d7fdb96dafeaf0689cdbbdef5c8e9dcc973b36ec2b3bfa\": rpc error: code = NotFound desc = could not find container \"982dad23ac4a904431d7fdb96dafeaf0689cdbbdef5c8e9dcc973b36ec2b3bfa\": container with ID starting with 982dad23ac4a904431d7fdb96dafeaf0689cdbbdef5c8e9dcc973b36ec2b3bfa not found: ID does not exist"
Jul 29 19:57:13.508417 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:13.508382    2583 scope.go:117] "RemoveContainer" containerID="77f545bd07ed04f714b4fc1b0319dddde1d04644a81e5ec4a494de82a866caf4"
Jul 29 19:57:13.515670 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:13.515653    2583 scope.go:117] "RemoveContainer" containerID="77f545bd07ed04f714b4fc1b0319dddde1d04644a81e5ec4a494de82a866caf4"
Jul 29 19:57:13.515927 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:57:13.515910    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77f545bd07ed04f714b4fc1b0319dddde1d04644a81e5ec4a494de82a866caf4\": container with ID starting with 77f545bd07ed04f714b4fc1b0319dddde1d04644a81e5ec4a494de82a866caf4 not found: ID does not exist" containerID="77f545bd07ed04f714b4fc1b0319dddde1d04644a81e5ec4a494de82a866caf4"
Jul 29 19:57:13.515990 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:13.515934    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77f545bd07ed04f714b4fc1b0319dddde1d04644a81e5ec4a494de82a866caf4"} err="failed to get container status \"77f545bd07ed04f714b4fc1b0319dddde1d04644a81e5ec4a494de82a866caf4\": rpc error: code = NotFound desc = could not find container \"77f545bd07ed04f714b4fc1b0319dddde1d04644a81e5ec4a494de82a866caf4\": container with ID starting with 77f545bd07ed04f714b4fc1b0319dddde1d04644a81e5ec4a494de82a866caf4 not found: ID does not exist"
Jul 29 19:57:13.521710 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:13.521688    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-accf7d-755887b768-nzs78"]
Jul 29 19:57:13.524954 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:13.524936    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f79a0c1-544a-45d3-be04-b3c02746391f-maas-api-tls\") pod \"4f79a0c1-544a-45d3-be04-b3c02746391f\" (UID: \"4f79a0c1-544a-45d3-be04-b3c02746391f\") "
Jul 29 19:57:13.525036 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:13.525012    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f79a0c1-544a-45d3-be04-b3c02746391f-openshift-service-ca\") pod \"4f79a0c1-544a-45d3-be04-b3c02746391f\" (UID: \"4f79a0c1-544a-45d3-be04-b3c02746391f\") "
Jul 29 19:57:13.525090 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:13.525052    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d86nj\" (UniqueName: \"kubernetes.io/projected/4f79a0c1-544a-45d3-be04-b3c02746391f-kube-api-access-d86nj\") pod \"4f79a0c1-544a-45d3-be04-b3c02746391f\" (UID: \"4f79a0c1-544a-45d3-be04-b3c02746391f\") "
Jul 29 19:57:13.525201 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:13.525187    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9b7899d-03b1-4fda-b0ff-900ed7be3b25-plugins-config-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:57:13.525264 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:13.525204    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t65h\" (UniqueName: \"kubernetes.io/projected/b9b7899d-03b1-4fda-b0ff-900ed7be3b25-kube-api-access-5t65h\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:57:13.525395 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:13.525371    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f79a0c1-544a-45d3-be04-b3c02746391f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4f79a0c1-544a-45d3-be04-b3c02746391f" (UID: "4f79a0c1-544a-45d3-be04-b3c02746391f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:57:13.525593 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:13.525578    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-accf7d-755887b768-nzs78"]
Jul 29 19:57:13.526822 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:13.526778    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f79a0c1-544a-45d3-be04-b3c02746391f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4f79a0c1-544a-45d3-be04-b3c02746391f" (UID: "4f79a0c1-544a-45d3-be04-b3c02746391f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:57:13.527087 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:13.527049    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f79a0c1-544a-45d3-be04-b3c02746391f-kube-api-access-d86nj" (OuterVolumeSpecName: "kube-api-access-d86nj") pod "4f79a0c1-544a-45d3-be04-b3c02746391f" (UID: "4f79a0c1-544a-45d3-be04-b3c02746391f"). InnerVolumeSpecName "kube-api-access-d86nj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:57:13.626504 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:13.626473    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d86nj\" (UniqueName: \"kubernetes.io/projected/4f79a0c1-544a-45d3-be04-b3c02746391f-kube-api-access-d86nj\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:57:13.626504 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:13.626504    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f79a0c1-544a-45d3-be04-b3c02746391f-maas-api-tls\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:57:13.626668 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:13.626518    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f79a0c1-544a-45d3-be04-b3c02746391f-openshift-service-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:57:13.825634 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:13.825602    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-accf7d-86d774c5f7-w5zpd"]
Jul 29 19:57:13.830768 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:13.830744    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-accf7d-86d774c5f7-w5zpd"]
Jul 29 19:57:15.340238 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:15.340196    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f79a0c1-544a-45d3-be04-b3c02746391f" path="/var/lib/kubelet/pods/4f79a0c1-544a-45d3-be04-b3c02746391f/volumes"
Jul 29 19:57:15.340710 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:15.340670    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9b7899d-03b1-4fda-b0ff-900ed7be3b25" path="/var/lib/kubelet/pods/b9b7899d-03b1-4fda-b0ff-900ed7be3b25/volumes"
Jul 29 19:57:16.093908 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:16.093872    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:57:23.817210 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:23.817180    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dc9ec3-7b95659fb-5bkfj"]
Jul 29 19:57:23.817610 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:23.817499    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f79a0c1-544a-45d3-be04-b3c02746391f" containerName="maas-api"
Jul 29 19:57:23.817610 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:23.817511    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f79a0c1-544a-45d3-be04-b3c02746391f" containerName="maas-api"
Jul 29 19:57:23.817610 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:23.817526    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9b7899d-03b1-4fda-b0ff-900ed7be3b25" containerName="payload-pre-processing"
Jul 29 19:57:23.817610 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:23.817531    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9b7899d-03b1-4fda-b0ff-900ed7be3b25" containerName="payload-pre-processing"
Jul 29 19:57:23.817610 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:23.817543    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9105397d-bbc4-463b-bc39-0a17f3c6a2fd" containerName="revoke-keys"
Jul 29 19:57:23.817610 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:23.817548    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9105397d-bbc4-463b-bc39-0a17f3c6a2fd" containerName="revoke-keys"
Jul 29 19:57:23.817610 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:23.817595    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9105397d-bbc4-463b-bc39-0a17f3c6a2fd" containerName="revoke-keys"
Jul 29 19:57:23.817610 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:23.817604    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9b7899d-03b1-4fda-b0ff-900ed7be3b25" containerName="payload-pre-processing"
Jul 29 19:57:23.817610 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:23.817610    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f79a0c1-544a-45d3-be04-b3c02746391f" containerName="maas-api"
Jul 29 19:57:23.820249 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:23.820233    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc9ec3-7b95659fb-5bkfj"
Jul 29 19:57:23.822553 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:23.822527    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6wzbz\""
Jul 29 19:57:23.822553 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:23.822535    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 29 19:57:23.822719 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:23.822559    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-dc9ec3-serving-cert\""
Jul 29 19:57:23.823129 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:23.823110    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 19:57:23.830861 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:23.830842    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dc9ec3-7b95659fb-5bkfj"]
Jul 29 19:57:23.917053 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:23.917019    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a56b8fe6-c110-4e52-be74-957ca6cd3236-openshift-service-ca\") pod \"maas-api-e2e-api-a-dc9ec3-7b95659fb-5bkfj\" (UID: \"a56b8fe6-c110-4e52-be74-957ca6cd3236\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc9ec3-7b95659fb-5bkfj"
Jul 29 19:57:23.917247 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:23.917067    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtqsr\" (UniqueName: \"kubernetes.io/projected/a56b8fe6-c110-4e52-be74-957ca6cd3236-kube-api-access-vtqsr\") pod \"maas-api-e2e-api-a-dc9ec3-7b95659fb-5bkfj\" (UID: \"a56b8fe6-c110-4e52-be74-957ca6cd3236\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc9ec3-7b95659fb-5bkfj"
Jul 29 19:57:23.917247 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:23.917089    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a56b8fe6-c110-4e52-be74-957ca6cd3236-maas-api-tls\") pod \"maas-api-e2e-api-a-dc9ec3-7b95659fb-5bkfj\" (UID: \"a56b8fe6-c110-4e52-be74-957ca6cd3236\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc9ec3-7b95659fb-5bkfj"
Jul 29 19:57:24.018357 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:24.018317    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a56b8fe6-c110-4e52-be74-957ca6cd3236-openshift-service-ca\") pod \"maas-api-e2e-api-a-dc9ec3-7b95659fb-5bkfj\" (UID: \"a56b8fe6-c110-4e52-be74-957ca6cd3236\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc9ec3-7b95659fb-5bkfj"
Jul 29 19:57:24.018531 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:24.018379    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtqsr\" (UniqueName: \"kubernetes.io/projected/a56b8fe6-c110-4e52-be74-957ca6cd3236-kube-api-access-vtqsr\") pod \"maas-api-e2e-api-a-dc9ec3-7b95659fb-5bkfj\" (UID: \"a56b8fe6-c110-4e52-be74-957ca6cd3236\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc9ec3-7b95659fb-5bkfj"
Jul 29 19:57:24.018531 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:24.018408    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a56b8fe6-c110-4e52-be74-957ca6cd3236-maas-api-tls\") pod \"maas-api-e2e-api-a-dc9ec3-7b95659fb-5bkfj\" (UID: \"a56b8fe6-c110-4e52-be74-957ca6cd3236\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc9ec3-7b95659fb-5bkfj"
Jul 29 19:57:24.018677 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:57:24.018540    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-dc9ec3-serving-cert: secret "maas-api-e2e-api-a-dc9ec3-serving-cert" not found
Jul 29 19:57:24.018677 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:57:24.018610    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a56b8fe6-c110-4e52-be74-957ca6cd3236-maas-api-tls podName:a56b8fe6-c110-4e52-be74-957ca6cd3236 nodeName:}" failed. No retries permitted until 2026-07-29 19:57:24.518588109 +0000 UTC m=+2411.746977218 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a56b8fe6-c110-4e52-be74-957ca6cd3236-maas-api-tls") pod "maas-api-e2e-api-a-dc9ec3-7b95659fb-5bkfj" (UID: "a56b8fe6-c110-4e52-be74-957ca6cd3236") : secret "maas-api-e2e-api-a-dc9ec3-serving-cert" not found
Jul 29 19:57:24.018977 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:24.018956    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a56b8fe6-c110-4e52-be74-957ca6cd3236-openshift-service-ca\") pod \"maas-api-e2e-api-a-dc9ec3-7b95659fb-5bkfj\" (UID: \"a56b8fe6-c110-4e52-be74-957ca6cd3236\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc9ec3-7b95659fb-5bkfj"
Jul 29 19:57:24.026832 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:24.026792    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtqsr\" (UniqueName: \"kubernetes.io/projected/a56b8fe6-c110-4e52-be74-957ca6cd3236-kube-api-access-vtqsr\") pod \"maas-api-e2e-api-a-dc9ec3-7b95659fb-5bkfj\" (UID: \"a56b8fe6-c110-4e52-be74-957ca6cd3236\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc9ec3-7b95659fb-5bkfj"
Jul 29 19:57:24.512001 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:24.511967    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dc9ec3-9ddc7bbd-mvjlf"]
Jul 29 19:57:24.515141 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:24.515126    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc9ec3-9ddc7bbd-mvjlf"
Jul 29 19:57:24.516990 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:24.516971    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-dc9ec3\""
Jul 29 19:57:24.517106 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:24.517023    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-dc9ec3-dockercfg-2lwdj\""
Jul 29 19:57:24.521927 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:24.521751    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a56b8fe6-c110-4e52-be74-957ca6cd3236-maas-api-tls\") pod \"maas-api-e2e-api-a-dc9ec3-7b95659fb-5bkfj\" (UID: \"a56b8fe6-c110-4e52-be74-957ca6cd3236\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc9ec3-7b95659fb-5bkfj"
Jul 29 19:57:24.524383 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:24.524362    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a56b8fe6-c110-4e52-be74-957ca6cd3236-maas-api-tls\") pod \"maas-api-e2e-api-a-dc9ec3-7b95659fb-5bkfj\" (UID: \"a56b8fe6-c110-4e52-be74-957ca6cd3236\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc9ec3-7b95659fb-5bkfj"
Jul 29 19:57:24.525447 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:24.525429    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dc9ec3-9ddc7bbd-mvjlf"]
Jul 29 19:57:24.547783 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:24.547757    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dc9ec3-b47cd6f6b-k2l78"]
Jul 29 19:57:24.550860 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:24.550843    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dc9ec3-b47cd6f6b-k2l78"
Jul 29 19:57:24.561154 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:24.561133    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dc9ec3-b47cd6f6b-k2l78"]
Jul 29 19:57:24.622627 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:24.622594    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08e56bd4-aa8d-4909-b01f-f98b8e5ace5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-dc9ec3-9ddc7bbd-mvjlf\" (UID: \"08e56bd4-aa8d-4909-b01f-f98b8e5ace5e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc9ec3-9ddc7bbd-mvjlf"
Jul 29 19:57:24.622828 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:24.622756    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m85h8\" (UniqueName: \"kubernetes.io/projected/08e56bd4-aa8d-4909-b01f-f98b8e5ace5e-kube-api-access-m85h8\") pod \"payload-pre-processing-e2e-api-a-dc9ec3-9ddc7bbd-mvjlf\" (UID: \"08e56bd4-aa8d-4909-b01f-f98b8e5ace5e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc9ec3-9ddc7bbd-mvjlf"
Jul 29 19:57:24.724131 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:24.724080    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f775ac3c-4fb8-49e9-b4a7-a61ed8398e2a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-dc9ec3-b47cd6f6b-k2l78\" (UID: \"f775ac3c-4fb8-49e9-b4a7-a61ed8398e2a\") " pod="openshift-ingress/payload-processing-e2e-api-a-dc9ec3-b47cd6f6b-k2l78"
Jul 29 19:57:24.724131 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:24.724142    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m85h8\" (UniqueName: \"kubernetes.io/projected/08e56bd4-aa8d-4909-b01f-f98b8e5ace5e-kube-api-access-m85h8\") pod \"payload-pre-processing-e2e-api-a-dc9ec3-9ddc7bbd-mvjlf\" (UID: \"08e56bd4-aa8d-4909-b01f-f98b8e5ace5e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc9ec3-9ddc7bbd-mvjlf"
Jul 29 19:57:24.724406 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:24.724162    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08e56bd4-aa8d-4909-b01f-f98b8e5ace5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-dc9ec3-9ddc7bbd-mvjlf\" (UID: \"08e56bd4-aa8d-4909-b01f-f98b8e5ace5e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc9ec3-9ddc7bbd-mvjlf"
Jul 29 19:57:24.724406 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:24.724182    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw7q2\" (UniqueName: \"kubernetes.io/projected/f775ac3c-4fb8-49e9-b4a7-a61ed8398e2a-kube-api-access-vw7q2\") pod \"payload-processing-e2e-api-a-dc9ec3-b47cd6f6b-k2l78\" (UID: \"f775ac3c-4fb8-49e9-b4a7-a61ed8398e2a\") " pod="openshift-ingress/payload-processing-e2e-api-a-dc9ec3-b47cd6f6b-k2l78"
Jul 29 19:57:24.724847 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:24.724827    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08e56bd4-aa8d-4909-b01f-f98b8e5ace5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-dc9ec3-9ddc7bbd-mvjlf\" (UID: \"08e56bd4-aa8d-4909-b01f-f98b8e5ace5e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc9ec3-9ddc7bbd-mvjlf"
Jul 29 19:57:24.730904 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:24.730882    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc9ec3-7b95659fb-5bkfj"
Jul 29 19:57:24.731831 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:24.731813    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m85h8\" (UniqueName: \"kubernetes.io/projected/08e56bd4-aa8d-4909-b01f-f98b8e5ace5e-kube-api-access-m85h8\") pod \"payload-pre-processing-e2e-api-a-dc9ec3-9ddc7bbd-mvjlf\" (UID: \"08e56bd4-aa8d-4909-b01f-f98b8e5ace5e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc9ec3-9ddc7bbd-mvjlf"
Jul 29 19:57:24.824941 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:24.824904    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f775ac3c-4fb8-49e9-b4a7-a61ed8398e2a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-dc9ec3-b47cd6f6b-k2l78\" (UID: \"f775ac3c-4fb8-49e9-b4a7-a61ed8398e2a\") " pod="openshift-ingress/payload-processing-e2e-api-a-dc9ec3-b47cd6f6b-k2l78"
Jul 29 19:57:24.825386 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:24.825080    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw7q2\" (UniqueName: \"kubernetes.io/projected/f775ac3c-4fb8-49e9-b4a7-a61ed8398e2a-kube-api-access-vw7q2\") pod \"payload-processing-e2e-api-a-dc9ec3-b47cd6f6b-k2l78\" (UID: \"f775ac3c-4fb8-49e9-b4a7-a61ed8398e2a\") " pod="openshift-ingress/payload-processing-e2e-api-a-dc9ec3-b47cd6f6b-k2l78"
Jul 29 19:57:24.825463 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:24.825409    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc9ec3-9ddc7bbd-mvjlf"
Jul 29 19:57:24.825858 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:24.825839    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f775ac3c-4fb8-49e9-b4a7-a61ed8398e2a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-dc9ec3-b47cd6f6b-k2l78\" (UID: \"f775ac3c-4fb8-49e9-b4a7-a61ed8398e2a\") " pod="openshift-ingress/payload-processing-e2e-api-a-dc9ec3-b47cd6f6b-k2l78"
Jul 29 19:57:24.832983 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:24.832936    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw7q2\" (UniqueName: \"kubernetes.io/projected/f775ac3c-4fb8-49e9-b4a7-a61ed8398e2a-kube-api-access-vw7q2\") pod \"payload-processing-e2e-api-a-dc9ec3-b47cd6f6b-k2l78\" (UID: \"f775ac3c-4fb8-49e9-b4a7-a61ed8398e2a\") " pod="openshift-ingress/payload-processing-e2e-api-a-dc9ec3-b47cd6f6b-k2l78"
Jul 29 19:57:24.860961 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:24.860913    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dc9ec3-b47cd6f6b-k2l78"
Jul 29 19:57:24.866888 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:24.866863    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dc9ec3-7b95659fb-5bkfj"]
Jul 29 19:57:24.871120 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:57:24.871094    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda56b8fe6_c110_4e52_be74_957ca6cd3236.slice/crio-3746e75ae09a803b5c471c7d66cc4412bea6d3a7e578417c54dea5fa45f2020a WatchSource:0}: Error finding container 3746e75ae09a803b5c471c7d66cc4412bea6d3a7e578417c54dea5fa45f2020a: Status 404 returned error can't find the container with id 3746e75ae09a803b5c471c7d66cc4412bea6d3a7e578417c54dea5fa45f2020a
Jul 29 19:57:24.872676 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:24.872658    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 19:57:24.975378 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:24.975259    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dc9ec3-9ddc7bbd-mvjlf"]
Jul 29 19:57:24.978345 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:57:24.978317    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08e56bd4_aa8d_4909_b01f_f98b8e5ace5e.slice/crio-6c2a92832db324c603963561059c894b0d809b698ad4797f49d761794e54be9f WatchSource:0}: Error finding container 6c2a92832db324c603963561059c894b0d809b698ad4797f49d761794e54be9f: Status 404 returned error can't find the container with id 6c2a92832db324c603963561059c894b0d809b698ad4797f49d761794e54be9f
Jul 29 19:57:25.014330 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:25.014304    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dc9ec3-b47cd6f6b-k2l78"]
Jul 29 19:57:25.017157 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:57:25.017131    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf775ac3c_4fb8_49e9_b4a7_a61ed8398e2a.slice/crio-352d38ea7b05aacc476b4e77ad94247716bb5bdecdf8bff1f22b705f3a1896de WatchSource:0}: Error finding container 352d38ea7b05aacc476b4e77ad94247716bb5bdecdf8bff1f22b705f3a1896de: Status 404 returned error can't find the container with id 352d38ea7b05aacc476b4e77ad94247716bb5bdecdf8bff1f22b705f3a1896de
Jul 29 19:57:25.555922 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:25.555815    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc9ec3-7b95659fb-5bkfj" event={"ID":"a56b8fe6-c110-4e52-be74-957ca6cd3236","Type":"ContainerStarted","Data":"56c5c69be8241ab0eac805201f32fe2f449c599b1586122ee35cc3ea77d78c90"}
Jul 29 19:57:25.555922 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:25.555898    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc9ec3-7b95659fb-5bkfj" event={"ID":"a56b8fe6-c110-4e52-be74-957ca6cd3236","Type":"ContainerStarted","Data":"3746e75ae09a803b5c471c7d66cc4412bea6d3a7e578417c54dea5fa45f2020a"}
Jul 29 19:57:25.555922 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:25.555931    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc9ec3-7b95659fb-5bkfj"
Jul 29 19:57:25.557187 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:25.557161    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc9ec3-9ddc7bbd-mvjlf" event={"ID":"08e56bd4-aa8d-4909-b01f-f98b8e5ace5e","Type":"ContainerStarted","Data":"968442f6bb2fad749e1edad4595494541babc68dcfeb1d367eff76627acc6ede"}
Jul 29 19:57:25.557316 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:25.557217    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc9ec3-9ddc7bbd-mvjlf"
Jul 29 19:57:25.557316 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:25.557243    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc9ec3-9ddc7bbd-mvjlf" event={"ID":"08e56bd4-aa8d-4909-b01f-f98b8e5ace5e","Type":"ContainerStarted","Data":"6c2a92832db324c603963561059c894b0d809b698ad4797f49d761794e54be9f"}
Jul 29 19:57:25.558543 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:25.558522    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dc9ec3-b47cd6f6b-k2l78" event={"ID":"f775ac3c-4fb8-49e9-b4a7-a61ed8398e2a","Type":"ContainerStarted","Data":"7edb6781419944389ac0de9fb19bbe1b562c0b531d910dfcc345c95eaa99472e"}
Jul 29 19:57:25.558622 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:25.558558    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dc9ec3-b47cd6f6b-k2l78" event={"ID":"f775ac3c-4fb8-49e9-b4a7-a61ed8398e2a","Type":"ContainerStarted","Data":"352d38ea7b05aacc476b4e77ad94247716bb5bdecdf8bff1f22b705f3a1896de"}
Jul 29 19:57:25.558662 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:25.558638    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-dc9ec3-b47cd6f6b-k2l78"
Jul 29 19:57:25.573063 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:25.573015    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc9ec3-7b95659fb-5bkfj" podStartSLOduration=2.140538774 podStartE2EDuration="2.572999194s" totalImagesPullingTime="432.46042ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:57:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:57:25.571526836 +0000 UTC m=+2412.799915961" watchObservedRunningTime="2026-07-29 19:57:25.572999194 +0000 UTC m=+2412.801388319"
Jul 29 19:57:25.585152 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:25.585103    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc9ec3-9ddc7bbd-mvjlf" podStartSLOduration=1.585088833 podStartE2EDuration="1.585088833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:57:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:57:25.584359627 +0000 UTC m=+2412.812748759" watchObservedRunningTime="2026-07-29 19:57:25.585088833 +0000 UTC m=+2412.813477958"
Jul 29 19:57:25.599699 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:25.599650    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-dc9ec3-b47cd6f6b-k2l78" podStartSLOduration=1.5996322429999998 podStartE2EDuration="1.599632243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:57:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:57:25.598942188 +0000 UTC m=+2412.827331314" watchObservedRunningTime="2026-07-29 19:57:25.599632243 +0000 UTC m=+2412.828021369"
Jul 29 19:57:30.594339 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:30.594300    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:57:31.569129 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:31.569096    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc9ec3-7b95659fb-5bkfj"
Jul 29 19:57:36.566034 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:36.565960    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc9ec3-9ddc7bbd-mvjlf"
Jul 29 19:57:36.566424 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:36.566043    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-dc9ec3-b47cd6f6b-k2l78"
Jul 29 19:57:52.304594 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:52.304556    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0e0fe5-54fc8cc669-rlfdj"]
Jul 29 19:57:52.308054 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:52.308037    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0fe5-54fc8cc669-rlfdj"
Jul 29 19:57:52.310605 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:52.310575    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-0e0fe5-serving-cert\""
Jul 29 19:57:52.319103 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:52.319079    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0e0fe5-54fc8cc669-rlfdj"]
Jul 29 19:57:52.356062 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:52.356028    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f344c67-12ab-493f-8d79-4acbd80d09d2-openshift-service-ca\") pod \"maas-api-e2e-api-b-0e0fe5-54fc8cc669-rlfdj\" (UID: \"2f344c67-12ab-493f-8d79-4acbd80d09d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0fe5-54fc8cc669-rlfdj"
Jul 29 19:57:52.356232 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:52.356090    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f344c67-12ab-493f-8d79-4acbd80d09d2-maas-api-tls\") pod \"maas-api-e2e-api-b-0e0fe5-54fc8cc669-rlfdj\" (UID: \"2f344c67-12ab-493f-8d79-4acbd80d09d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0fe5-54fc8cc669-rlfdj"
Jul 29 19:57:52.356232 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:52.356120    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpwm8\" (UniqueName: \"kubernetes.io/projected/2f344c67-12ab-493f-8d79-4acbd80d09d2-kube-api-access-gpwm8\") pod \"maas-api-e2e-api-b-0e0fe5-54fc8cc669-rlfdj\" (UID: \"2f344c67-12ab-493f-8d79-4acbd80d09d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0fe5-54fc8cc669-rlfdj"
Jul 29 19:57:52.457378 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:52.457328    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f344c67-12ab-493f-8d79-4acbd80d09d2-maas-api-tls\") pod \"maas-api-e2e-api-b-0e0fe5-54fc8cc669-rlfdj\" (UID: \"2f344c67-12ab-493f-8d79-4acbd80d09d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0fe5-54fc8cc669-rlfdj"
Jul 29 19:57:52.457558 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:52.457386    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpwm8\" (UniqueName: \"kubernetes.io/projected/2f344c67-12ab-493f-8d79-4acbd80d09d2-kube-api-access-gpwm8\") pod \"maas-api-e2e-api-b-0e0fe5-54fc8cc669-rlfdj\" (UID: \"2f344c67-12ab-493f-8d79-4acbd80d09d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0fe5-54fc8cc669-rlfdj"
Jul 29 19:57:52.457558 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:52.457433    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f344c67-12ab-493f-8d79-4acbd80d09d2-openshift-service-ca\") pod \"maas-api-e2e-api-b-0e0fe5-54fc8cc669-rlfdj\" (UID: \"2f344c67-12ab-493f-8d79-4acbd80d09d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0fe5-54fc8cc669-rlfdj"
Jul 29 19:57:52.458085 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:52.458066    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f344c67-12ab-493f-8d79-4acbd80d09d2-openshift-service-ca\") pod \"maas-api-e2e-api-b-0e0fe5-54fc8cc669-rlfdj\" (UID: \"2f344c67-12ab-493f-8d79-4acbd80d09d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0fe5-54fc8cc669-rlfdj"
Jul 29 19:57:52.459714 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:52.459695    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f344c67-12ab-493f-8d79-4acbd80d09d2-maas-api-tls\") pod \"maas-api-e2e-api-b-0e0fe5-54fc8cc669-rlfdj\" (UID: \"2f344c67-12ab-493f-8d79-4acbd80d09d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0fe5-54fc8cc669-rlfdj"
Jul 29 19:57:52.466488 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:52.466461    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpwm8\" (UniqueName: \"kubernetes.io/projected/2f344c67-12ab-493f-8d79-4acbd80d09d2-kube-api-access-gpwm8\") pod \"maas-api-e2e-api-b-0e0fe5-54fc8cc669-rlfdj\" (UID: \"2f344c67-12ab-493f-8d79-4acbd80d09d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0fe5-54fc8cc669-rlfdj"
Jul 29 19:57:52.618988 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:52.618888    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0fe5-54fc8cc669-rlfdj"
Jul 29 19:57:52.765943 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:52.765918    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0e0fe5-54fc8cc669-rlfdj"]
Jul 29 19:57:52.768113 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:57:52.768088    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f344c67_12ab_493f_8d79_4acbd80d09d2.slice/crio-d9b33658b293af3de05e6f1e87c1367116d97993a8264a561d026fecef190e84 WatchSource:0}: Error finding container d9b33658b293af3de05e6f1e87c1367116d97993a8264a561d026fecef190e84: Status 404 returned error can't find the container with id d9b33658b293af3de05e6f1e87c1367116d97993a8264a561d026fecef190e84
Jul 29 19:57:53.685506 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:53.685475    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0fe5-54fc8cc669-rlfdj" event={"ID":"2f344c67-12ab-493f-8d79-4acbd80d09d2","Type":"ContainerStarted","Data":"f70f532aea8871f0ff41dac787ac7b698b41e496b801448555752960249bd6c9"}
Jul 29 19:57:53.685905 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:53.685523    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0fe5-54fc8cc669-rlfdj" event={"ID":"2f344c67-12ab-493f-8d79-4acbd80d09d2","Type":"ContainerStarted","Data":"d9b33658b293af3de05e6f1e87c1367116d97993a8264a561d026fecef190e84"}
Jul 29 19:57:53.685905 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:53.685579    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0fe5-54fc8cc669-rlfdj"
Jul 29 19:57:53.743495 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:53.743435    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0fe5-54fc8cc669-rlfdj" podStartSLOduration=1.3590283730000001 podStartE2EDuration="1.743418555s" totalImagesPullingTime="384.390182ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:57:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:57:53.742910824 +0000 UTC m=+2440.971299950" watchObservedRunningTime="2026-07-29 19:57:53.743418555 +0000 UTC m=+2440.971807680"
Jul 29 19:57:59.694997 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:57:59.694965    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0fe5-54fc8cc669-rlfdj"
Jul 29 19:58:00.697978 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:00.697944    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:58:13.486250 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:13.486219    2583 scope.go:117] "RemoveContainer" containerID="2def6bb9a31417c98fd0a98d1e69eaa4e2ac8861f81c77a8a51915e3fd667674"
Jul 29 19:58:19.810028 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:19.809997    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc9ec3-svr4z"]
Jul 29 19:58:19.813184 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:19.813168    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc9ec3-svr4z"
Jul 29 19:58:19.815212 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:19.815189    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hq9l2\""
Jul 29 19:58:19.819880 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:19.819852    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc9ec3-svr4z"]
Jul 29 19:58:19.893626 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:19.893594    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90c48c96-e154-4bd6-81fd-0af24b2e987a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dc9ec3-svr4z\" (UID: \"90c48c96-e154-4bd6-81fd-0af24b2e987a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc9ec3-svr4z"
Jul 29 19:58:19.994049 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:19.994006    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90c48c96-e154-4bd6-81fd-0af24b2e987a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dc9ec3-svr4z\" (UID: \"90c48c96-e154-4bd6-81fd-0af24b2e987a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc9ec3-svr4z"
Jul 29 19:58:19.994628 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:19.994607    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90c48c96-e154-4bd6-81fd-0af24b2e987a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dc9ec3-svr4z\" (UID: \"90c48c96-e154-4bd6-81fd-0af24b2e987a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc9ec3-svr4z"
Jul 29 19:58:20.124611 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:20.124578    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc9ec3-svr4z"
Jul 29 19:58:20.272060 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:20.272033    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc9ec3-svr4z"]
Jul 29 19:58:20.273777 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:58:20.273750    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90c48c96_e154_4bd6_81fd_0af24b2e987a.slice/crio-6c5b94ce405176a03fe10cf6b7554577f28523738ebd29cef9156d6cb3cedc03 WatchSource:0}: Error finding container 6c5b94ce405176a03fe10cf6b7554577f28523738ebd29cef9156d6cb3cedc03: Status 404 returned error can't find the container with id 6c5b94ce405176a03fe10cf6b7554577f28523738ebd29cef9156d6cb3cedc03
Jul 29 19:58:20.806737 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:20.806653    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc9ec3-svr4z" event={"ID":"90c48c96-e154-4bd6-81fd-0af24b2e987a","Type":"ContainerStarted","Data":"2b433390b23f4a2eb92cbb763780427178f83ca53babb236a96e8e82091ac6fd"}
Jul 29 19:58:20.806737 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:20.806713    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc9ec3-svr4z" event={"ID":"90c48c96-e154-4bd6-81fd-0af24b2e987a","Type":"ContainerStarted","Data":"6c5b94ce405176a03fe10cf6b7554577f28523738ebd29cef9156d6cb3cedc03"}
Jul 29 19:58:20.820734 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:20.820680    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc9ec3-svr4z" podStartSLOduration=1.8206624169999999 podStartE2EDuration="1.820662417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:58:20.820323788 +0000 UTC m=+2468.048712948" watchObservedRunningTime="2026-07-29 19:58:20.820662417 +0000 UTC m=+2468.049051545"
Jul 29 19:58:21.812061 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:21.812025    2583 generic.go:356] "Generic (PLEG): container finished" podID="90c48c96-e154-4bd6-81fd-0af24b2e987a" containerID="2b433390b23f4a2eb92cbb763780427178f83ca53babb236a96e8e82091ac6fd" exitCode=0
Jul 29 19:58:21.812232 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:21.812110    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc9ec3-svr4z" event={"ID":"90c48c96-e154-4bd6-81fd-0af24b2e987a","Type":"ContainerDied","Data":"2b433390b23f4a2eb92cbb763780427178f83ca53babb236a96e8e82091ac6fd"}
Jul 29 19:58:22.949499 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:22.949471    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc9ec3-svr4z"
Jul 29 19:58:23.015334 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:23.015291    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90c48c96-e154-4bd6-81fd-0af24b2e987a-maas-api-service-ca\") pod \"90c48c96-e154-4bd6-81fd-0af24b2e987a\" (UID: \"90c48c96-e154-4bd6-81fd-0af24b2e987a\") "
Jul 29 19:58:23.015655 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:23.015632    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90c48c96-e154-4bd6-81fd-0af24b2e987a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "90c48c96-e154-4bd6-81fd-0af24b2e987a" (UID: "90c48c96-e154-4bd6-81fd-0af24b2e987a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:58:23.115960 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:23.115877    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90c48c96-e154-4bd6-81fd-0af24b2e987a-maas-api-service-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:58:23.822322 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:23.822290    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc9ec3-svr4z"
Jul 29 19:58:23.822510 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:23.822319    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc9ec3-svr4z" event={"ID":"90c48c96-e154-4bd6-81fd-0af24b2e987a","Type":"ContainerDied","Data":"6c5b94ce405176a03fe10cf6b7554577f28523738ebd29cef9156d6cb3cedc03"}
Jul 29 19:58:23.822510 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:23.822398    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c5b94ce405176a03fe10cf6b7554577f28523738ebd29cef9156d6cb3cedc03"
Jul 29 19:58:35.256036 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.255993    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dc9ec3-7b95659fb-5bkfj"]
Jul 29 19:58:35.256504 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.256298    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc9ec3-7b95659fb-5bkfj" podUID="a56b8fe6-c110-4e52-be74-957ca6cd3236" containerName="maas-api" containerID="cri-o://56c5c69be8241ab0eac805201f32fe2f449c599b1586122ee35cc3ea77d78c90" gracePeriod=30
Jul 29 19:58:35.306188 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.306152    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dc9ec3-b47cd6f6b-k2l78"]
Jul 29 19:58:35.306690 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.306654    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-dc9ec3-b47cd6f6b-k2l78" podUID="f775ac3c-4fb8-49e9-b4a7-a61ed8398e2a" containerName="payload-processing" containerID="cri-o://7edb6781419944389ac0de9fb19bbe1b562c0b531d910dfcc345c95eaa99472e" gracePeriod=30
Jul 29 19:58:35.314165 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.313879    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dc9ec3-9ddc7bbd-mvjlf"]
Jul 29 19:58:35.314165 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.314116    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc9ec3-9ddc7bbd-mvjlf" podUID="08e56bd4-aa8d-4909-b01f-f98b8e5ace5e" containerName="payload-pre-processing" containerID="cri-o://968442f6bb2fad749e1edad4595494541babc68dcfeb1d367eff76627acc6ede" gracePeriod=30
Jul 29 19:58:35.425730 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.425697    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc9ec3-svr4z"]
Jul 29 19:58:35.428211 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.428174    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc9ec3-svr4z"]
Jul 29 19:58:35.566774 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.566735    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc9ec3-7b95659fb-5bkfj"
Jul 29 19:58:35.569587 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.569567    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dc9ec3-b47cd6f6b-k2l78"
Jul 29 19:58:35.613763 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.613740    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc9ec3-9ddc7bbd-mvjlf"
Jul 29 19:58:35.619633 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.619610    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtqsr\" (UniqueName: \"kubernetes.io/projected/a56b8fe6-c110-4e52-be74-957ca6cd3236-kube-api-access-vtqsr\") pod \"a56b8fe6-c110-4e52-be74-957ca6cd3236\" (UID: \"a56b8fe6-c110-4e52-be74-957ca6cd3236\") "
Jul 29 19:58:35.619777 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.619657    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f775ac3c-4fb8-49e9-b4a7-a61ed8398e2a-plugins-config-volume\") pod \"f775ac3c-4fb8-49e9-b4a7-a61ed8398e2a\" (UID: \"f775ac3c-4fb8-49e9-b4a7-a61ed8398e2a\") "
Jul 29 19:58:35.619777 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.619695    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vw7q2\" (UniqueName: \"kubernetes.io/projected/f775ac3c-4fb8-49e9-b4a7-a61ed8398e2a-kube-api-access-vw7q2\") pod \"f775ac3c-4fb8-49e9-b4a7-a61ed8398e2a\" (UID: \"f775ac3c-4fb8-49e9-b4a7-a61ed8398e2a\") "
Jul 29 19:58:35.619777 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.619747    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a56b8fe6-c110-4e52-be74-957ca6cd3236-openshift-service-ca\") pod \"a56b8fe6-c110-4e52-be74-957ca6cd3236\" (UID: \"a56b8fe6-c110-4e52-be74-957ca6cd3236\") "
Jul 29 19:58:35.619969 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.619781    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a56b8fe6-c110-4e52-be74-957ca6cd3236-maas-api-tls\") pod \"a56b8fe6-c110-4e52-be74-957ca6cd3236\" (UID: \"a56b8fe6-c110-4e52-be74-957ca6cd3236\") "
Jul 29 19:58:35.620119 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.620090    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f775ac3c-4fb8-49e9-b4a7-a61ed8398e2a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f775ac3c-4fb8-49e9-b4a7-a61ed8398e2a" (UID: "f775ac3c-4fb8-49e9-b4a7-a61ed8398e2a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:58:35.620224 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.620127    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a56b8fe6-c110-4e52-be74-957ca6cd3236-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a56b8fe6-c110-4e52-be74-957ca6cd3236" (UID: "a56b8fe6-c110-4e52-be74-957ca6cd3236"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:58:35.621845 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.621818    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f775ac3c-4fb8-49e9-b4a7-a61ed8398e2a-kube-api-access-vw7q2" (OuterVolumeSpecName: "kube-api-access-vw7q2") pod "f775ac3c-4fb8-49e9-b4a7-a61ed8398e2a" (UID: "f775ac3c-4fb8-49e9-b4a7-a61ed8398e2a"). InnerVolumeSpecName "kube-api-access-vw7q2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:58:35.621936 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.621860    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a56b8fe6-c110-4e52-be74-957ca6cd3236-kube-api-access-vtqsr" (OuterVolumeSpecName: "kube-api-access-vtqsr") pod "a56b8fe6-c110-4e52-be74-957ca6cd3236" (UID: "a56b8fe6-c110-4e52-be74-957ca6cd3236"). InnerVolumeSpecName "kube-api-access-vtqsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:58:35.621936 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.621889    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a56b8fe6-c110-4e52-be74-957ca6cd3236-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a56b8fe6-c110-4e52-be74-957ca6cd3236" (UID: "a56b8fe6-c110-4e52-be74-957ca6cd3236"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:58:35.720723 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.720686    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08e56bd4-aa8d-4909-b01f-f98b8e5ace5e-plugins-config-volume\") pod \"08e56bd4-aa8d-4909-b01f-f98b8e5ace5e\" (UID: \"08e56bd4-aa8d-4909-b01f-f98b8e5ace5e\") "
Jul 29 19:58:35.720723 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.720729    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m85h8\" (UniqueName: \"kubernetes.io/projected/08e56bd4-aa8d-4909-b01f-f98b8e5ace5e-kube-api-access-m85h8\") pod \"08e56bd4-aa8d-4909-b01f-f98b8e5ace5e\" (UID: \"08e56bd4-aa8d-4909-b01f-f98b8e5ace5e\") "
Jul 29 19:58:35.721006 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.720899    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtqsr\" (UniqueName: \"kubernetes.io/projected/a56b8fe6-c110-4e52-be74-957ca6cd3236-kube-api-access-vtqsr\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:58:35.721006 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.720912    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f775ac3c-4fb8-49e9-b4a7-a61ed8398e2a-plugins-config-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:58:35.721006 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.720922    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw7q2\" (UniqueName: \"kubernetes.io/projected/f775ac3c-4fb8-49e9-b4a7-a61ed8398e2a-kube-api-access-vw7q2\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:58:35.721006 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.720930    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a56b8fe6-c110-4e52-be74-957ca6cd3236-openshift-service-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:58:35.721006 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.720938    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a56b8fe6-c110-4e52-be74-957ca6cd3236-maas-api-tls\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:58:35.721228 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.721083    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08e56bd4-aa8d-4909-b01f-f98b8e5ace5e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "08e56bd4-aa8d-4909-b01f-f98b8e5ace5e" (UID: "08e56bd4-aa8d-4909-b01f-f98b8e5ace5e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:58:35.722851 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.722826    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08e56bd4-aa8d-4909-b01f-f98b8e5ace5e-kube-api-access-m85h8" (OuterVolumeSpecName: "kube-api-access-m85h8") pod "08e56bd4-aa8d-4909-b01f-f98b8e5ace5e" (UID: "08e56bd4-aa8d-4909-b01f-f98b8e5ace5e"). InnerVolumeSpecName "kube-api-access-m85h8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:58:35.822103 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.822025    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08e56bd4-aa8d-4909-b01f-f98b8e5ace5e-plugins-config-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:58:35.822103 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.822051    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m85h8\" (UniqueName: \"kubernetes.io/projected/08e56bd4-aa8d-4909-b01f-f98b8e5ace5e-kube-api-access-m85h8\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:58:35.877775 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.877740    2583 generic.go:356] "Generic (PLEG): container finished" podID="f775ac3c-4fb8-49e9-b4a7-a61ed8398e2a" containerID="7edb6781419944389ac0de9fb19bbe1b562c0b531d910dfcc345c95eaa99472e" exitCode=0
Jul 29 19:58:35.877950 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.877780    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dc9ec3-b47cd6f6b-k2l78"
Jul 29 19:58:35.877950 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.877834    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dc9ec3-b47cd6f6b-k2l78" event={"ID":"f775ac3c-4fb8-49e9-b4a7-a61ed8398e2a","Type":"ContainerDied","Data":"7edb6781419944389ac0de9fb19bbe1b562c0b531d910dfcc345c95eaa99472e"}
Jul 29 19:58:35.877950 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.877890    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dc9ec3-b47cd6f6b-k2l78" event={"ID":"f775ac3c-4fb8-49e9-b4a7-a61ed8398e2a","Type":"ContainerDied","Data":"352d38ea7b05aacc476b4e77ad94247716bb5bdecdf8bff1f22b705f3a1896de"}
Jul 29 19:58:35.877950 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.877924    2583 scope.go:117] "RemoveContainer" containerID="7edb6781419944389ac0de9fb19bbe1b562c0b531d910dfcc345c95eaa99472e"
Jul 29 19:58:35.879314 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.879294    2583 generic.go:356] "Generic (PLEG): container finished" podID="a56b8fe6-c110-4e52-be74-957ca6cd3236" containerID="56c5c69be8241ab0eac805201f32fe2f449c599b1586122ee35cc3ea77d78c90" exitCode=0
Jul 29 19:58:35.879439 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.879323    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc9ec3-7b95659fb-5bkfj" event={"ID":"a56b8fe6-c110-4e52-be74-957ca6cd3236","Type":"ContainerDied","Data":"56c5c69be8241ab0eac805201f32fe2f449c599b1586122ee35cc3ea77d78c90"}
Jul 29 19:58:35.879439 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.879335    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc9ec3-7b95659fb-5bkfj"
Jul 29 19:58:35.879545 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.879448    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc9ec3-7b95659fb-5bkfj" event={"ID":"a56b8fe6-c110-4e52-be74-957ca6cd3236","Type":"ContainerDied","Data":"3746e75ae09a803b5c471c7d66cc4412bea6d3a7e578417c54dea5fa45f2020a"}
Jul 29 19:58:35.880895 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.880870    2583 generic.go:356] "Generic (PLEG): container finished" podID="08e56bd4-aa8d-4909-b01f-f98b8e5ace5e" containerID="968442f6bb2fad749e1edad4595494541babc68dcfeb1d367eff76627acc6ede" exitCode=0
Jul 29 19:58:35.881025 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.880909    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc9ec3-9ddc7bbd-mvjlf" event={"ID":"08e56bd4-aa8d-4909-b01f-f98b8e5ace5e","Type":"ContainerDied","Data":"968442f6bb2fad749e1edad4595494541babc68dcfeb1d367eff76627acc6ede"}
Jul 29 19:58:35.881025 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.880919    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc9ec3-9ddc7bbd-mvjlf"
Jul 29 19:58:35.881025 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.880950    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc9ec3-9ddc7bbd-mvjlf" event={"ID":"08e56bd4-aa8d-4909-b01f-f98b8e5ace5e","Type":"ContainerDied","Data":"6c2a92832db324c603963561059c894b0d809b698ad4797f49d761794e54be9f"}
Jul 29 19:58:35.887079 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.887057    2583 scope.go:117] "RemoveContainer" containerID="7edb6781419944389ac0de9fb19bbe1b562c0b531d910dfcc345c95eaa99472e"
Jul 29 19:58:35.887429 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:58:35.887405    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7edb6781419944389ac0de9fb19bbe1b562c0b531d910dfcc345c95eaa99472e\": container with ID starting with 7edb6781419944389ac0de9fb19bbe1b562c0b531d910dfcc345c95eaa99472e not found: ID does not exist" containerID="7edb6781419944389ac0de9fb19bbe1b562c0b531d910dfcc345c95eaa99472e"
Jul 29 19:58:35.887552 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.887435    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7edb6781419944389ac0de9fb19bbe1b562c0b531d910dfcc345c95eaa99472e"} err="failed to get container status \"7edb6781419944389ac0de9fb19bbe1b562c0b531d910dfcc345c95eaa99472e\": rpc error: code = NotFound desc = could not find container \"7edb6781419944389ac0de9fb19bbe1b562c0b531d910dfcc345c95eaa99472e\": container with ID starting with 7edb6781419944389ac0de9fb19bbe1b562c0b531d910dfcc345c95eaa99472e not found: ID does not exist"
Jul 29 19:58:35.887552 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.887454    2583 scope.go:117] "RemoveContainer" containerID="56c5c69be8241ab0eac805201f32fe2f449c599b1586122ee35cc3ea77d78c90"
Jul 29 19:58:35.896363 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.896342    2583 scope.go:117] "RemoveContainer" containerID="56c5c69be8241ab0eac805201f32fe2f449c599b1586122ee35cc3ea77d78c90"
Jul 29 19:58:35.896621 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:58:35.896603    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56c5c69be8241ab0eac805201f32fe2f449c599b1586122ee35cc3ea77d78c90\": container with ID starting with 56c5c69be8241ab0eac805201f32fe2f449c599b1586122ee35cc3ea77d78c90 not found: ID does not exist" containerID="56c5c69be8241ab0eac805201f32fe2f449c599b1586122ee35cc3ea77d78c90"
Jul 29 19:58:35.896667 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.896629    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56c5c69be8241ab0eac805201f32fe2f449c599b1586122ee35cc3ea77d78c90"} err="failed to get container status \"56c5c69be8241ab0eac805201f32fe2f449c599b1586122ee35cc3ea77d78c90\": rpc error: code = NotFound desc = could not find container \"56c5c69be8241ab0eac805201f32fe2f449c599b1586122ee35cc3ea77d78c90\": container with ID starting with 56c5c69be8241ab0eac805201f32fe2f449c599b1586122ee35cc3ea77d78c90 not found: ID does not exist"
Jul 29 19:58:35.896667 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.896646    2583 scope.go:117] "RemoveContainer" containerID="968442f6bb2fad749e1edad4595494541babc68dcfeb1d367eff76627acc6ede"
Jul 29 19:58:35.906108 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.906085    2583 scope.go:117] "RemoveContainer" containerID="968442f6bb2fad749e1edad4595494541babc68dcfeb1d367eff76627acc6ede"
Jul 29 19:58:35.906205 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.906165    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dc9ec3-b47cd6f6b-k2l78"]
Jul 29 19:58:35.906483 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:58:35.906465    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"968442f6bb2fad749e1edad4595494541babc68dcfeb1d367eff76627acc6ede\": container with ID starting with 968442f6bb2fad749e1edad4595494541babc68dcfeb1d367eff76627acc6ede not found: ID does not exist" containerID="968442f6bb2fad749e1edad4595494541babc68dcfeb1d367eff76627acc6ede"
Jul 29 19:58:35.906527 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.906490    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"968442f6bb2fad749e1edad4595494541babc68dcfeb1d367eff76627acc6ede"} err="failed to get container status \"968442f6bb2fad749e1edad4595494541babc68dcfeb1d367eff76627acc6ede\": rpc error: code = NotFound desc = could not find container \"968442f6bb2fad749e1edad4595494541babc68dcfeb1d367eff76627acc6ede\": container with ID starting with 968442f6bb2fad749e1edad4595494541babc68dcfeb1d367eff76627acc6ede not found: ID does not exist"
Jul 29 19:58:35.909364 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.909334    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dc9ec3-b47cd6f6b-k2l78"]
Jul 29 19:58:35.918940 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.918915    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dc9ec3-7b95659fb-5bkfj"]
Jul 29 19:58:35.921076 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.921052    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dc9ec3-7b95659fb-5bkfj"]
Jul 29 19:58:35.930242 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.930213    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dc9ec3-9ddc7bbd-mvjlf"]
Jul 29 19:58:35.934464 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:35.934429    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dc9ec3-9ddc7bbd-mvjlf"]
Jul 29 19:58:37.341297 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:37.341261    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08e56bd4-aa8d-4909-b01f-f98b8e5ace5e" path="/var/lib/kubelet/pods/08e56bd4-aa8d-4909-b01f-f98b8e5ace5e/volumes"
Jul 29 19:58:37.341678 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:37.341657    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90c48c96-e154-4bd6-81fd-0af24b2e987a" path="/var/lib/kubelet/pods/90c48c96-e154-4bd6-81fd-0af24b2e987a/volumes"
Jul 29 19:58:37.342052 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:37.342038    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a56b8fe6-c110-4e52-be74-957ca6cd3236" path="/var/lib/kubelet/pods/a56b8fe6-c110-4e52-be74-957ca6cd3236/volumes"
Jul 29 19:58:37.342386 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:37.342373    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f775ac3c-4fb8-49e9-b4a7-a61ed8398e2a" path="/var/lib/kubelet/pods/f775ac3c-4fb8-49e9-b4a7-a61ed8398e2a/volumes"
Jul 29 19:58:41.393357 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:41.393312    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:58:42.820979 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:42.820942    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0e0fe5-wj2sw"]
Jul 29 19:58:42.821400 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:42.821237    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90c48c96-e154-4bd6-81fd-0af24b2e987a" containerName="revoke-keys"
Jul 29 19:58:42.821400 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:42.821248    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="90c48c96-e154-4bd6-81fd-0af24b2e987a" containerName="revoke-keys"
Jul 29 19:58:42.821400 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:42.821259    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f775ac3c-4fb8-49e9-b4a7-a61ed8398e2a" containerName="payload-processing"
Jul 29 19:58:42.821400 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:42.821265    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f775ac3c-4fb8-49e9-b4a7-a61ed8398e2a" containerName="payload-processing"
Jul 29 19:58:42.821400 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:42.821279    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08e56bd4-aa8d-4909-b01f-f98b8e5ace5e" containerName="payload-pre-processing"
Jul 29 19:58:42.821400 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:42.821285    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="08e56bd4-aa8d-4909-b01f-f98b8e5ace5e" containerName="payload-pre-processing"
Jul 29 19:58:42.821400 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:42.821292    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a56b8fe6-c110-4e52-be74-957ca6cd3236" containerName="maas-api"
Jul 29 19:58:42.821400 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:42.821298    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a56b8fe6-c110-4e52-be74-957ca6cd3236" containerName="maas-api"
Jul 29 19:58:42.821400 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:42.821356    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="08e56bd4-aa8d-4909-b01f-f98b8e5ace5e" containerName="payload-pre-processing"
Jul 29 19:58:42.821400 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:42.821365    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a56b8fe6-c110-4e52-be74-957ca6cd3236" containerName="maas-api"
Jul 29 19:58:42.821400 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:42.821372    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="90c48c96-e154-4bd6-81fd-0af24b2e987a" containerName="revoke-keys"
Jul 29 19:58:42.821400 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:42.821379    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f775ac3c-4fb8-49e9-b4a7-a61ed8398e2a" containerName="payload-processing"
Jul 29 19:58:42.825761 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:42.825741    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0e0fe5-wj2sw"
Jul 29 19:58:42.827986 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:42.827962    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hq9l2\""
Jul 29 19:58:42.831600 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:42.831574    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0e0fe5-wj2sw"]
Jul 29 19:58:42.880466 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:42.880430    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51af2303-bd3d-43d0-a58f-b35082df8f76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0e0fe5-wj2sw\" (UID: \"51af2303-bd3d-43d0-a58f-b35082df8f76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0e0fe5-wj2sw"
Jul 29 19:58:42.981060 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:42.981017    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51af2303-bd3d-43d0-a58f-b35082df8f76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0e0fe5-wj2sw\" (UID: \"51af2303-bd3d-43d0-a58f-b35082df8f76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0e0fe5-wj2sw"
Jul 29 19:58:42.981657 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:42.981634    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51af2303-bd3d-43d0-a58f-b35082df8f76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0e0fe5-wj2sw\" (UID: \"51af2303-bd3d-43d0-a58f-b35082df8f76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0e0fe5-wj2sw"
Jul 29 19:58:43.137004 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:43.136973    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0e0fe5-wj2sw"
Jul 29 19:58:43.268470 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:43.268391    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0e0fe5-wj2sw"]
Jul 29 19:58:43.270948 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:58:43.270912    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51af2303_bd3d_43d0_a58f_b35082df8f76.slice/crio-1135d2042fe4b76e95daf91b54aeac31305770c4b5e91ec346720a5dd0ac8f8d WatchSource:0}: Error finding container 1135d2042fe4b76e95daf91b54aeac31305770c4b5e91ec346720a5dd0ac8f8d: Status 404 returned error can't find the container with id 1135d2042fe4b76e95daf91b54aeac31305770c4b5e91ec346720a5dd0ac8f8d
Jul 29 19:58:43.921894 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:43.921858    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0e0fe5-wj2sw" event={"ID":"51af2303-bd3d-43d0-a58f-b35082df8f76","Type":"ContainerStarted","Data":"962ee9e0db7aac4486f5fbe93d1f20fd78798862e9153e503d04e08c9b5107ac"}
Jul 29 19:58:43.922363 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:43.921909    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0e0fe5-wj2sw" event={"ID":"51af2303-bd3d-43d0-a58f-b35082df8f76","Type":"ContainerStarted","Data":"1135d2042fe4b76e95daf91b54aeac31305770c4b5e91ec346720a5dd0ac8f8d"}
Jul 29 19:58:43.946484 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:43.946434    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0e0fe5-wj2sw" podStartSLOduration=1.946416754 podStartE2EDuration="1.946416754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:58:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:58:43.943859194 +0000 UTC m=+2491.172248320" watchObservedRunningTime="2026-07-29 19:58:43.946416754 +0000 UTC m=+2491.174805879"
Jul 29 19:58:44.927723 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:44.927691    2583 generic.go:356] "Generic (PLEG): container finished" podID="51af2303-bd3d-43d0-a58f-b35082df8f76" containerID="962ee9e0db7aac4486f5fbe93d1f20fd78798862e9153e503d04e08c9b5107ac" exitCode=0
Jul 29 19:58:44.928109 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:44.927775    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0e0fe5-wj2sw" event={"ID":"51af2303-bd3d-43d0-a58f-b35082df8f76","Type":"ContainerDied","Data":"962ee9e0db7aac4486f5fbe93d1f20fd78798862e9153e503d04e08c9b5107ac"}
Jul 29 19:58:46.070428 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:46.070391    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0e0fe5-wj2sw"
Jul 29 19:58:46.107760 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:46.107723    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51af2303-bd3d-43d0-a58f-b35082df8f76-maas-api-service-ca\") pod \"51af2303-bd3d-43d0-a58f-b35082df8f76\" (UID: \"51af2303-bd3d-43d0-a58f-b35082df8f76\") "
Jul 29 19:58:46.108157 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:46.108137    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51af2303-bd3d-43d0-a58f-b35082df8f76-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "51af2303-bd3d-43d0-a58f-b35082df8f76" (UID: "51af2303-bd3d-43d0-a58f-b35082df8f76"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:58:46.209107 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:46.209026    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51af2303-bd3d-43d0-a58f-b35082df8f76-maas-api-service-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:58:46.938514 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:46.938479    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0e0fe5-wj2sw"
Jul 29 19:58:46.938514 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:46.938488    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0e0fe5-wj2sw" event={"ID":"51af2303-bd3d-43d0-a58f-b35082df8f76","Type":"ContainerDied","Data":"1135d2042fe4b76e95daf91b54aeac31305770c4b5e91ec346720a5dd0ac8f8d"}
Jul 29 19:58:46.938774 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:46.938544    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1135d2042fe4b76e95daf91b54aeac31305770c4b5e91ec346720a5dd0ac8f8d"
Jul 29 19:58:58.213199 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:58.213169    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0e0fe5-54fc8cc669-rlfdj"]
Jul 29 19:58:58.213871 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:58.213437    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0fe5-54fc8cc669-rlfdj" podUID="2f344c67-12ab-493f-8d79-4acbd80d09d2" containerName="maas-api" containerID="cri-o://f70f532aea8871f0ff41dac787ac7b698b41e496b801448555752960249bd6c9" gracePeriod=30
Jul 29 19:58:58.415973 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:58.415938    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0e0fe5-wj2sw"]
Jul 29 19:58:58.417436 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:58.417413    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0e0fe5-wj2sw"]
Jul 29 19:58:58.456326 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:58.456303    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0fe5-54fc8cc669-rlfdj"
Jul 29 19:58:58.519528 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:58.519459    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f344c67-12ab-493f-8d79-4acbd80d09d2-openshift-service-ca\") pod \"2f344c67-12ab-493f-8d79-4acbd80d09d2\" (UID: \"2f344c67-12ab-493f-8d79-4acbd80d09d2\") "
Jul 29 19:58:58.519528 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:58.519490    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f344c67-12ab-493f-8d79-4acbd80d09d2-maas-api-tls\") pod \"2f344c67-12ab-493f-8d79-4acbd80d09d2\" (UID: \"2f344c67-12ab-493f-8d79-4acbd80d09d2\") "
Jul 29 19:58:58.519732 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:58.519539    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpwm8\" (UniqueName: \"kubernetes.io/projected/2f344c67-12ab-493f-8d79-4acbd80d09d2-kube-api-access-gpwm8\") pod \"2f344c67-12ab-493f-8d79-4acbd80d09d2\" (UID: \"2f344c67-12ab-493f-8d79-4acbd80d09d2\") "
Jul 29 19:58:58.519876 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:58.519857    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f344c67-12ab-493f-8d79-4acbd80d09d2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2f344c67-12ab-493f-8d79-4acbd80d09d2" (UID: "2f344c67-12ab-493f-8d79-4acbd80d09d2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:58:58.521497 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:58.521478    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f344c67-12ab-493f-8d79-4acbd80d09d2-kube-api-access-gpwm8" (OuterVolumeSpecName: "kube-api-access-gpwm8") pod "2f344c67-12ab-493f-8d79-4acbd80d09d2" (UID: "2f344c67-12ab-493f-8d79-4acbd80d09d2"). InnerVolumeSpecName "kube-api-access-gpwm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:58:58.521603 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:58.521509    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f344c67-12ab-493f-8d79-4acbd80d09d2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2f344c67-12ab-493f-8d79-4acbd80d09d2" (UID: "2f344c67-12ab-493f-8d79-4acbd80d09d2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:58:58.620258 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:58.620235    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f344c67-12ab-493f-8d79-4acbd80d09d2-openshift-service-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:58:58.620258 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:58.620258    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f344c67-12ab-493f-8d79-4acbd80d09d2-maas-api-tls\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:58:58.620403 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:58.620269    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpwm8\" (UniqueName: \"kubernetes.io/projected/2f344c67-12ab-493f-8d79-4acbd80d09d2-kube-api-access-gpwm8\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 19:58:59.000948 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:59.000903    2583 generic.go:356] "Generic (PLEG): container finished" podID="2f344c67-12ab-493f-8d79-4acbd80d09d2" containerID="f70f532aea8871f0ff41dac787ac7b698b41e496b801448555752960249bd6c9" exitCode=0
Jul 29 19:58:59.000948 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:59.000946    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0fe5-54fc8cc669-rlfdj"
Jul 29 19:58:59.001167 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:59.000983    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0fe5-54fc8cc669-rlfdj" event={"ID":"2f344c67-12ab-493f-8d79-4acbd80d09d2","Type":"ContainerDied","Data":"f70f532aea8871f0ff41dac787ac7b698b41e496b801448555752960249bd6c9"}
Jul 29 19:58:59.001167 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:59.001047    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0fe5-54fc8cc669-rlfdj" event={"ID":"2f344c67-12ab-493f-8d79-4acbd80d09d2","Type":"ContainerDied","Data":"d9b33658b293af3de05e6f1e87c1367116d97993a8264a561d026fecef190e84"}
Jul 29 19:58:59.001167 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:59.001077    2583 scope.go:117] "RemoveContainer" containerID="f70f532aea8871f0ff41dac787ac7b698b41e496b801448555752960249bd6c9"
Jul 29 19:58:59.010533 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:59.010517    2583 scope.go:117] "RemoveContainer" containerID="f70f532aea8871f0ff41dac787ac7b698b41e496b801448555752960249bd6c9"
Jul 29 19:58:59.010826 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:58:59.010783    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f70f532aea8871f0ff41dac787ac7b698b41e496b801448555752960249bd6c9\": container with ID starting with f70f532aea8871f0ff41dac787ac7b698b41e496b801448555752960249bd6c9 not found: ID does not exist" containerID="f70f532aea8871f0ff41dac787ac7b698b41e496b801448555752960249bd6c9"
Jul 29 19:58:59.010882 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:59.010835    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f70f532aea8871f0ff41dac787ac7b698b41e496b801448555752960249bd6c9"} err="failed to get container status \"f70f532aea8871f0ff41dac787ac7b698b41e496b801448555752960249bd6c9\": rpc error: code = NotFound desc = could not find container \"f70f532aea8871f0ff41dac787ac7b698b41e496b801448555752960249bd6c9\": container with ID starting with f70f532aea8871f0ff41dac787ac7b698b41e496b801448555752960249bd6c9 not found: ID does not exist"
Jul 29 19:58:59.021612 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:59.021591    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0e0fe5-54fc8cc669-rlfdj"]
Jul 29 19:58:59.025232 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:59.025213    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0e0fe5-54fc8cc669-rlfdj"]
Jul 29 19:58:59.345504 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:59.345426    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f344c67-12ab-493f-8d79-4acbd80d09d2" path="/var/lib/kubelet/pods/2f344c67-12ab-493f-8d79-4acbd80d09d2/volumes"
Jul 29 19:58:59.345849 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:58:59.345779    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51af2303-bd3d-43d0-a58f-b35082df8f76" path="/var/lib/kubelet/pods/51af2303-bd3d-43d0-a58f-b35082df8f76/volumes"
Jul 29 19:59:02.703333 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:02.703296    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:59:09.393465 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:09.393383    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a60880-75cbcb67fc-mwsfk"]
Jul 29 19:59:09.393836 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:09.393660    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f344c67-12ab-493f-8d79-4acbd80d09d2" containerName="maas-api"
Jul 29 19:59:09.393836 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:09.393671    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f344c67-12ab-493f-8d79-4acbd80d09d2" containerName="maas-api"
Jul 29 19:59:09.393836 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:09.393685    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51af2303-bd3d-43d0-a58f-b35082df8f76" containerName="revoke-keys"
Jul 29 19:59:09.393836 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:09.393691    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="51af2303-bd3d-43d0-a58f-b35082df8f76" containerName="revoke-keys"
Jul 29 19:59:09.393836 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:09.393739    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f344c67-12ab-493f-8d79-4acbd80d09d2" containerName="maas-api"
Jul 29 19:59:09.393836 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:09.393746    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="51af2303-bd3d-43d0-a58f-b35082df8f76" containerName="revoke-keys"
Jul 29 19:59:09.397838 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:09.397818    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a60880-75cbcb67fc-mwsfk"
Jul 29 19:59:09.399838 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:09.399792    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 19:59:09.399838 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:09.399825    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6wzbz\""
Jul 29 19:59:09.399965 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:09.399926    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-a60880-serving-cert\""
Jul 29 19:59:09.400339 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:09.400326    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 29 19:59:09.408089 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:09.408065    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a60880-75cbcb67fc-mwsfk"]
Jul 29 19:59:09.508555 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:09.508521    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cmjj\" (UniqueName: \"kubernetes.io/projected/c3645361-80d7-4487-a79d-033a296e9016-kube-api-access-7cmjj\") pod \"maas-api-e2e-shared-a-a60880-75cbcb67fc-mwsfk\" (UID: \"c3645361-80d7-4487-a79d-033a296e9016\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a60880-75cbcb67fc-mwsfk"
Jul 29 19:59:09.508706 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:09.508588    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3645361-80d7-4487-a79d-033a296e9016-maas-api-tls\") pod \"maas-api-e2e-shared-a-a60880-75cbcb67fc-mwsfk\" (UID: \"c3645361-80d7-4487-a79d-033a296e9016\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a60880-75cbcb67fc-mwsfk"
Jul 29 19:59:09.508754 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:09.508709    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3645361-80d7-4487-a79d-033a296e9016-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a60880-75cbcb67fc-mwsfk\" (UID: \"c3645361-80d7-4487-a79d-033a296e9016\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a60880-75cbcb67fc-mwsfk"
Jul 29 19:59:09.610116 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:09.610082    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3645361-80d7-4487-a79d-033a296e9016-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a60880-75cbcb67fc-mwsfk\" (UID: \"c3645361-80d7-4487-a79d-033a296e9016\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a60880-75cbcb67fc-mwsfk"
Jul 29 19:59:09.610255 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:09.610133    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cmjj\" (UniqueName: \"kubernetes.io/projected/c3645361-80d7-4487-a79d-033a296e9016-kube-api-access-7cmjj\") pod \"maas-api-e2e-shared-a-a60880-75cbcb67fc-mwsfk\" (UID: \"c3645361-80d7-4487-a79d-033a296e9016\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a60880-75cbcb67fc-mwsfk"
Jul 29 19:59:09.610255 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:09.610169    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3645361-80d7-4487-a79d-033a296e9016-maas-api-tls\") pod \"maas-api-e2e-shared-a-a60880-75cbcb67fc-mwsfk\" (UID: \"c3645361-80d7-4487-a79d-033a296e9016\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a60880-75cbcb67fc-mwsfk"
Jul 29 19:59:09.610331 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:59:09.610253    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-a60880-serving-cert: secret "maas-api-e2e-shared-a-a60880-serving-cert" not found
Jul 29 19:59:09.610331 ip-10-0-128-101 kubenswrapper[2583]: E0729 19:59:09.610312    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3645361-80d7-4487-a79d-033a296e9016-maas-api-tls podName:c3645361-80d7-4487-a79d-033a296e9016 nodeName:}" failed. No retries permitted until 2026-07-29 19:59:10.110296525 +0000 UTC m=+2517.338685630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c3645361-80d7-4487-a79d-033a296e9016-maas-api-tls") pod "maas-api-e2e-shared-a-a60880-75cbcb67fc-mwsfk" (UID: "c3645361-80d7-4487-a79d-033a296e9016") : secret "maas-api-e2e-shared-a-a60880-serving-cert" not found
Jul 29 19:59:09.610677 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:09.610659    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3645361-80d7-4487-a79d-033a296e9016-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a60880-75cbcb67fc-mwsfk\" (UID: \"c3645361-80d7-4487-a79d-033a296e9016\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a60880-75cbcb67fc-mwsfk"
Jul 29 19:59:09.617860 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:09.617842    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cmjj\" (UniqueName: \"kubernetes.io/projected/c3645361-80d7-4487-a79d-033a296e9016-kube-api-access-7cmjj\") pod \"maas-api-e2e-shared-a-a60880-75cbcb67fc-mwsfk\" (UID: \"c3645361-80d7-4487-a79d-033a296e9016\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a60880-75cbcb67fc-mwsfk"
Jul 29 19:59:10.034597 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:10.034565    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-a60880-7846f4db76-nc6j5"]
Jul 29 19:59:10.037922 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:10.037907    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a60880-7846f4db76-nc6j5"
Jul 29 19:59:10.039956 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:10.039933    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-a60880-dockercfg-tsnhl\""
Jul 29 19:59:10.039956 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:10.039954    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-a60880\""
Jul 29 19:59:10.046222 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:10.046201    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-a60880-7846f4db76-nc6j5"]
Jul 29 19:59:10.056091 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:10.056070    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-a60880-6c5fff5886-mcrkd"]
Jul 29 19:59:10.059078 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:10.059064    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-a60880-6c5fff5886-mcrkd"
Jul 29 19:59:10.066922 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:10.066902    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-a60880-6c5fff5886-mcrkd"]
Jul 29 19:59:10.113664 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:10.113635    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3645361-80d7-4487-a79d-033a296e9016-maas-api-tls\") pod \"maas-api-e2e-shared-a-a60880-75cbcb67fc-mwsfk\" (UID: \"c3645361-80d7-4487-a79d-033a296e9016\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a60880-75cbcb67fc-mwsfk"
Jul 29 19:59:10.116108 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:10.116092    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3645361-80d7-4487-a79d-033a296e9016-maas-api-tls\") pod \"maas-api-e2e-shared-a-a60880-75cbcb67fc-mwsfk\" (UID: \"c3645361-80d7-4487-a79d-033a296e9016\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a60880-75cbcb67fc-mwsfk"
Jul 29 19:59:10.214943 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:10.214911    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb22fa7-1bf4-46c4-b936-9f5941caacc7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-a60880-6c5fff5886-mcrkd\" (UID: \"5eb22fa7-1bf4-46c4-b936-9f5941caacc7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-a60880-6c5fff5886-mcrkd"
Jul 29 19:59:10.214943 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:10.214949    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d7dv\" (UniqueName: \"kubernetes.io/projected/5eb22fa7-1bf4-46c4-b936-9f5941caacc7-kube-api-access-8d7dv\") pod \"payload-processing-e2e-shared-a-a60880-6c5fff5886-mcrkd\" (UID: \"5eb22fa7-1bf4-46c4-b936-9f5941caacc7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-a60880-6c5fff5886-mcrkd"
Jul 29 19:59:10.215138 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:10.214982    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8ba63e5-214a-4485-9174-43b4a7dd6f0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-a60880-7846f4db76-nc6j5\" (UID: \"c8ba63e5-214a-4485-9174-43b4a7dd6f0d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a60880-7846f4db76-nc6j5"
Jul 29 19:59:10.215138 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:10.215000    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftzlv\" (UniqueName: \"kubernetes.io/projected/c8ba63e5-214a-4485-9174-43b4a7dd6f0d-kube-api-access-ftzlv\") pod \"payload-pre-processing-e2e-shared-a-a60880-7846f4db76-nc6j5\" (UID: \"c8ba63e5-214a-4485-9174-43b4a7dd6f0d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a60880-7846f4db76-nc6j5"
Jul 29 19:59:10.308113 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:10.308042    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a60880-75cbcb67fc-mwsfk"
Jul 29 19:59:10.316231 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:10.316201    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb22fa7-1bf4-46c4-b936-9f5941caacc7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-a60880-6c5fff5886-mcrkd\" (UID: \"5eb22fa7-1bf4-46c4-b936-9f5941caacc7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-a60880-6c5fff5886-mcrkd"
Jul 29 19:59:10.316333 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:10.316251    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d7dv\" (UniqueName: \"kubernetes.io/projected/5eb22fa7-1bf4-46c4-b936-9f5941caacc7-kube-api-access-8d7dv\") pod \"payload-processing-e2e-shared-a-a60880-6c5fff5886-mcrkd\" (UID: \"5eb22fa7-1bf4-46c4-b936-9f5941caacc7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-a60880-6c5fff5886-mcrkd"
Jul 29 19:59:10.316333 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:10.316301    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8ba63e5-214a-4485-9174-43b4a7dd6f0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-a60880-7846f4db76-nc6j5\" (UID: \"c8ba63e5-214a-4485-9174-43b4a7dd6f0d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a60880-7846f4db76-nc6j5"
Jul 29 19:59:10.316333 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:10.316328    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftzlv\" (UniqueName: \"kubernetes.io/projected/c8ba63e5-214a-4485-9174-43b4a7dd6f0d-kube-api-access-ftzlv\") pod \"payload-pre-processing-e2e-shared-a-a60880-7846f4db76-nc6j5\" (UID: \"c8ba63e5-214a-4485-9174-43b4a7dd6f0d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a60880-7846f4db76-nc6j5"
Jul 29 19:59:10.317009 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:10.316988    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8ba63e5-214a-4485-9174-43b4a7dd6f0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-a60880-7846f4db76-nc6j5\" (UID: \"c8ba63e5-214a-4485-9174-43b4a7dd6f0d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a60880-7846f4db76-nc6j5"
Jul 29 19:59:10.317092 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:10.317072    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb22fa7-1bf4-46c4-b936-9f5941caacc7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-a60880-6c5fff5886-mcrkd\" (UID: \"5eb22fa7-1bf4-46c4-b936-9f5941caacc7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-a60880-6c5fff5886-mcrkd"
Jul 29 19:59:10.323926 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:10.323907    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftzlv\" (UniqueName: \"kubernetes.io/projected/c8ba63e5-214a-4485-9174-43b4a7dd6f0d-kube-api-access-ftzlv\") pod \"payload-pre-processing-e2e-shared-a-a60880-7846f4db76-nc6j5\" (UID: \"c8ba63e5-214a-4485-9174-43b4a7dd6f0d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a60880-7846f4db76-nc6j5"
Jul 29 19:59:10.324384 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:10.324361    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d7dv\" (UniqueName: \"kubernetes.io/projected/5eb22fa7-1bf4-46c4-b936-9f5941caacc7-kube-api-access-8d7dv\") pod \"payload-processing-e2e-shared-a-a60880-6c5fff5886-mcrkd\" (UID: \"5eb22fa7-1bf4-46c4-b936-9f5941caacc7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-a60880-6c5fff5886-mcrkd"
Jul 29 19:59:10.349463 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:10.349439    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a60880-7846f4db76-nc6j5"
Jul 29 19:59:10.369835 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:10.369779    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-a60880-6c5fff5886-mcrkd"
Jul 29 19:59:10.454829 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:10.454739    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a60880-75cbcb67fc-mwsfk"]
Jul 29 19:59:10.459413 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:59:10.459372    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3645361_80d7_4487_a79d_033a296e9016.slice/crio-db31c760436788e2c56c4f7c86f97b6cc79bb3ad60642b502d59f8630729ffd4 WatchSource:0}: Error finding container db31c760436788e2c56c4f7c86f97b6cc79bb3ad60642b502d59f8630729ffd4: Status 404 returned error can't find the container with id db31c760436788e2c56c4f7c86f97b6cc79bb3ad60642b502d59f8630729ffd4
Jul 29 19:59:10.507102 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:10.507072    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-a60880-7846f4db76-nc6j5"]
Jul 29 19:59:10.509017 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:59:10.508991    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8ba63e5_214a_4485_9174_43b4a7dd6f0d.slice/crio-eefa3f9e57f018a108c645ebbfb3b13f0e63a3ec6b2970d51738c5f2b64e9d6a WatchSource:0}: Error finding container eefa3f9e57f018a108c645ebbfb3b13f0e63a3ec6b2970d51738c5f2b64e9d6a: Status 404 returned error can't find the container with id eefa3f9e57f018a108c645ebbfb3b13f0e63a3ec6b2970d51738c5f2b64e9d6a
Jul 29 19:59:10.731387 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:10.731364    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-a60880-6c5fff5886-mcrkd"]
Jul 29 19:59:10.734058 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:59:10.734032    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eb22fa7_1bf4_46c4_b936_9f5941caacc7.slice/crio-d6b097e9640cd71e816cd9d54cd72e7623208c3aaeee08486df56f534dafe02b WatchSource:0}: Error finding container d6b097e9640cd71e816cd9d54cd72e7623208c3aaeee08486df56f534dafe02b: Status 404 returned error can't find the container with id d6b097e9640cd71e816cd9d54cd72e7623208c3aaeee08486df56f534dafe02b
Jul 29 19:59:11.055023 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:11.054994    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a60880-75cbcb67fc-mwsfk" event={"ID":"c3645361-80d7-4487-a79d-033a296e9016","Type":"ContainerStarted","Data":"28da31b7d30400eaca61ec92d4b2dac8a4e475f9ce0432fde57177767499a038"}
Jul 29 19:59:11.055163 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:11.055051    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a60880-75cbcb67fc-mwsfk" event={"ID":"c3645361-80d7-4487-a79d-033a296e9016","Type":"ContainerStarted","Data":"db31c760436788e2c56c4f7c86f97b6cc79bb3ad60642b502d59f8630729ffd4"}
Jul 29 19:59:11.055163 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:11.055080    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a60880-75cbcb67fc-mwsfk"
Jul 29 19:59:11.056481 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:11.056458    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a60880-7846f4db76-nc6j5" event={"ID":"c8ba63e5-214a-4485-9174-43b4a7dd6f0d","Type":"ContainerStarted","Data":"adc93ba0a19ae1339c989111d38f06f67702ae8d65a829105043d7bf1140ee52"}
Jul 29 19:59:11.056578 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:11.056501    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a60880-7846f4db76-nc6j5" event={"ID":"c8ba63e5-214a-4485-9174-43b4a7dd6f0d","Type":"ContainerStarted","Data":"eefa3f9e57f018a108c645ebbfb3b13f0e63a3ec6b2970d51738c5f2b64e9d6a"}
Jul 29 19:59:11.056578 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:11.056531    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a60880-7846f4db76-nc6j5"
Jul 29 19:59:11.058024 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:11.058003    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-a60880-6c5fff5886-mcrkd" event={"ID":"5eb22fa7-1bf4-46c4-b936-9f5941caacc7","Type":"ContainerStarted","Data":"02e1f7c1f89075c8df9c8791e8a3567ab45b3d873b54c41d9c1a8ea882065cce"}
Jul 29 19:59:11.058116 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:11.058046    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-a60880-6c5fff5886-mcrkd" event={"ID":"5eb22fa7-1bf4-46c4-b936-9f5941caacc7","Type":"ContainerStarted","Data":"d6b097e9640cd71e816cd9d54cd72e7623208c3aaeee08486df56f534dafe02b"}
Jul 29 19:59:11.058116 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:11.058074    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-a60880-6c5fff5886-mcrkd"
Jul 29 19:59:11.086831 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:11.086770    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a60880-75cbcb67fc-mwsfk" podStartSLOduration=1.5780710610000002 podStartE2EDuration="2.086755406s" totalImagesPullingTime="508.684345ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:59:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:59:11.084756052 +0000 UTC m=+2518.313145191" watchObservedRunningTime="2026-07-29 19:59:11.086755406 +0000 UTC m=+2518.315144532"
Jul 29 19:59:11.112161 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:11.108973    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-a60880-6c5fff5886-mcrkd" podStartSLOduration=1.108956055 podStartE2EDuration="1.108956055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:59:11.102508216 +0000 UTC m=+2518.330897342" watchObservedRunningTime="2026-07-29 19:59:11.108956055 +0000 UTC m=+2518.337345182"
Jul 29 19:59:11.121835 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:11.121767    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a60880-7846f4db76-nc6j5" podStartSLOduration=1.121751154 podStartE2EDuration="1.121751154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:59:11.12115877 +0000 UTC m=+2518.349547922" watchObservedRunningTime="2026-07-29 19:59:11.121751154 +0000 UTC m=+2518.350140282"
Jul 29 19:59:13.397939 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:13.397907    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:59:17.069156 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:17.069128    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a60880-75cbcb67fc-mwsfk"
Jul 29 19:59:22.065440 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:22.065407    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-a60880-6c5fff5886-mcrkd"
Jul 29 19:59:22.065904 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:22.065493    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a60880-7846f4db76-nc6j5"
Jul 29 19:59:37.930134 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:37.930098    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-601aa9-74fdb4b86c-vdmcl"]
Jul 29 19:59:37.933433 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:37.933413    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-601aa9-74fdb4b86c-vdmcl"
Jul 29 19:59:37.935996 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:37.935967    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-601aa9-serving-cert\""
Jul 29 19:59:37.952011 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:37.951986    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-601aa9-74fdb4b86c-vdmcl"]
Jul 29 19:59:38.029030 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:38.029001    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvqk6\" (UniqueName: \"kubernetes.io/projected/9d879ed6-e8a6-4784-9e23-74fdd10eb93a-kube-api-access-vvqk6\") pod \"maas-api-e2e-shared-b-601aa9-74fdb4b86c-vdmcl\" (UID: \"9d879ed6-e8a6-4784-9e23-74fdd10eb93a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-601aa9-74fdb4b86c-vdmcl"
Jul 29 19:59:38.029174 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:38.029036    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d879ed6-e8a6-4784-9e23-74fdd10eb93a-maas-api-tls\") pod \"maas-api-e2e-shared-b-601aa9-74fdb4b86c-vdmcl\" (UID: \"9d879ed6-e8a6-4784-9e23-74fdd10eb93a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-601aa9-74fdb4b86c-vdmcl"
Jul 29 19:59:38.029174 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:38.029053    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d879ed6-e8a6-4784-9e23-74fdd10eb93a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-601aa9-74fdb4b86c-vdmcl\" (UID: \"9d879ed6-e8a6-4784-9e23-74fdd10eb93a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-601aa9-74fdb4b86c-vdmcl"
Jul 29 19:59:38.129862 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:38.129827    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvqk6\" (UniqueName: \"kubernetes.io/projected/9d879ed6-e8a6-4784-9e23-74fdd10eb93a-kube-api-access-vvqk6\") pod \"maas-api-e2e-shared-b-601aa9-74fdb4b86c-vdmcl\" (UID: \"9d879ed6-e8a6-4784-9e23-74fdd10eb93a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-601aa9-74fdb4b86c-vdmcl"
Jul 29 19:59:38.130052 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:38.129876    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d879ed6-e8a6-4784-9e23-74fdd10eb93a-maas-api-tls\") pod \"maas-api-e2e-shared-b-601aa9-74fdb4b86c-vdmcl\" (UID: \"9d879ed6-e8a6-4784-9e23-74fdd10eb93a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-601aa9-74fdb4b86c-vdmcl"
Jul 29 19:59:38.130052 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:38.129901    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d879ed6-e8a6-4784-9e23-74fdd10eb93a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-601aa9-74fdb4b86c-vdmcl\" (UID: \"9d879ed6-e8a6-4784-9e23-74fdd10eb93a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-601aa9-74fdb4b86c-vdmcl"
Jul 29 19:59:38.130565 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:38.130541    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d879ed6-e8a6-4784-9e23-74fdd10eb93a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-601aa9-74fdb4b86c-vdmcl\" (UID: \"9d879ed6-e8a6-4784-9e23-74fdd10eb93a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-601aa9-74fdb4b86c-vdmcl"
Jul 29 19:59:38.132416 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:38.132394    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d879ed6-e8a6-4784-9e23-74fdd10eb93a-maas-api-tls\") pod \"maas-api-e2e-shared-b-601aa9-74fdb4b86c-vdmcl\" (UID: \"9d879ed6-e8a6-4784-9e23-74fdd10eb93a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-601aa9-74fdb4b86c-vdmcl"
Jul 29 19:59:38.137296 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:38.137275    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvqk6\" (UniqueName: \"kubernetes.io/projected/9d879ed6-e8a6-4784-9e23-74fdd10eb93a-kube-api-access-vvqk6\") pod \"maas-api-e2e-shared-b-601aa9-74fdb4b86c-vdmcl\" (UID: \"9d879ed6-e8a6-4784-9e23-74fdd10eb93a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-601aa9-74fdb4b86c-vdmcl"
Jul 29 19:59:38.243396 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:38.243309    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-601aa9-74fdb4b86c-vdmcl"
Jul 29 19:59:38.577700 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:38.577523    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-601aa9-74fdb4b86c-vdmcl"]
Jul 29 19:59:38.580423 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:59:38.580397    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d879ed6_e8a6_4784_9e23_74fdd10eb93a.slice/crio-daba1786d1b64b2bedd3de96a3d9ef9a0f68ec0ffdab34380156bc21dffbd147 WatchSource:0}: Error finding container daba1786d1b64b2bedd3de96a3d9ef9a0f68ec0ffdab34380156bc21dffbd147: Status 404 returned error can't find the container with id daba1786d1b64b2bedd3de96a3d9ef9a0f68ec0ffdab34380156bc21dffbd147
Jul 29 19:59:38.625697 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:38.625674    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-601aa9-685f844dfd-fljl2"]
Jul 29 19:59:38.630085 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:38.630066    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-601aa9-685f844dfd-fljl2"
Jul 29 19:59:38.632053 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:38.632038    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-601aa9\""
Jul 29 19:59:38.632141 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:38.632096    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-601aa9-dockercfg-rr8bw\""
Jul 29 19:59:38.637915 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:38.637892    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-601aa9-685f844dfd-fljl2"]
Jul 29 19:59:38.740881 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:38.740848    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9x9n\" (UniqueName: \"kubernetes.io/projected/ba56ecab-018c-4f0f-a801-0141f5995f15-kube-api-access-p9x9n\") pod \"payload-processing-e2e-shared-b-601aa9-685f844dfd-fljl2\" (UID: \"ba56ecab-018c-4f0f-a801-0141f5995f15\") " pod="openshift-ingress/payload-processing-e2e-shared-b-601aa9-685f844dfd-fljl2"
Jul 29 19:59:38.741035 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:38.740890    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba56ecab-018c-4f0f-a801-0141f5995f15-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-601aa9-685f844dfd-fljl2\" (UID: \"ba56ecab-018c-4f0f-a801-0141f5995f15\") " pod="openshift-ingress/payload-processing-e2e-shared-b-601aa9-685f844dfd-fljl2"
Jul 29 19:59:38.841835 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:38.841733    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9x9n\" (UniqueName: \"kubernetes.io/projected/ba56ecab-018c-4f0f-a801-0141f5995f15-kube-api-access-p9x9n\") pod \"payload-processing-e2e-shared-b-601aa9-685f844dfd-fljl2\" (UID: \"ba56ecab-018c-4f0f-a801-0141f5995f15\") " pod="openshift-ingress/payload-processing-e2e-shared-b-601aa9-685f844dfd-fljl2"
Jul 29 19:59:38.841835 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:38.841776    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba56ecab-018c-4f0f-a801-0141f5995f15-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-601aa9-685f844dfd-fljl2\" (UID: \"ba56ecab-018c-4f0f-a801-0141f5995f15\") " pod="openshift-ingress/payload-processing-e2e-shared-b-601aa9-685f844dfd-fljl2"
Jul 29 19:59:38.842409 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:38.842388    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba56ecab-018c-4f0f-a801-0141f5995f15-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-601aa9-685f844dfd-fljl2\" (UID: \"ba56ecab-018c-4f0f-a801-0141f5995f15\") " pod="openshift-ingress/payload-processing-e2e-shared-b-601aa9-685f844dfd-fljl2"
Jul 29 19:59:38.852158 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:38.852136    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9x9n\" (UniqueName: \"kubernetes.io/projected/ba56ecab-018c-4f0f-a801-0141f5995f15-kube-api-access-p9x9n\") pod \"payload-processing-e2e-shared-b-601aa9-685f844dfd-fljl2\" (UID: \"ba56ecab-018c-4f0f-a801-0141f5995f15\") " pod="openshift-ingress/payload-processing-e2e-shared-b-601aa9-685f844dfd-fljl2"
Jul 29 19:59:38.942174 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:38.942148    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-601aa9-685f844dfd-fljl2"
Jul 29 19:59:39.070985 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:39.070961    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-601aa9-685f844dfd-fljl2"]
Jul 29 19:59:39.072977 ip-10-0-128-101 kubenswrapper[2583]: W0729 19:59:39.072949    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba56ecab_018c_4f0f_a801_0141f5995f15.slice/crio-02f445a85680c690e3189d6bfc1246eec7a7b5ad7b47a1169436739e17bad83b WatchSource:0}: Error finding container 02f445a85680c690e3189d6bfc1246eec7a7b5ad7b47a1169436739e17bad83b: Status 404 returned error can't find the container with id 02f445a85680c690e3189d6bfc1246eec7a7b5ad7b47a1169436739e17bad83b
Jul 29 19:59:39.191178 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:39.191143    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-601aa9-685f844dfd-fljl2" event={"ID":"ba56ecab-018c-4f0f-a801-0141f5995f15","Type":"ContainerStarted","Data":"885fc74d8681a6862ec118d5099081f7c0c199a389f34b29ad269d7e43bde617"}
Jul 29 19:59:39.191312 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:39.191195    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-601aa9-685f844dfd-fljl2" event={"ID":"ba56ecab-018c-4f0f-a801-0141f5995f15","Type":"ContainerStarted","Data":"02f445a85680c690e3189d6bfc1246eec7a7b5ad7b47a1169436739e17bad83b"}
Jul 29 19:59:39.191312 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:39.191249    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-601aa9-685f844dfd-fljl2"
Jul 29 19:59:39.192634 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:39.192610    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-601aa9-74fdb4b86c-vdmcl" event={"ID":"9d879ed6-e8a6-4784-9e23-74fdd10eb93a","Type":"ContainerStarted","Data":"d92940c50ed5b744a0be300cfb6a47353c7a11b2dd9da44a208eef626c3a1ece"}
Jul 29 19:59:39.192845 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:39.192649    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-601aa9-74fdb4b86c-vdmcl" event={"ID":"9d879ed6-e8a6-4784-9e23-74fdd10eb93a","Type":"ContainerStarted","Data":"daba1786d1b64b2bedd3de96a3d9ef9a0f68ec0ffdab34380156bc21dffbd147"}
Jul 29 19:59:39.192845 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:39.192669    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-601aa9-74fdb4b86c-vdmcl"
Jul 29 19:59:39.207789 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:39.207705    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-601aa9-685f844dfd-fljl2" podStartSLOduration=1.20768943 podStartE2EDuration="1.20768943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:59:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:59:39.206830215 +0000 UTC m=+2546.435219338" watchObservedRunningTime="2026-07-29 19:59:39.20768943 +0000 UTC m=+2546.436078553"
Jul 29 19:59:39.223689 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:39.223647    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-601aa9-74fdb4b86c-vdmcl" podStartSLOduration=1.8263981139999999 podStartE2EDuration="2.223632231s" totalImagesPullingTime="397.234117ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:59:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:59:39.222208776 +0000 UTC m=+2546.450597904" watchObservedRunningTime="2026-07-29 19:59:39.223632231 +0000 UTC m=+2546.452021392"
Jul 29 19:59:45.202836 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:45.202791    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-601aa9-74fdb4b86c-vdmcl"
Jul 29 19:59:45.493925 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:45.493840    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 19:59:50.199965 ip-10-0-128-101 kubenswrapper[2583]: I0729 19:59:50.199933    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-601aa9-685f844dfd-fljl2"
Jul 29 20:00:00.142780 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:00.142746    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755920-4f7k8"]
Jul 29 20:00:00.145960 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:00.145936    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a60880-29755920-zbzss"]
Jul 29 20:00:00.146116 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:00.146071    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755920-4f7k8"
Jul 29 20:00:00.148266 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:00.148243    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hq9l2\""
Jul 29 20:00:00.149178 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:00.149159    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-601aa9-29755920-dlwgt"]
Jul 29 20:00:00.149294 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:00.149279    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a60880-29755920-zbzss"
Jul 29 20:00:00.152465 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:00.152446    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-601aa9-29755920-dlwgt"
Jul 29 20:00:00.158200 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:00.158174    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755920-4f7k8"
Jul 29 20:00:00.159272 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:00.159246    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a60880-29755920-zbzss"]
Jul 29 20:00:00.162252 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:00.162221    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755920-4f7k8"]
Jul 29 20:00:00.164772 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:00.164750    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-601aa9-29755920-dlwgt"]
Jul 29 20:00:00.170984 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:00.170959    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a60880-29755920-zbzss"
Jul 29 20:00:00.205956 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:00.205928    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-601aa9-29755920-dlwgt"
Jul 29 20:00:00.351071 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:00.351027    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755920-4f7k8"]
Jul 29 20:00:00.355682 ip-10-0-128-101 kubenswrapper[2583]: W0729 20:00:00.355656    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51c05aa5_0bb1_40eb_bf19_e85668da7db1.slice/crio-06464358a81368cef23d5faae3c992f181eaea7905355631f593a9ba66de5d2c WatchSource:0}: Error finding container 06464358a81368cef23d5faae3c992f181eaea7905355631f593a9ba66de5d2c: Status 404 returned error can't find the container with id 06464358a81368cef23d5faae3c992f181eaea7905355631f593a9ba66de5d2c
Jul 29 20:00:00.399492 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:00.399470    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-601aa9-29755920-dlwgt"]
Jul 29 20:00:00.402340 ip-10-0-128-101 kubenswrapper[2583]: W0729 20:00:00.402316    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dd467c4_e5cc_405a_8c3f_6448abda4632.slice/crio-8f495533fe29d384382c02aa7cb2230aec79448c3cea2d9c27ea49739d5cd106 WatchSource:0}: Error finding container 8f495533fe29d384382c02aa7cb2230aec79448c3cea2d9c27ea49739d5cd106: Status 404 returned error can't find the container with id 8f495533fe29d384382c02aa7cb2230aec79448c3cea2d9c27ea49739d5cd106
Jul 29 20:00:00.599084 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:00.599060    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a60880-29755920-zbzss"]
Jul 29 20:00:00.601140 ip-10-0-128-101 kubenswrapper[2583]: W0729 20:00:00.601103    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eb30076_3571_46d5_a81f_d792cdc4b6c6.slice/crio-a4bf6095e162fac51f6a74c9b33b5dc6e9a91fedb806a350c4c301162270e277 WatchSource:0}: Error finding container a4bf6095e162fac51f6a74c9b33b5dc6e9a91fedb806a350c4c301162270e277: Status 404 returned error can't find the container with id a4bf6095e162fac51f6a74c9b33b5dc6e9a91fedb806a350c4c301162270e277
Jul 29 20:00:01.309868 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:01.309832    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a60880-29755920-zbzss" event={"ID":"4eb30076-3571-46d5-a81f-d792cdc4b6c6","Type":"ContainerStarted","Data":"3753054139f31640adeafa07f3e9487cb7957bdb0d3c9d35452c90c8bb17d734"}
Jul 29 20:00:01.310324 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:01.309936    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a60880-29755920-zbzss" event={"ID":"4eb30076-3571-46d5-a81f-d792cdc4b6c6","Type":"ContainerStarted","Data":"a4bf6095e162fac51f6a74c9b33b5dc6e9a91fedb806a350c4c301162270e277"}
Jul 29 20:00:01.311593 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:01.311562    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-601aa9-29755920-dlwgt" event={"ID":"3dd467c4-e5cc-405a-8c3f-6448abda4632","Type":"ContainerStarted","Data":"e0be6f49a02632c46f82cd8b1dafe9a0a1f2d35ff38b542bcdbf83c45cd09346"}
Jul 29 20:00:01.311687 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:01.311618    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-601aa9-29755920-dlwgt" event={"ID":"3dd467c4-e5cc-405a-8c3f-6448abda4632","Type":"ContainerStarted","Data":"8f495533fe29d384382c02aa7cb2230aec79448c3cea2d9c27ea49739d5cd106"}
Jul 29 20:00:01.313293 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:01.313270    2583 generic.go:356] "Generic (PLEG): container finished" podID="51c05aa5-0bb1-40eb-bf19-e85668da7db1" containerID="34dfabc8447c04fb9d0dca89c76bc424c0d99729579556e49502663b0b19e3b4" exitCode=0
Jul 29 20:00:01.313397 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:01.313365    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755920-4f7k8" event={"ID":"51c05aa5-0bb1-40eb-bf19-e85668da7db1","Type":"ContainerDied","Data":"34dfabc8447c04fb9d0dca89c76bc424c0d99729579556e49502663b0b19e3b4"}
Jul 29 20:00:01.313448 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:01.313403    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755920-4f7k8" event={"ID":"51c05aa5-0bb1-40eb-bf19-e85668da7db1","Type":"ContainerStarted","Data":"06464358a81368cef23d5faae3c992f181eaea7905355631f593a9ba66de5d2c"}
Jul 29 20:00:01.325209 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:01.325160    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a60880-29755920-zbzss" podStartSLOduration=1.325144319 podStartE2EDuration="1.325144319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:00:01.324283753 +0000 UTC m=+2568.552672879" watchObservedRunningTime="2026-07-29 20:00:01.325144319 +0000 UTC m=+2568.553533455"
Jul 29 20:00:01.338016 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:01.337967    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-601aa9-29755920-dlwgt" podStartSLOduration=1.337952797 podStartE2EDuration="1.337952797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:00:01.337607096 +0000 UTC m=+2568.565996224" watchObservedRunningTime="2026-07-29 20:00:01.337952797 +0000 UTC m=+2568.566341924"
Jul 29 20:00:02.319985 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:02.319948    2583 generic.go:356] "Generic (PLEG): container finished" podID="4eb30076-3571-46d5-a81f-d792cdc4b6c6" containerID="3753054139f31640adeafa07f3e9487cb7957bdb0d3c9d35452c90c8bb17d734" exitCode=0
Jul 29 20:00:02.320442 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:02.320003    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a60880-29755920-zbzss" event={"ID":"4eb30076-3571-46d5-a81f-d792cdc4b6c6","Type":"ContainerDied","Data":"3753054139f31640adeafa07f3e9487cb7957bdb0d3c9d35452c90c8bb17d734"}
Jul 29 20:00:02.321418 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:02.321394    2583 generic.go:356] "Generic (PLEG): container finished" podID="3dd467c4-e5cc-405a-8c3f-6448abda4632" containerID="e0be6f49a02632c46f82cd8b1dafe9a0a1f2d35ff38b542bcdbf83c45cd09346" exitCode=0
Jul 29 20:00:02.321509 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:02.321477    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-601aa9-29755920-dlwgt" event={"ID":"3dd467c4-e5cc-405a-8c3f-6448abda4632","Type":"ContainerDied","Data":"e0be6f49a02632c46f82cd8b1dafe9a0a1f2d35ff38b542bcdbf83c45cd09346"}
Jul 29 20:00:02.450658 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:02.450632    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755920-4f7k8"
Jul 29 20:00:03.327879 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:03.327846    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755920-4f7k8"
Jul 29 20:00:03.327879 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:03.327860    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755920-4f7k8" event={"ID":"51c05aa5-0bb1-40eb-bf19-e85668da7db1","Type":"ContainerDied","Data":"06464358a81368cef23d5faae3c992f181eaea7905355631f593a9ba66de5d2c"}
Jul 29 20:00:03.328386 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:03.327906    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06464358a81368cef23d5faae3c992f181eaea7905355631f593a9ba66de5d2c"
Jul 29 20:00:03.462227 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:03.462178    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a60880-29755920-zbzss"
Jul 29 20:00:03.499016 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:03.498982    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-601aa9-29755920-dlwgt"
Jul 29 20:00:04.334118 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:04.334088    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a60880-29755920-zbzss"
Jul 29 20:00:04.334118 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:04.334114    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a60880-29755920-zbzss" event={"ID":"4eb30076-3571-46d5-a81f-d792cdc4b6c6","Type":"ContainerDied","Data":"a4bf6095e162fac51f6a74c9b33b5dc6e9a91fedb806a350c4c301162270e277"}
Jul 29 20:00:04.334620 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:04.334173    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4bf6095e162fac51f6a74c9b33b5dc6e9a91fedb806a350c4c301162270e277"
Jul 29 20:00:04.335392 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:04.335374    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-601aa9-29755920-dlwgt"
Jul 29 20:00:04.335491 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:04.335424    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-601aa9-29755920-dlwgt" event={"ID":"3dd467c4-e5cc-405a-8c3f-6448abda4632","Type":"ContainerDied","Data":"8f495533fe29d384382c02aa7cb2230aec79448c3cea2d9c27ea49739d5cd106"}
Jul 29 20:00:04.335491 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:04.335466    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f495533fe29d384382c02aa7cb2230aec79448c3cea2d9c27ea49739d5cd106"
Jul 29 20:00:21.991958 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:21.991922    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 20:00:48.824853 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:48.824819    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-601aa9/auth-test-model-601aa9-kserve-6bc657579-gxbw5"]
Jul 29 20:00:48.825213 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:48.825159    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dd467c4-e5cc-405a-8c3f-6448abda4632" containerName="cleanup"
Jul 29 20:00:48.825213 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:48.825169    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dd467c4-e5cc-405a-8c3f-6448abda4632" containerName="cleanup"
Jul 29 20:00:48.825213 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:48.825179    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4eb30076-3571-46d5-a81f-d792cdc4b6c6" containerName="cleanup"
Jul 29 20:00:48.825213 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:48.825185    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4eb30076-3571-46d5-a81f-d792cdc4b6c6" containerName="cleanup"
Jul 29 20:00:48.825213 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:48.825192    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51c05aa5-0bb1-40eb-bf19-e85668da7db1" containerName="cleanup"
Jul 29 20:00:48.825213 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:48.825198    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="51c05aa5-0bb1-40eb-bf19-e85668da7db1" containerName="cleanup"
Jul 29 20:00:48.825416 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:48.825255    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dd467c4-e5cc-405a-8c3f-6448abda4632" containerName="cleanup"
Jul 29 20:00:48.825416 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:48.825264    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4eb30076-3571-46d5-a81f-d792cdc4b6c6" containerName="cleanup"
Jul 29 20:00:48.825416 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:48.825271    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="51c05aa5-0bb1-40eb-bf19-e85668da7db1" containerName="cleanup"
Jul 29 20:00:48.829344 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:48.829327    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-601aa9/auth-test-model-601aa9-kserve-6bc657579-gxbw5"
Jul 29 20:00:48.831898 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:48.831878    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-601aa9\"/\"default-dockercfg-k8c42\""
Jul 29 20:00:48.832004 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:48.831941    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-601aa9\"/\"kube-root-ca.crt\""
Jul 29 20:00:48.832509 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:48.832486    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-601aa9\"/\"openshift-service-ca.crt\""
Jul 29 20:00:48.832862 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:48.832849    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-601aa9\"/\"auth-test-model-601aa9-kserve-self-signed-certs\""
Jul 29 20:00:48.848499 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:48.848473    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-601aa9/auth-test-model-601aa9-kserve-6bc657579-gxbw5"]
Jul 29 20:00:48.908342 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:48.908303    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/105a60d0-ec47-412e-a017-0b1c26ee47c2-dshm\") pod \"auth-test-model-601aa9-kserve-6bc657579-gxbw5\" (UID: \"105a60d0-ec47-412e-a017-0b1c26ee47c2\") " pod="ai-tenant-e2e-shared-b-601aa9/auth-test-model-601aa9-kserve-6bc657579-gxbw5"
Jul 29 20:00:48.908342 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:48.908342    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/105a60d0-ec47-412e-a017-0b1c26ee47c2-tmp-dir\") pod \"auth-test-model-601aa9-kserve-6bc657579-gxbw5\" (UID: \"105a60d0-ec47-412e-a017-0b1c26ee47c2\") " pod="ai-tenant-e2e-shared-b-601aa9/auth-test-model-601aa9-kserve-6bc657579-gxbw5"
Jul 29 20:00:48.908580 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:48.908399    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/105a60d0-ec47-412e-a017-0b1c26ee47c2-home\") pod \"auth-test-model-601aa9-kserve-6bc657579-gxbw5\" (UID: \"105a60d0-ec47-412e-a017-0b1c26ee47c2\") " pod="ai-tenant-e2e-shared-b-601aa9/auth-test-model-601aa9-kserve-6bc657579-gxbw5"
Jul 29 20:00:48.908580 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:48.908466    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fk5d\" (UniqueName: \"kubernetes.io/projected/105a60d0-ec47-412e-a017-0b1c26ee47c2-kube-api-access-6fk5d\") pod \"auth-test-model-601aa9-kserve-6bc657579-gxbw5\" (UID: \"105a60d0-ec47-412e-a017-0b1c26ee47c2\") " pod="ai-tenant-e2e-shared-b-601aa9/auth-test-model-601aa9-kserve-6bc657579-gxbw5"
Jul 29 20:00:48.908580 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:48.908494    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/105a60d0-ec47-412e-a017-0b1c26ee47c2-model-cache\") pod \"auth-test-model-601aa9-kserve-6bc657579-gxbw5\" (UID: \"105a60d0-ec47-412e-a017-0b1c26ee47c2\") " pod="ai-tenant-e2e-shared-b-601aa9/auth-test-model-601aa9-kserve-6bc657579-gxbw5"
Jul 29 20:00:48.908580 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:48.908519    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/105a60d0-ec47-412e-a017-0b1c26ee47c2-tls-certs\") pod \"auth-test-model-601aa9-kserve-6bc657579-gxbw5\" (UID: \"105a60d0-ec47-412e-a017-0b1c26ee47c2\") " pod="ai-tenant-e2e-shared-b-601aa9/auth-test-model-601aa9-kserve-6bc657579-gxbw5"
Jul 29 20:00:49.009215 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:49.009180    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fk5d\" (UniqueName: \"kubernetes.io/projected/105a60d0-ec47-412e-a017-0b1c26ee47c2-kube-api-access-6fk5d\") pod \"auth-test-model-601aa9-kserve-6bc657579-gxbw5\" (UID: \"105a60d0-ec47-412e-a017-0b1c26ee47c2\") " pod="ai-tenant-e2e-shared-b-601aa9/auth-test-model-601aa9-kserve-6bc657579-gxbw5"
Jul 29 20:00:49.009417 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:49.009221    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/105a60d0-ec47-412e-a017-0b1c26ee47c2-model-cache\") pod \"auth-test-model-601aa9-kserve-6bc657579-gxbw5\" (UID: \"105a60d0-ec47-412e-a017-0b1c26ee47c2\") " pod="ai-tenant-e2e-shared-b-601aa9/auth-test-model-601aa9-kserve-6bc657579-gxbw5"
Jul 29 20:00:49.009417 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:49.009247    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/105a60d0-ec47-412e-a017-0b1c26ee47c2-tls-certs\") pod \"auth-test-model-601aa9-kserve-6bc657579-gxbw5\" (UID: \"105a60d0-ec47-412e-a017-0b1c26ee47c2\") " pod="ai-tenant-e2e-shared-b-601aa9/auth-test-model-601aa9-kserve-6bc657579-gxbw5"
Jul 29 20:00:49.009417 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:49.009281    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/105a60d0-ec47-412e-a017-0b1c26ee47c2-dshm\") pod \"auth-test-model-601aa9-kserve-6bc657579-gxbw5\" (UID: \"105a60d0-ec47-412e-a017-0b1c26ee47c2\") " pod="ai-tenant-e2e-shared-b-601aa9/auth-test-model-601aa9-kserve-6bc657579-gxbw5"
Jul 29 20:00:49.009417 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:49.009301    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/105a60d0-ec47-412e-a017-0b1c26ee47c2-tmp-dir\") pod \"auth-test-model-601aa9-kserve-6bc657579-gxbw5\" (UID: \"105a60d0-ec47-412e-a017-0b1c26ee47c2\") " pod="ai-tenant-e2e-shared-b-601aa9/auth-test-model-601aa9-kserve-6bc657579-gxbw5"
Jul 29 20:00:49.009417 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:49.009340    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/105a60d0-ec47-412e-a017-0b1c26ee47c2-home\") pod \"auth-test-model-601aa9-kserve-6bc657579-gxbw5\" (UID: \"105a60d0-ec47-412e-a017-0b1c26ee47c2\") " pod="ai-tenant-e2e-shared-b-601aa9/auth-test-model-601aa9-kserve-6bc657579-gxbw5"
Jul 29 20:00:49.009671 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:49.009626    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/105a60d0-ec47-412e-a017-0b1c26ee47c2-home\") pod \"auth-test-model-601aa9-kserve-6bc657579-gxbw5\" (UID: \"105a60d0-ec47-412e-a017-0b1c26ee47c2\") " pod="ai-tenant-e2e-shared-b-601aa9/auth-test-model-601aa9-kserve-6bc657579-gxbw5"
Jul 29 20:00:49.009730 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:49.009707    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/105a60d0-ec47-412e-a017-0b1c26ee47c2-model-cache\") pod \"auth-test-model-601aa9-kserve-6bc657579-gxbw5\" (UID: \"105a60d0-ec47-412e-a017-0b1c26ee47c2\") " pod="ai-tenant-e2e-shared-b-601aa9/auth-test-model-601aa9-kserve-6bc657579-gxbw5"
Jul 29 20:00:49.009730 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:49.009716    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/105a60d0-ec47-412e-a017-0b1c26ee47c2-tmp-dir\") pod \"auth-test-model-601aa9-kserve-6bc657579-gxbw5\" (UID: \"105a60d0-ec47-412e-a017-0b1c26ee47c2\") " pod="ai-tenant-e2e-shared-b-601aa9/auth-test-model-601aa9-kserve-6bc657579-gxbw5"
Jul 29 20:00:49.011582 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:49.011555    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/105a60d0-ec47-412e-a017-0b1c26ee47c2-dshm\") pod \"auth-test-model-601aa9-kserve-6bc657579-gxbw5\" (UID: \"105a60d0-ec47-412e-a017-0b1c26ee47c2\") " pod="ai-tenant-e2e-shared-b-601aa9/auth-test-model-601aa9-kserve-6bc657579-gxbw5"
Jul 29 20:00:49.011868 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:49.011846    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/105a60d0-ec47-412e-a017-0b1c26ee47c2-tls-certs\") pod \"auth-test-model-601aa9-kserve-6bc657579-gxbw5\" (UID: \"105a60d0-ec47-412e-a017-0b1c26ee47c2\") " pod="ai-tenant-e2e-shared-b-601aa9/auth-test-model-601aa9-kserve-6bc657579-gxbw5"
Jul 29 20:00:49.017225 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:49.017202    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fk5d\" (UniqueName: \"kubernetes.io/projected/105a60d0-ec47-412e-a017-0b1c26ee47c2-kube-api-access-6fk5d\") pod \"auth-test-model-601aa9-kserve-6bc657579-gxbw5\" (UID: \"105a60d0-ec47-412e-a017-0b1c26ee47c2\") " pod="ai-tenant-e2e-shared-b-601aa9/auth-test-model-601aa9-kserve-6bc657579-gxbw5"
Jul 29 20:00:49.138235 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:49.138206    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-601aa9/auth-test-model-601aa9-kserve-6bc657579-gxbw5"
Jul 29 20:00:49.273898 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:49.273855    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-601aa9/auth-test-model-601aa9-kserve-6bc657579-gxbw5"]
Jul 29 20:00:49.276222 ip-10-0-128-101 kubenswrapper[2583]: W0729 20:00:49.276196    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod105a60d0_ec47_412e_a017_0b1c26ee47c2.slice/crio-6ef883f97eddd86c1b21bda49c53a9fcac09c4246b1b27c055654899e76c9226 WatchSource:0}: Error finding container 6ef883f97eddd86c1b21bda49c53a9fcac09c4246b1b27c055654899e76c9226: Status 404 returned error can't find the container with id 6ef883f97eddd86c1b21bda49c53a9fcac09c4246b1b27c055654899e76c9226
Jul 29 20:00:49.556188 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:49.556105    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-601aa9/auth-test-model-601aa9-kserve-6bc657579-gxbw5" event={"ID":"105a60d0-ec47-412e-a017-0b1c26ee47c2","Type":"ContainerStarted","Data":"49b7f570a0da32031f2f4c250df6902bde5f311c4340e6c0bf48ff26527021f4"}
Jul 29 20:00:49.556188 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:49.556145    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-601aa9/auth-test-model-601aa9-kserve-6bc657579-gxbw5" event={"ID":"105a60d0-ec47-412e-a017-0b1c26ee47c2","Type":"ContainerStarted","Data":"6ef883f97eddd86c1b21bda49c53a9fcac09c4246b1b27c055654899e76c9226"}
Jul 29 20:00:49.572351 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:49.572294    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-601aa9/auth-test-model-601aa9-kserve-6bc657579-gxbw5" podStartSLOduration=1.5722744290000001 podStartE2EDuration="1.572274429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:00:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:00:49.571716465 +0000 UTC m=+2616.800105592" watchObservedRunningTime="2026-07-29 20:00:49.572274429 +0000 UTC m=+2616.800663556"
Jul 29 20:00:52.404611 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:52.404566    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 20:00:59.138500 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:59.138451    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-601aa9/auth-test-model-601aa9-kserve-6bc657579-gxbw5"
Jul 29 20:00:59.138500 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:59.138514    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-601aa9/auth-test-model-601aa9-kserve-6bc657579-gxbw5"
Jul 29 20:00:59.151602 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:59.151573    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-601aa9/auth-test-model-601aa9-kserve-6bc657579-gxbw5"
Jul 29 20:00:59.618966 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:00:59.618876    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-601aa9/auth-test-model-601aa9-kserve-6bc657579-gxbw5"
Jul 29 20:01:13.536874 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:01:13.536842    2583 scope.go:117] "RemoveContainer" containerID="d7be693b137691a2fe29fc6d48c4075f41c9d9e3531c6092141bc3c4fa03947b"
Jul 29 20:01:20.496068 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:01:20.496032    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 20:01:47.691057 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:01:47.691021    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 20:02:05.592971 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:02:05.592877    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 20:02:28.006354 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:02:28.006319    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 20:02:46.294154 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:02:46.294114    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 20:03:08.004937 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:03:08.004896    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 20:03:13.562884 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:03:13.562853    2583 scope.go:117] "RemoveContainer" containerID="762f7c74e9144ea8c4bdd7e18eb014bd64f8e1e31a69ee567fe2902f96de4965"
Jul 29 20:03:21.701722 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:03:21.701684    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 20:03:32.709664 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:03:32.709626    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 20:03:47.800128 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:03:47.800047    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 20:04:03.998206 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:04:03.998170    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 20:04:17.303414 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:04:17.303375    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 20:04:34.293925 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:04:34.293890    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 20:04:52.606709 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:04:52.606676    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 20:05:13.589085 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:05:13.589010    2583 scope.go:117] "RemoveContainer" containerID="2b433390b23f4a2eb92cbb763780427178f83ca53babb236a96e8e82091ac6fd"
Jul 29 20:05:13.597083 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:05:13.597062    2583 scope.go:117] "RemoveContainer" containerID="962ee9e0db7aac4486f5fbe93d1f20fd78798862e9153e503d04e08c9b5107ac"
Jul 29 20:05:15.093206 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:05:15.093168    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 20:05:20.448665 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:05:20.448623    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-601aa9/auth-test-model-601aa9-kserve-6bc657579-gxbw5"]
Jul 29 20:05:31.996318 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:05:31.996286    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 20:05:35.467833 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:05:35.467757    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-601aa9/auth-test-model-601aa9-kserve-6bc657579-gxbw5" podUID="105a60d0-ec47-412e-a017-0b1c26ee47c2" containerName="main" containerID="cri-o://49b7f570a0da32031f2f4c250df6902bde5f311c4340e6c0bf48ff26527021f4" gracePeriod=45
Jul 29 20:05:35.711252 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:05:35.711227    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-601aa9/auth-test-model-601aa9-kserve-6bc657579-gxbw5"
Jul 29 20:05:35.727135 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:05:35.727085    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/105a60d0-ec47-412e-a017-0b1c26ee47c2-model-cache\") pod \"105a60d0-ec47-412e-a017-0b1c26ee47c2\" (UID: \"105a60d0-ec47-412e-a017-0b1c26ee47c2\") "
Jul 29 20:05:35.727135 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:05:35.727119    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/105a60d0-ec47-412e-a017-0b1c26ee47c2-dshm\") pod \"105a60d0-ec47-412e-a017-0b1c26ee47c2\" (UID: \"105a60d0-ec47-412e-a017-0b1c26ee47c2\") "
Jul 29 20:05:35.727279 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:05:35.727180    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/105a60d0-ec47-412e-a017-0b1c26ee47c2-tmp-dir\") pod \"105a60d0-ec47-412e-a017-0b1c26ee47c2\" (UID: \"105a60d0-ec47-412e-a017-0b1c26ee47c2\") "
Jul 29 20:05:35.727279 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:05:35.727208    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/105a60d0-ec47-412e-a017-0b1c26ee47c2-tls-certs\") pod \"105a60d0-ec47-412e-a017-0b1c26ee47c2\" (UID: \"105a60d0-ec47-412e-a017-0b1c26ee47c2\") "
Jul 29 20:05:35.727279 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:05:35.727245    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/105a60d0-ec47-412e-a017-0b1c26ee47c2-home\") pod \"105a60d0-ec47-412e-a017-0b1c26ee47c2\" (UID: \"105a60d0-ec47-412e-a017-0b1c26ee47c2\") "
Jul 29 20:05:35.727433 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:05:35.727295    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fk5d\" (UniqueName: \"kubernetes.io/projected/105a60d0-ec47-412e-a017-0b1c26ee47c2-kube-api-access-6fk5d\") pod \"105a60d0-ec47-412e-a017-0b1c26ee47c2\" (UID: \"105a60d0-ec47-412e-a017-0b1c26ee47c2\") "
Jul 29 20:05:35.727433 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:05:35.727342    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105a60d0-ec47-412e-a017-0b1c26ee47c2-model-cache" (OuterVolumeSpecName: "model-cache") pod "105a60d0-ec47-412e-a017-0b1c26ee47c2" (UID: "105a60d0-ec47-412e-a017-0b1c26ee47c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:05:35.727539 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:05:35.727409    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105a60d0-ec47-412e-a017-0b1c26ee47c2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "105a60d0-ec47-412e-a017-0b1c26ee47c2" (UID: "105a60d0-ec47-412e-a017-0b1c26ee47c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:05:35.727539 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:05:35.727479    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105a60d0-ec47-412e-a017-0b1c26ee47c2-home" (OuterVolumeSpecName: "home") pod "105a60d0-ec47-412e-a017-0b1c26ee47c2" (UID: "105a60d0-ec47-412e-a017-0b1c26ee47c2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:05:35.727658 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:05:35.727598    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/105a60d0-ec47-412e-a017-0b1c26ee47c2-model-cache\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:05:35.727658 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:05:35.727614    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/105a60d0-ec47-412e-a017-0b1c26ee47c2-tmp-dir\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:05:35.727658 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:05:35.727625    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/105a60d0-ec47-412e-a017-0b1c26ee47c2-home\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:05:35.729987 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:05:35.729916    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105a60d0-ec47-412e-a017-0b1c26ee47c2-dshm" (OuterVolumeSpecName: "dshm") pod "105a60d0-ec47-412e-a017-0b1c26ee47c2" (UID: "105a60d0-ec47-412e-a017-0b1c26ee47c2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:05:35.730106 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:05:35.730027    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/105a60d0-ec47-412e-a017-0b1c26ee47c2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "105a60d0-ec47-412e-a017-0b1c26ee47c2" (UID: "105a60d0-ec47-412e-a017-0b1c26ee47c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:05:35.730206 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:05:35.730187    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/105a60d0-ec47-412e-a017-0b1c26ee47c2-kube-api-access-6fk5d" (OuterVolumeSpecName: "kube-api-access-6fk5d") pod "105a60d0-ec47-412e-a017-0b1c26ee47c2" (UID: "105a60d0-ec47-412e-a017-0b1c26ee47c2"). InnerVolumeSpecName "kube-api-access-6fk5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:05:35.828722 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:05:35.828697    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/105a60d0-ec47-412e-a017-0b1c26ee47c2-tls-certs\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:05:35.828722 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:05:35.828719    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fk5d\" (UniqueName: \"kubernetes.io/projected/105a60d0-ec47-412e-a017-0b1c26ee47c2-kube-api-access-6fk5d\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:05:35.828889 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:05:35.828728    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/105a60d0-ec47-412e-a017-0b1c26ee47c2-dshm\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:05:35.924196 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:05:35.924170    2583 generic.go:356] "Generic (PLEG): container finished" podID="105a60d0-ec47-412e-a017-0b1c26ee47c2" containerID="49b7f570a0da32031f2f4c250df6902bde5f311c4340e6c0bf48ff26527021f4" exitCode=0
Jul 29 20:05:35.924313 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:05:35.924216    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-601aa9/auth-test-model-601aa9-kserve-6bc657579-gxbw5"
Jul 29 20:05:35.924313 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:05:35.924261    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-601aa9/auth-test-model-601aa9-kserve-6bc657579-gxbw5" event={"ID":"105a60d0-ec47-412e-a017-0b1c26ee47c2","Type":"ContainerDied","Data":"49b7f570a0da32031f2f4c250df6902bde5f311c4340e6c0bf48ff26527021f4"}
Jul 29 20:05:35.924313 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:05:35.924303    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-601aa9/auth-test-model-601aa9-kserve-6bc657579-gxbw5" event={"ID":"105a60d0-ec47-412e-a017-0b1c26ee47c2","Type":"ContainerDied","Data":"6ef883f97eddd86c1b21bda49c53a9fcac09c4246b1b27c055654899e76c9226"}
Jul 29 20:05:35.924419 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:05:35.924321    2583 scope.go:117] "RemoveContainer" containerID="49b7f570a0da32031f2f4c250df6902bde5f311c4340e6c0bf48ff26527021f4"
Jul 29 20:05:35.932896 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:05:35.932875    2583 scope.go:117] "RemoveContainer" containerID="49b7f570a0da32031f2f4c250df6902bde5f311c4340e6c0bf48ff26527021f4"
Jul 29 20:05:35.933129 ip-10-0-128-101 kubenswrapper[2583]: E0729 20:05:35.933111    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49b7f570a0da32031f2f4c250df6902bde5f311c4340e6c0bf48ff26527021f4\": container with ID starting with 49b7f570a0da32031f2f4c250df6902bde5f311c4340e6c0bf48ff26527021f4 not found: ID does not exist" containerID="49b7f570a0da32031f2f4c250df6902bde5f311c4340e6c0bf48ff26527021f4"
Jul 29 20:05:35.933187 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:05:35.933136    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49b7f570a0da32031f2f4c250df6902bde5f311c4340e6c0bf48ff26527021f4"} err="failed to get container status \"49b7f570a0da32031f2f4c250df6902bde5f311c4340e6c0bf48ff26527021f4\": rpc error: code = NotFound desc = could not find container \"49b7f570a0da32031f2f4c250df6902bde5f311c4340e6c0bf48ff26527021f4\": container with ID starting with 49b7f570a0da32031f2f4c250df6902bde5f311c4340e6c0bf48ff26527021f4 not found: ID does not exist"
Jul 29 20:05:35.944673 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:05:35.944650    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-601aa9/auth-test-model-601aa9-kserve-6bc657579-gxbw5"]
Jul 29 20:05:35.948345 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:05:35.948321    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-601aa9/auth-test-model-601aa9-kserve-6bc657579-gxbw5"]
Jul 29 20:05:37.339924 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:05:37.339887    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="105a60d0-ec47-412e-a017-0b1c26ee47c2" path="/var/lib/kubelet/pods/105a60d0-ec47-412e-a017-0b1c26ee47c2/volumes"
Jul 29 20:05:40.395409 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:05:40.395371    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 20:06:13.281943 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:06:13.281909    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-601aa9/sub-test-model-601aa9-kserve-76b6c79d7-d74nd"]
Jul 29 20:06:13.282414 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:06:13.282232    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="105a60d0-ec47-412e-a017-0b1c26ee47c2" containerName="main"
Jul 29 20:06:13.282414 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:06:13.282241    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="105a60d0-ec47-412e-a017-0b1c26ee47c2" containerName="main"
Jul 29 20:06:13.282414 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:06:13.282301    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="105a60d0-ec47-412e-a017-0b1c26ee47c2" containerName="main"
Jul 29 20:06:13.285320 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:06:13.285303    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-601aa9/sub-test-model-601aa9-kserve-76b6c79d7-d74nd"
Jul 29 20:06:13.288378 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:06:13.288358    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-601aa9\"/\"default-dockercfg-k8c42\""
Jul 29 20:06:13.288378 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:06:13.288368    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-601aa9\"/\"sub-test-model-601aa9-kserve-self-signed-certs\""
Jul 29 20:06:13.288543 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:06:13.288359    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-601aa9\"/\"openshift-service-ca.crt\""
Jul 29 20:06:13.288543 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:06:13.288361    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-601aa9\"/\"kube-root-ca.crt\""
Jul 29 20:06:13.293219 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:06:13.293199    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-601aa9/sub-test-model-601aa9-kserve-76b6c79d7-d74nd"]
Jul 29 20:06:13.438321 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:06:13.438288    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/562ac560-83d7-4a37-be60-f527003d8d2f-tls-certs\") pod \"sub-test-model-601aa9-kserve-76b6c79d7-d74nd\" (UID: \"562ac560-83d7-4a37-be60-f527003d8d2f\") " pod="ai-tenant-e2e-shared-b-601aa9/sub-test-model-601aa9-kserve-76b6c79d7-d74nd"
Jul 29 20:06:13.438495 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:06:13.438336    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/562ac560-83d7-4a37-be60-f527003d8d2f-dshm\") pod \"sub-test-model-601aa9-kserve-76b6c79d7-d74nd\" (UID: \"562ac560-83d7-4a37-be60-f527003d8d2f\") " pod="ai-tenant-e2e-shared-b-601aa9/sub-test-model-601aa9-kserve-76b6c79d7-d74nd"
Jul 29 20:06:13.438495 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:06:13.438359    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzd77\" (UniqueName: \"kubernetes.io/projected/562ac560-83d7-4a37-be60-f527003d8d2f-kube-api-access-wzd77\") pod \"sub-test-model-601aa9-kserve-76b6c79d7-d74nd\" (UID: \"562ac560-83d7-4a37-be60-f527003d8d2f\") " pod="ai-tenant-e2e-shared-b-601aa9/sub-test-model-601aa9-kserve-76b6c79d7-d74nd"
Jul 29 20:06:13.438495 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:06:13.438389    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/562ac560-83d7-4a37-be60-f527003d8d2f-model-cache\") pod \"sub-test-model-601aa9-kserve-76b6c79d7-d74nd\" (UID: \"562ac560-83d7-4a37-be60-f527003d8d2f\") " pod="ai-tenant-e2e-shared-b-601aa9/sub-test-model-601aa9-kserve-76b6c79d7-d74nd"
Jul 29 20:06:13.438495 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:06:13.438410    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/562ac560-83d7-4a37-be60-f527003d8d2f-tmp-dir\") pod \"sub-test-model-601aa9-kserve-76b6c79d7-d74nd\" (UID: \"562ac560-83d7-4a37-be60-f527003d8d2f\") " pod="ai-tenant-e2e-shared-b-601aa9/sub-test-model-601aa9-kserve-76b6c79d7-d74nd"
Jul 29 20:06:13.438495 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:06:13.438440    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/562ac560-83d7-4a37-be60-f527003d8d2f-home\") pod \"sub-test-model-601aa9-kserve-76b6c79d7-d74nd\" (UID: \"562ac560-83d7-4a37-be60-f527003d8d2f\") " pod="ai-tenant-e2e-shared-b-601aa9/sub-test-model-601aa9-kserve-76b6c79d7-d74nd"
Jul 29 20:06:13.539548 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:06:13.539457    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/562ac560-83d7-4a37-be60-f527003d8d2f-dshm\") pod \"sub-test-model-601aa9-kserve-76b6c79d7-d74nd\" (UID: \"562ac560-83d7-4a37-be60-f527003d8d2f\") " pod="ai-tenant-e2e-shared-b-601aa9/sub-test-model-601aa9-kserve-76b6c79d7-d74nd"
Jul 29 20:06:13.539548 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:06:13.539502    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzd77\" (UniqueName: \"kubernetes.io/projected/562ac560-83d7-4a37-be60-f527003d8d2f-kube-api-access-wzd77\") pod \"sub-test-model-601aa9-kserve-76b6c79d7-d74nd\" (UID: \"562ac560-83d7-4a37-be60-f527003d8d2f\") " pod="ai-tenant-e2e-shared-b-601aa9/sub-test-model-601aa9-kserve-76b6c79d7-d74nd"
Jul 29 20:06:13.539548 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:06:13.539533    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/562ac560-83d7-4a37-be60-f527003d8d2f-model-cache\") pod \"sub-test-model-601aa9-kserve-76b6c79d7-d74nd\" (UID: \"562ac560-83d7-4a37-be60-f527003d8d2f\") " pod="ai-tenant-e2e-shared-b-601aa9/sub-test-model-601aa9-kserve-76b6c79d7-d74nd"
Jul 29 20:06:13.539860 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:06:13.539567    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/562ac560-83d7-4a37-be60-f527003d8d2f-tmp-dir\") pod \"sub-test-model-601aa9-kserve-76b6c79d7-d74nd\" (UID: \"562ac560-83d7-4a37-be60-f527003d8d2f\") " pod="ai-tenant-e2e-shared-b-601aa9/sub-test-model-601aa9-kserve-76b6c79d7-d74nd"
Jul 29 20:06:13.539860 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:06:13.539610    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/562ac560-83d7-4a37-be60-f527003d8d2f-home\") pod \"sub-test-model-601aa9-kserve-76b6c79d7-d74nd\" (UID: \"562ac560-83d7-4a37-be60-f527003d8d2f\") " pod="ai-tenant-e2e-shared-b-601aa9/sub-test-model-601aa9-kserve-76b6c79d7-d74nd"
Jul 29 20:06:13.539860 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:06:13.539652    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/562ac560-83d7-4a37-be60-f527003d8d2f-tls-certs\") pod \"sub-test-model-601aa9-kserve-76b6c79d7-d74nd\" (UID: \"562ac560-83d7-4a37-be60-f527003d8d2f\") " pod="ai-tenant-e2e-shared-b-601aa9/sub-test-model-601aa9-kserve-76b6c79d7-d74nd"
Jul 29 20:06:13.540058 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:06:13.540034    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/562ac560-83d7-4a37-be60-f527003d8d2f-tmp-dir\") pod \"sub-test-model-601aa9-kserve-76b6c79d7-d74nd\" (UID: \"562ac560-83d7-4a37-be60-f527003d8d2f\") " pod="ai-tenant-e2e-shared-b-601aa9/sub-test-model-601aa9-kserve-76b6c79d7-d74nd"
Jul 29 20:06:13.540129 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:06:13.540065    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/562ac560-83d7-4a37-be60-f527003d8d2f-model-cache\") pod \"sub-test-model-601aa9-kserve-76b6c79d7-d74nd\" (UID: \"562ac560-83d7-4a37-be60-f527003d8d2f\") " pod="ai-tenant-e2e-shared-b-601aa9/sub-test-model-601aa9-kserve-76b6c79d7-d74nd"
Jul 29 20:06:13.540129 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:06:13.540100    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/562ac560-83d7-4a37-be60-f527003d8d2f-home\") pod \"sub-test-model-601aa9-kserve-76b6c79d7-d74nd\" (UID: \"562ac560-83d7-4a37-be60-f527003d8d2f\") " pod="ai-tenant-e2e-shared-b-601aa9/sub-test-model-601aa9-kserve-76b6c79d7-d74nd"
Jul 29 20:06:13.541658 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:06:13.541638    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/562ac560-83d7-4a37-be60-f527003d8d2f-dshm\") pod \"sub-test-model-601aa9-kserve-76b6c79d7-d74nd\" (UID: \"562ac560-83d7-4a37-be60-f527003d8d2f\") " pod="ai-tenant-e2e-shared-b-601aa9/sub-test-model-601aa9-kserve-76b6c79d7-d74nd"
Jul 29 20:06:13.542117 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:06:13.542095    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/562ac560-83d7-4a37-be60-f527003d8d2f-tls-certs\") pod \"sub-test-model-601aa9-kserve-76b6c79d7-d74nd\" (UID: \"562ac560-83d7-4a37-be60-f527003d8d2f\") " pod="ai-tenant-e2e-shared-b-601aa9/sub-test-model-601aa9-kserve-76b6c79d7-d74nd"
Jul 29 20:06:13.547409 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:06:13.547387    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzd77\" (UniqueName: \"kubernetes.io/projected/562ac560-83d7-4a37-be60-f527003d8d2f-kube-api-access-wzd77\") pod \"sub-test-model-601aa9-kserve-76b6c79d7-d74nd\" (UID: \"562ac560-83d7-4a37-be60-f527003d8d2f\") " pod="ai-tenant-e2e-shared-b-601aa9/sub-test-model-601aa9-kserve-76b6c79d7-d74nd"
Jul 29 20:06:13.595378 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:06:13.595343    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-601aa9/sub-test-model-601aa9-kserve-76b6c79d7-d74nd"
Jul 29 20:06:13.733215 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:06:13.733186    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-601aa9/sub-test-model-601aa9-kserve-76b6c79d7-d74nd"]
Jul 29 20:06:13.734816 ip-10-0-128-101 kubenswrapper[2583]: W0729 20:06:13.734772    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod562ac560_83d7_4a37_be60_f527003d8d2f.slice/crio-c56672cccb519aa0d9122347cbcf16c51676901e7402f96eb91326e3dfad3dac WatchSource:0}: Error finding container c56672cccb519aa0d9122347cbcf16c51676901e7402f96eb91326e3dfad3dac: Status 404 returned error can't find the container with id c56672cccb519aa0d9122347cbcf16c51676901e7402f96eb91326e3dfad3dac
Jul 29 20:06:14.099756 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:06:14.099719    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-601aa9/sub-test-model-601aa9-kserve-76b6c79d7-d74nd" event={"ID":"562ac560-83d7-4a37-be60-f527003d8d2f","Type":"ContainerStarted","Data":"8d26a7c0b8b7ca87b3d7d90b0c961a3dac198ffaac9b6eecccd9c3dee45dc4db"}
Jul 29 20:06:14.099756 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:06:14.099761    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-601aa9/sub-test-model-601aa9-kserve-76b6c79d7-d74nd" event={"ID":"562ac560-83d7-4a37-be60-f527003d8d2f","Type":"ContainerStarted","Data":"c56672cccb519aa0d9122347cbcf16c51676901e7402f96eb91326e3dfad3dac"}
Jul 29 20:06:14.115443 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:06:14.115395    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-601aa9/sub-test-model-601aa9-kserve-76b6c79d7-d74nd" podStartSLOduration=1.115379724 podStartE2EDuration="1.115379724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:06:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:06:14.114873615 +0000 UTC m=+2941.343262759" watchObservedRunningTime="2026-07-29 20:06:14.115379724 +0000 UTC m=+2941.343768849"
Jul 29 20:06:17.206129 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:06:17.206073    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 20:06:23.596385 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:06:23.596343    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-601aa9/sub-test-model-601aa9-kserve-76b6c79d7-d74nd"
Jul 29 20:06:23.596902 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:06:23.596398    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-601aa9/sub-test-model-601aa9-kserve-76b6c79d7-d74nd"
Jul 29 20:06:23.613044 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:06:23.613009    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-601aa9/sub-test-model-601aa9-kserve-76b6c79d7-d74nd"
Jul 29 20:06:24.164740 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:06:24.164711    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-601aa9/sub-test-model-601aa9-kserve-76b6c79d7-d74nd"
Jul 29 20:06:54.501307 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:06:54.501227    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 20:07:13.497402 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:13.497369    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 20:07:20.535703 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:20.535663    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-601aa9/sub-test-model-601aa9-kserve-76b6c79d7-d74nd"]
Jul 29 20:07:32.496047 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:32.496007    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 20:07:35.554924 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:35.554879    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-601aa9/sub-test-model-601aa9-kserve-76b6c79d7-d74nd" podUID="562ac560-83d7-4a37-be60-f527003d8d2f" containerName="main" containerID="cri-o://8d26a7c0b8b7ca87b3d7d90b0c961a3dac198ffaac9b6eecccd9c3dee45dc4db" gracePeriod=45
Jul 29 20:07:35.811601 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:35.811525    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-601aa9/sub-test-model-601aa9-kserve-76b6c79d7-d74nd"
Jul 29 20:07:35.841075 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:35.841042    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/562ac560-83d7-4a37-be60-f527003d8d2f-model-cache\") pod \"562ac560-83d7-4a37-be60-f527003d8d2f\" (UID: \"562ac560-83d7-4a37-be60-f527003d8d2f\") "
Jul 29 20:07:35.841264 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:35.841104    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/562ac560-83d7-4a37-be60-f527003d8d2f-home\") pod \"562ac560-83d7-4a37-be60-f527003d8d2f\" (UID: \"562ac560-83d7-4a37-be60-f527003d8d2f\") "
Jul 29 20:07:35.841264 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:35.841139    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/562ac560-83d7-4a37-be60-f527003d8d2f-tls-certs\") pod \"562ac560-83d7-4a37-be60-f527003d8d2f\" (UID: \"562ac560-83d7-4a37-be60-f527003d8d2f\") "
Jul 29 20:07:35.841264 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:35.841182    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/562ac560-83d7-4a37-be60-f527003d8d2f-tmp-dir\") pod \"562ac560-83d7-4a37-be60-f527003d8d2f\" (UID: \"562ac560-83d7-4a37-be60-f527003d8d2f\") "
Jul 29 20:07:35.841449 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:35.841267    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/562ac560-83d7-4a37-be60-f527003d8d2f-dshm\") pod \"562ac560-83d7-4a37-be60-f527003d8d2f\" (UID: \"562ac560-83d7-4a37-be60-f527003d8d2f\") "
Jul 29 20:07:35.841449 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:35.841293    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzd77\" (UniqueName: \"kubernetes.io/projected/562ac560-83d7-4a37-be60-f527003d8d2f-kube-api-access-wzd77\") pod \"562ac560-83d7-4a37-be60-f527003d8d2f\" (UID: \"562ac560-83d7-4a37-be60-f527003d8d2f\") "
Jul 29 20:07:35.841449 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:35.841315    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/562ac560-83d7-4a37-be60-f527003d8d2f-model-cache" (OuterVolumeSpecName: "model-cache") pod "562ac560-83d7-4a37-be60-f527003d8d2f" (UID: "562ac560-83d7-4a37-be60-f527003d8d2f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:07:35.841449 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:35.841434    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/562ac560-83d7-4a37-be60-f527003d8d2f-home" (OuterVolumeSpecName: "home") pod "562ac560-83d7-4a37-be60-f527003d8d2f" (UID: "562ac560-83d7-4a37-be60-f527003d8d2f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:07:35.841657 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:35.841448    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/562ac560-83d7-4a37-be60-f527003d8d2f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "562ac560-83d7-4a37-be60-f527003d8d2f" (UID: "562ac560-83d7-4a37-be60-f527003d8d2f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:07:35.841657 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:35.841564    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/562ac560-83d7-4a37-be60-f527003d8d2f-model-cache\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:07:35.841657 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:35.841581    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/562ac560-83d7-4a37-be60-f527003d8d2f-home\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:07:35.841657 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:35.841593    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/562ac560-83d7-4a37-be60-f527003d8d2f-tmp-dir\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:07:35.843305 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:35.843273    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/562ac560-83d7-4a37-be60-f527003d8d2f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "562ac560-83d7-4a37-be60-f527003d8d2f" (UID: "562ac560-83d7-4a37-be60-f527003d8d2f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:07:35.843517 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:35.843499    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/562ac560-83d7-4a37-be60-f527003d8d2f-kube-api-access-wzd77" (OuterVolumeSpecName: "kube-api-access-wzd77") pod "562ac560-83d7-4a37-be60-f527003d8d2f" (UID: "562ac560-83d7-4a37-be60-f527003d8d2f"). InnerVolumeSpecName "kube-api-access-wzd77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:07:35.843715 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:35.843696    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/562ac560-83d7-4a37-be60-f527003d8d2f-dshm" (OuterVolumeSpecName: "dshm") pod "562ac560-83d7-4a37-be60-f527003d8d2f" (UID: "562ac560-83d7-4a37-be60-f527003d8d2f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:07:35.942475 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:35.942432    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/562ac560-83d7-4a37-be60-f527003d8d2f-dshm\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:07:35.942475 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:35.942468    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzd77\" (UniqueName: \"kubernetes.io/projected/562ac560-83d7-4a37-be60-f527003d8d2f-kube-api-access-wzd77\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:07:35.942475 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:35.942478    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/562ac560-83d7-4a37-be60-f527003d8d2f-tls-certs\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:07:36.485131 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:36.485100    2583 generic.go:356] "Generic (PLEG): container finished" podID="562ac560-83d7-4a37-be60-f527003d8d2f" containerID="8d26a7c0b8b7ca87b3d7d90b0c961a3dac198ffaac9b6eecccd9c3dee45dc4db" exitCode=0
Jul 29 20:07:36.485303 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:36.485147    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-601aa9/sub-test-model-601aa9-kserve-76b6c79d7-d74nd"
Jul 29 20:07:36.485303 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:36.485178    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-601aa9/sub-test-model-601aa9-kserve-76b6c79d7-d74nd" event={"ID":"562ac560-83d7-4a37-be60-f527003d8d2f","Type":"ContainerDied","Data":"8d26a7c0b8b7ca87b3d7d90b0c961a3dac198ffaac9b6eecccd9c3dee45dc4db"}
Jul 29 20:07:36.485303 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:36.485219    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-601aa9/sub-test-model-601aa9-kserve-76b6c79d7-d74nd" event={"ID":"562ac560-83d7-4a37-be60-f527003d8d2f","Type":"ContainerDied","Data":"c56672cccb519aa0d9122347cbcf16c51676901e7402f96eb91326e3dfad3dac"}
Jul 29 20:07:36.485303 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:36.485237    2583 scope.go:117] "RemoveContainer" containerID="8d26a7c0b8b7ca87b3d7d90b0c961a3dac198ffaac9b6eecccd9c3dee45dc4db"
Jul 29 20:07:36.496273 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:36.496220    2583 scope.go:117] "RemoveContainer" containerID="8d26a7c0b8b7ca87b3d7d90b0c961a3dac198ffaac9b6eecccd9c3dee45dc4db"
Jul 29 20:07:36.496685 ip-10-0-128-101 kubenswrapper[2583]: E0729 20:07:36.496661    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d26a7c0b8b7ca87b3d7d90b0c961a3dac198ffaac9b6eecccd9c3dee45dc4db\": container with ID starting with 8d26a7c0b8b7ca87b3d7d90b0c961a3dac198ffaac9b6eecccd9c3dee45dc4db not found: ID does not exist" containerID="8d26a7c0b8b7ca87b3d7d90b0c961a3dac198ffaac9b6eecccd9c3dee45dc4db"
Jul 29 20:07:36.496762 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:36.496699    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d26a7c0b8b7ca87b3d7d90b0c961a3dac198ffaac9b6eecccd9c3dee45dc4db"} err="failed to get container status \"8d26a7c0b8b7ca87b3d7d90b0c961a3dac198ffaac9b6eecccd9c3dee45dc4db\": rpc error: code = NotFound desc = could not find container \"8d26a7c0b8b7ca87b3d7d90b0c961a3dac198ffaac9b6eecccd9c3dee45dc4db\": container with ID starting with 8d26a7c0b8b7ca87b3d7d90b0c961a3dac198ffaac9b6eecccd9c3dee45dc4db not found: ID does not exist"
Jul 29 20:07:36.506416 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:36.506384    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-601aa9/sub-test-model-601aa9-kserve-76b6c79d7-d74nd"]
Jul 29 20:07:36.512305 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:36.512270    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-601aa9/sub-test-model-601aa9-kserve-76b6c79d7-d74nd"]
Jul 29 20:07:37.340635 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:37.340594    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="562ac560-83d7-4a37-be60-f527003d8d2f" path="/var/lib/kubelet/pods/562ac560-83d7-4a37-be60-f527003d8d2f/volumes"
Jul 29 20:07:44.806953 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:44.806913    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 20:07:59.070945 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:59.070909    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-601aa9/rate-test-model-601aa9-kserve-f8bc498dc-zfq9z"]
Jul 29 20:07:59.071348 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:59.071267    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="562ac560-83d7-4a37-be60-f527003d8d2f" containerName="main"
Jul 29 20:07:59.071348 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:59.071277    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="562ac560-83d7-4a37-be60-f527003d8d2f" containerName="main"
Jul 29 20:07:59.071348 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:59.071332    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="562ac560-83d7-4a37-be60-f527003d8d2f" containerName="main"
Jul 29 20:07:59.074107 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:59.074091    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-601aa9/rate-test-model-601aa9-kserve-f8bc498dc-zfq9z"
Jul 29 20:07:59.076223 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:59.076203    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-601aa9\"/\"kube-root-ca.crt\""
Jul 29 20:07:59.077010 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:59.076989    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-601aa9\"/\"rate-test-model-601aa9-kserve-self-signed-certs\""
Jul 29 20:07:59.077098 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:59.077021    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-601aa9\"/\"openshift-service-ca.crt\""
Jul 29 20:07:59.077162 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:59.077101    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-601aa9\"/\"default-dockercfg-k8c42\""
Jul 29 20:07:59.081795 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:59.081774    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-601aa9/rate-test-model-601aa9-kserve-f8bc498dc-zfq9z"]
Jul 29 20:07:59.134160 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:59.134134    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/275320b5-9a5f-45b7-bb64-8f661730a395-tls-certs\") pod \"rate-test-model-601aa9-kserve-f8bc498dc-zfq9z\" (UID: \"275320b5-9a5f-45b7-bb64-8f661730a395\") " pod="ai-tenant-e2e-shared-b-601aa9/rate-test-model-601aa9-kserve-f8bc498dc-zfq9z"
Jul 29 20:07:59.134281 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:59.134179    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/275320b5-9a5f-45b7-bb64-8f661730a395-home\") pod \"rate-test-model-601aa9-kserve-f8bc498dc-zfq9z\" (UID: \"275320b5-9a5f-45b7-bb64-8f661730a395\") " pod="ai-tenant-e2e-shared-b-601aa9/rate-test-model-601aa9-kserve-f8bc498dc-zfq9z"
Jul 29 20:07:59.134281 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:59.134205    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/275320b5-9a5f-45b7-bb64-8f661730a395-tmp-dir\") pod \"rate-test-model-601aa9-kserve-f8bc498dc-zfq9z\" (UID: \"275320b5-9a5f-45b7-bb64-8f661730a395\") " pod="ai-tenant-e2e-shared-b-601aa9/rate-test-model-601aa9-kserve-f8bc498dc-zfq9z"
Jul 29 20:07:59.134281 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:59.134264    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/275320b5-9a5f-45b7-bb64-8f661730a395-model-cache\") pod \"rate-test-model-601aa9-kserve-f8bc498dc-zfq9z\" (UID: \"275320b5-9a5f-45b7-bb64-8f661730a395\") " pod="ai-tenant-e2e-shared-b-601aa9/rate-test-model-601aa9-kserve-f8bc498dc-zfq9z"
Jul 29 20:07:59.134391 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:59.134288    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/275320b5-9a5f-45b7-bb64-8f661730a395-dshm\") pod \"rate-test-model-601aa9-kserve-f8bc498dc-zfq9z\" (UID: \"275320b5-9a5f-45b7-bb64-8f661730a395\") " pod="ai-tenant-e2e-shared-b-601aa9/rate-test-model-601aa9-kserve-f8bc498dc-zfq9z"
Jul 29 20:07:59.134391 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:59.134321    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5tj9\" (UniqueName: \"kubernetes.io/projected/275320b5-9a5f-45b7-bb64-8f661730a395-kube-api-access-q5tj9\") pod \"rate-test-model-601aa9-kserve-f8bc498dc-zfq9z\" (UID: \"275320b5-9a5f-45b7-bb64-8f661730a395\") " pod="ai-tenant-e2e-shared-b-601aa9/rate-test-model-601aa9-kserve-f8bc498dc-zfq9z"
Jul 29 20:07:59.234958 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:59.234932    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/275320b5-9a5f-45b7-bb64-8f661730a395-home\") pod \"rate-test-model-601aa9-kserve-f8bc498dc-zfq9z\" (UID: \"275320b5-9a5f-45b7-bb64-8f661730a395\") " pod="ai-tenant-e2e-shared-b-601aa9/rate-test-model-601aa9-kserve-f8bc498dc-zfq9z"
Jul 29 20:07:59.235107 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:59.234970    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/275320b5-9a5f-45b7-bb64-8f661730a395-tmp-dir\") pod \"rate-test-model-601aa9-kserve-f8bc498dc-zfq9z\" (UID: \"275320b5-9a5f-45b7-bb64-8f661730a395\") " pod="ai-tenant-e2e-shared-b-601aa9/rate-test-model-601aa9-kserve-f8bc498dc-zfq9z"
Jul 29 20:07:59.235107 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:59.235001    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/275320b5-9a5f-45b7-bb64-8f661730a395-model-cache\") pod \"rate-test-model-601aa9-kserve-f8bc498dc-zfq9z\" (UID: \"275320b5-9a5f-45b7-bb64-8f661730a395\") " pod="ai-tenant-e2e-shared-b-601aa9/rate-test-model-601aa9-kserve-f8bc498dc-zfq9z"
Jul 29 20:07:59.235197 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:59.235121    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/275320b5-9a5f-45b7-bb64-8f661730a395-dshm\") pod \"rate-test-model-601aa9-kserve-f8bc498dc-zfq9z\" (UID: \"275320b5-9a5f-45b7-bb64-8f661730a395\") " pod="ai-tenant-e2e-shared-b-601aa9/rate-test-model-601aa9-kserve-f8bc498dc-zfq9z"
Jul 29 20:07:59.235197 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:59.235180    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5tj9\" (UniqueName: \"kubernetes.io/projected/275320b5-9a5f-45b7-bb64-8f661730a395-kube-api-access-q5tj9\") pod \"rate-test-model-601aa9-kserve-f8bc498dc-zfq9z\" (UID: \"275320b5-9a5f-45b7-bb64-8f661730a395\") " pod="ai-tenant-e2e-shared-b-601aa9/rate-test-model-601aa9-kserve-f8bc498dc-zfq9z"
Jul 29 20:07:59.235264 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:59.235239    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/275320b5-9a5f-45b7-bb64-8f661730a395-tls-certs\") pod \"rate-test-model-601aa9-kserve-f8bc498dc-zfq9z\" (UID: \"275320b5-9a5f-45b7-bb64-8f661730a395\") " pod="ai-tenant-e2e-shared-b-601aa9/rate-test-model-601aa9-kserve-f8bc498dc-zfq9z"
Jul 29 20:07:59.235329 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:59.235309    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/275320b5-9a5f-45b7-bb64-8f661730a395-tmp-dir\") pod \"rate-test-model-601aa9-kserve-f8bc498dc-zfq9z\" (UID: \"275320b5-9a5f-45b7-bb64-8f661730a395\") " pod="ai-tenant-e2e-shared-b-601aa9/rate-test-model-601aa9-kserve-f8bc498dc-zfq9z"
Jul 29 20:07:59.235393 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:59.235374    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/275320b5-9a5f-45b7-bb64-8f661730a395-home\") pod \"rate-test-model-601aa9-kserve-f8bc498dc-zfq9z\" (UID: \"275320b5-9a5f-45b7-bb64-8f661730a395\") " pod="ai-tenant-e2e-shared-b-601aa9/rate-test-model-601aa9-kserve-f8bc498dc-zfq9z"
Jul 29 20:07:59.235429 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:59.235395    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/275320b5-9a5f-45b7-bb64-8f661730a395-model-cache\") pod \"rate-test-model-601aa9-kserve-f8bc498dc-zfq9z\" (UID: \"275320b5-9a5f-45b7-bb64-8f661730a395\") " pod="ai-tenant-e2e-shared-b-601aa9/rate-test-model-601aa9-kserve-f8bc498dc-zfq9z"
Jul 29 20:07:59.237359 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:59.237327    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/275320b5-9a5f-45b7-bb64-8f661730a395-dshm\") pod \"rate-test-model-601aa9-kserve-f8bc498dc-zfq9z\" (UID: \"275320b5-9a5f-45b7-bb64-8f661730a395\") " pod="ai-tenant-e2e-shared-b-601aa9/rate-test-model-601aa9-kserve-f8bc498dc-zfq9z"
Jul 29 20:07:59.237593 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:59.237578    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/275320b5-9a5f-45b7-bb64-8f661730a395-tls-certs\") pod \"rate-test-model-601aa9-kserve-f8bc498dc-zfq9z\" (UID: \"275320b5-9a5f-45b7-bb64-8f661730a395\") " pod="ai-tenant-e2e-shared-b-601aa9/rate-test-model-601aa9-kserve-f8bc498dc-zfq9z"
Jul 29 20:07:59.242732 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:59.242713    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5tj9\" (UniqueName: \"kubernetes.io/projected/275320b5-9a5f-45b7-bb64-8f661730a395-kube-api-access-q5tj9\") pod \"rate-test-model-601aa9-kserve-f8bc498dc-zfq9z\" (UID: \"275320b5-9a5f-45b7-bb64-8f661730a395\") " pod="ai-tenant-e2e-shared-b-601aa9/rate-test-model-601aa9-kserve-f8bc498dc-zfq9z"
Jul 29 20:07:59.384261 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:59.384233    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-601aa9/rate-test-model-601aa9-kserve-f8bc498dc-zfq9z"
Jul 29 20:07:59.512148 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:59.512127    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-601aa9/rate-test-model-601aa9-kserve-f8bc498dc-zfq9z"]
Jul 29 20:07:59.515026 ip-10-0-128-101 kubenswrapper[2583]: W0729 20:07:59.514994    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod275320b5_9a5f_45b7_bb64_8f661730a395.slice/crio-acbaacd96a481bafeb0b9af9939cec98d2f3485a6784c481f39467d75442ae45 WatchSource:0}: Error finding container acbaacd96a481bafeb0b9af9939cec98d2f3485a6784c481f39467d75442ae45: Status 404 returned error can't find the container with id acbaacd96a481bafeb0b9af9939cec98d2f3485a6784c481f39467d75442ae45
Jul 29 20:07:59.595328 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:59.595290    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-601aa9/rate-test-model-601aa9-kserve-f8bc498dc-zfq9z" event={"ID":"275320b5-9a5f-45b7-bb64-8f661730a395","Type":"ContainerStarted","Data":"449a6c0dff3b0d48d5509e5f5d5ec172e18092d47e38639de0e29b2b9c25732d"}
Jul 29 20:07:59.595459 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:59.595332    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-601aa9/rate-test-model-601aa9-kserve-f8bc498dc-zfq9z" event={"ID":"275320b5-9a5f-45b7-bb64-8f661730a395","Type":"ContainerStarted","Data":"acbaacd96a481bafeb0b9af9939cec98d2f3485a6784c481f39467d75442ae45"}
Jul 29 20:07:59.612604 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:07:59.612560    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-601aa9/rate-test-model-601aa9-kserve-f8bc498dc-zfq9z" podStartSLOduration=0.612545241 podStartE2EDuration="612.545241ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:07:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:07:59.610971228 +0000 UTC m=+3046.839360354" watchObservedRunningTime="2026-07-29 20:07:59.612545241 +0000 UTC m=+3046.840934367"
Jul 29 20:08:02.793713 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:08:02.793680    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 20:08:09.384379 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:08:09.384298    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-601aa9/rate-test-model-601aa9-kserve-f8bc498dc-zfq9z"
Jul 29 20:08:09.384379 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:08:09.384341    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-601aa9/rate-test-model-601aa9-kserve-f8bc498dc-zfq9z"
Jul 29 20:08:09.396795 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:08:09.396771    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-601aa9/rate-test-model-601aa9-kserve-f8bc498dc-zfq9z"
Jul 29 20:08:09.651175 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:08:09.651088    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-601aa9/rate-test-model-601aa9-kserve-f8bc498dc-zfq9z"
Jul 29 20:08:39.507131 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:08:39.507094    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 20:08:53.997978 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:08:53.997943    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 20:09:12.207702 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:12.207666    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 20:09:17.557467 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:17.557431    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-601aa9/rate-test-model-601aa9-kserve-f8bc498dc-zfq9z"]
Jul 29 20:09:28.300550 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:28.300515    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 20:09:32.576273 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:32.576222    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-601aa9/rate-test-model-601aa9-kserve-f8bc498dc-zfq9z" podUID="275320b5-9a5f-45b7-bb64-8f661730a395" containerName="main" containerID="cri-o://449a6c0dff3b0d48d5509e5f5d5ec172e18092d47e38639de0e29b2b9c25732d" gracePeriod=45
Jul 29 20:09:32.835273 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:32.835207    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-601aa9/rate-test-model-601aa9-kserve-f8bc498dc-zfq9z"
Jul 29 20:09:33.021846 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:33.021782    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/275320b5-9a5f-45b7-bb64-8f661730a395-model-cache\") pod \"275320b5-9a5f-45b7-bb64-8f661730a395\" (UID: \"275320b5-9a5f-45b7-bb64-8f661730a395\") "
Jul 29 20:09:33.021846 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:33.021851    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/275320b5-9a5f-45b7-bb64-8f661730a395-home\") pod \"275320b5-9a5f-45b7-bb64-8f661730a395\" (UID: \"275320b5-9a5f-45b7-bb64-8f661730a395\") "
Jul 29 20:09:33.022084 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:33.021903    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/275320b5-9a5f-45b7-bb64-8f661730a395-dshm\") pod \"275320b5-9a5f-45b7-bb64-8f661730a395\" (UID: \"275320b5-9a5f-45b7-bb64-8f661730a395\") "
Jul 29 20:09:33.022084 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:33.021926    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/275320b5-9a5f-45b7-bb64-8f661730a395-tmp-dir\") pod \"275320b5-9a5f-45b7-bb64-8f661730a395\" (UID: \"275320b5-9a5f-45b7-bb64-8f661730a395\") "
Jul 29 20:09:33.022084 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:33.021948    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5tj9\" (UniqueName: \"kubernetes.io/projected/275320b5-9a5f-45b7-bb64-8f661730a395-kube-api-access-q5tj9\") pod \"275320b5-9a5f-45b7-bb64-8f661730a395\" (UID: \"275320b5-9a5f-45b7-bb64-8f661730a395\") "
Jul 29 20:09:33.022084 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:33.021998    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/275320b5-9a5f-45b7-bb64-8f661730a395-tls-certs\") pod \"275320b5-9a5f-45b7-bb64-8f661730a395\" (UID: \"275320b5-9a5f-45b7-bb64-8f661730a395\") "
Jul 29 20:09:33.022296 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:33.022159    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/275320b5-9a5f-45b7-bb64-8f661730a395-home" (OuterVolumeSpecName: "home") pod "275320b5-9a5f-45b7-bb64-8f661730a395" (UID: "275320b5-9a5f-45b7-bb64-8f661730a395"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:09:33.022296 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:33.022199    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/275320b5-9a5f-45b7-bb64-8f661730a395-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "275320b5-9a5f-45b7-bb64-8f661730a395" (UID: "275320b5-9a5f-45b7-bb64-8f661730a395"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:09:33.022296 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:33.022133    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/275320b5-9a5f-45b7-bb64-8f661730a395-model-cache" (OuterVolumeSpecName: "model-cache") pod "275320b5-9a5f-45b7-bb64-8f661730a395" (UID: "275320b5-9a5f-45b7-bb64-8f661730a395"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:09:33.022431 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:33.022324    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/275320b5-9a5f-45b7-bb64-8f661730a395-model-cache\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:09:33.022431 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:33.022336    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/275320b5-9a5f-45b7-bb64-8f661730a395-home\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:09:33.022431 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:33.022345    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/275320b5-9a5f-45b7-bb64-8f661730a395-tmp-dir\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:09:33.024000 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:33.023976    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/275320b5-9a5f-45b7-bb64-8f661730a395-dshm" (OuterVolumeSpecName: "dshm") pod "275320b5-9a5f-45b7-bb64-8f661730a395" (UID: "275320b5-9a5f-45b7-bb64-8f661730a395"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:09:33.024119 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:33.024060    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/275320b5-9a5f-45b7-bb64-8f661730a395-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "275320b5-9a5f-45b7-bb64-8f661730a395" (UID: "275320b5-9a5f-45b7-bb64-8f661730a395"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:09:33.024237 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:33.024205    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/275320b5-9a5f-45b7-bb64-8f661730a395-kube-api-access-q5tj9" (OuterVolumeSpecName: "kube-api-access-q5tj9") pod "275320b5-9a5f-45b7-bb64-8f661730a395" (UID: "275320b5-9a5f-45b7-bb64-8f661730a395"). InnerVolumeSpecName "kube-api-access-q5tj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:09:33.029461 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:33.029442    2583 generic.go:356] "Generic (PLEG): container finished" podID="275320b5-9a5f-45b7-bb64-8f661730a395" containerID="449a6c0dff3b0d48d5509e5f5d5ec172e18092d47e38639de0e29b2b9c25732d" exitCode=0
Jul 29 20:09:33.029527 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:33.029500    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-601aa9/rate-test-model-601aa9-kserve-f8bc498dc-zfq9z"
Jul 29 20:09:33.029585 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:33.029529    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-601aa9/rate-test-model-601aa9-kserve-f8bc498dc-zfq9z" event={"ID":"275320b5-9a5f-45b7-bb64-8f661730a395","Type":"ContainerDied","Data":"449a6c0dff3b0d48d5509e5f5d5ec172e18092d47e38639de0e29b2b9c25732d"}
Jul 29 20:09:33.029640 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:33.029584    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-601aa9/rate-test-model-601aa9-kserve-f8bc498dc-zfq9z" event={"ID":"275320b5-9a5f-45b7-bb64-8f661730a395","Type":"ContainerDied","Data":"acbaacd96a481bafeb0b9af9939cec98d2f3485a6784c481f39467d75442ae45"}
Jul 29 20:09:33.029640 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:33.029610    2583 scope.go:117] "RemoveContainer" containerID="449a6c0dff3b0d48d5509e5f5d5ec172e18092d47e38639de0e29b2b9c25732d"
Jul 29 20:09:33.044208 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:33.044189    2583 scope.go:117] "RemoveContainer" containerID="449a6c0dff3b0d48d5509e5f5d5ec172e18092d47e38639de0e29b2b9c25732d"
Jul 29 20:09:33.044639 ip-10-0-128-101 kubenswrapper[2583]: E0729 20:09:33.044612    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"449a6c0dff3b0d48d5509e5f5d5ec172e18092d47e38639de0e29b2b9c25732d\": container with ID starting with 449a6c0dff3b0d48d5509e5f5d5ec172e18092d47e38639de0e29b2b9c25732d not found: ID does not exist" containerID="449a6c0dff3b0d48d5509e5f5d5ec172e18092d47e38639de0e29b2b9c25732d"
Jul 29 20:09:33.044737 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:33.044646    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"449a6c0dff3b0d48d5509e5f5d5ec172e18092d47e38639de0e29b2b9c25732d"} err="failed to get container status \"449a6c0dff3b0d48d5509e5f5d5ec172e18092d47e38639de0e29b2b9c25732d\": rpc error: code = NotFound desc = could not find container \"449a6c0dff3b0d48d5509e5f5d5ec172e18092d47e38639de0e29b2b9c25732d\": container with ID starting with 449a6c0dff3b0d48d5509e5f5d5ec172e18092d47e38639de0e29b2b9c25732d not found: ID does not exist"
Jul 29 20:09:33.054023 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:33.053997    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-601aa9/rate-test-model-601aa9-kserve-f8bc498dc-zfq9z"]
Jul 29 20:09:33.057962 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:33.057941    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-601aa9/rate-test-model-601aa9-kserve-f8bc498dc-zfq9z"]
Jul 29 20:09:33.123069 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:33.122988    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/275320b5-9a5f-45b7-bb64-8f661730a395-dshm\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:09:33.123069 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:33.123013    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5tj9\" (UniqueName: \"kubernetes.io/projected/275320b5-9a5f-45b7-bb64-8f661730a395-kube-api-access-q5tj9\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:09:33.123069 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:33.123022    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/275320b5-9a5f-45b7-bb64-8f661730a395-tls-certs\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:09:33.341134 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:33.341101    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="275320b5-9a5f-45b7-bb64-8f661730a395" path="/var/lib/kubelet/pods/275320b5-9a5f-45b7-bb64-8f661730a395/volumes"
Jul 29 20:09:41.198544 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:41.198460    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb0399-bf6746987-pvcts"]
Jul 29 20:09:41.198997 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:41.198975    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="275320b5-9a5f-45b7-bb64-8f661730a395" containerName="main"
Jul 29 20:09:41.198997 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:41.198994    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="275320b5-9a5f-45b7-bb64-8f661730a395" containerName="main"
Jul 29 20:09:41.199133 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:41.199092    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="275320b5-9a5f-45b7-bb64-8f661730a395" containerName="main"
Jul 29 20:09:41.203647 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:41.203626    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb0399-bf6746987-pvcts"
Jul 29 20:09:41.205851 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:41.205830    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-fb0399-serving-cert\""
Jul 29 20:09:41.213269 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:41.213243    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb0399-bf6746987-pvcts"]
Jul 29 20:09:41.292117 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:41.292078    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2219514b-7cbd-4839-9282-4639dad3ba28-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-fb0399-bf6746987-pvcts\" (UID: \"2219514b-7cbd-4839-9282-4639dad3ba28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb0399-bf6746987-pvcts"
Jul 29 20:09:41.292296 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:41.292147    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcpzl\" (UniqueName: \"kubernetes.io/projected/2219514b-7cbd-4839-9282-4639dad3ba28-kube-api-access-lcpzl\") pod \"maas-api-e2e-ipp-a-fb0399-bf6746987-pvcts\" (UID: \"2219514b-7cbd-4839-9282-4639dad3ba28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb0399-bf6746987-pvcts"
Jul 29 20:09:41.292296 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:41.292186    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2219514b-7cbd-4839-9282-4639dad3ba28-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fb0399-bf6746987-pvcts\" (UID: \"2219514b-7cbd-4839-9282-4639dad3ba28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb0399-bf6746987-pvcts"
Jul 29 20:09:41.393450 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:41.393411    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2219514b-7cbd-4839-9282-4639dad3ba28-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-fb0399-bf6746987-pvcts\" (UID: \"2219514b-7cbd-4839-9282-4639dad3ba28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb0399-bf6746987-pvcts"
Jul 29 20:09:41.393668 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:41.393495    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcpzl\" (UniqueName: \"kubernetes.io/projected/2219514b-7cbd-4839-9282-4639dad3ba28-kube-api-access-lcpzl\") pod \"maas-api-e2e-ipp-a-fb0399-bf6746987-pvcts\" (UID: \"2219514b-7cbd-4839-9282-4639dad3ba28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb0399-bf6746987-pvcts"
Jul 29 20:09:41.393796 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:41.393753    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2219514b-7cbd-4839-9282-4639dad3ba28-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fb0399-bf6746987-pvcts\" (UID: \"2219514b-7cbd-4839-9282-4639dad3ba28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb0399-bf6746987-pvcts"
Jul 29 20:09:41.393893 ip-10-0-128-101 kubenswrapper[2583]: E0729 20:09:41.393874    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb0399-serving-cert: secret "maas-api-e2e-ipp-a-fb0399-serving-cert" not found
Jul 29 20:09:41.393956 ip-10-0-128-101 kubenswrapper[2583]: E0729 20:09:41.393946    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2219514b-7cbd-4839-9282-4639dad3ba28-maas-api-tls podName:2219514b-7cbd-4839-9282-4639dad3ba28 nodeName:}" failed. No retries permitted until 2026-07-29 20:09:41.893929252 +0000 UTC m=+3149.122318358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2219514b-7cbd-4839-9282-4639dad3ba28-maas-api-tls") pod "maas-api-e2e-ipp-a-fb0399-bf6746987-pvcts" (UID: "2219514b-7cbd-4839-9282-4639dad3ba28") : secret "maas-api-e2e-ipp-a-fb0399-serving-cert" not found
Jul 29 20:09:41.394109 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:41.394090    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2219514b-7cbd-4839-9282-4639dad3ba28-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-fb0399-bf6746987-pvcts\" (UID: \"2219514b-7cbd-4839-9282-4639dad3ba28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb0399-bf6746987-pvcts"
Jul 29 20:09:41.403726 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:41.403703    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcpzl\" (UniqueName: \"kubernetes.io/projected/2219514b-7cbd-4839-9282-4639dad3ba28-kube-api-access-lcpzl\") pod \"maas-api-e2e-ipp-a-fb0399-bf6746987-pvcts\" (UID: \"2219514b-7cbd-4839-9282-4639dad3ba28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb0399-bf6746987-pvcts"
Jul 29 20:09:41.897540 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:41.897502    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2219514b-7cbd-4839-9282-4639dad3ba28-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fb0399-bf6746987-pvcts\" (UID: \"2219514b-7cbd-4839-9282-4639dad3ba28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb0399-bf6746987-pvcts"
Jul 29 20:09:41.900265 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:41.900240    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2219514b-7cbd-4839-9282-4639dad3ba28-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fb0399-bf6746987-pvcts\" (UID: \"2219514b-7cbd-4839-9282-4639dad3ba28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb0399-bf6746987-pvcts"
Jul 29 20:09:42.114565 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:42.114518    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb0399-bf6746987-pvcts"
Jul 29 20:09:42.248827 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:42.248785    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb0399-bf6746987-pvcts"]
Jul 29 20:09:42.251092 ip-10-0-128-101 kubenswrapper[2583]: W0729 20:09:42.251040    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2219514b_7cbd_4839_9282_4639dad3ba28.slice/crio-25b99a0e8a0e19d5afcac0a69cab8517f5afbb130d1932ea82099444695c79fa WatchSource:0}: Error finding container 25b99a0e8a0e19d5afcac0a69cab8517f5afbb130d1932ea82099444695c79fa: Status 404 returned error can't find the container with id 25b99a0e8a0e19d5afcac0a69cab8517f5afbb130d1932ea82099444695c79fa
Jul 29 20:09:42.252369 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:42.252348    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 20:09:43.079525 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:43.079433    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb0399-bf6746987-pvcts" event={"ID":"2219514b-7cbd-4839-9282-4639dad3ba28","Type":"ContainerStarted","Data":"1e799a8699b96817d6c0b08dff6a0abeb9b313b4a54199948a7aa03925097c5f"}
Jul 29 20:09:43.079525 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:43.079490    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb0399-bf6746987-pvcts" event={"ID":"2219514b-7cbd-4839-9282-4639dad3ba28","Type":"ContainerStarted","Data":"25b99a0e8a0e19d5afcac0a69cab8517f5afbb130d1932ea82099444695c79fa"}
Jul 29 20:09:43.079836 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:43.079561    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb0399-bf6746987-pvcts"
Jul 29 20:09:43.096418 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:43.096365    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb0399-bf6746987-pvcts" podStartSLOduration=1.578475173 podStartE2EDuration="2.096350392s" totalImagesPullingTime="517.875219ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:09:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:09:43.09407275 +0000 UTC m=+3150.322461876" watchObservedRunningTime="2026-07-29 20:09:43.096350392 +0000 UTC m=+3150.324739517"
Jul 29 20:09:49.090177 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:49.090145    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb0399-bf6746987-pvcts"
Jul 29 20:09:49.996385 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:09:49.996355    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 20:10:09.710540 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:10:09.710501    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6488bc-86578d55f8-tqxwz"]
Jul 29 20:10:09.714277 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:10:09.714256    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6488bc-86578d55f8-tqxwz"
Jul 29 20:10:09.716591 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:10:09.716569    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-6488bc-serving-cert\""
Jul 29 20:10:09.724410 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:10:09.724386    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6488bc-86578d55f8-tqxwz"]
Jul 29 20:10:09.847848 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:10:09.847779    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95221161-2bd1-405c-b856-8c51f3dff04d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6488bc-86578d55f8-tqxwz\" (UID: \"95221161-2bd1-405c-b856-8c51f3dff04d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6488bc-86578d55f8-tqxwz"
Jul 29 20:10:09.848025 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:10:09.847866    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grq5t\" (UniqueName: \"kubernetes.io/projected/95221161-2bd1-405c-b856-8c51f3dff04d-kube-api-access-grq5t\") pod \"maas-api-e2e-ipp-b-6488bc-86578d55f8-tqxwz\" (UID: \"95221161-2bd1-405c-b856-8c51f3dff04d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6488bc-86578d55f8-tqxwz"
Jul 29 20:10:09.848025 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:10:09.847893    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95221161-2bd1-405c-b856-8c51f3dff04d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6488bc-86578d55f8-tqxwz\" (UID: \"95221161-2bd1-405c-b856-8c51f3dff04d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6488bc-86578d55f8-tqxwz"
Jul 29 20:10:09.948512 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:10:09.948471    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grq5t\" (UniqueName: \"kubernetes.io/projected/95221161-2bd1-405c-b856-8c51f3dff04d-kube-api-access-grq5t\") pod \"maas-api-e2e-ipp-b-6488bc-86578d55f8-tqxwz\" (UID: \"95221161-2bd1-405c-b856-8c51f3dff04d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6488bc-86578d55f8-tqxwz"
Jul 29 20:10:09.948698 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:10:09.948523    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95221161-2bd1-405c-b856-8c51f3dff04d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6488bc-86578d55f8-tqxwz\" (UID: \"95221161-2bd1-405c-b856-8c51f3dff04d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6488bc-86578d55f8-tqxwz"
Jul 29 20:10:09.948698 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:10:09.948591    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95221161-2bd1-405c-b856-8c51f3dff04d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6488bc-86578d55f8-tqxwz\" (UID: \"95221161-2bd1-405c-b856-8c51f3dff04d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6488bc-86578d55f8-tqxwz"
Jul 29 20:10:09.948698 ip-10-0-128-101 kubenswrapper[2583]: E0729 20:10:09.948685    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-6488bc-serving-cert: secret "maas-api-e2e-ipp-b-6488bc-serving-cert" not found
Jul 29 20:10:09.948829 ip-10-0-128-101 kubenswrapper[2583]: E0729 20:10:09.948747    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95221161-2bd1-405c-b856-8c51f3dff04d-maas-api-tls podName:95221161-2bd1-405c-b856-8c51f3dff04d nodeName:}" failed. No retries permitted until 2026-07-29 20:10:10.448730098 +0000 UTC m=+3177.677119203 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/95221161-2bd1-405c-b856-8c51f3dff04d-maas-api-tls") pod "maas-api-e2e-ipp-b-6488bc-86578d55f8-tqxwz" (UID: "95221161-2bd1-405c-b856-8c51f3dff04d") : secret "maas-api-e2e-ipp-b-6488bc-serving-cert" not found
Jul 29 20:10:09.949278 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:10:09.949260    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95221161-2bd1-405c-b856-8c51f3dff04d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6488bc-86578d55f8-tqxwz\" (UID: \"95221161-2bd1-405c-b856-8c51f3dff04d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6488bc-86578d55f8-tqxwz"
Jul 29 20:10:09.960657 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:10:09.960588    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grq5t\" (UniqueName: \"kubernetes.io/projected/95221161-2bd1-405c-b856-8c51f3dff04d-kube-api-access-grq5t\") pod \"maas-api-e2e-ipp-b-6488bc-86578d55f8-tqxwz\" (UID: \"95221161-2bd1-405c-b856-8c51f3dff04d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6488bc-86578d55f8-tqxwz"
Jul 29 20:10:10.452831 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:10:10.452782    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95221161-2bd1-405c-b856-8c51f3dff04d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6488bc-86578d55f8-tqxwz\" (UID: \"95221161-2bd1-405c-b856-8c51f3dff04d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6488bc-86578d55f8-tqxwz"
Jul 29 20:10:10.453762 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:10:10.453739    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6488bc-6dbf565b78-5cptz"]
Jul 29 20:10:10.455487 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:10:10.455464    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95221161-2bd1-405c-b856-8c51f3dff04d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6488bc-86578d55f8-tqxwz\" (UID: \"95221161-2bd1-405c-b856-8c51f3dff04d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6488bc-86578d55f8-tqxwz"
Jul 29 20:10:10.457285 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:10:10.457271    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6488bc-6dbf565b78-5cptz"
Jul 29 20:10:10.459472 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:10:10.459454    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-6488bc\""
Jul 29 20:10:10.459472 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:10:10.459461    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-6488bc-dockercfg-5grbh\""
Jul 29 20:10:10.465658 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:10:10.465632    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6488bc-6dbf565b78-5cptz"]
Jul 29 20:10:10.554239 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:10:10.554204    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/297f43e6-7f3c-4bc5-8675-ab57c4728032-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6488bc-6dbf565b78-5cptz\" (UID: \"297f43e6-7f3c-4bc5-8675-ab57c4728032\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6488bc-6dbf565b78-5cptz"
Jul 29 20:10:10.554425 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:10:10.554245    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhkjk\" (UniqueName: \"kubernetes.io/projected/297f43e6-7f3c-4bc5-8675-ab57c4728032-kube-api-access-jhkjk\") pod \"payload-processing-e2e-ipp-b-6488bc-6dbf565b78-5cptz\" (UID: \"297f43e6-7f3c-4bc5-8675-ab57c4728032\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6488bc-6dbf565b78-5cptz"
Jul 29 20:10:10.625911 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:10:10.625881    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6488bc-86578d55f8-tqxwz"
Jul 29 20:10:10.655070 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:10:10.655040    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/297f43e6-7f3c-4bc5-8675-ab57c4728032-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6488bc-6dbf565b78-5cptz\" (UID: \"297f43e6-7f3c-4bc5-8675-ab57c4728032\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6488bc-6dbf565b78-5cptz"
Jul 29 20:10:10.655226 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:10:10.655081    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhkjk\" (UniqueName: \"kubernetes.io/projected/297f43e6-7f3c-4bc5-8675-ab57c4728032-kube-api-access-jhkjk\") pod \"payload-processing-e2e-ipp-b-6488bc-6dbf565b78-5cptz\" (UID: \"297f43e6-7f3c-4bc5-8675-ab57c4728032\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6488bc-6dbf565b78-5cptz"
Jul 29 20:10:10.655757 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:10:10.655739    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/297f43e6-7f3c-4bc5-8675-ab57c4728032-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6488bc-6dbf565b78-5cptz\" (UID: \"297f43e6-7f3c-4bc5-8675-ab57c4728032\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6488bc-6dbf565b78-5cptz"
Jul 29 20:10:10.662894 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:10:10.662866    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhkjk\" (UniqueName: \"kubernetes.io/projected/297f43e6-7f3c-4bc5-8675-ab57c4728032-kube-api-access-jhkjk\") pod \"payload-processing-e2e-ipp-b-6488bc-6dbf565b78-5cptz\" (UID: \"297f43e6-7f3c-4bc5-8675-ab57c4728032\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6488bc-6dbf565b78-5cptz"
Jul 29 20:10:10.756119 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:10:10.756084    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6488bc-86578d55f8-tqxwz"]
Jul 29 20:10:10.758912 ip-10-0-128-101 kubenswrapper[2583]: W0729 20:10:10.758887    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95221161_2bd1_405c_b856_8c51f3dff04d.slice/crio-31e63d509e8c1b6877c86bb5f69bc3fc797149b7a4835e0ba510dccd3709bcad WatchSource:0}: Error finding container 31e63d509e8c1b6877c86bb5f69bc3fc797149b7a4835e0ba510dccd3709bcad: Status 404 returned error can't find the container with id 31e63d509e8c1b6877c86bb5f69bc3fc797149b7a4835e0ba510dccd3709bcad
Jul 29 20:10:10.772546 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:10:10.772515    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6488bc-6dbf565b78-5cptz"
Jul 29 20:10:10.904457 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:10:10.904430    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6488bc-6dbf565b78-5cptz"]
Jul 29 20:10:10.906368 ip-10-0-128-101 kubenswrapper[2583]: W0729 20:10:10.906335    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod297f43e6_7f3c_4bc5_8675_ab57c4728032.slice/crio-c9424dcd082364abd07b978c1e2ad91267bba104d2a2fddffd109756f298afc4 WatchSource:0}: Error finding container c9424dcd082364abd07b978c1e2ad91267bba104d2a2fddffd109756f298afc4: Status 404 returned error can't find the container with id c9424dcd082364abd07b978c1e2ad91267bba104d2a2fddffd109756f298afc4
Jul 29 20:10:11.227240 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:10:11.227128    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6488bc-86578d55f8-tqxwz" event={"ID":"95221161-2bd1-405c-b856-8c51f3dff04d","Type":"ContainerStarted","Data":"80d377a3c25a65d0260a442394fdde59fbc6d2934bf6de5110388971a5a55d57"}
Jul 29 20:10:11.227413 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:10:11.227305    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6488bc-86578d55f8-tqxwz"
Jul 29 20:10:11.227413 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:10:11.227348    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6488bc-86578d55f8-tqxwz" event={"ID":"95221161-2bd1-405c-b856-8c51f3dff04d","Type":"ContainerStarted","Data":"31e63d509e8c1b6877c86bb5f69bc3fc797149b7a4835e0ba510dccd3709bcad"}
Jul 29 20:10:11.228757 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:10:11.228735    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6488bc-6dbf565b78-5cptz" event={"ID":"297f43e6-7f3c-4bc5-8675-ab57c4728032","Type":"ContainerStarted","Data":"f6cd1fae7130a2f9e64978a5e991cfde96f784c28b9e1c4e2a03eb244b01903a"}
Jul 29 20:10:11.228890 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:10:11.228773    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6488bc-6dbf565b78-5cptz" event={"ID":"297f43e6-7f3c-4bc5-8675-ab57c4728032","Type":"ContainerStarted","Data":"c9424dcd082364abd07b978c1e2ad91267bba104d2a2fddffd109756f298afc4"}
Jul 29 20:10:11.228890 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:10:11.228841    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6488bc-6dbf565b78-5cptz"
Jul 29 20:10:11.245337 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:10:11.245294    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6488bc-86578d55f8-tqxwz" podStartSLOduration=1.8953303620000002 podStartE2EDuration="2.245280094s" totalImagesPullingTime="349.949732ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:10:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:10:11.24372445 +0000 UTC m=+3178.472113576" watchObservedRunningTime="2026-07-29 20:10:11.245280094 +0000 UTC m=+3178.473669221"
Jul 29 20:10:11.260675 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:10:11.260630    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-6488bc-6dbf565b78-5cptz" podStartSLOduration=1.260617303 podStartE2EDuration="1.260617303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:10:11.259519544 +0000 UTC m=+3178.487908683" watchObservedRunningTime="2026-07-29 20:10:11.260617303 +0000 UTC m=+3178.489006427"
Jul 29 20:10:17.239730 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:10:17.239701    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6488bc-86578d55f8-tqxwz"
Jul 29 20:10:18.806477 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:10:18.806440    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 20:10:22.236607 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:10:22.236582    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6488bc-6dbf565b78-5cptz"
Jul 29 20:10:41.521435 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:10:41.521399    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-6488bc-6dbf565b78-5cptz_297f43e6-7f3c-4bc5-8675-ab57c4728032/payload-processing/0.log"
Jul 29 20:10:58.396208 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:10:58.396175    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 20:11:26.205676 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:11:26.205586    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 20:11:36.733610 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:11:36.733581    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-6488bc-6dbf565b78-5cptz_297f43e6-7f3c-4bc5-8675-ab57c4728032/payload-processing/0.log"
Jul 29 20:11:40.037877 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:11:40.037837    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7720d1-5d4889bcd-885bz"]
Jul 29 20:11:40.041189 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:11:40.041172    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7720d1-5d4889bcd-885bz"
Jul 29 20:11:40.043789 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:11:40.043766    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-7720d1-serving-cert\""
Jul 29 20:11:40.062062 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:11:40.062028    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7720d1-5d4889bcd-885bz"]
Jul 29 20:11:40.099603 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:11:40.099572    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2sqt\" (UniqueName: \"kubernetes.io/projected/904748ff-ca23-4ce7-aa4b-f63bed95888e-kube-api-access-m2sqt\") pod \"maas-api-e2e-ipp-cleanup-7720d1-5d4889bcd-885bz\" (UID: \"904748ff-ca23-4ce7-aa4b-f63bed95888e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7720d1-5d4889bcd-885bz"
Jul 29 20:11:40.099780 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:11:40.099636    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/904748ff-ca23-4ce7-aa4b-f63bed95888e-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-7720d1-5d4889bcd-885bz\" (UID: \"904748ff-ca23-4ce7-aa4b-f63bed95888e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7720d1-5d4889bcd-885bz"
Jul 29 20:11:40.099780 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:11:40.099686    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/904748ff-ca23-4ce7-aa4b-f63bed95888e-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-7720d1-5d4889bcd-885bz\" (UID: \"904748ff-ca23-4ce7-aa4b-f63bed95888e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7720d1-5d4889bcd-885bz"
Jul 29 20:11:40.200692 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:11:40.200650    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/904748ff-ca23-4ce7-aa4b-f63bed95888e-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-7720d1-5d4889bcd-885bz\" (UID: \"904748ff-ca23-4ce7-aa4b-f63bed95888e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7720d1-5d4889bcd-885bz"
Jul 29 20:11:40.200910 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:11:40.200699    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/904748ff-ca23-4ce7-aa4b-f63bed95888e-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-7720d1-5d4889bcd-885bz\" (UID: \"904748ff-ca23-4ce7-aa4b-f63bed95888e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7720d1-5d4889bcd-885bz"
Jul 29 20:11:40.200910 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:11:40.200813    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2sqt\" (UniqueName: \"kubernetes.io/projected/904748ff-ca23-4ce7-aa4b-f63bed95888e-kube-api-access-m2sqt\") pod \"maas-api-e2e-ipp-cleanup-7720d1-5d4889bcd-885bz\" (UID: \"904748ff-ca23-4ce7-aa4b-f63bed95888e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7720d1-5d4889bcd-885bz"
Jul 29 20:11:40.200910 ip-10-0-128-101 kubenswrapper[2583]: E0729 20:11:40.200875    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7720d1-serving-cert: secret "maas-api-e2e-ipp-cleanup-7720d1-serving-cert" not found
Jul 29 20:11:40.201054 ip-10-0-128-101 kubenswrapper[2583]: E0729 20:11:40.200936    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/904748ff-ca23-4ce7-aa4b-f63bed95888e-maas-api-tls podName:904748ff-ca23-4ce7-aa4b-f63bed95888e nodeName:}" failed. No retries permitted until 2026-07-29 20:11:40.700920419 +0000 UTC m=+3267.929309523 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/904748ff-ca23-4ce7-aa4b-f63bed95888e-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-7720d1-5d4889bcd-885bz" (UID: "904748ff-ca23-4ce7-aa4b-f63bed95888e") : secret "maas-api-e2e-ipp-cleanup-7720d1-serving-cert" not found
Jul 29 20:11:40.201417 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:11:40.201389    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/904748ff-ca23-4ce7-aa4b-f63bed95888e-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-7720d1-5d4889bcd-885bz\" (UID: \"904748ff-ca23-4ce7-aa4b-f63bed95888e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7720d1-5d4889bcd-885bz"
Jul 29 20:11:40.213420 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:11:40.213398    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2sqt\" (UniqueName: \"kubernetes.io/projected/904748ff-ca23-4ce7-aa4b-f63bed95888e-kube-api-access-m2sqt\") pod \"maas-api-e2e-ipp-cleanup-7720d1-5d4889bcd-885bz\" (UID: \"904748ff-ca23-4ce7-aa4b-f63bed95888e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7720d1-5d4889bcd-885bz"
Jul 29 20:11:40.704987 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:11:40.704949    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/904748ff-ca23-4ce7-aa4b-f63bed95888e-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-7720d1-5d4889bcd-885bz\" (UID: \"904748ff-ca23-4ce7-aa4b-f63bed95888e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7720d1-5d4889bcd-885bz"
Jul 29 20:11:40.707357 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:11:40.707331    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/904748ff-ca23-4ce7-aa4b-f63bed95888e-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-7720d1-5d4889bcd-885bz\" (UID: \"904748ff-ca23-4ce7-aa4b-f63bed95888e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7720d1-5d4889bcd-885bz"
Jul 29 20:11:40.802317 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:11:40.802275    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-7720d1-8576c6fd58-xc6pr"]
Jul 29 20:11:40.806028 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:11:40.806006    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7720d1-8576c6fd58-xc6pr"
Jul 29 20:11:40.808196 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:11:40.808174    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-7720d1\""
Jul 29 20:11:40.808323 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:11:40.808185    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-7720d1-dockercfg-qd4qs\""
Jul 29 20:11:40.817359 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:11:40.817337    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-7720d1-8576c6fd58-xc6pr"]
Jul 29 20:11:40.906606 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:11:40.906574    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aaaff35-1d8c-48d0-bb80-9daf321b965d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-7720d1-8576c6fd58-xc6pr\" (UID: \"2aaaff35-1d8c-48d0-bb80-9daf321b965d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7720d1-8576c6fd58-xc6pr"
Jul 29 20:11:40.906783 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:11:40.906621    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r86mf\" (UniqueName: \"kubernetes.io/projected/2aaaff35-1d8c-48d0-bb80-9daf321b965d-kube-api-access-r86mf\") pod \"payload-processing-e2e-ipp-cleanup-7720d1-8576c6fd58-xc6pr\" (UID: \"2aaaff35-1d8c-48d0-bb80-9daf321b965d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7720d1-8576c6fd58-xc6pr"
Jul 29 20:11:40.951593 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:11:40.951555    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7720d1-5d4889bcd-885bz"
Jul 29 20:11:41.007446 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:11:41.007405    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aaaff35-1d8c-48d0-bb80-9daf321b965d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-7720d1-8576c6fd58-xc6pr\" (UID: \"2aaaff35-1d8c-48d0-bb80-9daf321b965d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7720d1-8576c6fd58-xc6pr"
Jul 29 20:11:41.007605 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:11:41.007492    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r86mf\" (UniqueName: \"kubernetes.io/projected/2aaaff35-1d8c-48d0-bb80-9daf321b965d-kube-api-access-r86mf\") pod \"payload-processing-e2e-ipp-cleanup-7720d1-8576c6fd58-xc6pr\" (UID: \"2aaaff35-1d8c-48d0-bb80-9daf321b965d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7720d1-8576c6fd58-xc6pr"
Jul 29 20:11:41.008227 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:11:41.008205    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aaaff35-1d8c-48d0-bb80-9daf321b965d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-7720d1-8576c6fd58-xc6pr\" (UID: \"2aaaff35-1d8c-48d0-bb80-9daf321b965d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7720d1-8576c6fd58-xc6pr"
Jul 29 20:11:41.022202 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:11:41.022170    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r86mf\" (UniqueName: \"kubernetes.io/projected/2aaaff35-1d8c-48d0-bb80-9daf321b965d-kube-api-access-r86mf\") pod \"payload-processing-e2e-ipp-cleanup-7720d1-8576c6fd58-xc6pr\" (UID: \"2aaaff35-1d8c-48d0-bb80-9daf321b965d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7720d1-8576c6fd58-xc6pr"
Jul 29 20:11:41.110155 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:11:41.110129    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7720d1-5d4889bcd-885bz"]
Jul 29 20:11:41.112054 ip-10-0-128-101 kubenswrapper[2583]: W0729 20:11:41.112028    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod904748ff_ca23_4ce7_aa4b_f63bed95888e.slice/crio-e2ae8732bb3e2159051833676b3a87f4d7dd4e069abe9db69f2172559958b74e WatchSource:0}: Error finding container e2ae8732bb3e2159051833676b3a87f4d7dd4e069abe9db69f2172559958b74e: Status 404 returned error can't find the container with id e2ae8732bb3e2159051833676b3a87f4d7dd4e069abe9db69f2172559958b74e
Jul 29 20:11:41.117935 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:11:41.117914    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7720d1-8576c6fd58-xc6pr"
Jul 29 20:11:41.264392 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:11:41.264366    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-7720d1-8576c6fd58-xc6pr"]
Jul 29 20:11:41.266291 ip-10-0-128-101 kubenswrapper[2583]: W0729 20:11:41.266266    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aaaff35_1d8c_48d0_bb80_9daf321b965d.slice/crio-14c948c2a380705f7a76cd320c2eb67811a4f83b6c87dc3c6c9cf26cdb23a4d8 WatchSource:0}: Error finding container 14c948c2a380705f7a76cd320c2eb67811a4f83b6c87dc3c6c9cf26cdb23a4d8: Status 404 returned error can't find the container with id 14c948c2a380705f7a76cd320c2eb67811a4f83b6c87dc3c6c9cf26cdb23a4d8
Jul 29 20:11:41.651546 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:11:41.651511    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7720d1-8576c6fd58-xc6pr" event={"ID":"2aaaff35-1d8c-48d0-bb80-9daf321b965d","Type":"ContainerStarted","Data":"ce962aa5b052de14dd3a220894ef5633c62db9bf00707edbb33032c44816911d"}
Jul 29 20:11:41.651677 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:11:41.651571    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7720d1-8576c6fd58-xc6pr" event={"ID":"2aaaff35-1d8c-48d0-bb80-9daf321b965d","Type":"ContainerStarted","Data":"14c948c2a380705f7a76cd320c2eb67811a4f83b6c87dc3c6c9cf26cdb23a4d8"}
Jul 29 20:11:41.651677 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:11:41.651616    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7720d1-8576c6fd58-xc6pr"
Jul 29 20:11:41.652836 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:11:41.652788    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7720d1-5d4889bcd-885bz" event={"ID":"904748ff-ca23-4ce7-aa4b-f63bed95888e","Type":"ContainerStarted","Data":"e2ae8732bb3e2159051833676b3a87f4d7dd4e069abe9db69f2172559958b74e"}
Jul 29 20:11:41.680772 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:11:41.680721    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7720d1-8576c6fd58-xc6pr" podStartSLOduration=1.6807021070000001 podStartE2EDuration="1.680702107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:11:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:11:41.678989533 +0000 UTC m=+3268.907378663" watchObservedRunningTime="2026-07-29 20:11:41.680702107 +0000 UTC m=+3268.909091235"
Jul 29 20:11:42.660382 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:11:42.660349    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7720d1-5d4889bcd-885bz" event={"ID":"904748ff-ca23-4ce7-aa4b-f63bed95888e","Type":"ContainerStarted","Data":"2b0af7dd718ac02ad9fd9ac74ba0eef8d28b3df6242934eafec5f9988cc31040"}
Jul 29 20:11:42.660815 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:11:42.660675    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7720d1-5d4889bcd-885bz"
Jul 29 20:11:42.680261 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:11:42.680195    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7720d1-5d4889bcd-885bz" podStartSLOduration=3.150770661 podStartE2EDuration="3.680173303s" totalImagesPullingTime="529.402642ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:11:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:11:42.679307933 +0000 UTC m=+3269.907697101" watchObservedRunningTime="2026-07-29 20:11:42.680173303 +0000 UTC m=+3269.908562429"
Jul 29 20:11:46.303647 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:11:46.303609    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 20:11:48.678046 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:11:48.678017    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7720d1-5d4889bcd-885bz"
Jul 29 20:11:52.662433 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:11:52.662400    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7720d1-8576c6fd58-xc6pr"
Jul 29 20:12:05.391572 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:05.391532    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7720d1-9zk9f"]
Jul 29 20:12:05.395462 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:05.395437    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7720d1-9zk9f"
Jul 29 20:12:05.397720 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:05.397698    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hq9l2\""
Jul 29 20:12:05.403702 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:05.403677    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7720d1-9zk9f"]
Jul 29 20:12:05.528346 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:05.528303    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bff70b45-6546-4ec7-add4-e19e9fc07e5c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-7720d1-9zk9f\" (UID: \"bff70b45-6546-4ec7-add4-e19e9fc07e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7720d1-9zk9f"
Jul 29 20:12:05.628920 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:05.628881    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bff70b45-6546-4ec7-add4-e19e9fc07e5c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-7720d1-9zk9f\" (UID: \"bff70b45-6546-4ec7-add4-e19e9fc07e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7720d1-9zk9f"
Jul 29 20:12:05.629583 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:05.629562    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bff70b45-6546-4ec7-add4-e19e9fc07e5c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-7720d1-9zk9f\" (UID: \"bff70b45-6546-4ec7-add4-e19e9fc07e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7720d1-9zk9f"
Jul 29 20:12:05.707022 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:05.706922    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7720d1-9zk9f"
Jul 29 20:12:05.850554 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:05.850525    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7720d1-9zk9f"]
Jul 29 20:12:05.853093 ip-10-0-128-101 kubenswrapper[2583]: W0729 20:12:05.853061    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbff70b45_6546_4ec7_add4_e19e9fc07e5c.slice/crio-d12292ba3177634852de017fb36978bbb04304bcaa1fe180f0cb5afa127605e6 WatchSource:0}: Error finding container d12292ba3177634852de017fb36978bbb04304bcaa1fe180f0cb5afa127605e6: Status 404 returned error can't find the container with id d12292ba3177634852de017fb36978bbb04304bcaa1fe180f0cb5afa127605e6
Jul 29 20:12:06.784564 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:06.784478    2583 generic.go:356] "Generic (PLEG): container finished" podID="bff70b45-6546-4ec7-add4-e19e9fc07e5c" containerID="5e028beb93ed020c945fd0b37dc6ff30552c64f5a45e71b5bf1308517b6cbf4f" exitCode=0
Jul 29 20:12:06.784564 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:06.784529    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7720d1-9zk9f" event={"ID":"bff70b45-6546-4ec7-add4-e19e9fc07e5c","Type":"ContainerDied","Data":"5e028beb93ed020c945fd0b37dc6ff30552c64f5a45e71b5bf1308517b6cbf4f"}
Jul 29 20:12:06.784967 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:06.784567    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7720d1-9zk9f" event={"ID":"bff70b45-6546-4ec7-add4-e19e9fc07e5c","Type":"ContainerStarted","Data":"d12292ba3177634852de017fb36978bbb04304bcaa1fe180f0cb5afa127605e6"}
Jul 29 20:12:07.930675 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:07.930626    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7720d1-9zk9f"
Jul 29 20:12:08.050276 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:08.050183    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bff70b45-6546-4ec7-add4-e19e9fc07e5c-maas-api-service-ca\") pod \"bff70b45-6546-4ec7-add4-e19e9fc07e5c\" (UID: \"bff70b45-6546-4ec7-add4-e19e9fc07e5c\") "
Jul 29 20:12:08.050588 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:08.050568    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bff70b45-6546-4ec7-add4-e19e9fc07e5c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bff70b45-6546-4ec7-add4-e19e9fc07e5c" (UID: "bff70b45-6546-4ec7-add4-e19e9fc07e5c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:12:08.151869 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:08.151825    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bff70b45-6546-4ec7-add4-e19e9fc07e5c-maas-api-service-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:12:08.795345 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:08.795314    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7720d1-9zk9f"
Jul 29 20:12:08.795522 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:08.795359    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7720d1-9zk9f" event={"ID":"bff70b45-6546-4ec7-add4-e19e9fc07e5c","Type":"ContainerDied","Data":"d12292ba3177634852de017fb36978bbb04304bcaa1fe180f0cb5afa127605e6"}
Jul 29 20:12:08.795522 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:08.795401    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d12292ba3177634852de017fb36978bbb04304bcaa1fe180f0cb5afa127605e6"
Jul 29 20:12:09.474563 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:09.474525    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7720d1-5d4889bcd-885bz"]
Jul 29 20:12:09.474996 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:09.474730    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7720d1-5d4889bcd-885bz" podUID="904748ff-ca23-4ce7-aa4b-f63bed95888e" containerName="maas-api" containerID="cri-o://2b0af7dd718ac02ad9fd9ac74ba0eef8d28b3df6242934eafec5f9988cc31040" gracePeriod=30
Jul 29 20:12:09.526077 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:09.526030    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-7720d1-8576c6fd58-xc6pr"]
Jul 29 20:12:09.526924 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:09.526272    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7720d1-8576c6fd58-xc6pr" podUID="2aaaff35-1d8c-48d0-bb80-9daf321b965d" containerName="payload-processing" containerID="cri-o://ce962aa5b052de14dd3a220894ef5633c62db9bf00707edbb33032c44816911d" gracePeriod=30
Jul 29 20:12:09.681548 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:09.681516    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7720d1-9zk9f"]
Jul 29 20:12:09.691300 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:09.691267    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7720d1-9zk9f"]
Jul 29 20:12:09.802528 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:09.802498    2583 generic.go:356] "Generic (PLEG): container finished" podID="2aaaff35-1d8c-48d0-bb80-9daf321b965d" containerID="ce962aa5b052de14dd3a220894ef5633c62db9bf00707edbb33032c44816911d" exitCode=0
Jul 29 20:12:09.802674 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:09.802572    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7720d1-8576c6fd58-xc6pr" event={"ID":"2aaaff35-1d8c-48d0-bb80-9daf321b965d","Type":"ContainerDied","Data":"ce962aa5b052de14dd3a220894ef5633c62db9bf00707edbb33032c44816911d"}
Jul 29 20:12:09.802674 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:09.802635    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7720d1-8576c6fd58-xc6pr" event={"ID":"2aaaff35-1d8c-48d0-bb80-9daf321b965d","Type":"ContainerDied","Data":"14c948c2a380705f7a76cd320c2eb67811a4f83b6c87dc3c6c9cf26cdb23a4d8"}
Jul 29 20:12:09.802674 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:09.802658    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14c948c2a380705f7a76cd320c2eb67811a4f83b6c87dc3c6c9cf26cdb23a4d8"
Jul 29 20:12:09.804062 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:09.804041    2583 generic.go:356] "Generic (PLEG): container finished" podID="904748ff-ca23-4ce7-aa4b-f63bed95888e" containerID="2b0af7dd718ac02ad9fd9ac74ba0eef8d28b3df6242934eafec5f9988cc31040" exitCode=0
Jul 29 20:12:09.804157 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:09.804073    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7720d1-5d4889bcd-885bz" event={"ID":"904748ff-ca23-4ce7-aa4b-f63bed95888e","Type":"ContainerDied","Data":"2b0af7dd718ac02ad9fd9ac74ba0eef8d28b3df6242934eafec5f9988cc31040"}
Jul 29 20:12:09.804157 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:09.804106    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7720d1-5d4889bcd-885bz" event={"ID":"904748ff-ca23-4ce7-aa4b-f63bed95888e","Type":"ContainerDied","Data":"e2ae8732bb3e2159051833676b3a87f4d7dd4e069abe9db69f2172559958b74e"}
Jul 29 20:12:09.804157 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:09.804123    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2ae8732bb3e2159051833676b3a87f4d7dd4e069abe9db69f2172559958b74e"
Jul 29 20:12:09.811081 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:09.811053    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7720d1-8576c6fd58-xc6pr"
Jul 29 20:12:09.814999 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:09.814973    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7720d1-5d4889bcd-885bz"
Jul 29 20:12:09.967536 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:09.967492    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aaaff35-1d8c-48d0-bb80-9daf321b965d-plugins-config-volume\") pod \"2aaaff35-1d8c-48d0-bb80-9daf321b965d\" (UID: \"2aaaff35-1d8c-48d0-bb80-9daf321b965d\") "
Jul 29 20:12:09.967738 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:09.967594    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2sqt\" (UniqueName: \"kubernetes.io/projected/904748ff-ca23-4ce7-aa4b-f63bed95888e-kube-api-access-m2sqt\") pod \"904748ff-ca23-4ce7-aa4b-f63bed95888e\" (UID: \"904748ff-ca23-4ce7-aa4b-f63bed95888e\") "
Jul 29 20:12:09.967738 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:09.967613    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/904748ff-ca23-4ce7-aa4b-f63bed95888e-openshift-service-ca\") pod \"904748ff-ca23-4ce7-aa4b-f63bed95888e\" (UID: \"904748ff-ca23-4ce7-aa4b-f63bed95888e\") "
Jul 29 20:12:09.967738 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:09.967630    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r86mf\" (UniqueName: \"kubernetes.io/projected/2aaaff35-1d8c-48d0-bb80-9daf321b965d-kube-api-access-r86mf\") pod \"2aaaff35-1d8c-48d0-bb80-9daf321b965d\" (UID: \"2aaaff35-1d8c-48d0-bb80-9daf321b965d\") "
Jul 29 20:12:09.967738 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:09.967677    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/904748ff-ca23-4ce7-aa4b-f63bed95888e-maas-api-tls\") pod \"904748ff-ca23-4ce7-aa4b-f63bed95888e\" (UID: \"904748ff-ca23-4ce7-aa4b-f63bed95888e\") "
Jul 29 20:12:09.968043 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:09.968013    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aaaff35-1d8c-48d0-bb80-9daf321b965d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2aaaff35-1d8c-48d0-bb80-9daf321b965d" (UID: "2aaaff35-1d8c-48d0-bb80-9daf321b965d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:12:09.968103 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:09.968078    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/904748ff-ca23-4ce7-aa4b-f63bed95888e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "904748ff-ca23-4ce7-aa4b-f63bed95888e" (UID: "904748ff-ca23-4ce7-aa4b-f63bed95888e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:12:09.969931 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:09.969903    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/904748ff-ca23-4ce7-aa4b-f63bed95888e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "904748ff-ca23-4ce7-aa4b-f63bed95888e" (UID: "904748ff-ca23-4ce7-aa4b-f63bed95888e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:12:09.970011 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:09.969931    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aaaff35-1d8c-48d0-bb80-9daf321b965d-kube-api-access-r86mf" (OuterVolumeSpecName: "kube-api-access-r86mf") pod "2aaaff35-1d8c-48d0-bb80-9daf321b965d" (UID: "2aaaff35-1d8c-48d0-bb80-9daf321b965d"). InnerVolumeSpecName "kube-api-access-r86mf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:12:09.970011 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:09.969953    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/904748ff-ca23-4ce7-aa4b-f63bed95888e-kube-api-access-m2sqt" (OuterVolumeSpecName: "kube-api-access-m2sqt") pod "904748ff-ca23-4ce7-aa4b-f63bed95888e" (UID: "904748ff-ca23-4ce7-aa4b-f63bed95888e"). InnerVolumeSpecName "kube-api-access-m2sqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:12:10.068560 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:10.068471    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/904748ff-ca23-4ce7-aa4b-f63bed95888e-maas-api-tls\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:12:10.068560 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:10.068506    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aaaff35-1d8c-48d0-bb80-9daf321b965d-plugins-config-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:12:10.068560 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:10.068518    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2sqt\" (UniqueName: \"kubernetes.io/projected/904748ff-ca23-4ce7-aa4b-f63bed95888e-kube-api-access-m2sqt\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:12:10.068560 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:10.068526    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/904748ff-ca23-4ce7-aa4b-f63bed95888e-openshift-service-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:12:10.068560 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:10.068536    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r86mf\" (UniqueName: \"kubernetes.io/projected/2aaaff35-1d8c-48d0-bb80-9daf321b965d-kube-api-access-r86mf\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:12:10.810531 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:10.810449    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7720d1-8576c6fd58-xc6pr"
Jul 29 20:12:10.811017 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:10.810451    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7720d1-5d4889bcd-885bz"
Jul 29 20:12:10.833281 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:10.833250    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7720d1-5d4889bcd-885bz"]
Jul 29 20:12:10.840567 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:10.840544    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7720d1-5d4889bcd-885bz"]
Jul 29 20:12:10.850102 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:10.850076    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-7720d1-8576c6fd58-xc6pr"]
Jul 29 20:12:10.851669 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:10.851649    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-7720d1-8576c6fd58-xc6pr"]
Jul 29 20:12:11.341478 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:11.341446    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2aaaff35-1d8c-48d0-bb80-9daf321b965d" path="/var/lib/kubelet/pods/2aaaff35-1d8c-48d0-bb80-9daf321b965d/volumes"
Jul 29 20:12:11.341862 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:11.341849    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="904748ff-ca23-4ce7-aa4b-f63bed95888e" path="/var/lib/kubelet/pods/904748ff-ca23-4ce7-aa4b-f63bed95888e/volumes"
Jul 29 20:12:11.342135 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:11.342124    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bff70b45-6546-4ec7-add4-e19e9fc07e5c" path="/var/lib/kubelet/pods/bff70b45-6546-4ec7-add4-e19e9fc07e5c/volumes"
Jul 29 20:12:13.601767 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:13.601732    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 20:12:19.846582 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:19.846550    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb0399-glbmv"]
Jul 29 20:12:19.846987 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:19.846924    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2aaaff35-1d8c-48d0-bb80-9daf321b965d" containerName="payload-processing"
Jul 29 20:12:19.846987 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:19.846936    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2aaaff35-1d8c-48d0-bb80-9daf321b965d" containerName="payload-processing"
Jul 29 20:12:19.846987 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:19.846954    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="904748ff-ca23-4ce7-aa4b-f63bed95888e" containerName="maas-api"
Jul 29 20:12:19.846987 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:19.846959    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="904748ff-ca23-4ce7-aa4b-f63bed95888e" containerName="maas-api"
Jul 29 20:12:19.846987 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:19.846968    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bff70b45-6546-4ec7-add4-e19e9fc07e5c" containerName="revoke-keys"
Jul 29 20:12:19.846987 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:19.846976    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bff70b45-6546-4ec7-add4-e19e9fc07e5c" containerName="revoke-keys"
Jul 29 20:12:19.847181 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:19.847029    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="bff70b45-6546-4ec7-add4-e19e9fc07e5c" containerName="revoke-keys"
Jul 29 20:12:19.847181 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:19.847038    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2aaaff35-1d8c-48d0-bb80-9daf321b965d" containerName="payload-processing"
Jul 29 20:12:19.847181 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:19.847043    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="904748ff-ca23-4ce7-aa4b-f63bed95888e" containerName="maas-api"
Jul 29 20:12:19.851396 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:19.851375    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb0399-glbmv"
Jul 29 20:12:19.856672 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:19.856648    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hq9l2\""
Jul 29 20:12:19.859741 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:19.859718    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb0399-glbmv"]
Jul 29 20:12:19.955173 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:19.955139    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d780c5b-b9ac-42a3-a747-89887351a6f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fb0399-glbmv\" (UID: \"8d780c5b-b9ac-42a3-a747-89887351a6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb0399-glbmv"
Jul 29 20:12:20.055862 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:20.055786    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d780c5b-b9ac-42a3-a747-89887351a6f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fb0399-glbmv\" (UID: \"8d780c5b-b9ac-42a3-a747-89887351a6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb0399-glbmv"
Jul 29 20:12:20.056446 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:20.056422    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d780c5b-b9ac-42a3-a747-89887351a6f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fb0399-glbmv\" (UID: \"8d780c5b-b9ac-42a3-a747-89887351a6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb0399-glbmv"
Jul 29 20:12:20.163374 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:20.163336    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb0399-glbmv"
Jul 29 20:12:20.303118 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:20.303088    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb0399-glbmv"]
Jul 29 20:12:20.305705 ip-10-0-128-101 kubenswrapper[2583]: W0729 20:12:20.305677    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d780c5b_b9ac_42a3_a747_89887351a6f1.slice/crio-a174b3df0d9af5ed982fa8241c1688d5f4600982033935316b096b14633b33d6 WatchSource:0}: Error finding container a174b3df0d9af5ed982fa8241c1688d5f4600982033935316b096b14633b33d6: Status 404 returned error can't find the container with id a174b3df0d9af5ed982fa8241c1688d5f4600982033935316b096b14633b33d6
Jul 29 20:12:20.858340 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:20.858253    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb0399-glbmv" event={"ID":"8d780c5b-b9ac-42a3-a747-89887351a6f1","Type":"ContainerStarted","Data":"cf6dc3582ea06a08edec8941944d7c50549c3b6314c19bd2967c72453645a2b5"}
Jul 29 20:12:20.858340 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:20.858304    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb0399-glbmv" event={"ID":"8d780c5b-b9ac-42a3-a747-89887351a6f1","Type":"ContainerStarted","Data":"a174b3df0d9af5ed982fa8241c1688d5f4600982033935316b096b14633b33d6"}
Jul 29 20:12:20.875572 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:20.875523    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb0399-glbmv" podStartSLOduration=1.8755055889999999 podStartE2EDuration="1.875505589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:12:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:12:20.873745895 +0000 UTC m=+3308.102135020" watchObservedRunningTime="2026-07-29 20:12:20.875505589 +0000 UTC m=+3308.103894714"
Jul 29 20:12:21.864911 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:21.864870    2583 generic.go:356] "Generic (PLEG): container finished" podID="8d780c5b-b9ac-42a3-a747-89887351a6f1" containerID="cf6dc3582ea06a08edec8941944d7c50549c3b6314c19bd2967c72453645a2b5" exitCode=0
Jul 29 20:12:21.865324 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:21.864955    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb0399-glbmv" event={"ID":"8d780c5b-b9ac-42a3-a747-89887351a6f1","Type":"ContainerDied","Data":"cf6dc3582ea06a08edec8941944d7c50549c3b6314c19bd2967c72453645a2b5"}
Jul 29 20:12:23.008996 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:23.008968    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb0399-glbmv"
Jul 29 20:12:23.185900 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:23.185852    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d780c5b-b9ac-42a3-a747-89887351a6f1-maas-api-service-ca\") pod \"8d780c5b-b9ac-42a3-a747-89887351a6f1\" (UID: \"8d780c5b-b9ac-42a3-a747-89887351a6f1\") "
Jul 29 20:12:23.186217 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:23.186194    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d780c5b-b9ac-42a3-a747-89887351a6f1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8d780c5b-b9ac-42a3-a747-89887351a6f1" (UID: "8d780c5b-b9ac-42a3-a747-89887351a6f1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:12:23.287050 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:23.287002    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d780c5b-b9ac-42a3-a747-89887351a6f1-maas-api-service-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:12:23.875757 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:23.875732    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb0399-glbmv"
Jul 29 20:12:23.875757 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:23.875740    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb0399-glbmv" event={"ID":"8d780c5b-b9ac-42a3-a747-89887351a6f1","Type":"ContainerDied","Data":"a174b3df0d9af5ed982fa8241c1688d5f4600982033935316b096b14633b33d6"}
Jul 29 20:12:23.875987 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:23.875784    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a174b3df0d9af5ed982fa8241c1688d5f4600982033935316b096b14633b33d6"
Jul 29 20:12:35.313731 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:35.313637    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb0399-bf6746987-pvcts"]
Jul 29 20:12:35.314211 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:35.313928    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb0399-bf6746987-pvcts" podUID="2219514b-7cbd-4839-9282-4639dad3ba28" containerName="maas-api" containerID="cri-o://1e799a8699b96817d6c0b08dff6a0abeb9b313b4a54199948a7aa03925097c5f" gracePeriod=30
Jul 29 20:12:35.570215 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:35.570156    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb0399-bf6746987-pvcts"
Jul 29 20:12:35.670151 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:35.670116    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb0399-glbmv"]
Jul 29 20:12:35.674647 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:35.674625    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb0399-glbmv"]
Jul 29 20:12:35.678591 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:35.678569    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2219514b-7cbd-4839-9282-4639dad3ba28-openshift-service-ca\") pod \"2219514b-7cbd-4839-9282-4639dad3ba28\" (UID: \"2219514b-7cbd-4839-9282-4639dad3ba28\") "
Jul 29 20:12:35.678659 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:35.678602    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcpzl\" (UniqueName: \"kubernetes.io/projected/2219514b-7cbd-4839-9282-4639dad3ba28-kube-api-access-lcpzl\") pod \"2219514b-7cbd-4839-9282-4639dad3ba28\" (UID: \"2219514b-7cbd-4839-9282-4639dad3ba28\") "
Jul 29 20:12:35.678697 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:35.678666    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2219514b-7cbd-4839-9282-4639dad3ba28-maas-api-tls\") pod \"2219514b-7cbd-4839-9282-4639dad3ba28\" (UID: \"2219514b-7cbd-4839-9282-4639dad3ba28\") "
Jul 29 20:12:35.679049 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:35.679016    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2219514b-7cbd-4839-9282-4639dad3ba28-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2219514b-7cbd-4839-9282-4639dad3ba28" (UID: "2219514b-7cbd-4839-9282-4639dad3ba28"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:12:35.680737 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:35.680706    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2219514b-7cbd-4839-9282-4639dad3ba28-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2219514b-7cbd-4839-9282-4639dad3ba28" (UID: "2219514b-7cbd-4839-9282-4639dad3ba28"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:12:35.680827 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:35.680713    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2219514b-7cbd-4839-9282-4639dad3ba28-kube-api-access-lcpzl" (OuterVolumeSpecName: "kube-api-access-lcpzl") pod "2219514b-7cbd-4839-9282-4639dad3ba28" (UID: "2219514b-7cbd-4839-9282-4639dad3ba28"). InnerVolumeSpecName "kube-api-access-lcpzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:12:35.780103 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:35.780072    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2219514b-7cbd-4839-9282-4639dad3ba28-openshift-service-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:12:35.780103 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:35.780097    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcpzl\" (UniqueName: \"kubernetes.io/projected/2219514b-7cbd-4839-9282-4639dad3ba28-kube-api-access-lcpzl\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:12:35.780103 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:35.780107    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2219514b-7cbd-4839-9282-4639dad3ba28-maas-api-tls\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:12:35.940497 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:35.940462    2583 generic.go:356] "Generic (PLEG): container finished" podID="2219514b-7cbd-4839-9282-4639dad3ba28" containerID="1e799a8699b96817d6c0b08dff6a0abeb9b313b4a54199948a7aa03925097c5f" exitCode=0
Jul 29 20:12:35.940649 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:35.940506    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb0399-bf6746987-pvcts"
Jul 29 20:12:35.940649 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:35.940544    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb0399-bf6746987-pvcts" event={"ID":"2219514b-7cbd-4839-9282-4639dad3ba28","Type":"ContainerDied","Data":"1e799a8699b96817d6c0b08dff6a0abeb9b313b4a54199948a7aa03925097c5f"}
Jul 29 20:12:35.940649 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:35.940598    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb0399-bf6746987-pvcts" event={"ID":"2219514b-7cbd-4839-9282-4639dad3ba28","Type":"ContainerDied","Data":"25b99a0e8a0e19d5afcac0a69cab8517f5afbb130d1932ea82099444695c79fa"}
Jul 29 20:12:35.940649 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:35.940622    2583 scope.go:117] "RemoveContainer" containerID="1e799a8699b96817d6c0b08dff6a0abeb9b313b4a54199948a7aa03925097c5f"
Jul 29 20:12:35.950499 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:35.950482    2583 scope.go:117] "RemoveContainer" containerID="1e799a8699b96817d6c0b08dff6a0abeb9b313b4a54199948a7aa03925097c5f"
Jul 29 20:12:35.950751 ip-10-0-128-101 kubenswrapper[2583]: E0729 20:12:35.950733    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e799a8699b96817d6c0b08dff6a0abeb9b313b4a54199948a7aa03925097c5f\": container with ID starting with 1e799a8699b96817d6c0b08dff6a0abeb9b313b4a54199948a7aa03925097c5f not found: ID does not exist" containerID="1e799a8699b96817d6c0b08dff6a0abeb9b313b4a54199948a7aa03925097c5f"
Jul 29 20:12:35.950828 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:35.950763    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e799a8699b96817d6c0b08dff6a0abeb9b313b4a54199948a7aa03925097c5f"} err="failed to get container status \"1e799a8699b96817d6c0b08dff6a0abeb9b313b4a54199948a7aa03925097c5f\": rpc error: code = NotFound desc = could not find container \"1e799a8699b96817d6c0b08dff6a0abeb9b313b4a54199948a7aa03925097c5f\": container with ID starting with 1e799a8699b96817d6c0b08dff6a0abeb9b313b4a54199948a7aa03925097c5f not found: ID does not exist"
Jul 29 20:12:35.962752 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:35.962726    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb0399-bf6746987-pvcts"]
Jul 29 20:12:35.967319 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:35.967300    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb0399-bf6746987-pvcts"]
Jul 29 20:12:37.003043 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:37.003012    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 20:12:37.340350 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:37.340270    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2219514b-7cbd-4839-9282-4639dad3ba28" path="/var/lib/kubelet/pods/2219514b-7cbd-4839-9282-4639dad3ba28/volumes"
Jul 29 20:12:37.340591 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:37.340579    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d780c5b-b9ac-42a3-a747-89887351a6f1" path="/var/lib/kubelet/pods/8d780c5b-b9ac-42a3-a747-89887351a6f1/volumes"
Jul 29 20:12:47.605573 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:12:47.605534    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 20:13:28.053062 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:28.053027    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6488bc-zg7w7"]
Jul 29 20:13:28.053541 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:28.053380    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2219514b-7cbd-4839-9282-4639dad3ba28" containerName="maas-api"
Jul 29 20:13:28.053541 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:28.053391    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2219514b-7cbd-4839-9282-4639dad3ba28" containerName="maas-api"
Jul 29 20:13:28.053541 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:28.053403    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d780c5b-b9ac-42a3-a747-89887351a6f1" containerName="revoke-keys"
Jul 29 20:13:28.053541 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:28.053409    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d780c5b-b9ac-42a3-a747-89887351a6f1" containerName="revoke-keys"
Jul 29 20:13:28.053541 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:28.053474    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d780c5b-b9ac-42a3-a747-89887351a6f1" containerName="revoke-keys"
Jul 29 20:13:28.053541 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:28.053486    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2219514b-7cbd-4839-9282-4639dad3ba28" containerName="maas-api"
Jul 29 20:13:28.056519 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:28.056503    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6488bc-zg7w7"
Jul 29 20:13:28.059054 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:28.059028    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hq9l2\""
Jul 29 20:13:28.065913 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:28.065886    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6488bc-zg7w7"]
Jul 29 20:13:28.228042 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:28.227998    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ae898a3-d821-4cd4-8c52-58aeeb6a503a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6488bc-zg7w7\" (UID: \"8ae898a3-d821-4cd4-8c52-58aeeb6a503a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6488bc-zg7w7"
Jul 29 20:13:28.329075 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:28.328976    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ae898a3-d821-4cd4-8c52-58aeeb6a503a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6488bc-zg7w7\" (UID: \"8ae898a3-d821-4cd4-8c52-58aeeb6a503a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6488bc-zg7w7"
Jul 29 20:13:28.329585 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:28.329566    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ae898a3-d821-4cd4-8c52-58aeeb6a503a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6488bc-zg7w7\" (UID: \"8ae898a3-d821-4cd4-8c52-58aeeb6a503a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6488bc-zg7w7"
Jul 29 20:13:28.367083 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:28.367050    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6488bc-zg7w7"
Jul 29 20:13:28.511473 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:28.511450    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6488bc-zg7w7"]
Jul 29 20:13:28.513608 ip-10-0-128-101 kubenswrapper[2583]: W0729 20:13:28.513586    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ae898a3_d821_4cd4_8c52_58aeeb6a503a.slice/crio-cc1a010277b6f87079eb0eb5637f258a8fea801f1910f2c31d42b2858a5a1ebe WatchSource:0}: Error finding container cc1a010277b6f87079eb0eb5637f258a8fea801f1910f2c31d42b2858a5a1ebe: Status 404 returned error can't find the container with id cc1a010277b6f87079eb0eb5637f258a8fea801f1910f2c31d42b2858a5a1ebe
Jul 29 20:13:29.194607 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:29.194563    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6488bc-zg7w7" event={"ID":"8ae898a3-d821-4cd4-8c52-58aeeb6a503a","Type":"ContainerStarted","Data":"9d2cb9cc78c755d792a15f63469ee32adf5e94ef12b434aee34518b6af98a972"}
Jul 29 20:13:29.195080 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:29.194675    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6488bc-zg7w7" event={"ID":"8ae898a3-d821-4cd4-8c52-58aeeb6a503a","Type":"ContainerStarted","Data":"cc1a010277b6f87079eb0eb5637f258a8fea801f1910f2c31d42b2858a5a1ebe"}
Jul 29 20:13:29.214527 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:29.214476    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6488bc-zg7w7" podStartSLOduration=1.214460059 podStartE2EDuration="1.214460059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:13:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:13:29.213227854 +0000 UTC m=+3376.441616981" watchObservedRunningTime="2026-07-29 20:13:29.214460059 +0000 UTC m=+3376.442849185"
Jul 29 20:13:30.202211 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:30.202176    2583 generic.go:356] "Generic (PLEG): container finished" podID="8ae898a3-d821-4cd4-8c52-58aeeb6a503a" containerID="9d2cb9cc78c755d792a15f63469ee32adf5e94ef12b434aee34518b6af98a972" exitCode=0
Jul 29 20:13:30.202608 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:30.202260    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6488bc-zg7w7" event={"ID":"8ae898a3-d821-4cd4-8c52-58aeeb6a503a","Type":"ContainerDied","Data":"9d2cb9cc78c755d792a15f63469ee32adf5e94ef12b434aee34518b6af98a972"}
Jul 29 20:13:31.346618 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:31.346592    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6488bc-zg7w7"
Jul 29 20:13:31.350708 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:31.350689    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ae898a3-d821-4cd4-8c52-58aeeb6a503a-maas-api-service-ca\") pod \"8ae898a3-d821-4cd4-8c52-58aeeb6a503a\" (UID: \"8ae898a3-d821-4cd4-8c52-58aeeb6a503a\") "
Jul 29 20:13:31.351022 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:31.350999    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ae898a3-d821-4cd4-8c52-58aeeb6a503a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8ae898a3-d821-4cd4-8c52-58aeeb6a503a" (UID: "8ae898a3-d821-4cd4-8c52-58aeeb6a503a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:13:31.451689 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:31.451653    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ae898a3-d821-4cd4-8c52-58aeeb6a503a-maas-api-service-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:13:32.213677 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:32.213639    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6488bc-zg7w7" event={"ID":"8ae898a3-d821-4cd4-8c52-58aeeb6a503a","Type":"ContainerDied","Data":"cc1a010277b6f87079eb0eb5637f258a8fea801f1910f2c31d42b2858a5a1ebe"}
Jul 29 20:13:32.213677 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:32.213662    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6488bc-zg7w7"
Jul 29 20:13:32.213935 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:32.213696    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc1a010277b6f87079eb0eb5637f258a8fea801f1910f2c31d42b2858a5a1ebe"
Jul 29 20:13:38.458413 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:38.458379    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6488bc-86578d55f8-tqxwz"]
Jul 29 20:13:38.520520 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:38.520487    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6488bc-6dbf565b78-5cptz"]
Jul 29 20:13:38.520753 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:38.520729    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-6488bc-6dbf565b78-5cptz" podUID="297f43e6-7f3c-4bc5-8675-ab57c4728032" containerName="payload-processing" containerID="cri-o://f6cd1fae7130a2f9e64978a5e991cfde96f784c28b9e1c4e2a03eb244b01903a" gracePeriod=30
Jul 29 20:13:38.706090 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:38.706050    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6488bc-zg7w7"]
Jul 29 20:13:38.708994 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:38.708924    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6488bc-zg7w7"]
Jul 29 20:13:38.779673 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:38.779651    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6488bc-6dbf565b78-5cptz"
Jul 29 20:13:38.805056 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:38.805021    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhkjk\" (UniqueName: \"kubernetes.io/projected/297f43e6-7f3c-4bc5-8675-ab57c4728032-kube-api-access-jhkjk\") pod \"297f43e6-7f3c-4bc5-8675-ab57c4728032\" (UID: \"297f43e6-7f3c-4bc5-8675-ab57c4728032\") "
Jul 29 20:13:38.805230 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:38.805066    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/297f43e6-7f3c-4bc5-8675-ab57c4728032-plugins-config-volume\") pod \"297f43e6-7f3c-4bc5-8675-ab57c4728032\" (UID: \"297f43e6-7f3c-4bc5-8675-ab57c4728032\") "
Jul 29 20:13:38.805455 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:38.805432    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/297f43e6-7f3c-4bc5-8675-ab57c4728032-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "297f43e6-7f3c-4bc5-8675-ab57c4728032" (UID: "297f43e6-7f3c-4bc5-8675-ab57c4728032"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:13:38.807126 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:38.807095    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/297f43e6-7f3c-4bc5-8675-ab57c4728032-kube-api-access-jhkjk" (OuterVolumeSpecName: "kube-api-access-jhkjk") pod "297f43e6-7f3c-4bc5-8675-ab57c4728032" (UID: "297f43e6-7f3c-4bc5-8675-ab57c4728032"). InnerVolumeSpecName "kube-api-access-jhkjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:13:38.906176 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:38.906138    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhkjk\" (UniqueName: \"kubernetes.io/projected/297f43e6-7f3c-4bc5-8675-ab57c4728032-kube-api-access-jhkjk\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:13:38.906176 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:38.906170    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/297f43e6-7f3c-4bc5-8675-ab57c4728032-plugins-config-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:13:39.246736 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:39.246706    2583 generic.go:356] "Generic (PLEG): container finished" podID="297f43e6-7f3c-4bc5-8675-ab57c4728032" containerID="f6cd1fae7130a2f9e64978a5e991cfde96f784c28b9e1c4e2a03eb244b01903a" exitCode=0
Jul 29 20:13:39.246939 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:39.246751    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6488bc-6dbf565b78-5cptz"
Jul 29 20:13:39.246939 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:39.246828    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6488bc-6dbf565b78-5cptz" event={"ID":"297f43e6-7f3c-4bc5-8675-ab57c4728032","Type":"ContainerDied","Data":"f6cd1fae7130a2f9e64978a5e991cfde96f784c28b9e1c4e2a03eb244b01903a"}
Jul 29 20:13:39.246939 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:39.246885    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6488bc-6dbf565b78-5cptz" event={"ID":"297f43e6-7f3c-4bc5-8675-ab57c4728032","Type":"ContainerDied","Data":"c9424dcd082364abd07b978c1e2ad91267bba104d2a2fddffd109756f298afc4"}
Jul 29 20:13:39.246939 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:39.246909    2583 scope.go:117] "RemoveContainer" containerID="f6cd1fae7130a2f9e64978a5e991cfde96f784c28b9e1c4e2a03eb244b01903a"
Jul 29 20:13:39.247127 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:39.247065    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6488bc-86578d55f8-tqxwz" podUID="95221161-2bd1-405c-b856-8c51f3dff04d" containerName="maas-api" containerID="cri-o://80d377a3c25a65d0260a442394fdde59fbc6d2934bf6de5110388971a5a55d57" gracePeriod=30
Jul 29 20:13:39.256606 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:39.256586    2583 scope.go:117] "RemoveContainer" containerID="f6cd1fae7130a2f9e64978a5e991cfde96f784c28b9e1c4e2a03eb244b01903a"
Jul 29 20:13:39.256903 ip-10-0-128-101 kubenswrapper[2583]: E0729 20:13:39.256883    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6cd1fae7130a2f9e64978a5e991cfde96f784c28b9e1c4e2a03eb244b01903a\": container with ID starting with f6cd1fae7130a2f9e64978a5e991cfde96f784c28b9e1c4e2a03eb244b01903a not found: ID does not exist" containerID="f6cd1fae7130a2f9e64978a5e991cfde96f784c28b9e1c4e2a03eb244b01903a"
Jul 29 20:13:39.256981 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:39.256915    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6cd1fae7130a2f9e64978a5e991cfde96f784c28b9e1c4e2a03eb244b01903a"} err="failed to get container status \"f6cd1fae7130a2f9e64978a5e991cfde96f784c28b9e1c4e2a03eb244b01903a\": rpc error: code = NotFound desc = could not find container \"f6cd1fae7130a2f9e64978a5e991cfde96f784c28b9e1c4e2a03eb244b01903a\": container with ID starting with f6cd1fae7130a2f9e64978a5e991cfde96f784c28b9e1c4e2a03eb244b01903a not found: ID does not exist"
Jul 29 20:13:39.287781 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:39.287752    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6488bc-6dbf565b78-5cptz"]
Jul 29 20:13:39.294377 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:39.294351    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6488bc-6dbf565b78-5cptz"]
Jul 29 20:13:39.341257 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:39.341224    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="297f43e6-7f3c-4bc5-8675-ab57c4728032" path="/var/lib/kubelet/pods/297f43e6-7f3c-4bc5-8675-ab57c4728032/volumes"
Jul 29 20:13:39.341576 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:39.341562    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ae898a3-d821-4cd4-8c52-58aeeb6a503a" path="/var/lib/kubelet/pods/8ae898a3-d821-4cd4-8c52-58aeeb6a503a/volumes"
Jul 29 20:13:39.497368 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:39.497297    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6488bc-86578d55f8-tqxwz"
Jul 29 20:13:39.511323 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:39.511297    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-grq5t\" (UniqueName: \"kubernetes.io/projected/95221161-2bd1-405c-b856-8c51f3dff04d-kube-api-access-grq5t\") pod \"95221161-2bd1-405c-b856-8c51f3dff04d\" (UID: \"95221161-2bd1-405c-b856-8c51f3dff04d\") "
Jul 29 20:13:39.511460 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:39.511372    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95221161-2bd1-405c-b856-8c51f3dff04d-openshift-service-ca\") pod \"95221161-2bd1-405c-b856-8c51f3dff04d\" (UID: \"95221161-2bd1-405c-b856-8c51f3dff04d\") "
Jul 29 20:13:39.511537 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:39.511452    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95221161-2bd1-405c-b856-8c51f3dff04d-maas-api-tls\") pod \"95221161-2bd1-405c-b856-8c51f3dff04d\" (UID: \"95221161-2bd1-405c-b856-8c51f3dff04d\") "
Jul 29 20:13:39.511741 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:39.511715    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95221161-2bd1-405c-b856-8c51f3dff04d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "95221161-2bd1-405c-b856-8c51f3dff04d" (UID: "95221161-2bd1-405c-b856-8c51f3dff04d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:13:39.513454 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:39.513421    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95221161-2bd1-405c-b856-8c51f3dff04d-kube-api-access-grq5t" (OuterVolumeSpecName: "kube-api-access-grq5t") pod "95221161-2bd1-405c-b856-8c51f3dff04d" (UID: "95221161-2bd1-405c-b856-8c51f3dff04d"). InnerVolumeSpecName "kube-api-access-grq5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:13:39.513557 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:39.513469    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95221161-2bd1-405c-b856-8c51f3dff04d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "95221161-2bd1-405c-b856-8c51f3dff04d" (UID: "95221161-2bd1-405c-b856-8c51f3dff04d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:13:39.612731 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:39.612692    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95221161-2bd1-405c-b856-8c51f3dff04d-maas-api-tls\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:13:39.612731 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:39.612727    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grq5t\" (UniqueName: \"kubernetes.io/projected/95221161-2bd1-405c-b856-8c51f3dff04d-kube-api-access-grq5t\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:13:39.612731 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:39.612740    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95221161-2bd1-405c-b856-8c51f3dff04d-openshift-service-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:13:40.253219 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:40.253184    2583 generic.go:356] "Generic (PLEG): container finished" podID="95221161-2bd1-405c-b856-8c51f3dff04d" containerID="80d377a3c25a65d0260a442394fdde59fbc6d2934bf6de5110388971a5a55d57" exitCode=0
Jul 29 20:13:40.253417 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:40.253228    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6488bc-86578d55f8-tqxwz"
Jul 29 20:13:40.253417 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:40.253275    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6488bc-86578d55f8-tqxwz" event={"ID":"95221161-2bd1-405c-b856-8c51f3dff04d","Type":"ContainerDied","Data":"80d377a3c25a65d0260a442394fdde59fbc6d2934bf6de5110388971a5a55d57"}
Jul 29 20:13:40.253417 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:40.253325    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6488bc-86578d55f8-tqxwz" event={"ID":"95221161-2bd1-405c-b856-8c51f3dff04d","Type":"ContainerDied","Data":"31e63d509e8c1b6877c86bb5f69bc3fc797149b7a4835e0ba510dccd3709bcad"}
Jul 29 20:13:40.253417 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:40.253347    2583 scope.go:117] "RemoveContainer" containerID="80d377a3c25a65d0260a442394fdde59fbc6d2934bf6de5110388971a5a55d57"
Jul 29 20:13:40.262896 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:40.262874    2583 scope.go:117] "RemoveContainer" containerID="80d377a3c25a65d0260a442394fdde59fbc6d2934bf6de5110388971a5a55d57"
Jul 29 20:13:40.263187 ip-10-0-128-101 kubenswrapper[2583]: E0729 20:13:40.263168    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80d377a3c25a65d0260a442394fdde59fbc6d2934bf6de5110388971a5a55d57\": container with ID starting with 80d377a3c25a65d0260a442394fdde59fbc6d2934bf6de5110388971a5a55d57 not found: ID does not exist" containerID="80d377a3c25a65d0260a442394fdde59fbc6d2934bf6de5110388971a5a55d57"
Jul 29 20:13:40.263242 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:40.263196    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80d377a3c25a65d0260a442394fdde59fbc6d2934bf6de5110388971a5a55d57"} err="failed to get container status \"80d377a3c25a65d0260a442394fdde59fbc6d2934bf6de5110388971a5a55d57\": rpc error: code = NotFound desc = could not find container \"80d377a3c25a65d0260a442394fdde59fbc6d2934bf6de5110388971a5a55d57\": container with ID starting with 80d377a3c25a65d0260a442394fdde59fbc6d2934bf6de5110388971a5a55d57 not found: ID does not exist"
Jul 29 20:13:40.284021 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:40.283984    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6488bc-86578d55f8-tqxwz"]
Jul 29 20:13:40.290656 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:40.290623    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6488bc-86578d55f8-tqxwz"]
Jul 29 20:13:41.340548 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:41.340513    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95221161-2bd1-405c-b856-8c51f3dff04d" path="/var/lib/kubelet/pods/95221161-2bd1-405c-b856-8c51f3dff04d/volumes"
Jul 29 20:13:41.813777 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:41.813743    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 20:13:56.525060 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:56.525027    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a60880-75cbcb67fc-mwsfk"]
Jul 29 20:13:56.525649 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:56.525303    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a60880-75cbcb67fc-mwsfk" podUID="c3645361-80d7-4487-a79d-033a296e9016" containerName="maas-api" containerID="cri-o://28da31b7d30400eaca61ec92d4b2dac8a4e475f9ce0432fde57177767499a038" gracePeriod=30
Jul 29 20:13:56.556778 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:56.556750    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a60880-29755920-zbzss"]
Jul 29 20:13:56.559608 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:56.559587    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a60880-29755920-zbzss"]
Jul 29 20:13:56.584636 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:56.584613    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-a60880-6c5fff5886-mcrkd"]
Jul 29 20:13:56.584876 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:56.584853    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-a60880-6c5fff5886-mcrkd" podUID="5eb22fa7-1bf4-46c4-b936-9f5941caacc7" containerName="payload-processing" containerID="cri-o://02e1f7c1f89075c8df9c8791e8a3567ab45b3d873b54c41d9c1a8ea882065cce" gracePeriod=30
Jul 29 20:13:56.590175 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:56.590134    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-a60880-7846f4db76-nc6j5"]
Jul 29 20:13:56.590445 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:56.590390    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a60880-7846f4db76-nc6j5" podUID="c8ba63e5-214a-4485-9174-43b4a7dd6f0d" containerName="payload-pre-processing" containerID="cri-o://adc93ba0a19ae1339c989111d38f06f67702ae8d65a829105043d7bf1140ee52" gracePeriod=30
Jul 29 20:13:56.867324 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:56.867302    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-a60880-6c5fff5886-mcrkd"
Jul 29 20:13:56.869994 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:56.869978    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a60880-75cbcb67fc-mwsfk"
Jul 29 20:13:56.882644 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:56.882626    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a60880-7846f4db76-nc6j5"
Jul 29 20:13:56.954683 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:56.954655    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3645361-80d7-4487-a79d-033a296e9016-maas-api-tls\") pod \"c3645361-80d7-4487-a79d-033a296e9016\" (UID: \"c3645361-80d7-4487-a79d-033a296e9016\") "
Jul 29 20:13:56.954842 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:56.954734    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7cmjj\" (UniqueName: \"kubernetes.io/projected/c3645361-80d7-4487-a79d-033a296e9016-kube-api-access-7cmjj\") pod \"c3645361-80d7-4487-a79d-033a296e9016\" (UID: \"c3645361-80d7-4487-a79d-033a296e9016\") "
Jul 29 20:13:56.954842 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:56.954766    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3645361-80d7-4487-a79d-033a296e9016-openshift-service-ca\") pod \"c3645361-80d7-4487-a79d-033a296e9016\" (UID: \"c3645361-80d7-4487-a79d-033a296e9016\") "
Jul 29 20:13:56.954842 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:56.954791    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8ba63e5-214a-4485-9174-43b4a7dd6f0d-plugins-config-volume\") pod \"c8ba63e5-214a-4485-9174-43b4a7dd6f0d\" (UID: \"c8ba63e5-214a-4485-9174-43b4a7dd6f0d\") "
Jul 29 20:13:56.955009 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:56.954893    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftzlv\" (UniqueName: \"kubernetes.io/projected/c8ba63e5-214a-4485-9174-43b4a7dd6f0d-kube-api-access-ftzlv\") pod \"c8ba63e5-214a-4485-9174-43b4a7dd6f0d\" (UID: \"c8ba63e5-214a-4485-9174-43b4a7dd6f0d\") "
Jul 29 20:13:56.955009 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:56.954919    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8d7dv\" (UniqueName: \"kubernetes.io/projected/5eb22fa7-1bf4-46c4-b936-9f5941caacc7-kube-api-access-8d7dv\") pod \"5eb22fa7-1bf4-46c4-b936-9f5941caacc7\" (UID: \"5eb22fa7-1bf4-46c4-b936-9f5941caacc7\") "
Jul 29 20:13:56.955009 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:56.954959    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb22fa7-1bf4-46c4-b936-9f5941caacc7-plugins-config-volume\") pod \"5eb22fa7-1bf4-46c4-b936-9f5941caacc7\" (UID: \"5eb22fa7-1bf4-46c4-b936-9f5941caacc7\") "
Jul 29 20:13:56.955225 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:56.955127    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3645361-80d7-4487-a79d-033a296e9016-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c3645361-80d7-4487-a79d-033a296e9016" (UID: "c3645361-80d7-4487-a79d-033a296e9016"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:13:56.955288 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:56.955222    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3645361-80d7-4487-a79d-033a296e9016-openshift-service-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:13:56.955288 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:56.955221    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8ba63e5-214a-4485-9174-43b4a7dd6f0d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c8ba63e5-214a-4485-9174-43b4a7dd6f0d" (UID: "c8ba63e5-214a-4485-9174-43b4a7dd6f0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:13:56.955550 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:56.955524    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eb22fa7-1bf4-46c4-b936-9f5941caacc7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5eb22fa7-1bf4-46c4-b936-9f5941caacc7" (UID: "5eb22fa7-1bf4-46c4-b936-9f5941caacc7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:13:56.957062 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:56.957036    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8ba63e5-214a-4485-9174-43b4a7dd6f0d-kube-api-access-ftzlv" (OuterVolumeSpecName: "kube-api-access-ftzlv") pod "c8ba63e5-214a-4485-9174-43b4a7dd6f0d" (UID: "c8ba63e5-214a-4485-9174-43b4a7dd6f0d"). InnerVolumeSpecName "kube-api-access-ftzlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:13:56.957062 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:56.957043    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3645361-80d7-4487-a79d-033a296e9016-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c3645361-80d7-4487-a79d-033a296e9016" (UID: "c3645361-80d7-4487-a79d-033a296e9016"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:13:56.957198 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:56.957066    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3645361-80d7-4487-a79d-033a296e9016-kube-api-access-7cmjj" (OuterVolumeSpecName: "kube-api-access-7cmjj") pod "c3645361-80d7-4487-a79d-033a296e9016" (UID: "c3645361-80d7-4487-a79d-033a296e9016"). InnerVolumeSpecName "kube-api-access-7cmjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:13:56.957368 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:56.957346    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eb22fa7-1bf4-46c4-b936-9f5941caacc7-kube-api-access-8d7dv" (OuterVolumeSpecName: "kube-api-access-8d7dv") pod "5eb22fa7-1bf4-46c4-b936-9f5941caacc7" (UID: "5eb22fa7-1bf4-46c4-b936-9f5941caacc7"). InnerVolumeSpecName "kube-api-access-8d7dv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:13:57.056384 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:57.056299    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3645361-80d7-4487-a79d-033a296e9016-maas-api-tls\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:13:57.056384 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:57.056328    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cmjj\" (UniqueName: \"kubernetes.io/projected/c3645361-80d7-4487-a79d-033a296e9016-kube-api-access-7cmjj\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:13:57.056384 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:57.056341    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8ba63e5-214a-4485-9174-43b4a7dd6f0d-plugins-config-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:13:57.056384 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:57.056354    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftzlv\" (UniqueName: \"kubernetes.io/projected/c8ba63e5-214a-4485-9174-43b4a7dd6f0d-kube-api-access-ftzlv\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:13:57.056384 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:57.056366    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d7dv\" (UniqueName: \"kubernetes.io/projected/5eb22fa7-1bf4-46c4-b936-9f5941caacc7-kube-api-access-8d7dv\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:13:57.056384 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:57.056377    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb22fa7-1bf4-46c4-b936-9f5941caacc7-plugins-config-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:13:57.336450 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:57.336366    2583 generic.go:356] "Generic (PLEG): container finished" podID="c3645361-80d7-4487-a79d-033a296e9016" containerID="28da31b7d30400eaca61ec92d4b2dac8a4e475f9ce0432fde57177767499a038" exitCode=0
Jul 29 20:13:57.336620 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:57.336474    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a60880-75cbcb67fc-mwsfk"
Jul 29 20:13:57.337761 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:57.337735    2583 generic.go:356] "Generic (PLEG): container finished" podID="c8ba63e5-214a-4485-9174-43b4a7dd6f0d" containerID="adc93ba0a19ae1339c989111d38f06f67702ae8d65a829105043d7bf1140ee52" exitCode=0
Jul 29 20:13:57.337909 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:57.337854    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a60880-7846f4db76-nc6j5"
Jul 29 20:13:57.339177 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:57.339153    2583 generic.go:356] "Generic (PLEG): container finished" podID="5eb22fa7-1bf4-46c4-b936-9f5941caacc7" containerID="02e1f7c1f89075c8df9c8791e8a3567ab45b3d873b54c41d9c1a8ea882065cce" exitCode=0
Jul 29 20:13:57.339315 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:57.339212    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-a60880-6c5fff5886-mcrkd"
Jul 29 20:13:57.342244 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:57.342219    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4eb30076-3571-46d5-a81f-d792cdc4b6c6" path="/var/lib/kubelet/pods/4eb30076-3571-46d5-a81f-d792cdc4b6c6/volumes"
Jul 29 20:13:57.343089 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:57.342789    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a60880-75cbcb67fc-mwsfk" event={"ID":"c3645361-80d7-4487-a79d-033a296e9016","Type":"ContainerDied","Data":"28da31b7d30400eaca61ec92d4b2dac8a4e475f9ce0432fde57177767499a038"}
Jul 29 20:13:57.343089 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:57.342854    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a60880-75cbcb67fc-mwsfk" event={"ID":"c3645361-80d7-4487-a79d-033a296e9016","Type":"ContainerDied","Data":"db31c760436788e2c56c4f7c86f97b6cc79bb3ad60642b502d59f8630729ffd4"}
Jul 29 20:13:57.343089 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:57.342879    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a60880-7846f4db76-nc6j5" event={"ID":"c8ba63e5-214a-4485-9174-43b4a7dd6f0d","Type":"ContainerDied","Data":"adc93ba0a19ae1339c989111d38f06f67702ae8d65a829105043d7bf1140ee52"}
Jul 29 20:13:57.343089 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:57.342903    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a60880-7846f4db76-nc6j5" event={"ID":"c8ba63e5-214a-4485-9174-43b4a7dd6f0d","Type":"ContainerDied","Data":"eefa3f9e57f018a108c645ebbfb3b13f0e63a3ec6b2970d51738c5f2b64e9d6a"}
Jul 29 20:13:57.343089 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:57.342923    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-a60880-6c5fff5886-mcrkd" event={"ID":"5eb22fa7-1bf4-46c4-b936-9f5941caacc7","Type":"ContainerDied","Data":"02e1f7c1f89075c8df9c8791e8a3567ab45b3d873b54c41d9c1a8ea882065cce"}
Jul 29 20:13:57.343089 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:57.342945    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-a60880-6c5fff5886-mcrkd" event={"ID":"5eb22fa7-1bf4-46c4-b936-9f5941caacc7","Type":"ContainerDied","Data":"d6b097e9640cd71e816cd9d54cd72e7623208c3aaeee08486df56f534dafe02b"}
Jul 29 20:13:57.343089 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:57.342974    2583 scope.go:117] "RemoveContainer" containerID="28da31b7d30400eaca61ec92d4b2dac8a4e475f9ce0432fde57177767499a038"
Jul 29 20:13:57.354785 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:57.354765    2583 scope.go:117] "RemoveContainer" containerID="28da31b7d30400eaca61ec92d4b2dac8a4e475f9ce0432fde57177767499a038"
Jul 29 20:13:57.355205 ip-10-0-128-101 kubenswrapper[2583]: E0729 20:13:57.355185    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28da31b7d30400eaca61ec92d4b2dac8a4e475f9ce0432fde57177767499a038\": container with ID starting with 28da31b7d30400eaca61ec92d4b2dac8a4e475f9ce0432fde57177767499a038 not found: ID does not exist" containerID="28da31b7d30400eaca61ec92d4b2dac8a4e475f9ce0432fde57177767499a038"
Jul 29 20:13:57.355293 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:57.355210    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28da31b7d30400eaca61ec92d4b2dac8a4e475f9ce0432fde57177767499a038"} err="failed to get container status \"28da31b7d30400eaca61ec92d4b2dac8a4e475f9ce0432fde57177767499a038\": rpc error: code = NotFound desc = could not find container \"28da31b7d30400eaca61ec92d4b2dac8a4e475f9ce0432fde57177767499a038\": container with ID starting with 28da31b7d30400eaca61ec92d4b2dac8a4e475f9ce0432fde57177767499a038 not found: ID does not exist"
Jul 29 20:13:57.355293 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:57.355225    2583 scope.go:117] "RemoveContainer" containerID="adc93ba0a19ae1339c989111d38f06f67702ae8d65a829105043d7bf1140ee52"
Jul 29 20:13:57.362928 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:57.362911    2583 scope.go:117] "RemoveContainer" containerID="adc93ba0a19ae1339c989111d38f06f67702ae8d65a829105043d7bf1140ee52"
Jul 29 20:13:57.363164 ip-10-0-128-101 kubenswrapper[2583]: E0729 20:13:57.363147    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adc93ba0a19ae1339c989111d38f06f67702ae8d65a829105043d7bf1140ee52\": container with ID starting with adc93ba0a19ae1339c989111d38f06f67702ae8d65a829105043d7bf1140ee52 not found: ID does not exist" containerID="adc93ba0a19ae1339c989111d38f06f67702ae8d65a829105043d7bf1140ee52"
Jul 29 20:13:57.363227 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:57.363169    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adc93ba0a19ae1339c989111d38f06f67702ae8d65a829105043d7bf1140ee52"} err="failed to get container status \"adc93ba0a19ae1339c989111d38f06f67702ae8d65a829105043d7bf1140ee52\": rpc error: code = NotFound desc = could not find container \"adc93ba0a19ae1339c989111d38f06f67702ae8d65a829105043d7bf1140ee52\": container with ID starting with adc93ba0a19ae1339c989111d38f06f67702ae8d65a829105043d7bf1140ee52 not found: ID does not exist"
Jul 29 20:13:57.363227 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:57.363184    2583 scope.go:117] "RemoveContainer" containerID="02e1f7c1f89075c8df9c8791e8a3567ab45b3d873b54c41d9c1a8ea882065cce"
Jul 29 20:13:57.368100 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:57.368076    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-a60880-7846f4db76-nc6j5"]
Jul 29 20:13:57.370787 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:57.370775    2583 scope.go:117] "RemoveContainer" containerID="02e1f7c1f89075c8df9c8791e8a3567ab45b3d873b54c41d9c1a8ea882065cce"
Jul 29 20:13:57.371039 ip-10-0-128-101 kubenswrapper[2583]: E0729 20:13:57.371020    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02e1f7c1f89075c8df9c8791e8a3567ab45b3d873b54c41d9c1a8ea882065cce\": container with ID starting with 02e1f7c1f89075c8df9c8791e8a3567ab45b3d873b54c41d9c1a8ea882065cce not found: ID does not exist" containerID="02e1f7c1f89075c8df9c8791e8a3567ab45b3d873b54c41d9c1a8ea882065cce"
Jul 29 20:13:57.371088 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:57.371046    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02e1f7c1f89075c8df9c8791e8a3567ab45b3d873b54c41d9c1a8ea882065cce"} err="failed to get container status \"02e1f7c1f89075c8df9c8791e8a3567ab45b3d873b54c41d9c1a8ea882065cce\": rpc error: code = NotFound desc = could not find container \"02e1f7c1f89075c8df9c8791e8a3567ab45b3d873b54c41d9c1a8ea882065cce\": container with ID starting with 02e1f7c1f89075c8df9c8791e8a3567ab45b3d873b54c41d9c1a8ea882065cce not found: ID does not exist"
Jul 29 20:13:57.374241 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:57.374221    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-a60880-7846f4db76-nc6j5"]
Jul 29 20:13:57.385321 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:57.385299    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a60880-75cbcb67fc-mwsfk"]
Jul 29 20:13:57.389365 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:57.389345    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a60880-75cbcb67fc-mwsfk"]
Jul 29 20:13:57.402771 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:57.402747    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-a60880-6c5fff5886-mcrkd"]
Jul 29 20:13:57.406572 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:57.406555    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-a60880-6c5fff5886-mcrkd"]
Jul 29 20:13:59.340582 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:59.340551    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5eb22fa7-1bf4-46c4-b936-9f5941caacc7" path="/var/lib/kubelet/pods/5eb22fa7-1bf4-46c4-b936-9f5941caacc7/volumes"
Jul 29 20:13:59.340959 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:59.340934    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3645361-80d7-4487-a79d-033a296e9016" path="/var/lib/kubelet/pods/c3645361-80d7-4487-a79d-033a296e9016/volumes"
Jul 29 20:13:59.341244 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:59.341230    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8ba63e5-214a-4485-9174-43b4a7dd6f0d" path="/var/lib/kubelet/pods/c8ba63e5-214a-4485-9174-43b4a7dd6f0d/volumes"
Jul 29 20:13:59.608301 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:13:59.608217    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 20:14:04.318653 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:04.318621    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-601aa9-n7x94"]
Jul 29 20:14:04.319056 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:04.318943    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="297f43e6-7f3c-4bc5-8675-ab57c4728032" containerName="payload-processing"
Jul 29 20:14:04.319056 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:04.318955    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="297f43e6-7f3c-4bc5-8675-ab57c4728032" containerName="payload-processing"
Jul 29 20:14:04.319056 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:04.318970    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95221161-2bd1-405c-b856-8c51f3dff04d" containerName="maas-api"
Jul 29 20:14:04.319056 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:04.318975    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="95221161-2bd1-405c-b856-8c51f3dff04d" containerName="maas-api"
Jul 29 20:14:04.319056 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:04.318989    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5eb22fa7-1bf4-46c4-b936-9f5941caacc7" containerName="payload-processing"
Jul 29 20:14:04.319056 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:04.318996    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5eb22fa7-1bf4-46c4-b936-9f5941caacc7" containerName="payload-processing"
Jul 29 20:14:04.319056 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:04.319003    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3645361-80d7-4487-a79d-033a296e9016" containerName="maas-api"
Jul 29 20:14:04.319056 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:04.319008    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3645361-80d7-4487-a79d-033a296e9016" containerName="maas-api"
Jul 29 20:14:04.319056 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:04.319015    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8ba63e5-214a-4485-9174-43b4a7dd6f0d" containerName="payload-pre-processing"
Jul 29 20:14:04.319056 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:04.319021    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8ba63e5-214a-4485-9174-43b4a7dd6f0d" containerName="payload-pre-processing"
Jul 29 20:14:04.319056 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:04.319027    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ae898a3-d821-4cd4-8c52-58aeeb6a503a" containerName="revoke-keys"
Jul 29 20:14:04.319056 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:04.319031    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ae898a3-d821-4cd4-8c52-58aeeb6a503a" containerName="revoke-keys"
Jul 29 20:14:04.319420 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:04.319078    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8ba63e5-214a-4485-9174-43b4a7dd6f0d" containerName="payload-pre-processing"
Jul 29 20:14:04.319420 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:04.319084    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ae898a3-d821-4cd4-8c52-58aeeb6a503a" containerName="revoke-keys"
Jul 29 20:14:04.319420 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:04.319091    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="297f43e6-7f3c-4bc5-8675-ab57c4728032" containerName="payload-processing"
Jul 29 20:14:04.319420 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:04.319098    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3645361-80d7-4487-a79d-033a296e9016" containerName="maas-api"
Jul 29 20:14:04.319420 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:04.319104    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5eb22fa7-1bf4-46c4-b936-9f5941caacc7" containerName="payload-processing"
Jul 29 20:14:04.319420 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:04.319109    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="95221161-2bd1-405c-b856-8c51f3dff04d" containerName="maas-api"
Jul 29 20:14:04.323644 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:04.323626    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-601aa9-n7x94"
Jul 29 20:14:04.325895 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:04.325875    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hq9l2\""
Jul 29 20:14:04.331168 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:04.331024    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-601aa9-n7x94"]
Jul 29 20:14:04.417534 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:04.417492    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e1916a4-965c-4537-b529-18d1e087a654-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-601aa9-n7x94\" (UID: \"9e1916a4-965c-4537-b529-18d1e087a654\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-601aa9-n7x94"
Jul 29 20:14:04.518138 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:04.518060    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e1916a4-965c-4537-b529-18d1e087a654-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-601aa9-n7x94\" (UID: \"9e1916a4-965c-4537-b529-18d1e087a654\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-601aa9-n7x94"
Jul 29 20:14:04.518694 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:04.518674    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e1916a4-965c-4537-b529-18d1e087a654-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-601aa9-n7x94\" (UID: \"9e1916a4-965c-4537-b529-18d1e087a654\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-601aa9-n7x94"
Jul 29 20:14:04.635514 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:04.635486    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-601aa9-n7x94"
Jul 29 20:14:04.766309 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:04.766284    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-601aa9-n7x94"]
Jul 29 20:14:04.768320 ip-10-0-128-101 kubenswrapper[2583]: W0729 20:14:04.768287    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e1916a4_965c_4537_b529_18d1e087a654.slice/crio-3984e2d9d6afcddbbd2bcef7535f97fa4912c9ed1530053faa13850ecbd007ab WatchSource:0}: Error finding container 3984e2d9d6afcddbbd2bcef7535f97fa4912c9ed1530053faa13850ecbd007ab: Status 404 returned error can't find the container with id 3984e2d9d6afcddbbd2bcef7535f97fa4912c9ed1530053faa13850ecbd007ab
Jul 29 20:14:05.384350 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:05.384313    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-601aa9-n7x94" event={"ID":"9e1916a4-965c-4537-b529-18d1e087a654","Type":"ContainerStarted","Data":"d618bd5e444f9fa0117c0948aa86dc4926c27b4ede9a7dfe5f9ac75a8cdea4a5"}
Jul 29 20:14:05.384772 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:05.384366    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-601aa9-n7x94" event={"ID":"9e1916a4-965c-4537-b529-18d1e087a654","Type":"ContainerStarted","Data":"3984e2d9d6afcddbbd2bcef7535f97fa4912c9ed1530053faa13850ecbd007ab"}
Jul 29 20:14:05.399793 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:05.399732    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-601aa9-n7x94" podStartSLOduration=1.399711895 podStartE2EDuration="1.399711895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:14:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:14:05.399390261 +0000 UTC m=+3412.627779387" watchObservedRunningTime="2026-07-29 20:14:05.399711895 +0000 UTC m=+3412.628101022"
Jul 29 20:14:06.390449 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:06.390414    2583 generic.go:356] "Generic (PLEG): container finished" podID="9e1916a4-965c-4537-b529-18d1e087a654" containerID="d618bd5e444f9fa0117c0948aa86dc4926c27b4ede9a7dfe5f9ac75a8cdea4a5" exitCode=0
Jul 29 20:14:06.390861 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:06.390503    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-601aa9-n7x94" event={"ID":"9e1916a4-965c-4537-b529-18d1e087a654","Type":"ContainerDied","Data":"d618bd5e444f9fa0117c0948aa86dc4926c27b4ede9a7dfe5f9ac75a8cdea4a5"}
Jul 29 20:14:07.526092 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:07.526064    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-601aa9-n7x94"
Jul 29 20:14:07.645293 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:07.645250    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e1916a4-965c-4537-b529-18d1e087a654-maas-api-service-ca\") pod \"9e1916a4-965c-4537-b529-18d1e087a654\" (UID: \"9e1916a4-965c-4537-b529-18d1e087a654\") "
Jul 29 20:14:07.645653 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:07.645599    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e1916a4-965c-4537-b529-18d1e087a654-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9e1916a4-965c-4537-b529-18d1e087a654" (UID: "9e1916a4-965c-4537-b529-18d1e087a654"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:14:07.746770 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:07.746735    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e1916a4-965c-4537-b529-18d1e087a654-maas-api-service-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:14:08.400705 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:08.400675    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-601aa9-n7x94"
Jul 29 20:14:08.400917 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:08.400727    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-601aa9-n7x94" event={"ID":"9e1916a4-965c-4537-b529-18d1e087a654","Type":"ContainerDied","Data":"3984e2d9d6afcddbbd2bcef7535f97fa4912c9ed1530053faa13850ecbd007ab"}
Jul 29 20:14:08.400917 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:08.400776    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3984e2d9d6afcddbbd2bcef7535f97fa4912c9ed1530053faa13850ecbd007ab"
Jul 29 20:14:13.696709 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:13.696676    2583 scope.go:117] "RemoveContainer" containerID="3753054139f31640adeafa07f3e9487cb7957bdb0d3c9d35452c90c8bb17d734"
Jul 29 20:14:14.810523 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:14.810488    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-601aa9-74fdb4b86c-vdmcl"]
Jul 29 20:14:14.810983 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:14.810754    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-601aa9-74fdb4b86c-vdmcl" podUID="9d879ed6-e8a6-4784-9e23-74fdd10eb93a" containerName="maas-api" containerID="cri-o://d92940c50ed5b744a0be300cfb6a47353c7a11b2dd9da44a208eef626c3a1ece" gracePeriod=30
Jul 29 20:14:14.860307 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:14.860277    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-601aa9-29755920-dlwgt"]
Jul 29 20:14:14.865794 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:14.865767    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-601aa9-29755920-dlwgt"]
Jul 29 20:14:14.934305 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:14.934264    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-601aa9-685f844dfd-fljl2"]
Jul 29 20:14:14.934512 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:14.934483    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-601aa9-685f844dfd-fljl2" podUID="ba56ecab-018c-4f0f-a801-0141f5995f15" containerName="payload-processing" containerID="cri-o://885fc74d8681a6862ec118d5099081f7c0c199a389f34b29ad269d7e43bde617" gracePeriod=30
Jul 29 20:14:15.078294 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:15.078259    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-601aa9-74fdb4b86c-vdmcl"
Jul 29 20:14:15.153028 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:15.152996    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-601aa9-n7x94"]
Jul 29 20:14:15.156055 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:15.156032    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-601aa9-n7x94"]
Jul 29 20:14:15.204247 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:15.204222    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-601aa9-685f844dfd-fljl2"
Jul 29 20:14:15.209750 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:15.209728    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d879ed6-e8a6-4784-9e23-74fdd10eb93a-maas-api-tls\") pod \"9d879ed6-e8a6-4784-9e23-74fdd10eb93a\" (UID: \"9d879ed6-e8a6-4784-9e23-74fdd10eb93a\") "
Jul 29 20:14:15.209905 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:15.209764    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d879ed6-e8a6-4784-9e23-74fdd10eb93a-openshift-service-ca\") pod \"9d879ed6-e8a6-4784-9e23-74fdd10eb93a\" (UID: \"9d879ed6-e8a6-4784-9e23-74fdd10eb93a\") "
Jul 29 20:14:15.209905 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:15.209816    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvqk6\" (UniqueName: \"kubernetes.io/projected/9d879ed6-e8a6-4784-9e23-74fdd10eb93a-kube-api-access-vvqk6\") pod \"9d879ed6-e8a6-4784-9e23-74fdd10eb93a\" (UID: \"9d879ed6-e8a6-4784-9e23-74fdd10eb93a\") "
Jul 29 20:14:15.210179 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:15.210155    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d879ed6-e8a6-4784-9e23-74fdd10eb93a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9d879ed6-e8a6-4784-9e23-74fdd10eb93a" (UID: "9d879ed6-e8a6-4784-9e23-74fdd10eb93a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:14:15.211849 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:15.211822    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d879ed6-e8a6-4784-9e23-74fdd10eb93a-kube-api-access-vvqk6" (OuterVolumeSpecName: "kube-api-access-vvqk6") pod "9d879ed6-e8a6-4784-9e23-74fdd10eb93a" (UID: "9d879ed6-e8a6-4784-9e23-74fdd10eb93a"). InnerVolumeSpecName "kube-api-access-vvqk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:14:15.211966 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:15.211853    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d879ed6-e8a6-4784-9e23-74fdd10eb93a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9d879ed6-e8a6-4784-9e23-74fdd10eb93a" (UID: "9d879ed6-e8a6-4784-9e23-74fdd10eb93a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:14:15.311029 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:15.310989    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba56ecab-018c-4f0f-a801-0141f5995f15-plugins-config-volume\") pod \"ba56ecab-018c-4f0f-a801-0141f5995f15\" (UID: \"ba56ecab-018c-4f0f-a801-0141f5995f15\") "
Jul 29 20:14:15.311029 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:15.311033    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9x9n\" (UniqueName: \"kubernetes.io/projected/ba56ecab-018c-4f0f-a801-0141f5995f15-kube-api-access-p9x9n\") pod \"ba56ecab-018c-4f0f-a801-0141f5995f15\" (UID: \"ba56ecab-018c-4f0f-a801-0141f5995f15\") "
Jul 29 20:14:15.311346 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:15.311294    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d879ed6-e8a6-4784-9e23-74fdd10eb93a-maas-api-tls\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:14:15.311346 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:15.311317    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d879ed6-e8a6-4784-9e23-74fdd10eb93a-openshift-service-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:14:15.311346 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:15.311331    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvqk6\" (UniqueName: \"kubernetes.io/projected/9d879ed6-e8a6-4784-9e23-74fdd10eb93a-kube-api-access-vvqk6\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:14:15.311466 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:15.311431    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba56ecab-018c-4f0f-a801-0141f5995f15-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ba56ecab-018c-4f0f-a801-0141f5995f15" (UID: "ba56ecab-018c-4f0f-a801-0141f5995f15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:14:15.313171 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:15.313150    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba56ecab-018c-4f0f-a801-0141f5995f15-kube-api-access-p9x9n" (OuterVolumeSpecName: "kube-api-access-p9x9n") pod "ba56ecab-018c-4f0f-a801-0141f5995f15" (UID: "ba56ecab-018c-4f0f-a801-0141f5995f15"). InnerVolumeSpecName "kube-api-access-p9x9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:14:15.340640 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:15.340610    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3dd467c4-e5cc-405a-8c3f-6448abda4632" path="/var/lib/kubelet/pods/3dd467c4-e5cc-405a-8c3f-6448abda4632/volumes"
Jul 29 20:14:15.340916 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:15.340903    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e1916a4-965c-4537-b529-18d1e087a654" path="/var/lib/kubelet/pods/9e1916a4-965c-4537-b529-18d1e087a654/volumes"
Jul 29 20:14:15.412151 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:15.412110    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba56ecab-018c-4f0f-a801-0141f5995f15-plugins-config-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:14:15.412151 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:15.412146    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9x9n\" (UniqueName: \"kubernetes.io/projected/ba56ecab-018c-4f0f-a801-0141f5995f15-kube-api-access-p9x9n\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Jul 29 20:14:15.435202 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:15.435163    2583 generic.go:356] "Generic (PLEG): container finished" podID="ba56ecab-018c-4f0f-a801-0141f5995f15" containerID="885fc74d8681a6862ec118d5099081f7c0c199a389f34b29ad269d7e43bde617" exitCode=0
Jul 29 20:14:15.435354 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:15.435200    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-601aa9-685f844dfd-fljl2" event={"ID":"ba56ecab-018c-4f0f-a801-0141f5995f15","Type":"ContainerDied","Data":"885fc74d8681a6862ec118d5099081f7c0c199a389f34b29ad269d7e43bde617"}
Jul 29 20:14:15.435354 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:15.435256    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-601aa9-685f844dfd-fljl2" event={"ID":"ba56ecab-018c-4f0f-a801-0141f5995f15","Type":"ContainerDied","Data":"02f445a85680c690e3189d6bfc1246eec7a7b5ad7b47a1169436739e17bad83b"}
Jul 29 20:14:15.435354 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:15.435206    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-601aa9-685f844dfd-fljl2"
Jul 29 20:14:15.435354 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:15.435281    2583 scope.go:117] "RemoveContainer" containerID="885fc74d8681a6862ec118d5099081f7c0c199a389f34b29ad269d7e43bde617"
Jul 29 20:14:15.436737 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:15.436708    2583 generic.go:356] "Generic (PLEG): container finished" podID="9d879ed6-e8a6-4784-9e23-74fdd10eb93a" containerID="d92940c50ed5b744a0be300cfb6a47353c7a11b2dd9da44a208eef626c3a1ece" exitCode=0
Jul 29 20:14:15.436896 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:15.436765    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-601aa9-74fdb4b86c-vdmcl"
Jul 29 20:14:15.436896 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:15.436785    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-601aa9-74fdb4b86c-vdmcl" event={"ID":"9d879ed6-e8a6-4784-9e23-74fdd10eb93a","Type":"ContainerDied","Data":"d92940c50ed5b744a0be300cfb6a47353c7a11b2dd9da44a208eef626c3a1ece"}
Jul 29 20:14:15.436896 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:15.436847    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-601aa9-74fdb4b86c-vdmcl" event={"ID":"9d879ed6-e8a6-4784-9e23-74fdd10eb93a","Type":"ContainerDied","Data":"daba1786d1b64b2bedd3de96a3d9ef9a0f68ec0ffdab34380156bc21dffbd147"}
Jul 29 20:14:15.444031 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:15.444006    2583 scope.go:117] "RemoveContainer" containerID="885fc74d8681a6862ec118d5099081f7c0c199a389f34b29ad269d7e43bde617"
Jul 29 20:14:15.444315 ip-10-0-128-101 kubenswrapper[2583]: E0729 20:14:15.444297    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"885fc74d8681a6862ec118d5099081f7c0c199a389f34b29ad269d7e43bde617\": container with ID starting with 885fc74d8681a6862ec118d5099081f7c0c199a389f34b29ad269d7e43bde617 not found: ID does not exist" containerID="885fc74d8681a6862ec118d5099081f7c0c199a389f34b29ad269d7e43bde617"
Jul 29 20:14:15.444407 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:15.444321    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"885fc74d8681a6862ec118d5099081f7c0c199a389f34b29ad269d7e43bde617"} err="failed to get container status \"885fc74d8681a6862ec118d5099081f7c0c199a389f34b29ad269d7e43bde617\": rpc error: code = NotFound desc = could not find container \"885fc74d8681a6862ec118d5099081f7c0c199a389f34b29ad269d7e43bde617\": container with ID starting with 885fc74d8681a6862ec118d5099081f7c0c199a389f34b29ad269d7e43bde617 not found: ID does not exist"
Jul 29 20:14:15.444407 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:15.444337    2583 scope.go:117] "RemoveContainer" containerID="d92940c50ed5b744a0be300cfb6a47353c7a11b2dd9da44a208eef626c3a1ece"
Jul 29 20:14:15.452014 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:15.451992    2583 scope.go:117] "RemoveContainer" containerID="d92940c50ed5b744a0be300cfb6a47353c7a11b2dd9da44a208eef626c3a1ece"
Jul 29 20:14:15.452272 ip-10-0-128-101 kubenswrapper[2583]: E0729 20:14:15.452248    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d92940c50ed5b744a0be300cfb6a47353c7a11b2dd9da44a208eef626c3a1ece\": container with ID starting with d92940c50ed5b744a0be300cfb6a47353c7a11b2dd9da44a208eef626c3a1ece not found: ID does not exist" containerID="d92940c50ed5b744a0be300cfb6a47353c7a11b2dd9da44a208eef626c3a1ece"
Jul 29 20:14:15.452384 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:15.452282    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d92940c50ed5b744a0be300cfb6a47353c7a11b2dd9da44a208eef626c3a1ece"} err="failed to get container status \"d92940c50ed5b744a0be300cfb6a47353c7a11b2dd9da44a208eef626c3a1ece\": rpc error: code = NotFound desc = could not find container \"d92940c50ed5b744a0be300cfb6a47353c7a11b2dd9da44a208eef626c3a1ece\": container with ID starting with d92940c50ed5b744a0be300cfb6a47353c7a11b2dd9da44a208eef626c3a1ece not found: ID does not exist"
Jul 29 20:14:15.454505 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:15.454487    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-601aa9-685f844dfd-fljl2"]
Jul 29 20:14:15.456687 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:15.456666    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-601aa9-685f844dfd-fljl2"]
Jul 29 20:14:15.465182 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:15.465151    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-601aa9-74fdb4b86c-vdmcl"]
Jul 29 20:14:15.469523 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:15.469499    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-601aa9-74fdb4b86c-vdmcl"]
Jul 29 20:14:17.339543 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:17.339509    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d879ed6-e8a6-4784-9e23-74fdd10eb93a" path="/var/lib/kubelet/pods/9d879ed6-e8a6-4784-9e23-74fdd10eb93a/volumes"
Jul 29 20:14:17.339929 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:17.339872    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba56ecab-018c-4f0f-a801-0141f5995f15" path="/var/lib/kubelet/pods/ba56ecab-018c-4f0f-a801-0141f5995f15/volumes"
Jul 29 20:14:18.108064 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:18.108027    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sd7wr"]
Jul 29 20:14:28.252253 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:28.252201    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-q94h6_7ec5dddd-2a18-46ce-bbd3-ff5f0c447915/manager/0.log"
Jul 29 20:14:28.371606 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:28.371576    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-t9vcv_538180db-c08e-4b16-b2b8-ded900ea01e1/manager/0.log"
Jul 29 20:14:28.623285 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:28.623204    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-dbbkd_3915492a-6290-43a6-b7a0-3643b5085183/server/0.log"
Jul 29 20:14:28.742272 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:28.742233    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-gqj8n_9a18c2d9-4c11-42de-bdea-16ff6e458049/manager/0.log"
Jul 29 20:14:29.110634 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:29.110589    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-667b7bc75f-g6jzm_cae6c80b-633e-43ba-b0e9-0045ebd3fe37/manager/0.log"
Jul 29 20:14:29.117749 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:29.117725    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-667b7bc75f-g6jzm_cae6c80b-633e-43ba-b0e9-0045ebd3fe37/copy-manifests/0.log"
Jul 29 20:14:29.556547 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:29.556520    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29755890-2wzkf_c3979261-f3fb-47da-ba69-d7ea080f6f48/cleanup/0.log"
Jul 29 20:14:29.660299 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:29.660273    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29755905-69g49_269c6be3-441e-43e8-8931-eec7457dfb01/cleanup/0.log"
Jul 29 20:14:29.766188 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:29.766160    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29755920-4f7k8_51c05aa5-0bb1-40eb-bf19-e85668da7db1/cleanup/0.log"
Jul 29 20:14:31.221623 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:31.221594    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-6g6tj_f7cfc0f9-8792-41bb-82a7-fdc902cc5506/manager/0.log"
Jul 29 20:14:31.791434 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:31.791408    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-sd7wr_382a805c-5973-428e-888c-1c30e0226476/limitador/0.log"
Jul 29 20:14:32.355045 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:32.355015    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-srr62_6d180d8b-da5e-4cfb-a409-eef0f14ebe9b/discovery/0.log"
Jul 29 20:14:32.566978 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:32.566946    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b6ffddf6-tcktk_54036b22-87b4-4104-bd40-56da3fb1d01f/kube-auth-proxy/0.log"
Jul 29 20:14:33.661565 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:33.661530    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbkfs7_8defa80b-2b41-49b3-921b-57792d5fb3f2/main/0.log"
Jul 29 20:14:41.768333 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:41.768299    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pp24q_2485c438-0e6d-4aae-bab6-8dcda418e188/global-pull-secret-syncer/0.log"
Jul 29 20:14:41.893209 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:41.893180    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-m8f8f_efd6982e-9638-4562-9ff3-9aab71caae71/konnectivity-agent/0.log"
Jul 29 20:14:41.957322 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:41.957299    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-101.ec2.internal_b19b16543121e99642de34dc778bf29b/haproxy/0.log"
Jul 29 20:14:46.868723 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:46.868696    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-6g6tj_f7cfc0f9-8792-41bb-82a7-fdc902cc5506/manager/0.log"
Jul 29 20:14:47.235640 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:47.235615    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-sd7wr_382a805c-5973-428e-888c-1c30e0226476/limitador/0.log"
Jul 29 20:14:49.523383 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:49.523312    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d5m5t_b6c4faa3-5b24-4394-a73d-915b1b9911f5/node-exporter/0.log"
Jul 29 20:14:49.545637 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:49.545592    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d5m5t_b6c4faa3-5b24-4394-a73d-915b1b9911f5/kube-rbac-proxy/0.log"
Jul 29 20:14:49.567480 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:49.567461    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d5m5t_b6c4faa3-5b24-4394-a73d-915b1b9911f5/init-textfile/0.log"
Jul 29 20:14:51.446420 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:51.446395    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-sz59c_78662159-1110-4413-ab45-219ed9d4ede9/networking-console-plugin/0.log"
Jul 29 20:14:51.684886 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:51.684855    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ttn7v/perf-node-gather-daemonset-9plb8"]
Jul 29 20:14:51.685149 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:51.685137    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d879ed6-e8a6-4784-9e23-74fdd10eb93a" containerName="maas-api"
Jul 29 20:14:51.685199 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:51.685150    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d879ed6-e8a6-4784-9e23-74fdd10eb93a" containerName="maas-api"
Jul 29 20:14:51.685199 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:51.685166    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba56ecab-018c-4f0f-a801-0141f5995f15" containerName="payload-processing"
Jul 29 20:14:51.685199 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:51.685173    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba56ecab-018c-4f0f-a801-0141f5995f15" containerName="payload-processing"
Jul 29 20:14:51.685199 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:51.685184    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e1916a4-965c-4537-b529-18d1e087a654" containerName="revoke-keys"
Jul 29 20:14:51.685199 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:51.685190    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e1916a4-965c-4537-b529-18d1e087a654" containerName="revoke-keys"
Jul 29 20:14:51.685364 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:51.685238    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba56ecab-018c-4f0f-a801-0141f5995f15" containerName="payload-processing"
Jul 29 20:14:51.685364 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:51.685247    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d879ed6-e8a6-4784-9e23-74fdd10eb93a" containerName="maas-api"
Jul 29 20:14:51.685364 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:51.685255    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e1916a4-965c-4537-b529-18d1e087a654" containerName="revoke-keys"
Jul 29 20:14:51.688206 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:51.688192    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ttn7v/perf-node-gather-daemonset-9plb8"
Jul 29 20:14:51.690518 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:51.690494    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ttn7v\"/\"kube-root-ca.crt\""
Jul 29 20:14:51.691046 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:51.691027    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ttn7v\"/\"openshift-service-ca.crt\""
Jul 29 20:14:51.691046 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:51.691035    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ttn7v\"/\"default-dockercfg-4bzhx\""
Jul 29 20:14:51.696899 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:51.696854    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ttn7v/perf-node-gather-daemonset-9plb8"]
Jul 29 20:14:51.791346 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:51.791321    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4de8dcca-677d-42e6-89d8-b009d42a2eac-podres\") pod \"perf-node-gather-daemonset-9plb8\" (UID: \"4de8dcca-677d-42e6-89d8-b009d42a2eac\") " pod="openshift-must-gather-ttn7v/perf-node-gather-daemonset-9plb8"
Jul 29 20:14:51.791493 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:51.791351    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4de8dcca-677d-42e6-89d8-b009d42a2eac-lib-modules\") pod \"perf-node-gather-daemonset-9plb8\" (UID: \"4de8dcca-677d-42e6-89d8-b009d42a2eac\") " pod="openshift-must-gather-ttn7v/perf-node-gather-daemonset-9plb8"
Jul 29 20:14:51.791493 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:51.791377    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w92hl\" (UniqueName: \"kubernetes.io/projected/4de8dcca-677d-42e6-89d8-b009d42a2eac-kube-api-access-w92hl\") pod \"perf-node-gather-daemonset-9plb8\" (UID: \"4de8dcca-677d-42e6-89d8-b009d42a2eac\") " pod="openshift-must-gather-ttn7v/perf-node-gather-daemonset-9plb8"
Jul 29 20:14:51.791611 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:51.791510    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4de8dcca-677d-42e6-89d8-b009d42a2eac-sys\") pod \"perf-node-gather-daemonset-9plb8\" (UID: \"4de8dcca-677d-42e6-89d8-b009d42a2eac\") " pod="openshift-must-gather-ttn7v/perf-node-gather-daemonset-9plb8"
Jul 29 20:14:51.791611 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:51.791541    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4de8dcca-677d-42e6-89d8-b009d42a2eac-proc\") pod \"perf-node-gather-daemonset-9plb8\" (UID: \"4de8dcca-677d-42e6-89d8-b009d42a2eac\") " pod="openshift-must-gather-ttn7v/perf-node-gather-daemonset-9plb8"
Jul 29 20:14:51.892438 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:51.892415    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4de8dcca-677d-42e6-89d8-b009d42a2eac-sys\") pod \"perf-node-gather-daemonset-9plb8\" (UID: \"4de8dcca-677d-42e6-89d8-b009d42a2eac\") " pod="openshift-must-gather-ttn7v/perf-node-gather-daemonset-9plb8"
Jul 29 20:14:51.892555 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:51.892444    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4de8dcca-677d-42e6-89d8-b009d42a2eac-proc\") pod \"perf-node-gather-daemonset-9plb8\" (UID: \"4de8dcca-677d-42e6-89d8-b009d42a2eac\") " pod="openshift-must-gather-ttn7v/perf-node-gather-daemonset-9plb8"
Jul 29 20:14:51.892555 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:51.892469    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4de8dcca-677d-42e6-89d8-b009d42a2eac-podres\") pod \"perf-node-gather-daemonset-9plb8\" (UID: \"4de8dcca-677d-42e6-89d8-b009d42a2eac\") " pod="openshift-must-gather-ttn7v/perf-node-gather-daemonset-9plb8"
Jul 29 20:14:51.892555 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:51.892519    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4de8dcca-677d-42e6-89d8-b009d42a2eac-proc\") pod \"perf-node-gather-daemonset-9plb8\" (UID: \"4de8dcca-677d-42e6-89d8-b009d42a2eac\") " pod="openshift-must-gather-ttn7v/perf-node-gather-daemonset-9plb8"
Jul 29 20:14:51.892555 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:51.892534    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4de8dcca-677d-42e6-89d8-b009d42a2eac-sys\") pod \"perf-node-gather-daemonset-9plb8\" (UID: \"4de8dcca-677d-42e6-89d8-b009d42a2eac\") " pod="openshift-must-gather-ttn7v/perf-node-gather-daemonset-9plb8"
Jul 29 20:14:51.892759 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:51.892574    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4de8dcca-677d-42e6-89d8-b009d42a2eac-lib-modules\") pod \"perf-node-gather-daemonset-9plb8\" (UID: \"4de8dcca-677d-42e6-89d8-b009d42a2eac\") " pod="openshift-must-gather-ttn7v/perf-node-gather-daemonset-9plb8"
Jul 29 20:14:51.892759 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:51.892624    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w92hl\" (UniqueName: \"kubernetes.io/projected/4de8dcca-677d-42e6-89d8-b009d42a2eac-kube-api-access-w92hl\") pod \"perf-node-gather-daemonset-9plb8\" (UID: \"4de8dcca-677d-42e6-89d8-b009d42a2eac\") " pod="openshift-must-gather-ttn7v/perf-node-gather-daemonset-9plb8"
Jul 29 20:14:51.892759 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:51.892677    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4de8dcca-677d-42e6-89d8-b009d42a2eac-podres\") pod \"perf-node-gather-daemonset-9plb8\" (UID: \"4de8dcca-677d-42e6-89d8-b009d42a2eac\") " pod="openshift-must-gather-ttn7v/perf-node-gather-daemonset-9plb8"
Jul 29 20:14:51.892759 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:51.892694    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4de8dcca-677d-42e6-89d8-b009d42a2eac-lib-modules\") pod \"perf-node-gather-daemonset-9plb8\" (UID: \"4de8dcca-677d-42e6-89d8-b009d42a2eac\") " pod="openshift-must-gather-ttn7v/perf-node-gather-daemonset-9plb8"
Jul 29 20:14:51.900505 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:51.900481    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w92hl\" (UniqueName: \"kubernetes.io/projected/4de8dcca-677d-42e6-89d8-b009d42a2eac-kube-api-access-w92hl\") pod \"perf-node-gather-daemonset-9plb8\" (UID: \"4de8dcca-677d-42e6-89d8-b009d42a2eac\") " pod="openshift-must-gather-ttn7v/perf-node-gather-daemonset-9plb8"
Jul 29 20:14:51.998964 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:51.998908    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ttn7v/perf-node-gather-daemonset-9plb8"
Jul 29 20:14:52.123135 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:52.123109    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ttn7v/perf-node-gather-daemonset-9plb8"]
Jul 29 20:14:52.125563 ip-10-0-128-101 kubenswrapper[2583]: W0729 20:14:52.125533    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4de8dcca_677d_42e6_89d8_b009d42a2eac.slice/crio-a512056a850fdda60603a6ab8ca090fa8cbd95f9d3eafa14562c274a05b2071e WatchSource:0}: Error finding container a512056a850fdda60603a6ab8ca090fa8cbd95f9d3eafa14562c274a05b2071e: Status 404 returned error can't find the container with id a512056a850fdda60603a6ab8ca090fa8cbd95f9d3eafa14562c274a05b2071e
Jul 29 20:14:52.618149 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:52.618113    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ttn7v/perf-node-gather-daemonset-9plb8" event={"ID":"4de8dcca-677d-42e6-89d8-b009d42a2eac","Type":"ContainerStarted","Data":"0585c03a431a531d2e770f91fb98fb5c7fa1b7de8682820f7f1b2081c381c827"}
Jul 29 20:14:52.618565 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:52.618165    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ttn7v/perf-node-gather-daemonset-9plb8" event={"ID":"4de8dcca-677d-42e6-89d8-b009d42a2eac","Type":"ContainerStarted","Data":"a512056a850fdda60603a6ab8ca090fa8cbd95f9d3eafa14562c274a05b2071e"}
Jul 29 20:14:52.618565 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:52.618203    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ttn7v/perf-node-gather-daemonset-9plb8"
Jul 29 20:14:52.634899 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:52.634858    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ttn7v/perf-node-gather-daemonset-9plb8" podStartSLOduration=1.6348456 podStartE2EDuration="1.6348456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:14:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:14:52.633844432 +0000 UTC m=+3459.862233560" watchObservedRunningTime="2026-07-29 20:14:52.6348456 +0000 UTC m=+3459.863234723"
Jul 29 20:14:54.462896 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:54.462857    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r9gkq_93412f51-94fe-40a4-b1bb-10f5bb4ee2a9/dns/0.log"
Jul 29 20:14:54.490987 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:54.490957    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r9gkq_93412f51-94fe-40a4-b1bb-10f5bb4ee2a9/kube-rbac-proxy/0.log"
Jul 29 20:14:54.560562 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:54.560537    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mlk48_558f3f9a-7e78-4810-ba73-42063864201e/dns-node-resolver/0.log"
Jul 29 20:14:55.149695 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:55.149666    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-98vz6_4f2d56b2-16f1-460a-b498-1bad91e77dd1/node-ca/0.log"
Jul 29 20:14:56.234711 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:56.234674    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-srr62_6d180d8b-da5e-4cfb-a409-eef0f14ebe9b/discovery/0.log"
Jul 29 20:14:56.283438 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:56.283413    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b6ffddf6-tcktk_54036b22-87b4-4104-bd40-56da3fb1d01f/kube-auth-proxy/0.log"
Jul 29 20:14:57.221722 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:57.221690    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hr2f6_38b8311f-6cea-4246-ac0a-619be43452e0/serve-healthcheck-canary/0.log"
Jul 29 20:14:57.815573 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:57.815543    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nzzqf_7bafa333-3be5-44ae-b6a2-255fd658a919/kube-rbac-proxy/0.log"
Jul 29 20:14:57.835673 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:57.835644    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nzzqf_7bafa333-3be5-44ae-b6a2-255fd658a919/exporter/0.log"
Jul 29 20:14:57.860216 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:57.860189    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nzzqf_7bafa333-3be5-44ae-b6a2-255fd658a919/extractor/0.log"
Jul 29 20:14:58.631726 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:14:58.631699    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ttn7v/perf-node-gather-daemonset-9plb8"
Jul 29 20:15:00.141332 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:15:00.141298    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755935-nwrws"]
Jul 29 20:15:00.144293 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:15:00.144272    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-q94h6_7ec5dddd-2a18-46ce-bbd3-ff5f0c447915/manager/0.log"
Jul 29 20:15:00.144395 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:15:00.144357    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755935-nwrws"
Jul 29 20:15:00.146502 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:15:00.146488    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hq9l2\""
Jul 29 20:15:00.152040 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:15:00.152020    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755935-nwrws"]
Jul 29 20:15:00.155019 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:15:00.155000    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755935-nwrws"
Jul 29 20:15:00.317187 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:15:00.317094    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755935-nwrws"]
Jul 29 20:15:00.319463 ip-10-0-128-101 kubenswrapper[2583]: W0729 20:15:00.319433    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod894aa4d3_45d2_41f3_91fa_860b98f5095d.slice/crio-1957d7884f6e55f64e829c00b31903f1b1b497b2b16f7056987d51b11a048941 WatchSource:0}: Error finding container 1957d7884f6e55f64e829c00b31903f1b1b497b2b16f7056987d51b11a048941: Status 404 returned error can't find the container with id 1957d7884f6e55f64e829c00b31903f1b1b497b2b16f7056987d51b11a048941
Jul 29 20:15:00.403919 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:15:00.403819    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-t9vcv_538180db-c08e-4b16-b2b8-ded900ea01e1/manager/0.log"
Jul 29 20:15:00.601480 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:15:00.601457    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-dbbkd_3915492a-6290-43a6-b7a0-3643b5085183/server/0.log"
Jul 29 20:15:00.657373 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:15:00.657270    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755935-nwrws" event={"ID":"894aa4d3-45d2-41f3-91fa-860b98f5095d","Type":"ContainerStarted","Data":"9d7a50f651a3987d5fef2b82f89c2654e2bd042e27a0a5ec27b1524d9af8e5b7"}
Jul 29 20:15:00.657553 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:15:00.657386    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755935-nwrws" event={"ID":"894aa4d3-45d2-41f3-91fa-860b98f5095d","Type":"ContainerStarted","Data":"1957d7884f6e55f64e829c00b31903f1b1b497b2b16f7056987d51b11a048941"}
Jul 29 20:15:00.672375 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:15:00.672346    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-gqj8n_9a18c2d9-4c11-42de-bdea-16ff6e458049/manager/0.log"
Jul 29 20:15:00.673762 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:15:00.673712    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755935-nwrws" podStartSLOduration=0.673696905 podStartE2EDuration="673.696905ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:15:00.672840654 +0000 UTC m=+3467.901229774" watchObservedRunningTime="2026-07-29 20:15:00.673696905 +0000 UTC m=+3467.902086032"
Jul 29 20:15:00.898140 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:15:00.898110    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-667b7bc75f-g6jzm_cae6c80b-633e-43ba-b0e9-0045ebd3fe37/manager/0.log"
Jul 29 20:15:00.920993 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:15:00.920919    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-667b7bc75f-g6jzm_cae6c80b-633e-43ba-b0e9-0045ebd3fe37/copy-manifests/0.log"
Jul 29 20:15:01.663352 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:15:01.663317    2583 generic.go:356] "Generic (PLEG): container finished" podID="894aa4d3-45d2-41f3-91fa-860b98f5095d" containerID="9d7a50f651a3987d5fef2b82f89c2654e2bd042e27a0a5ec27b1524d9af8e5b7" exitCode=0
Jul 29 20:15:01.663352 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:15:01.663354    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755935-nwrws" event={"ID":"894aa4d3-45d2-41f3-91fa-860b98f5095d","Type":"ContainerDied","Data":"9d7a50f651a3987d5fef2b82f89c2654e2bd042e27a0a5ec27b1524d9af8e5b7"}
Jul 29 20:15:02.320249 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:15:02.320213    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-hlznp_426278c0-ce22-43df-8830-06789715ae93/openshift-lws-operator/0.log"
Jul 29 20:15:02.417984 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:15:02.417945    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-srr62_6d180d8b-da5e-4cfb-a409-eef0f14ebe9b/discovery/0.log"
Jul 29 20:15:02.446006 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:15:02.445972    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b6ffddf6-tcktk_54036b22-87b4-4104-bd40-56da3fb1d01f/kube-auth-proxy/0.log"
Jul 29 20:15:02.805463 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:15:02.805438    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755935-nwrws"
Jul 29 20:15:03.197001 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:15:03.196968    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755890-2wzkf"]
Jul 29 20:15:03.199193 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:15:03.199165    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755890-2wzkf"]
Jul 29 20:15:03.340643 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:15:03.340562    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3979261-f3fb-47da-ba69-d7ea080f6f48" path="/var/lib/kubelet/pods/c3979261-f3fb-47da-ba69-d7ea080f6f48/volumes"
Jul 29 20:15:03.676766 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:15:03.676743    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755935-nwrws"
Jul 29 20:15:03.676766 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:15:03.676758    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755935-nwrws" event={"ID":"894aa4d3-45d2-41f3-91fa-860b98f5095d","Type":"ContainerDied","Data":"1957d7884f6e55f64e829c00b31903f1b1b497b2b16f7056987d51b11a048941"}
Jul 29 20:15:03.676992 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:15:03.676823    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1957d7884f6e55f64e829c00b31903f1b1b497b2b16f7056987d51b11a048941"
Jul 29 20:15:04.078824 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:15:04.078721    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-srr62_6d180d8b-da5e-4cfb-a409-eef0f14ebe9b/discovery/0.log"
Jul 29 20:15:04.106664 ip-10-0-128-101 kubenswrapper[2583]: I0729 20:15:04.106604    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b6ffddf6-tcktk_54036b22-87b4-4104-bd40-56da3fb1d01f/kube-auth-proxy/0.log"