Jul 29 18:32:04.134057 ip-10-0-138-152 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 29 18:32:04.134217 ip-10-0-138-152 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 29 18:32:04.134237 ip-10-0-138-152 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 29 18:32:04.134521 ip-10-0-138-152 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 29 18:32:13.715677 ip-10-0-138-152 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found.
Jul 29 18:32:13.715709 ip-10-0-138-152 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 2e5eebcd0ba94d7490463871276dd390 --
Jul 29 18:33:54.057895 ip-10-0-138-152 systemd[1]: Starting Kubernetes Kubelet...
Jul 29 18:33:54.377225 ip-10-0-138-152 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 18:33:54.377225 ip-10-0-138-152 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 29 18:33:54.377225 ip-10-0-138-152 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 18:33:54.377225 ip-10-0-138-152 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 29 18:33:54.377225 ip-10-0-138-152 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 18:33:54.378841 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.378692    2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 29 18:33:54.382758 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382728    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 18:33:54.382758 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382757    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 18:33:54.382758 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382761    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 18:33:54.382758 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382765    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 18:33:54.382758 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382768    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 18:33:54.382758 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382771    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 18:33:54.382938 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382774    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 18:33:54.382938 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382776    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 18:33:54.382938 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382780    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 18:33:54.382938 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382783    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 18:33:54.382938 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382786    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 18:33:54.382938 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382788    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 18:33:54.382938 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382791    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 18:33:54.382938 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382794    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 18:33:54.382938 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382797    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 18:33:54.382938 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382799    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 18:33:54.382938 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382801    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 18:33:54.382938 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382810    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 18:33:54.382938 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382813    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 18:33:54.382938 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382816    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 18:33:54.382938 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382818    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 18:33:54.382938 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382821    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 18:33:54.382938 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382823    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 18:33:54.382938 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382826    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 18:33:54.382938 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382828    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 18:33:54.382938 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382831    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 18:33:54.383410 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382833    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 18:33:54.383410 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382836    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 18:33:54.383410 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382838    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 18:33:54.383410 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382842    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 18:33:54.383410 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382846    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 18:33:54.383410 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382849    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 18:33:54.383410 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382852    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 18:33:54.383410 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382854    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 18:33:54.383410 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382857    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 18:33:54.383410 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382860    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 18:33:54.383410 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382862    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 18:33:54.383410 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382866    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 18:33:54.383410 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382868    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 18:33:54.383410 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382871    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 18:33:54.383410 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382874    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 18:33:54.383410 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382877    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 18:33:54.383410 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382880    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 18:33:54.383410 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382882    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 18:33:54.383410 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382885    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 18:33:54.383928 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382887    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 18:33:54.383928 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382891    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 18:33:54.383928 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382894    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 18:33:54.383928 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382897    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 18:33:54.383928 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382899    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 18:33:54.383928 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382902    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 18:33:54.383928 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382904    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 18:33:54.383928 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382906    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 18:33:54.383928 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382909    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 18:33:54.383928 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382911    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 18:33:54.383928 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382914    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 18:33:54.383928 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382916    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 18:33:54.383928 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382918    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 18:33:54.383928 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382921    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 18:33:54.383928 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382924    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 18:33:54.383928 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382926    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 18:33:54.383928 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382929    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 18:33:54.383928 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382932    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 18:33:54.383928 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382934    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 18:33:54.384371 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382937    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 18:33:54.384371 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382940    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 18:33:54.384371 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382942    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 18:33:54.384371 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382945    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 18:33:54.384371 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382948    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 18:33:54.384371 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382950    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 18:33:54.384371 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382953    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 18:33:54.384371 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382956    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 18:33:54.384371 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382959    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 18:33:54.384371 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382961    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 18:33:54.384371 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382966    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 18:33:54.384371 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382970    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 18:33:54.384371 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382974    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 18:33:54.384371 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382976    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 18:33:54.384371 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382979    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 18:33:54.384371 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382982    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 18:33:54.384371 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382985    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 18:33:54.384371 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382988    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 18:33:54.384371 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382992    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 18:33:54.384371 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382994    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 18:33:54.384946 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382997    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 18:33:54.384946 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.382999    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 18:33:54.384946 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.383009    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 18:33:54.384946 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.383013    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 18:33:54.384946 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.383016    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 18:33:54.384946 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.383018    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 18:33:54.384946 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.383021    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 18:33:54.384946 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.383023    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 18:33:54.384946 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.383026    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 18:33:54.384946 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.383030    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 18:33:54.384946 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.383034    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 18:33:54.384946 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.383037    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 18:33:54.384946 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.383040    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 18:33:54.384946 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.383044    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 18:33:54.384946 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.383046    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 18:33:54.384946 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384430    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 18:33:54.384946 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384438    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 18:33:54.384946 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384441    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 18:33:54.384946 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384444    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 18:33:54.385392 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384447    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 18:33:54.385392 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384450    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 18:33:54.385392 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384453    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 18:33:54.385392 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384456    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 18:33:54.385392 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384458    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 18:33:54.385392 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384461    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 18:33:54.385392 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384463    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 18:33:54.385392 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384466    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 18:33:54.385392 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384468    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 18:33:54.385392 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384471    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 18:33:54.385392 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384475    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 18:33:54.385392 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384477    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 18:33:54.385392 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384480    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 18:33:54.385392 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384483    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 18:33:54.385392 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384486    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 18:33:54.385392 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384488    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 18:33:54.385392 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384491    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 18:33:54.385392 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384494    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 18:33:54.385392 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384496    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 18:33:54.385879 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384499    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 18:33:54.385879 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384502    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 18:33:54.385879 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384506    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 18:33:54.385879 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384510    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 18:33:54.385879 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384514    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 18:33:54.385879 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384517    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 18:33:54.385879 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384520    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 18:33:54.385879 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384523    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 18:33:54.385879 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384526    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 18:33:54.385879 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384529    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 18:33:54.385879 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384532    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 18:33:54.385879 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384535    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 18:33:54.385879 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384537    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 18:33:54.385879 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384540    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 18:33:54.385879 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384558    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 18:33:54.385879 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384561    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 18:33:54.385879 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384563    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 18:33:54.385879 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384566    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 18:33:54.385879 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384570    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 18:33:54.385879 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384572    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 18:33:54.386394 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384575    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 18:33:54.386394 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384579    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 18:33:54.386394 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384583    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 18:33:54.386394 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384586    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 18:33:54.386394 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384589    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 18:33:54.386394 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384593    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 18:33:54.386394 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384595    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 18:33:54.386394 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384598    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 18:33:54.386394 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384600    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 18:33:54.386394 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384604    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 18:33:54.386394 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384606    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 18:33:54.386394 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384609    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 18:33:54.386394 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384611    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 18:33:54.386394 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384614    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 18:33:54.386394 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384617    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 18:33:54.386394 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384620    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 18:33:54.386394 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384623    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 18:33:54.386394 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384626    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 18:33:54.386394 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384628    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 18:33:54.386884 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384631    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 18:33:54.386884 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384633    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 18:33:54.386884 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384636    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 18:33:54.386884 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384639    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 18:33:54.386884 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384643    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 18:33:54.386884 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384646    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 18:33:54.386884 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384649    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 18:33:54.386884 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384652    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 18:33:54.386884 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384654    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 18:33:54.386884 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384657    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 18:33:54.386884 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384660    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 18:33:54.386884 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384662    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 18:33:54.386884 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384665    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 18:33:54.386884 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384668    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 18:33:54.386884 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384670    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 18:33:54.386884 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384673    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 18:33:54.386884 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384676    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 18:33:54.386884 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384678    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 18:33:54.386884 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384681    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 18:33:54.386884 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384684    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 18:33:54.387393 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384687    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 18:33:54.387393 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384690    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 18:33:54.387393 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384692    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 18:33:54.387393 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384695    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 18:33:54.387393 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384698    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 18:33:54.387393 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384701    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 18:33:54.387393 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384703    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 18:33:54.387393 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384709    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 18:33:54.387393 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384712    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 18:33:54.387393 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384715    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 18:33:54.387393 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384717    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 18:33:54.387393 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384720    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 18:33:54.387393 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384722    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 18:33:54.387393 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384725    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 18:33:54.387393 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384727    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 18:33:54.387393 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384730    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 18:33:54.387393 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.384732    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 18:33:54.387393 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385116    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 18:33:54.387393 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385122    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 18:33:54.387393 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385125    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 18:33:54.387917 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385127    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 18:33:54.387917 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385130    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 18:33:54.387917 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385132    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 18:33:54.387917 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385135    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 18:33:54.387917 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385137    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 18:33:54.387917 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385140    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 18:33:54.387917 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385142    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 18:33:54.387917 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385145    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 18:33:54.387917 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385147    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 18:33:54.387917 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385150    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 18:33:54.387917 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385153    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 18:33:54.387917 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385156    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 18:33:54.387917 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385158    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 18:33:54.387917 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385161    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 18:33:54.387917 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385163    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 18:33:54.387917 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385166    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 18:33:54.387917 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385168    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 18:33:54.387917 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385170    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 18:33:54.387917 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385173    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 18:33:54.387917 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385177    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 18:33:54.388410 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385180    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 18:33:54.388410 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385183    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 18:33:54.388410 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385186    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 18:33:54.388410 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385188    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 18:33:54.388410 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385191    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 18:33:54.388410 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385195    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 18:33:54.388410 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385197    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 18:33:54.388410 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385200    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 18:33:54.388410 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385202    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 18:33:54.388410 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385204    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 18:33:54.388410 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385207    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 18:33:54.388410 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385209    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 18:33:54.388410 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385212    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 18:33:54.388410 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385231    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 18:33:54.388410 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385234    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 18:33:54.388410 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385237    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 18:33:54.388410 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385240    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 18:33:54.388410 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385243    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 18:33:54.388410 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385245    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 18:33:54.388410 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385247    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 18:33:54.388920 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385250    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 18:33:54.388920 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385253    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 18:33:54.388920 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385256    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 18:33:54.388920 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385259    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 18:33:54.388920 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385261    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 18:33:54.388920 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385264    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 18:33:54.388920 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385266    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 18:33:54.388920 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385269    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 18:33:54.388920 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385271    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 18:33:54.388920 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385274    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 18:33:54.388920 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385276    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 18:33:54.388920 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385280    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 18:33:54.388920 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385282    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 18:33:54.388920 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385285    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 18:33:54.388920 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385287    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 18:33:54.388920 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385290    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 18:33:54.388920 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385292    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 18:33:54.388920 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385295    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 18:33:54.388920 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385298    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 18:33:54.389380 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385303    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 18:33:54.389380 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385306    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 18:33:54.389380 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385309    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 18:33:54.389380 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385312    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 18:33:54.389380 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385314    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 18:33:54.389380 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385317    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 18:33:54.389380 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385319    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 18:33:54.389380 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385322    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 18:33:54.389380 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385324    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 18:33:54.389380 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385327    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 18:33:54.389380 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385329    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 18:33:54.389380 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385331    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 18:33:54.389380 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385334    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 18:33:54.389380 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385337    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 18:33:54.389380 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385339    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 18:33:54.389380 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385343    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 18:33:54.389380 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385346    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 18:33:54.389380 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385349    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 18:33:54.389380 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385352    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 18:33:54.389854 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385354    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 18:33:54.389854 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385357    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 18:33:54.389854 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385360    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 18:33:54.389854 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385363    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 18:33:54.389854 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385365    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 18:33:54.389854 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385370    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 18:33:54.389854 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385373    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 18:33:54.389854 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385375    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 18:33:54.389854 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385378    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 18:33:54.389854 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385380    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 18:33:54.389854 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385383    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 18:33:54.389854 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385385    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 18:33:54.389854 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385387    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 18:33:54.389854 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385390    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 18:33:54.389854 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385393    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 18:33:54.389854 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385396    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 18:33:54.389854 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385399    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 18:33:54.389854 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.385402    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 18:33:54.389854 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385471    2559 flags.go:64] FLAG: --address="0.0.0.0"
Jul 29 18:33:54.389854 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385478    2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 29 18:33:54.390335 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385485    2559 flags.go:64] FLAG: --anonymous-auth="true"
Jul 29 18:33:54.390335 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385490    2559 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 29 18:33:54.390335 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385496    2559 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 29 18:33:54.390335 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385501    2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 29 18:33:54.390335 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385507    2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 29 18:33:54.390335 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385514    2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 29 18:33:54.390335 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385518    2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 29 18:33:54.390335 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385523    2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 29 18:33:54.390335 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385528    2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 29 18:33:54.390335 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385532    2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 29 18:33:54.390335 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385535    2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 29 18:33:54.390335 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385539    2559 flags.go:64] FLAG: --cgroup-root=""
Jul 29 18:33:54.390335 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385555    2559 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 29 18:33:54.390335 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385558    2559 flags.go:64] FLAG: --client-ca-file=""
Jul 29 18:33:54.390335 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385561    2559 flags.go:64] FLAG: --cloud-provider="external"
Jul 29 18:33:54.390335 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385564    2559 flags.go:64] FLAG: --cluster-dns="[]"
Jul 29 18:33:54.390335 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385569    2559 flags.go:64] FLAG: --cluster-domain=""
Jul 29 18:33:54.390335 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385573    2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 29 18:33:54.390335 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385576    2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 29 18:33:54.390335 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385579    2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 29 18:33:54.390335 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385583    2559 flags.go:64] FLAG: --container-log-max-files="5"
Jul 29 18:33:54.390335 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385587    2559 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 29 18:33:54.390335 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385590    2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 29 18:33:54.390887 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385593    2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 29 18:33:54.390887 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385596    2559 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 29 18:33:54.390887 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385599    2559 flags.go:64] FLAG: --contention-profiling="false"
Jul 29 18:33:54.390887 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385602    2559 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 29 18:33:54.390887 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385604    2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 29 18:33:54.390887 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385608    2559 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 29 18:33:54.390887 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385610    2559 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 29 18:33:54.390887 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385614    2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 29 18:33:54.390887 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385617    2559 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 29 18:33:54.390887 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385620    2559 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 29 18:33:54.390887 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385623    2559 flags.go:64] FLAG: --enable-load-reader="false"
Jul 29 18:33:54.390887 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385626    2559 flags.go:64] FLAG: --enable-server="true"
Jul 29 18:33:54.390887 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385629    2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 29 18:33:54.390887 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385634    2559 flags.go:64] FLAG: --event-burst="100"
Jul 29 18:33:54.390887 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385637    2559 flags.go:64] FLAG: --event-qps="50"
Jul 29 18:33:54.390887 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385639    2559 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 29 18:33:54.390887 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385642    2559 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 29 18:33:54.390887 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385645    2559 flags.go:64] FLAG: --eviction-hard=""
Jul 29 18:33:54.390887 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385649    2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 29 18:33:54.390887 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385652    2559 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 29 18:33:54.390887 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385655    2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 29 18:33:54.390887 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385658    2559 flags.go:64] FLAG: --eviction-soft=""
Jul 29 18:33:54.390887 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385662    2559 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 29 18:33:54.390887 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385664    2559 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 29 18:33:54.390887 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385669    2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 29 18:33:54.391559 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385672    2559 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 29 18:33:54.391559 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385676    2559 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 29 18:33:54.391559 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385679    2559 flags.go:64] FLAG: --fail-swap-on="true"
Jul 29 18:33:54.391559 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385682    2559 flags.go:64] FLAG: --feature-gates=""
Jul 29 18:33:54.391559 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385686    2559 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 29 18:33:54.391559 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385689    2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 29 18:33:54.391559 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385692    2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 29 18:33:54.391559 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385695    2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 29 18:33:54.391559 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385698    2559 flags.go:64] FLAG: --healthz-port="10248"
Jul 29 18:33:54.391559 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385701    2559 flags.go:64] FLAG: --help="false"
Jul 29 18:33:54.391559 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385704    2559 flags.go:64] FLAG: --hostname-override="ip-10-0-138-152.ec2.internal"
Jul 29 18:33:54.391559 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385707    2559 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 29 18:33:54.391559 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385711    2559 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 29 18:33:54.391559 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385714    2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 29 18:33:54.391559 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385717    2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 29 18:33:54.391559 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385721    2559 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 29 18:33:54.391559 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385723    2559 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 29 18:33:54.391559 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385726    2559 flags.go:64] FLAG: --image-service-endpoint=""
Jul 29 18:33:54.391559 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385729    2559 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 29 18:33:54.391559 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385732    2559 flags.go:64] FLAG: --kube-api-burst="100"
Jul 29 18:33:54.391559 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385734    2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 29 18:33:54.391559 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385737    2559 flags.go:64] FLAG: --kube-api-qps="50"
Jul 29 18:33:54.391559 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385740    2559 flags.go:64] FLAG: --kube-reserved=""
Jul 29 18:33:54.391559 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385743    2559 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 29 18:33:54.392185 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385746    2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 29 18:33:54.392185 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385749    2559 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 29 18:33:54.392185 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385751    2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 29 18:33:54.392185 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385754    2559 flags.go:64] FLAG: --lock-file=""
Jul 29 18:33:54.392185 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.385757    2559 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 29 18:33:54.392185 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386127    2559 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 29 18:33:54.392185 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386131    2559 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 29 18:33:54.392185 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386138    2559 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 29 18:33:54.392185 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386141    2559 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 29 18:33:54.392185 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386145    2559 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 29 18:33:54.392185 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386148    2559 flags.go:64] FLAG: --logging-format="text"
Jul 29 18:33:54.392185 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386150    2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 29 18:33:54.392185 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386154    2559 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 29 18:33:54.392185 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386156    2559 flags.go:64] FLAG: --manifest-url=""
Jul 29 18:33:54.392185 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386159    2559 flags.go:64] FLAG: --manifest-url-header=""
Jul 29 18:33:54.392185 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386163    2559 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 29 18:33:54.392185 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386166    2559 flags.go:64] FLAG: --max-open-files="1000000"
Jul 29 18:33:54.392185 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386170    2559 flags.go:64] FLAG: --max-pods="110"
Jul 29 18:33:54.392185 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386173    2559 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 29 18:33:54.392185 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386176    2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 29 18:33:54.392185 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386179    2559 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 29 18:33:54.392185 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386181    2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 29 18:33:54.392185 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386184    2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 29 18:33:54.392185 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386187    2559 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 29 18:33:54.392185 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386190    2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 29 18:33:54.392962 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386198    2559 flags.go:64] FLAG: --node-status-max-images="50"
Jul 29 18:33:54.392962 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386201    2559 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 29 18:33:54.392962 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386204    2559 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 29 18:33:54.392962 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386207    2559 flags.go:64] FLAG: --pod-cidr=""
Jul 29 18:33:54.392962 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386209    2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 29 18:33:54.392962 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386215    2559 flags.go:64] FLAG: --pod-manifest-path=""
Jul 29 18:33:54.392962 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386218    2559 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 29 18:33:54.392962 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386221    2559 flags.go:64] FLAG: --pods-per-core="0"
Jul 29 18:33:54.392962 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386224    2559 flags.go:64] FLAG: --port="10250"
Jul 29 18:33:54.392962 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386227    2559 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 29 18:33:54.392962 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386230    2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-088b2f0fa6451da4b"
Jul 29 18:33:54.392962 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386233    2559 flags.go:64] FLAG: --qos-reserved=""
Jul 29 18:33:54.392962 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386236    2559 flags.go:64] FLAG: --read-only-port="10255"
Jul 29 18:33:54.392962 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386239    2559 flags.go:64] FLAG: --register-node="true"
Jul 29 18:33:54.392962 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386243    2559 flags.go:64] FLAG: --register-with-taints=""
Jul 29 18:33:54.392962 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386247    2559 flags.go:64] FLAG: --registry-burst="10"
Jul 29 18:33:54.392962 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386249    2559 flags.go:64] FLAG: --registry-qps="5"
Jul 29 18:33:54.392962 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386252    2559 flags.go:64] FLAG: --reserved-cpus=""
Jul 29 18:33:54.392962 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386255    2559 flags.go:64] FLAG: --reserved-memory=""
Jul 29 18:33:54.392962 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386259    2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 29 18:33:54.392962 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386262    2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 29 18:33:54.392962 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386265    2559 flags.go:64] FLAG: --rotate-certificates="false"
Jul 29 18:33:54.392962 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386268    2559 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 29 18:33:54.392962 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386270    2559 flags.go:64] FLAG: --runonce="false"
Jul 29 18:33:54.392962 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386273    2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 29 18:33:54.393863 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386276    2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 29 18:33:54.393863 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386279    2559 flags.go:64] FLAG: --seccomp-default="false"
Jul 29 18:33:54.393863 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386282    2559 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 29 18:33:54.393863 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386285    2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 29 18:33:54.393863 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386287    2559 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 29 18:33:54.393863 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386290    2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 29 18:33:54.393863 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386293    2559 flags.go:64] FLAG: --storage-driver-password="root"
Jul 29 18:33:54.393863 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386296    2559 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 29 18:33:54.393863 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386300    2559 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 29 18:33:54.393863 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386302    2559 flags.go:64] FLAG: --storage-driver-user="root"
Jul 29 18:33:54.393863 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386305    2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 29 18:33:54.393863 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386308    2559 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 29 18:33:54.393863 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386311    2559 flags.go:64] FLAG: --system-cgroups=""
Jul 29 18:33:54.393863 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386313    2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 29 18:33:54.393863 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386319    2559 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 29 18:33:54.393863 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386322    2559 flags.go:64] FLAG: --tls-cert-file=""
Jul 29 18:33:54.393863 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386325    2559 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 29 18:33:54.393863 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386329    2559 flags.go:64] FLAG: --tls-min-version=""
Jul 29 18:33:54.393863 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386331    2559 flags.go:64] FLAG: --tls-private-key-file=""
Jul 29 18:33:54.393863 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386334    2559 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 29 18:33:54.393863 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386337    2559 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 29 18:33:54.393863 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386341    2559 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 29 18:33:54.393863 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386344    2559 flags.go:64] FLAG: --v="2"
Jul 29 18:33:54.393863 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386348    2559 flags.go:64] FLAG: --version="false"
Jul 29 18:33:54.393863 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386352    2559 flags.go:64] FLAG: --vmodule=""
Jul 29 18:33:54.394464 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386356    2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 29 18:33:54.394464 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386359    2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 29 18:33:54.394464 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386479    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 18:33:54.394464 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386483    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 18:33:54.394464 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386486    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 18:33:54.394464 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386489    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 18:33:54.394464 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386492    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 18:33:54.394464 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386494    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 18:33:54.394464 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386497    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 18:33:54.394464 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386499    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 18:33:54.394464 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386502    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 18:33:54.394464 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386504    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 18:33:54.394464 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386507    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 18:33:54.394464 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386509    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 18:33:54.394464 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386511    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 18:33:54.394464 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386514    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 18:33:54.394464 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386516    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 18:33:54.394464 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386519    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 18:33:54.394464 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386521    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 18:33:54.394464 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386524    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 18:33:54.394980 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386526    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 18:33:54.394980 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386529    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 18:33:54.394980 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386531    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 18:33:54.394980 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386534    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 18:33:54.394980 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386538    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 18:33:54.394980 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386541    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 18:33:54.394980 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386558    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 18:33:54.394980 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386562    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 18:33:54.394980 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386565    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 18:33:54.394980 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386569    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 18:33:54.394980 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386572    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 18:33:54.394980 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386576    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 18:33:54.394980 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386580    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 18:33:54.394980 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386583    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 18:33:54.394980 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386586    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 18:33:54.394980 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386588    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 18:33:54.394980 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386591    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 18:33:54.394980 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386594    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 18:33:54.394980 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386596    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 18:33:54.395447 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386599    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 18:33:54.395447 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386601    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 18:33:54.395447 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386604    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 18:33:54.395447 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386606    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 18:33:54.395447 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386609    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 18:33:54.395447 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386611    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 18:33:54.395447 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386614    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 18:33:54.395447 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386616    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 18:33:54.395447 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386618    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 18:33:54.395447 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386621    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 18:33:54.395447 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386624    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 18:33:54.395447 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386626    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 18:33:54.395447 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386630    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 18:33:54.395447 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386633    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 18:33:54.395447 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386638    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 18:33:54.395447 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386641    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 18:33:54.395447 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386644    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 18:33:54.395447 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386646    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 18:33:54.395447 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386649    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 18:33:54.395447 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386652    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 18:33:54.396004 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386655    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 18:33:54.396004 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386657    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 18:33:54.396004 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386660    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 18:33:54.396004 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386663    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 18:33:54.396004 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386666    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 18:33:54.396004 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386669    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 18:33:54.396004 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386672    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 18:33:54.396004 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386675    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 18:33:54.396004 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386677    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 18:33:54.396004 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386680    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 18:33:54.396004 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386682    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 18:33:54.396004 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386685    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 18:33:54.396004 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386687    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 18:33:54.396004 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386690    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 18:33:54.396004 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386692    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 18:33:54.396004 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386695    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 18:33:54.396004 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386697    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 18:33:54.396004 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386699    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 18:33:54.396004 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386702    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 18:33:54.396004 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386704    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 18:33:54.396496 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386706    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 18:33:54.396496 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386709    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 18:33:54.396496 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386711    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 18:33:54.396496 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386714    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 18:33:54.396496 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386716    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 18:33:54.396496 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386719    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 18:33:54.396496 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386723    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 18:33:54.396496 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386725    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 18:33:54.396496 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386728    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 18:33:54.396496 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386730    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 18:33:54.396496 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386733    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 18:33:54.396496 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386736    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 18:33:54.396496 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386738    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 18:33:54.396496 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386741    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 18:33:54.396496 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386743    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 18:33:54.396496 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386746    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 18:33:54.396496 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386749    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 18:33:54.396496 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386751    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 18:33:54.396496 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386754    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 18:33:54.396496 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386757    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 18:33:54.397012 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386759    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 18:33:54.397012 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.386762    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 18:33:54.397012 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.386767    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 18:33:54.397012 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.394897    2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 29 18:33:54.397012 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.394922    2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 29 18:33:54.397012 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.394942    2559 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 29 18:33:54.397012 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.394948    2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 29 18:33:54.397012 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395018    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 18:33:54.397012 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395023    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 18:33:54.397012 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395026    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 18:33:54.397012 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395029    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 18:33:54.397012 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395032    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 18:33:54.397012 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395035    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 18:33:54.397012 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395038    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 18:33:54.397012 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395040    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 18:33:54.397398 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395043    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 18:33:54.397398 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395046    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 18:33:54.397398 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395048    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 18:33:54.397398 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395051    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 18:33:54.397398 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395054    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 18:33:54.397398 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395056    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 18:33:54.397398 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395059    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 18:33:54.397398 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395062    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 18:33:54.397398 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395064    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 18:33:54.397398 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395067    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 18:33:54.397398 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395070    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 18:33:54.397398 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395073    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 18:33:54.397398 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395077    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 18:33:54.397398 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395081    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 18:33:54.397398 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395084    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 18:33:54.397398 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395087    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 18:33:54.397398 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395090    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 18:33:54.397398 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395093    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 18:33:54.397398 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395095    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 18:33:54.397969 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395098    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 18:33:54.397969 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395102    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 18:33:54.397969 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395106    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 18:33:54.397969 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395109    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 18:33:54.397969 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395112    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 18:33:54.397969 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395114    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 18:33:54.397969 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395117    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 18:33:54.397969 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395120    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 18:33:54.397969 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395122    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 18:33:54.397969 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395125    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 18:33:54.397969 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395127    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 18:33:54.397969 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395130    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 18:33:54.397969 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395132    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 18:33:54.397969 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395135    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 18:33:54.397969 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395137    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 18:33:54.397969 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395140    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 18:33:54.397969 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395142    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 18:33:54.397969 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395145    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 18:33:54.397969 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395147    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 18:33:54.397969 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395150    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 18:33:54.398476 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395153    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 18:33:54.398476 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395155    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 18:33:54.398476 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395158    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 18:33:54.398476 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395160    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 18:33:54.398476 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395163    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 18:33:54.398476 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395165    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 18:33:54.398476 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395168    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 18:33:54.398476 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395170    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 18:33:54.398476 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395172    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 18:33:54.398476 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395175    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 18:33:54.398476 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395178    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 18:33:54.398476 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395181    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 18:33:54.398476 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395184    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 18:33:54.398476 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395188    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 18:33:54.398476 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395192    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 18:33:54.398476 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395195    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 18:33:54.398476 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395197    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 18:33:54.398476 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395200    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 18:33:54.398476 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395202    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 18:33:54.398967 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395205    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 18:33:54.398967 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395208    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 18:33:54.398967 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395210    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 18:33:54.398967 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395213    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 18:33:54.398967 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395215    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 18:33:54.398967 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395217    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 18:33:54.398967 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395220    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 18:33:54.398967 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395223    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 18:33:54.398967 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395226    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 18:33:54.398967 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395228    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 18:33:54.398967 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395231    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 18:33:54.398967 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395233    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 18:33:54.398967 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395236    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 18:33:54.398967 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395239    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 18:33:54.398967 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395242    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 18:33:54.398967 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395244    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 18:33:54.398967 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395247    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 18:33:54.398967 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395249    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 18:33:54.398967 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395252    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 18:33:54.398967 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395255    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 18:33:54.399520 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395257    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 18:33:54.399520 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395259    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 18:33:54.399520 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395262    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 18:33:54.399520 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395264    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 18:33:54.399520 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395267    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 18:33:54.399520 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395269    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 18:33:54.399520 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395274    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 18:33:54.399520 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395278    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 18:33:54.399520 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395282    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 18:33:54.399520 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395285    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 18:33:54.399520 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395287    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 18:33:54.399520 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395290    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 18:33:54.399520 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395293    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 18:33:54.399520 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.395298    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 18:33:54.399887 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395418    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 18:33:54.399887 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395423    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 18:33:54.399887 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395426    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 18:33:54.399887 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395429    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 18:33:54.399887 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395431    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 18:33:54.399887 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395434    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 18:33:54.399887 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395437    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 18:33:54.399887 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395439    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 18:33:54.399887 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395442    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 18:33:54.399887 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395444    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 18:33:54.399887 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395447    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 18:33:54.399887 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395449    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 18:33:54.399887 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395453    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 18:33:54.399887 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395457    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 18:33:54.399887 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395461    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 18:33:54.399887 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395465    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 18:33:54.399887 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395467    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 18:33:54.399887 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395470    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 18:33:54.399887 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395472    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 18:33:54.399887 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395475    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 18:33:54.400389 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395478    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 18:33:54.400389 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395480    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 18:33:54.400389 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395482    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 18:33:54.400389 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395485    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 18:33:54.400389 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395488    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 18:33:54.400389 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395491    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 18:33:54.400389 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395494    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 18:33:54.400389 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395497    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 18:33:54.400389 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395500    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 18:33:54.400389 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395502    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 18:33:54.400389 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395505    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 18:33:54.400389 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395507    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 18:33:54.400389 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395510    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 18:33:54.400389 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395512    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 18:33:54.400389 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395515    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 18:33:54.400389 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395519    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 18:33:54.400389 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395522    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 18:33:54.400389 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395526    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 18:33:54.400872 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395529    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 18:33:54.400872 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395532    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 18:33:54.400872 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395534    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 18:33:54.400872 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395537    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 18:33:54.400872 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395540    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 18:33:54.400872 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395556    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 18:33:54.400872 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395560    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 18:33:54.400872 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395566    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 18:33:54.400872 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395569    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 18:33:54.400872 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395572    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 18:33:54.400872 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395574    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 18:33:54.400872 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395577    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 18:33:54.400872 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395579    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 18:33:54.400872 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395582    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 18:33:54.400872 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395585    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 18:33:54.400872 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395587    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 18:33:54.400872 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395590    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 18:33:54.400872 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395592    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 18:33:54.400872 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395595    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 18:33:54.400872 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395598    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 18:33:54.401347 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395600    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 18:33:54.401347 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395603    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 18:33:54.401347 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395606    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 18:33:54.401347 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395608    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 18:33:54.401347 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395611    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 18:33:54.401347 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395613    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 18:33:54.401347 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395616    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 18:33:54.401347 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395618    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 18:33:54.401347 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395621    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 18:33:54.401347 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395623    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 18:33:54.401347 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395626    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 18:33:54.401347 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395628    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 18:33:54.401347 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395631    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 18:33:54.401347 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395633    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 18:33:54.401347 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395635    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 18:33:54.401347 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395638    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 18:33:54.401347 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395640    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 18:33:54.401347 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395642    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 18:33:54.401347 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395645    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 18:33:54.401347 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395647    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 18:33:54.401858 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395650    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 18:33:54.401858 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395652    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 18:33:54.401858 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395655    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 18:33:54.401858 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395657    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 18:33:54.401858 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395660    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 18:33:54.401858 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395662    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 18:33:54.401858 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395664    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 18:33:54.401858 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395667    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 18:33:54.401858 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395669    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 18:33:54.401858 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395671    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 18:33:54.401858 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395675    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 18:33:54.401858 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395678    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 18:33:54.401858 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395680    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 18:33:54.401858 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395683    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 18:33:54.401858 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395685    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 18:33:54.401858 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395688    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 18:33:54.401858 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395690    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 18:33:54.401858 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395692    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 18:33:54.401858 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395695    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 18:33:54.401858 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395697    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 18:33:54.402378 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:54.395700    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 18:33:54.402378 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.395704    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 18:33:54.402378 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.395829    2559 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 29 18:33:54.402378 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.398465    2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 29 18:33:54.402378 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.399659    2559 server.go:1019] "Starting client certificate rotation"
Jul 29 18:33:54.402378 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.399767    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 18:33:54.402378 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.399811    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 18:33:54.402378 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.401869    2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 18:33:54.403416 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.403396    2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 18:33:54.413032 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.413009    2559 log.go:25] "Validated CRI v1 runtime API"
Jul 29 18:33:54.415111 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.415088    2559 log.go:25] "Validated CRI v1 image API"
Jul 29 18:33:54.416080 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.416060    2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 29 18:33:54.418266 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.418245    2559 fs.go:135] Filesystem UUIDs: map[4a99f94c-7ee4-4513-9e5b-6b38520e4b19:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 7c4c6b59-2d2d-4834-bc27-f18b353729f7:/dev/nvme0n1p4]
Jul 29 18:33:54.418341 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.418264    2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 29 18:33:54.423873 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.423260    2559 manager.go:216] Machine: {Timestamp:2026-07-29 18:33:54.422742495 +0000 UTC m=+0.266343760 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3112664 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21c6d62ad97f580d8d0f90574a56cf SystemUUID:ec21c6d6-2ad9-7f58-0d8d-0f90574a56cf BootID:2e5eebcd-0ba9-4d74-9046-3871276dd390 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:70:14:47:af:2f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:70:14:47:af:2f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0a:23:2d:24:a8:ed Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 29 18:33:54.423873 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.423857    2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 29 18:33:54.424061 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.424003    2559 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 29 18:33:54.427254 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.427051    2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 29 18:33:54.427401 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.427258    2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-152.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 29 18:33:54.428118 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.428097    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 18:33:54.428384 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.428372    2559 topology_manager.go:138] "Creating topology manager with none policy"
Jul 29 18:33:54.428414 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.428387    2559 container_manager_linux.go:309] "Creating device plugin manager"
Jul 29 18:33:54.428414 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.428395    2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 18:33:54.429010 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.429000    2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 18:33:54.429816 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.429806    2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 29 18:33:54.433018 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.433004    2559 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 18:33:54.433177 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.433167    2559 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 29 18:33:54.433210 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.433204    2559 kubelet.go:486] "Attempting to sync node with API server"
Jul 29 18:33:54.433240 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.433218    2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 29 18:33:54.433240 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.433234    2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 29 18:33:54.433294 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.433244    2559 kubelet.go:398] "Adding apiserver pod source"
Jul 29 18:33:54.433294 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.433253    2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 29 18:33:54.433663 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.433643    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 18:33:54.433663 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.433668    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 18:33:54.435616 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.435597    2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 29 18:33:54.437463 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.437443    2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 29 18:33:54.437585 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.437483    2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 29 18:33:54.438945 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.438931    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 29 18:33:54.439030 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.438951    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 29 18:33:54.439030 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.438960    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 29 18:33:54.439030 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.438968    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 29 18:33:54.439030 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.438980    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 29 18:33:54.439030 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.438989    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 29 18:33:54.439030 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.439001    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 29 18:33:54.439030 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.439019    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 29 18:33:54.439030 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.439031    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 29 18:33:54.439301 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.439040    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 29 18:33:54.439301 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.439054    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 29 18:33:54.439301 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.439062    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 29 18:33:54.439645 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.439634    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 29 18:33:54.439694 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.439650    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 29 18:33:54.443027 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.443000    2559 server.go:1268] "Started kubelet"
Jul 29 18:33:54.443126 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.443089    2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 29 18:33:54.443203 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.443147    2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 29 18:33:54.443318 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.443302    2559 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 29 18:33:54.443918 ip-10-0-138-152 systemd[1]: Started Kubernetes Kubelet.
Jul 29 18:33:54.444070 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:54.443980    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 29 18:33:54.444070 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:54.444019    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-152.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 29 18:33:54.444304 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.444283    2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-152.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 29 18:33:54.445425 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.445411    2559 server.go:310] "Adding debug handlers to kubelet server"
Jul 29 18:33:54.445425 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.445421    2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 29 18:33:54.449943 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:54.449922    2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 29 18:33:54.451694 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:54.450952    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-152.ec2.internal.18c6d665aab45d31  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-152.ec2.internal,UID:ip-10-0-138-152.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-152.ec2.internal,},FirstTimestamp:2026-07-29 18:33:54.442968369 +0000 UTC m=+0.286569634,LastTimestamp:2026-07-29 18:33:54.442968369 +0000 UTC m=+0.286569634,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-152.ec2.internal,}"
Jul 29 18:33:54.453901 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.453882    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 29 18:33:54.453957 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.453902    2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 29 18:33:54.454141 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:54.454036    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-152.ec2.internal\" not found"
Jul 29 18:33:54.454774 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.454588    2559 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 29 18:33:54.454774 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.454608    2559 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 29 18:33:54.454774 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.454709    2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 29 18:33:54.454774 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.454727    2559 reconstruct.go:97] "Volume reconstruction finished"
Jul 29 18:33:54.454774 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.454741    2559 reconciler.go:29] "Reconciler: start to sync state"
Jul 29 18:33:54.454990 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.454871    2559 factory.go:55] Registering systemd factory
Jul 29 18:33:54.454990 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.454884    2559 factory.go:223] Registration of the systemd container factory successfully
Jul 29 18:33:54.455154 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.455136    2559 factory.go:153] Registering CRI-O factory
Jul 29 18:33:54.455154 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.455156    2559 factory.go:223] Registration of the crio container factory successfully
Jul 29 18:33:54.455279 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.455206    2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 29 18:33:54.455279 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.455230    2559 factory.go:103] Registering Raw factory
Jul 29 18:33:54.455279 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.455243    2559 manager.go:1196] Started watching for new ooms in manager
Jul 29 18:33:54.456274 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:54.456171    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 29 18:33:54.456338 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:54.456275    2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-152.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 29 18:33:54.457048 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.456560    2559 manager.go:349] Starting recovery of all containers
Jul 29 18:33:54.468367 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.468349    2559 manager.go:354] Recovery completed
Jul 29 18:33:54.472870 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.472851    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 18:33:54.474833 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.474806    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-152.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 18:33:54.474919 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.474852    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-152.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 18:33:54.474919 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.474868    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-152.ec2.internal" event="NodeHasSufficientPID"
Jul 29 18:33:54.475467 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.475450    2559 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 29 18:33:54.475467 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.475464    2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 29 18:33:54.475598 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.475467    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 18:33:54.475598 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.475481    2559 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 18:33:54.476131 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.476118    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-152.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 18:33:54.476185 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.476144    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-152.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 18:33:54.476185 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.476153    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-152.ec2.internal" event="NodeHasSufficientPID"
Jul 29 18:33:54.478325 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.478313    2559 policy_none.go:49] "None policy: Start"
Jul 29 18:33:54.478363 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.478329    2559 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 29 18:33:54.478363 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.478340    2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 29 18:33:54.480420 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.480409    2559 policy_none.go:47] "Start"
Jul 29 18:33:54.485043 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.485023    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tm7hv"
Jul 29 18:33:54.486053 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:54.485968    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-152.ec2.internal.18c6d665ac9a982d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-152.ec2.internal,UID:ip-10-0-138-152.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-152.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-152.ec2.internal,},FirstTimestamp:2026-07-29 18:33:54.474833965 +0000 UTC m=+0.318435235,LastTimestamp:2026-07-29 18:33:54.474833965 +0000 UTC m=+0.318435235,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-152.ec2.internal,}"
Jul 29 18:33:54.490472 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.490453    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tm7hv"
Jul 29 18:33:54.496249 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:54.496121    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-152.ec2.internal.18c6d665ac9afac3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-152.ec2.internal,UID:ip-10-0-138-152.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-152.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-152.ec2.internal,},FirstTimestamp:2026-07-29 18:33:54.474859203 +0000 UTC m=+0.318460469,LastTimestamp:2026-07-29 18:33:54.474859203 +0000 UTC m=+0.318460469,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-152.ec2.internal,}"
Jul 29 18:33:54.520285 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.520265    2559 manager.go:337] "Starting Device Plugin manager"
Jul 29 18:33:54.520453 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:54.520303    2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 29 18:33:54.520453 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.520317    2559 server.go:87] "Starting device plugin registration server"
Jul 29 18:33:54.520650 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.520636    2559 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 29 18:33:54.520691 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.520653    2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 29 18:33:54.520759 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.520742    2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 29 18:33:54.520870 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.520828    2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 29 18:33:54.520870 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.520837    2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 29 18:33:54.521319 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:54.521281    2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 29 18:33:54.521423 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:54.521333    2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-152.ec2.internal\" not found"
Jul 29 18:33:54.568468 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.568423    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 29 18:33:54.570211 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.569802    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 29 18:33:54.570211 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.569838    2559 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 29 18:33:54.570211 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.569864    2559 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 29 18:33:54.570211 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:54.569907    2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 29 18:33:54.572335 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.572318    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 18:33:54.621342 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.621309    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 18:33:54.622302 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.622284    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-152.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 18:33:54.622403 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.622320    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-152.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 18:33:54.622403 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.622334    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-152.ec2.internal" event="NodeHasSufficientPID"
Jul 29 18:33:54.622403 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.622365    2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-152.ec2.internal"
Jul 29 18:33:54.632278 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.632205    2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-152.ec2.internal"
Jul 29 18:33:54.632278 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:54.632235    2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-152.ec2.internal\": node \"ip-10-0-138-152.ec2.internal\" not found"
Jul 29 18:33:54.660337 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:54.660306    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-152.ec2.internal\" not found"
Jul 29 18:33:54.670851 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.670815    2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-152.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-152.ec2.internal"]
Jul 29 18:33:54.671020 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.670923    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 18:33:54.672848 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.672825    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-152.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 18:33:54.672949 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.672868    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-152.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 18:33:54.672949 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.672881    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-152.ec2.internal" event="NodeHasSufficientPID"
Jul 29 18:33:54.675307 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.675292    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 18:33:54.675463 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.675447    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-152.ec2.internal"
Jul 29 18:33:54.675509 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.675500    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 18:33:54.676134 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.676119    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-152.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 18:33:54.676219 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.676130    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-152.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 18:33:54.676219 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.676145    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-152.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 18:33:54.676219 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.676156    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-152.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 18:33:54.676219 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.676170    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-152.ec2.internal" event="NodeHasSufficientPID"
Jul 29 18:33:54.676334 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.676157    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-152.ec2.internal" event="NodeHasSufficientPID"
Jul 29 18:33:54.678680 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.678661    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-152.ec2.internal"
Jul 29 18:33:54.678782 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.678689    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 18:33:54.679460 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.679442    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-152.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 18:33:54.679535 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.679478    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-152.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 18:33:54.679535 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.679493    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-152.ec2.internal" event="NodeHasSufficientPID"
Jul 29 18:33:54.702470 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:54.702435    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-152.ec2.internal\" not found" node="ip-10-0-138-152.ec2.internal"
Jul 29 18:33:54.707141 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:54.707119    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-152.ec2.internal\" not found" node="ip-10-0-138-152.ec2.internal"
Jul 29 18:33:54.756057 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.756013    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2c2c229256a0f446f156df43183000bc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-152.ec2.internal\" (UID: \"2c2c229256a0f446f156df43183000bc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-152.ec2.internal"
Jul 29 18:33:54.756057 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.756055    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c2c229256a0f446f156df43183000bc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-152.ec2.internal\" (UID: \"2c2c229256a0f446f156df43183000bc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-152.ec2.internal"
Jul 29 18:33:54.756245 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.756084    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8923e83396bb7ee7f5fbbb29dfb80487-config\") pod \"kube-apiserver-proxy-ip-10-0-138-152.ec2.internal\" (UID: \"8923e83396bb7ee7f5fbbb29dfb80487\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-152.ec2.internal"
Jul 29 18:33:54.761113 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:54.761090    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-152.ec2.internal\" not found"
Jul 29 18:33:54.856701 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.856669    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2c2c229256a0f446f156df43183000bc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-152.ec2.internal\" (UID: \"2c2c229256a0f446f156df43183000bc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-152.ec2.internal"
Jul 29 18:33:54.856768 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.856714    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c2c229256a0f446f156df43183000bc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-152.ec2.internal\" (UID: \"2c2c229256a0f446f156df43183000bc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-152.ec2.internal"
Jul 29 18:33:54.856768 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.856731    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8923e83396bb7ee7f5fbbb29dfb80487-config\") pod \"kube-apiserver-proxy-ip-10-0-138-152.ec2.internal\" (UID: \"8923e83396bb7ee7f5fbbb29dfb80487\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-152.ec2.internal"
Jul 29 18:33:54.856833 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.856778    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8923e83396bb7ee7f5fbbb29dfb80487-config\") pod \"kube-apiserver-proxy-ip-10-0-138-152.ec2.internal\" (UID: \"8923e83396bb7ee7f5fbbb29dfb80487\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-152.ec2.internal"
Jul 29 18:33:54.856833 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.856778    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2c2c229256a0f446f156df43183000bc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-152.ec2.internal\" (UID: \"2c2c229256a0f446f156df43183000bc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-152.ec2.internal"
Jul 29 18:33:54.856833 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:54.856764    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c2c229256a0f446f156df43183000bc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-152.ec2.internal\" (UID: \"2c2c229256a0f446f156df43183000bc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-152.ec2.internal"
Jul 29 18:33:54.861764 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:54.861741    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-152.ec2.internal\" not found"
Jul 29 18:33:54.962579 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:54.962483    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-152.ec2.internal\" not found"
Jul 29 18:33:55.005705 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.005678    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-152.ec2.internal"
Jul 29 18:33:55.009874 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.009847    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-152.ec2.internal"
Jul 29 18:33:55.062777 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:55.062739    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-152.ec2.internal\" not found"
Jul 29 18:33:55.163325 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:55.163294    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-152.ec2.internal\" not found"
Jul 29 18:33:55.263927 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:55.263893    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-152.ec2.internal\" not found"
Jul 29 18:33:55.339537 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.339508    2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 18:33:55.354391 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.354358    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-152.ec2.internal"
Jul 29 18:33:55.363035 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.363010    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 18:33:55.364622 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.364604    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-152.ec2.internal"
Jul 29 18:33:55.380262 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.380237    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 18:33:55.399409 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.399374    2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 29 18:33:55.399631 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.399600    2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 29 18:33:55.399702 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.399609    2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 29 18:33:55.433716 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.433678    2559 apiserver.go:52] "Watching apiserver"
Jul 29 18:33:55.452525 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.452498    2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 29 18:33:55.454480 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.453665    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-jjdrx","kube-system/global-pull-secret-syncer-f5w95","kube-system/konnectivity-agent-bp28x","kube-system/kube-apiserver-proxy-ip-10-0-138-152.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd95n","openshift-dns/node-resolver-t4wjl","openshift-image-registry/node-ca-ldlwt","openshift-multus/multus-6vd4x","openshift-network-diagnostics/network-check-target-jh2dv","openshift-cluster-node-tuning-operator/tuned-qjztn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-152.ec2.internal","openshift-multus/multus-additional-cni-plugins-4wgbf","openshift-multus/network-metrics-daemon-9zxr5","openshift-network-operator/iptables-alerter-b6rhb"]
Jul 29 18:33:55.454480 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.454174    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 29 18:33:55.456479 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.456459    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.458678 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.458656    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5w95"
Jul 29 18:33:55.458780 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:55.458744    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5w95" podUID="faa3a79b-75fb-4890-b169-09674f6d00db"
Jul 29 18:33:55.459231 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.459207    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 29 18:33:55.459388 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.459363    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 29 18:33:55.459463 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.459367    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 29 18:33:55.459514 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.459368    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 29 18:33:55.459740 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.459719    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-host-run-netns\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.459796 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.459760    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-run-systemd\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.459834 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.459808    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-run-openvswitch\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.459878 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.459838    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-host-cni-netd\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.459921 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.459878    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-var-lib-openvswitch\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.459921 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.459916    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-etc-openvswitch\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.460044 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.459970    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-node-log\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.460044 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.460005    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-log-socket\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.460044 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.460037    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-host-cni-bin\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.460181 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.460038    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 29 18:33:55.460181 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.460064    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.460181 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.460076    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zhk5w\""
Jul 29 18:33:55.460181 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.460090    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-ovn-node-metrics-cert\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.460181 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.460039    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 29 18:33:55.460181 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.460119    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-host-kubelet\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.460181 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.460145    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-systemd-units\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.460181 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.460168    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-run-ovn\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.460571 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.460209    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-ovnkube-config\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.460571 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.460226    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-ovnkube-script-lib\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.460571 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.460242    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-host-slash\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.460571 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.460265    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-host-run-ovn-kubernetes\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.460571 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.460286    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-env-overrides\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.460571 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.460308    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrd55\" (UniqueName: \"kubernetes.io/projected/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-kube-api-access-xrd55\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.460882 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.460857    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bp28x"
Jul 29 18:33:55.463053 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.463033    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd95n"
Jul 29 18:33:55.463609 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.463582    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 29 18:33:55.463781 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.463764    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 29 18:33:55.463867 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.463787    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8blk5\""
Jul 29 18:33:55.465274 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.465251    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t4wjl"
Jul 29 18:33:55.465506 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.465347    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-czv7p\""
Jul 29 18:33:55.465684 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.465661    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 29 18:33:55.466028 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.466009    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 29 18:33:55.466028 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.466011    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 29 18:33:55.467400 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.467381    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 29 18:33:55.467659 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.467642    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 29 18:33:55.467791 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.467773    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qbddp\""
Jul 29 18:33:55.467937 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.467920    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ldlwt"
Jul 29 18:33:55.468583 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.468559    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 18:33:55.470228 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.470108    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 29 18:33:55.470323 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.470303    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 29 18:33:55.470425 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.470408    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:55.470509 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.470488    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 29 18:33:55.470914 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.470539    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-l4fsz\""
Jul 29 18:33:55.475040 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.472883    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 29 18:33:55.475040 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.472996    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-cvhs5\""
Jul 29 18:33:55.475040 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.473016    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh2dv"
Jul 29 18:33:55.475040 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:55.473089    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh2dv" podUID="df3fd46a-27e2-4557-bc40-09a9fbaa2e8f"
Jul 29 18:33:55.475040 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.473473    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 29 18:33:55.475040 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.473592    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 29 18:33:55.475040 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.473624    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 29 18:33:55.476142 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.475951    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qjztn"
Jul 29 18:33:55.478259 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.478234    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 29 18:33:55.478379 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.478274    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wgzfr\""
Jul 29 18:33:55.478379 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.478301    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 29 18:33:55.478568 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.478523    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4wgbf"
Jul 29 18:33:55.480916 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.480899    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 29 18:33:55.480916 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.480914    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zxr5"
Jul 29 18:33:55.481074 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.480956    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 29 18:33:55.481074 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:55.480991    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zxr5" podUID="4b86e30e-def7-4bf4-90da-de4fef47f433"
Jul 29 18:33:55.481312 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.481295    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tvhvm\""
Jul 29 18:33:55.483228 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.483212    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-b6rhb"
Jul 29 18:33:55.485333 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.485312    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 29 18:33:55.485418 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.485401    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 29 18:33:55.485601 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.485573    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 29 18:33:55.485722 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.485657    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ckvns\""
Jul 29 18:33:55.496059 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.496014    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 18:28:56 +0000 UTC" deadline="2028-04-06 00:44:44.788266112 +0000 UTC"
Jul 29 18:33:55.496059 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.496054    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14790h10m49.292214544s"
Jul 29 18:33:55.515414 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.515361    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-h4p68"
Jul 29 18:33:55.525382 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.525342    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-h4p68"
Jul 29 18:33:55.555948 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.555931    2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 29 18:33:55.560953 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.560929    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30397e41-d823-4e74-9485-66f74e907f8b-etc-kubernetes\") pod \"tuned-qjztn\" (UID: \"30397e41-d823-4e74-9485-66f74e907f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-qjztn"
Jul 29 18:33:55.561074 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.560963    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6dad0cba-079a-4ec1-9f2b-8399f34fb80c-os-release\") pod \"multus-additional-cni-plugins-4wgbf\" (UID: \"6dad0cba-079a-4ec1-9f2b-8399f34fb80c\") " pod="openshift-multus/multus-additional-cni-plugins-4wgbf"
Jul 29 18:33:55.561074 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.560996    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-run-systemd\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.561074 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.561021    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/faa3a79b-75fb-4890-b169-09674f6d00db-kubelet-config\") pod \"global-pull-secret-syncer-f5w95\" (UID: \"faa3a79b-75fb-4890-b169-09674f6d00db\") " pod="kube-system/global-pull-secret-syncer-f5w95"
Jul 29 18:33:55.561074 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.561048    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2201b8bf-575c-4958-b909-29ff288ed75e-host\") pod \"node-ca-ldlwt\" (UID: \"2201b8bf-575c-4958-b909-29ff288ed75e\") " pod="openshift-image-registry/node-ca-ldlwt"
Jul 29 18:33:55.561225 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.561073    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/37ec2c6c-1216-4d35-8d37-95071fddf90a-multus-daemon-config\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:55.561225 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.561082    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-run-systemd\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.561225 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.561110    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b03a3682-95be-4852-b28a-8e19fee4cbe7-tmp-dir\") pod \"node-resolver-t4wjl\" (UID: \"b03a3682-95be-4852-b28a-8e19fee4cbe7\") " pod="openshift-dns/node-resolver-t4wjl"
Jul 29 18:33:55.561225 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.561134    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b86e30e-def7-4bf4-90da-de4fef47f433-metrics-certs\") pod \"network-metrics-daemon-9zxr5\" (UID: \"4b86e30e-def7-4bf4-90da-de4fef47f433\") " pod="openshift-multus/network-metrics-daemon-9zxr5"
Jul 29 18:33:55.561225 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.561160    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4h9f\" (UniqueName: \"kubernetes.io/projected/2201b8bf-575c-4958-b909-29ff288ed75e-kube-api-access-v4h9f\") pod \"node-ca-ldlwt\" (UID: \"2201b8bf-575c-4958-b909-29ff288ed75e\") " pod="openshift-image-registry/node-ca-ldlwt"
Jul 29 18:33:55.561225 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.561196    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37ec2c6c-1216-4d35-8d37-95071fddf90a-cnibin\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:55.561225 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.561211    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37ec2c6c-1216-4d35-8d37-95071fddf90a-multus-conf-dir\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:55.561465 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.561227    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/30397e41-d823-4e74-9485-66f74e907f8b-etc-tuned\") pod \"tuned-qjztn\" (UID: \"30397e41-d823-4e74-9485-66f74e907f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-qjztn"
Jul 29 18:33:55.561465 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.561285    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6dad0cba-079a-4ec1-9f2b-8399f34fb80c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4wgbf\" (UID: \"6dad0cba-079a-4ec1-9f2b-8399f34fb80c\") " pod="openshift-multus/multus-additional-cni-plugins-4wgbf"
Jul 29 18:33:55.561465 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.561324    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6dad0cba-079a-4ec1-9f2b-8399f34fb80c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4wgbf\" (UID: \"6dad0cba-079a-4ec1-9f2b-8399f34fb80c\") " pod="openshift-multus/multus-additional-cni-plugins-4wgbf"
Jul 29 18:33:55.561465 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.561364    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8fa7a55e-1003-4cc1-918e-655ccd91f071-etc-selinux\") pod \"aws-ebs-csi-driver-node-gd95n\" (UID: \"8fa7a55e-1003-4cc1-918e-655ccd91f071\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd95n"
Jul 29 18:33:55.561465 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.561392    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6m89\" (UniqueName: \"kubernetes.io/projected/37ec2c6c-1216-4d35-8d37-95071fddf90a-kube-api-access-t6m89\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:55.561465 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.561418    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/30397e41-d823-4e74-9485-66f74e907f8b-run\") pod \"tuned-qjztn\" (UID: \"30397e41-d823-4e74-9485-66f74e907f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-qjztn"
Jul 29 18:33:55.561465 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.561459    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-host-slash\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.561811 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.561483    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8fa7a55e-1003-4cc1-918e-655ccd91f071-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gd95n\" (UID: \"8fa7a55e-1003-4cc1-918e-655ccd91f071\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd95n"
Jul 29 18:33:55.561811 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.561509    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztfbt\" (UniqueName: \"kubernetes.io/projected/df3fd46a-27e2-4557-bc40-09a9fbaa2e8f-kube-api-access-ztfbt\") pod \"network-check-target-jh2dv\" (UID: \"df3fd46a-27e2-4557-bc40-09a9fbaa2e8f\") " pod="openshift-network-diagnostics/network-check-target-jh2dv"
Jul 29 18:33:55.561811 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.561536    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-host-slash\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.561811 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.561644    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6dad0cba-079a-4ec1-9f2b-8399f34fb80c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4wgbf\" (UID: \"6dad0cba-079a-4ec1-9f2b-8399f34fb80c\") " pod="openshift-multus/multus-additional-cni-plugins-4wgbf"
Jul 29 18:33:55.561811 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.561679    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-run-openvswitch\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.561811 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.561735    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/37ec2c6c-1216-4d35-8d37-95071fddf90a-multus-socket-dir-parent\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:55.561811 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.561789    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-run-openvswitch\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.562307 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.562286    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37ec2c6c-1216-4d35-8d37-95071fddf90a-etc-kubernetes\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:55.562348 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.562324    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30397e41-d823-4e74-9485-66f74e907f8b-tmp\") pod \"tuned-qjztn\" (UID: \"30397e41-d823-4e74-9485-66f74e907f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-qjztn"
Jul 29 18:33:55.562376 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.562351    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-544jm\" (UniqueName: \"kubernetes.io/projected/6dad0cba-079a-4ec1-9f2b-8399f34fb80c-kube-api-access-544jm\") pod \"multus-additional-cni-plugins-4wgbf\" (UID: \"6dad0cba-079a-4ec1-9f2b-8399f34fb80c\") " pod="openshift-multus/multus-additional-cni-plugins-4wgbf"
Jul 29 18:33:55.562408 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.562377    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/44c368d9-c4c2-481e-bbca-3bfa6eae44d1-iptables-alerter-script\") pod \"iptables-alerter-b6rhb\" (UID: \"44c368d9-c4c2-481e-bbca-3bfa6eae44d1\") " pod="openshift-network-operator/iptables-alerter-b6rhb"
Jul 29 18:33:55.562445 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.562408    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-etc-openvswitch\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.562445 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.562433    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-log-socket\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.562518 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.562455    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-etc-openvswitch\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.562518 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.562457    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz64d\" (UniqueName: \"kubernetes.io/projected/8fa7a55e-1003-4cc1-918e-655ccd91f071-kube-api-access-qz64d\") pod \"aws-ebs-csi-driver-node-gd95n\" (UID: \"8fa7a55e-1003-4cc1-918e-655ccd91f071\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd95n"
Jul 29 18:33:55.562518 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.562500    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/30397e41-d823-4e74-9485-66f74e907f8b-etc-sysctl-conf\") pod \"tuned-qjztn\" (UID: \"30397e41-d823-4e74-9485-66f74e907f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-qjztn"
Jul 29 18:33:55.562648 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.562515    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-log-socket\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.562648 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.562525    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30397e41-d823-4e74-9485-66f74e907f8b-lib-modules\") pod \"tuned-qjztn\" (UID: \"30397e41-d823-4e74-9485-66f74e907f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-qjztn"
Jul 29 18:33:55.562648 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.562572    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6a62d29d-5bf6-444f-bd28-bf2d1e54ac87-konnectivity-ca\") pod \"konnectivity-agent-bp28x\" (UID: \"6a62d29d-5bf6-444f-bd28-bf2d1e54ac87\") " pod="kube-system/konnectivity-agent-bp28x"
Jul 29 18:33:55.562648 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.562605    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-ovnkube-script-lib\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.562648 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.562640    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/30397e41-d823-4e74-9485-66f74e907f8b-etc-sysctl-d\") pod \"tuned-qjztn\" (UID: \"30397e41-d823-4e74-9485-66f74e907f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-qjztn"
Jul 29 18:33:55.562875 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.562746    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b03a3682-95be-4852-b28a-8e19fee4cbe7-hosts-file\") pod \"node-resolver-t4wjl\" (UID: \"b03a3682-95be-4852-b28a-8e19fee4cbe7\") " pod="openshift-dns/node-resolver-t4wjl"
Jul 29 18:33:55.562875 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.562771    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m4ld\" (UniqueName: \"kubernetes.io/projected/b03a3682-95be-4852-b28a-8e19fee4cbe7-kube-api-access-6m4ld\") pod \"node-resolver-t4wjl\" (UID: \"b03a3682-95be-4852-b28a-8e19fee4cbe7\") " pod="openshift-dns/node-resolver-t4wjl"
Jul 29 18:33:55.562875 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.562791    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6dad0cba-079a-4ec1-9f2b-8399f34fb80c-cni-binary-copy\") pod \"multus-additional-cni-plugins-4wgbf\" (UID: \"6dad0cba-079a-4ec1-9f2b-8399f34fb80c\") " pod="openshift-multus/multus-additional-cni-plugins-4wgbf"
Jul 29 18:33:55.562875 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.562816    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8fa7a55e-1003-4cc1-918e-655ccd91f071-socket-dir\") pod \"aws-ebs-csi-driver-node-gd95n\" (UID: \"8fa7a55e-1003-4cc1-918e-655ccd91f071\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd95n"
Jul 29 18:33:55.562875 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.562834    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8fa7a55e-1003-4cc1-918e-655ccd91f071-device-dir\") pod \"aws-ebs-csi-driver-node-gd95n\" (UID: \"8fa7a55e-1003-4cc1-918e-655ccd91f071\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd95n"
Jul 29 18:33:55.562875 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.562850    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2201b8bf-575c-4958-b909-29ff288ed75e-serviceca\") pod \"node-ca-ldlwt\" (UID: \"2201b8bf-575c-4958-b909-29ff288ed75e\") " pod="openshift-image-registry/node-ca-ldlwt"
Jul 29 18:33:55.562875 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.562864    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/37ec2c6c-1216-4d35-8d37-95071fddf90a-host-run-k8s-cni-cncf-io\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:55.563289 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.562902    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lglkr\" (UniqueName: \"kubernetes.io/projected/30397e41-d823-4e74-9485-66f74e907f8b-kube-api-access-lglkr\") pod \"tuned-qjztn\" (UID: \"30397e41-d823-4e74-9485-66f74e907f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-qjztn"
Jul 29 18:33:55.563289 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.562928    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6a62d29d-5bf6-444f-bd28-bf2d1e54ac87-agent-certs\") pod \"konnectivity-agent-bp28x\" (UID: \"6a62d29d-5bf6-444f-bd28-bf2d1e54ac87\") " pod="kube-system/konnectivity-agent-bp28x"
Jul 29 18:33:55.563289 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.562955    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/44c368d9-c4c2-481e-bbca-3bfa6eae44d1-host-slash\") pod \"iptables-alerter-b6rhb\" (UID: \"44c368d9-c4c2-481e-bbca-3bfa6eae44d1\") " pod="openshift-network-operator/iptables-alerter-b6rhb"
Jul 29 18:33:55.563289 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.562980    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-host-run-netns\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.563289 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563012    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-host-cni-netd\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.563289 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563024    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-host-run-netns\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.563289 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563039    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6dad0cba-079a-4ec1-9f2b-8399f34fb80c-cnibin\") pod \"multus-additional-cni-plugins-4wgbf\" (UID: \"6dad0cba-079a-4ec1-9f2b-8399f34fb80c\") " pod="openshift-multus/multus-additional-cni-plugins-4wgbf"
Jul 29 18:33:55.563289 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563060    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-node-log\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.563289 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563064    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-host-cni-netd\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.563289 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563087    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/faa3a79b-75fb-4890-b169-09674f6d00db-dbus\") pod \"global-pull-secret-syncer-f5w95\" (UID: \"faa3a79b-75fb-4890-b169-09674f6d00db\") " pod="kube-system/global-pull-secret-syncer-f5w95"
Jul 29 18:33:55.563289 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563104    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-node-log\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.563289 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563108    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37ec2c6c-1216-4d35-8d37-95071fddf90a-multus-cni-dir\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:55.563289 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563135    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37ec2c6c-1216-4d35-8d37-95071fddf90a-host-var-lib-kubelet\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:55.563289 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563167    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/30397e41-d823-4e74-9485-66f74e907f8b-etc-systemd\") pod \"tuned-qjztn\" (UID: \"30397e41-d823-4e74-9485-66f74e907f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-qjztn"
Jul 29 18:33:55.563289 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563191    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30397e41-d823-4e74-9485-66f74e907f8b-sys\") pod \"tuned-qjztn\" (UID: \"30397e41-d823-4e74-9485-66f74e907f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-qjztn"
Jul 29 18:33:55.563289 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563206    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6dad0cba-079a-4ec1-9f2b-8399f34fb80c-system-cni-dir\") pod \"multus-additional-cni-plugins-4wgbf\" (UID: \"6dad0cba-079a-4ec1-9f2b-8399f34fb80c\") " pod="openshift-multus/multus-additional-cni-plugins-4wgbf"
Jul 29 18:33:55.563289 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563216    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-ovnkube-script-lib\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.563956 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563227    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt2qp\" (UniqueName: \"kubernetes.io/projected/44c368d9-c4c2-481e-bbca-3bfa6eae44d1-kube-api-access-bt2qp\") pod \"iptables-alerter-b6rhb\" (UID: \"44c368d9-c4c2-481e-bbca-3bfa6eae44d1\") " pod="openshift-network-operator/iptables-alerter-b6rhb"
Jul 29 18:33:55.563956 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563254    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-systemd-units\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.563956 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563294    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-systemd-units\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.563956 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563292    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/faa3a79b-75fb-4890-b169-09674f6d00db-original-pull-secret\") pod \"global-pull-secret-syncer-f5w95\" (UID: \"faa3a79b-75fb-4890-b169-09674f6d00db\") " pod="kube-system/global-pull-secret-syncer-f5w95"
Jul 29 18:33:55.563956 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563326    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8fa7a55e-1003-4cc1-918e-655ccd91f071-sys-fs\") pod \"aws-ebs-csi-driver-node-gd95n\" (UID: \"8fa7a55e-1003-4cc1-918e-655ccd91f071\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd95n"
Jul 29 18:33:55.563956 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563351    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37ec2c6c-1216-4d35-8d37-95071fddf90a-host-var-lib-cni-bin\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:55.563956 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563366    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/37ec2c6c-1216-4d35-8d37-95071fddf90a-hostroot\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:55.563956 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563378    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/30397e41-d823-4e74-9485-66f74e907f8b-host\") pod \"tuned-qjztn\" (UID: \"30397e41-d823-4e74-9485-66f74e907f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-qjztn"
Jul 29 18:33:55.563956 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563392    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30397e41-d823-4e74-9485-66f74e907f8b-var-lib-kubelet\") pod \"tuned-qjztn\" (UID: \"30397e41-d823-4e74-9485-66f74e907f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-qjztn"
Jul 29 18:33:55.563956 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563408    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/37ec2c6c-1216-4d35-8d37-95071fddf90a-host-var-lib-cni-multus\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:55.563956 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563450    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-var-lib-openvswitch\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.563956 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563472    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-host-cni-bin\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.563956 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563496    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.563956 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563523    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-var-lib-openvswitch\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.563956 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563531    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-host-cni-bin\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.563956 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563525    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-ovn-node-metrics-cert\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.563956 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563584    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.564505 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563584    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37ec2c6c-1216-4d35-8d37-95071fddf90a-system-cni-dir\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:55.564505 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563605    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37ec2c6c-1216-4d35-8d37-95071fddf90a-host-run-netns\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:55.564505 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563638    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/37ec2c6c-1216-4d35-8d37-95071fddf90a-host-run-multus-certs\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:55.564505 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563658    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/30397e41-d823-4e74-9485-66f74e907f8b-etc-modprobe-d\") pod \"tuned-qjztn\" (UID: \"30397e41-d823-4e74-9485-66f74e907f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-qjztn"
Jul 29 18:33:55.564505 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563677    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/30397e41-d823-4e74-9485-66f74e907f8b-etc-sysconfig\") pod \"tuned-qjztn\" (UID: \"30397e41-d823-4e74-9485-66f74e907f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-qjztn"
Jul 29 18:33:55.564505 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563693    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-host-kubelet\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.564505 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563709    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-run-ovn\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.564505 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563732    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-ovnkube-config\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.564505 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563758    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/37ec2c6c-1216-4d35-8d37-95071fddf90a-os-release\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:55.564505 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563771    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-host-kubelet\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.564505 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563781    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn5k9\" (UniqueName: \"kubernetes.io/projected/4b86e30e-def7-4bf4-90da-de4fef47f433-kube-api-access-sn5k9\") pod \"network-metrics-daemon-9zxr5\" (UID: \"4b86e30e-def7-4bf4-90da-de4fef47f433\") " pod="openshift-multus/network-metrics-daemon-9zxr5"
Jul 29 18:33:55.564505 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563781    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-run-ovn\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.564505 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563804    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-host-run-ovn-kubernetes\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.564505 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563845    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-env-overrides\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.564505 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563862    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrd55\" (UniqueName: \"kubernetes.io/projected/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-kube-api-access-xrd55\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.564505 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563877    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8fa7a55e-1003-4cc1-918e-655ccd91f071-registration-dir\") pod \"aws-ebs-csi-driver-node-gd95n\" (UID: \"8fa7a55e-1003-4cc1-918e-655ccd91f071\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd95n"
Jul 29 18:33:55.564505 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563897    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-host-run-ovn-kubernetes\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.564997 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563903    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37ec2c6c-1216-4d35-8d37-95071fddf90a-cni-binary-copy\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:55.564997 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.563883    2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 29 18:33:55.564997 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:55.563876    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c2c229256a0f446f156df43183000bc.slice/crio-5dc8b984b987ea41f5da6af7cfd3bd21d2de6e5bee0ed08d9b33e238b64ecad5 WatchSource:0}: Error finding container 5dc8b984b987ea41f5da6af7cfd3bd21d2de6e5bee0ed08d9b33e238b64ecad5: Status 404 returned error can't find the container with id 5dc8b984b987ea41f5da6af7cfd3bd21d2de6e5bee0ed08d9b33e238b64ecad5
Jul 29 18:33:55.564997 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.564240    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-ovnkube-config\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.564997 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.564341    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-env-overrides\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.567228 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.567206    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-ovn-node-metrics-cert\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.570152 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.570137    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 18:33:55.573857 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.573808    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-152.ec2.internal" event={"ID":"2c2c229256a0f446f156df43183000bc","Type":"ContainerStarted","Data":"5dc8b984b987ea41f5da6af7cfd3bd21d2de6e5bee0ed08d9b33e238b64ecad5"}
Jul 29 18:33:55.575231 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.575196    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrd55\" (UniqueName: \"kubernetes.io/projected/60f444e1-7e31-443c-8ec7-a4d04c64bf4b-kube-api-access-xrd55\") pod \"ovnkube-node-jjdrx\" (UID: \"60f444e1-7e31-443c-8ec7-a4d04c64bf4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:55.664345 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.664298    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/37ec2c6c-1216-4d35-8d37-95071fddf90a-host-var-lib-cni-multus\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:55.664345 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.664340    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37ec2c6c-1216-4d35-8d37-95071fddf90a-system-cni-dir\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:55.664617 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.664356    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37ec2c6c-1216-4d35-8d37-95071fddf90a-host-run-netns\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:55.664617 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.664371    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/37ec2c6c-1216-4d35-8d37-95071fddf90a-host-run-multus-certs\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:55.664617 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.664386    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/30397e41-d823-4e74-9485-66f74e907f8b-etc-modprobe-d\") pod \"tuned-qjztn\" (UID: \"30397e41-d823-4e74-9485-66f74e907f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-qjztn"
Jul 29 18:33:55.664617 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.664413    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/30397e41-d823-4e74-9485-66f74e907f8b-etc-sysconfig\") pod \"tuned-qjztn\" (UID: \"30397e41-d823-4e74-9485-66f74e907f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-qjztn"
Jul 29 18:33:55.664617 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.664425    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37ec2c6c-1216-4d35-8d37-95071fddf90a-host-run-netns\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:55.664617 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.664427    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/37ec2c6c-1216-4d35-8d37-95071fddf90a-host-var-lib-cni-multus\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:55.664617 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.664456    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/37ec2c6c-1216-4d35-8d37-95071fddf90a-os-release\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:55.664617 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.664481    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn5k9\" (UniqueName: \"kubernetes.io/projected/4b86e30e-def7-4bf4-90da-de4fef47f433-kube-api-access-sn5k9\") pod \"network-metrics-daemon-9zxr5\" (UID: \"4b86e30e-def7-4bf4-90da-de4fef47f433\") " pod="openshift-multus/network-metrics-daemon-9zxr5"
Jul 29 18:33:55.664617 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.664497    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/30397e41-d823-4e74-9485-66f74e907f8b-etc-sysconfig\") pod \"tuned-qjztn\" (UID: \"30397e41-d823-4e74-9485-66f74e907f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-qjztn"
Jul 29 18:33:55.664617 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.664499    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8fa7a55e-1003-4cc1-918e-655ccd91f071-registration-dir\") pod \"aws-ebs-csi-driver-node-gd95n\" (UID: \"8fa7a55e-1003-4cc1-918e-655ccd91f071\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd95n"
Jul 29 18:33:55.664617 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.664452    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37ec2c6c-1216-4d35-8d37-95071fddf90a-system-cni-dir\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:55.664617 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.664534    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37ec2c6c-1216-4d35-8d37-95071fddf90a-cni-binary-copy\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:55.664617 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.664562    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8fa7a55e-1003-4cc1-918e-655ccd91f071-registration-dir\") pod \"aws-ebs-csi-driver-node-gd95n\" (UID: \"8fa7a55e-1003-4cc1-918e-655ccd91f071\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd95n"
Jul 29 18:33:55.664617 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.664455    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/37ec2c6c-1216-4d35-8d37-95071fddf90a-host-run-multus-certs\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:55.664617 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.664576    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/30397e41-d823-4e74-9485-66f74e907f8b-etc-modprobe-d\") pod \"tuned-qjztn\" (UID: \"30397e41-d823-4e74-9485-66f74e907f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-qjztn"
Jul 29 18:33:55.664617 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.664580    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30397e41-d823-4e74-9485-66f74e907f8b-etc-kubernetes\") pod \"tuned-qjztn\" (UID: \"30397e41-d823-4e74-9485-66f74e907f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-qjztn"
Jul 29 18:33:55.664617 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.664617    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/37ec2c6c-1216-4d35-8d37-95071fddf90a-os-release\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:57.239627 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.664641    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6dad0cba-079a-4ec1-9f2b-8399f34fb80c-os-release\") pod \"multus-additional-cni-plugins-4wgbf\" (UID: \"6dad0cba-079a-4ec1-9f2b-8399f34fb80c\") " pod="openshift-multus/multus-additional-cni-plugins-4wgbf"
Jul 29 18:33:57.239627 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.664676    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/faa3a79b-75fb-4890-b169-09674f6d00db-kubelet-config\") pod \"global-pull-secret-syncer-f5w95\" (UID: \"faa3a79b-75fb-4890-b169-09674f6d00db\") " pod="kube-system/global-pull-secret-syncer-f5w95"
Jul 29 18:33:57.239627 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.664700    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30397e41-d823-4e74-9485-66f74e907f8b-etc-kubernetes\") pod \"tuned-qjztn\" (UID: \"30397e41-d823-4e74-9485-66f74e907f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-qjztn"
Jul 29 18:33:57.239627 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.664706    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2201b8bf-575c-4958-b909-29ff288ed75e-host\") pod \"node-ca-ldlwt\" (UID: \"2201b8bf-575c-4958-b909-29ff288ed75e\") " pod="openshift-image-registry/node-ca-ldlwt"
Jul 29 18:33:57.239627 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.664730    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6dad0cba-079a-4ec1-9f2b-8399f34fb80c-os-release\") pod \"multus-additional-cni-plugins-4wgbf\" (UID: \"6dad0cba-079a-4ec1-9f2b-8399f34fb80c\") " pod="openshift-multus/multus-additional-cni-plugins-4wgbf"
Jul 29 18:33:57.239627 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.664732    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/37ec2c6c-1216-4d35-8d37-95071fddf90a-multus-daemon-config\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:57.239627 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.664752    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/faa3a79b-75fb-4890-b169-09674f6d00db-kubelet-config\") pod \"global-pull-secret-syncer-f5w95\" (UID: \"faa3a79b-75fb-4890-b169-09674f6d00db\") " pod="kube-system/global-pull-secret-syncer-f5w95"
Jul 29 18:33:57.239627 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.664777    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2201b8bf-575c-4958-b909-29ff288ed75e-host\") pod \"node-ca-ldlwt\" (UID: \"2201b8bf-575c-4958-b909-29ff288ed75e\") " pod="openshift-image-registry/node-ca-ldlwt"
Jul 29 18:33:57.239627 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.664779    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b03a3682-95be-4852-b28a-8e19fee4cbe7-tmp-dir\") pod \"node-resolver-t4wjl\" (UID: \"b03a3682-95be-4852-b28a-8e19fee4cbe7\") " pod="openshift-dns/node-resolver-t4wjl"
Jul 29 18:33:57.239627 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.664810    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b86e30e-def7-4bf4-90da-de4fef47f433-metrics-certs\") pod \"network-metrics-daemon-9zxr5\" (UID: \"4b86e30e-def7-4bf4-90da-de4fef47f433\") " pod="openshift-multus/network-metrics-daemon-9zxr5"
Jul 29 18:33:57.239627 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.664835    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4h9f\" (UniqueName: \"kubernetes.io/projected/2201b8bf-575c-4958-b909-29ff288ed75e-kube-api-access-v4h9f\") pod \"node-ca-ldlwt\" (UID: \"2201b8bf-575c-4958-b909-29ff288ed75e\") " pod="openshift-image-registry/node-ca-ldlwt"
Jul 29 18:33:57.239627 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.664857    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37ec2c6c-1216-4d35-8d37-95071fddf90a-cnibin\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:57.239627 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.664877    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37ec2c6c-1216-4d35-8d37-95071fddf90a-multus-conf-dir\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:57.239627 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.664900    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/30397e41-d823-4e74-9485-66f74e907f8b-etc-tuned\") pod \"tuned-qjztn\" (UID: \"30397e41-d823-4e74-9485-66f74e907f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-qjztn"
Jul 29 18:33:57.239627 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:55.664919    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 18:33:57.239627 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.664952    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37ec2c6c-1216-4d35-8d37-95071fddf90a-multus-conf-dir\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:57.239627 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.664922    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6dad0cba-079a-4ec1-9f2b-8399f34fb80c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4wgbf\" (UID: \"6dad0cba-079a-4ec1-9f2b-8399f34fb80c\") " pod="openshift-multus/multus-additional-cni-plugins-4wgbf"
Jul 29 18:33:57.240821 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:55.665003    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b86e30e-def7-4bf4-90da-de4fef47f433-metrics-certs podName:4b86e30e-def7-4bf4-90da-de4fef47f433 nodeName:}" failed. No retries permitted until 2026-07-29 18:33:56.164981906 +0000 UTC m=+2.008583158 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b86e30e-def7-4bf4-90da-de4fef47f433-metrics-certs") pod "network-metrics-daemon-9zxr5" (UID: "4b86e30e-def7-4bf4-90da-de4fef47f433") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 18:33:57.240821 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.664921    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37ec2c6c-1216-4d35-8d37-95071fddf90a-cnibin\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:57.240821 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.665026    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6dad0cba-079a-4ec1-9f2b-8399f34fb80c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4wgbf\" (UID: \"6dad0cba-079a-4ec1-9f2b-8399f34fb80c\") " pod="openshift-multus/multus-additional-cni-plugins-4wgbf"
Jul 29 18:33:57.240821 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:55.665060    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8fa7a55e-1003-4cc1-918e-655ccd91f071-etc-selinux\") pod \"aws-ebs-csi-driver-node-gd95n\" (UID: \"8fa7a55e-1003-4cc1-918e-655ccd91f071\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd95n"
Jul 29 18:33:57.240821 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.238916    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6m89\" (UniqueName: \"kubernetes.io/projected/37ec2c6c-1216-4d35-8d37-95071fddf90a-kube-api-access-t6m89\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:57.240821 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.238939    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/30397e41-d823-4e74-9485-66f74e907f8b-run\") pod \"tuned-qjztn\" (UID: \"30397e41-d823-4e74-9485-66f74e907f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-qjztn"
Jul 29 18:33:57.240821 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.238962    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b03a3682-95be-4852-b28a-8e19fee4cbe7-tmp-dir\") pod \"node-resolver-t4wjl\" (UID: \"b03a3682-95be-4852-b28a-8e19fee4cbe7\") " pod="openshift-dns/node-resolver-t4wjl"
Jul 29 18:33:57.240821 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.238965    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8fa7a55e-1003-4cc1-918e-655ccd91f071-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gd95n\" (UID: \"8fa7a55e-1003-4cc1-918e-655ccd91f071\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd95n"
Jul 29 18:33:57.240821 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.239025    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztfbt\" (UniqueName: \"kubernetes.io/projected/df3fd46a-27e2-4557-bc40-09a9fbaa2e8f-kube-api-access-ztfbt\") pod \"network-check-target-jh2dv\" (UID: \"df3fd46a-27e2-4557-bc40-09a9fbaa2e8f\") " pod="openshift-network-diagnostics/network-check-target-jh2dv"
Jul 29 18:33:57.240821 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.239036    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8fa7a55e-1003-4cc1-918e-655ccd91f071-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gd95n\" (UID: \"8fa7a55e-1003-4cc1-918e-655ccd91f071\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd95n"
Jul 29 18:33:57.240821 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.239053    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6dad0cba-079a-4ec1-9f2b-8399f34fb80c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4wgbf\" (UID: \"6dad0cba-079a-4ec1-9f2b-8399f34fb80c\") " pod="openshift-multus/multus-additional-cni-plugins-4wgbf"
Jul 29 18:33:57.240821 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.239055    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37ec2c6c-1216-4d35-8d37-95071fddf90a-cni-binary-copy\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:57.240821 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.239086    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/37ec2c6c-1216-4d35-8d37-95071fddf90a-multus-socket-dir-parent\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:57.240821 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.239087    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8fa7a55e-1003-4cc1-918e-655ccd91f071-etc-selinux\") pod \"aws-ebs-csi-driver-node-gd95n\" (UID: \"8fa7a55e-1003-4cc1-918e-655ccd91f071\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd95n"
Jul 29 18:33:57.240821 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.239112    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37ec2c6c-1216-4d35-8d37-95071fddf90a-etc-kubernetes\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:57.240821 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.239119    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/37ec2c6c-1216-4d35-8d37-95071fddf90a-multus-daemon-config\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:57.241513 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.239134    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30397e41-d823-4e74-9485-66f74e907f8b-tmp\") pod \"tuned-qjztn\" (UID: \"30397e41-d823-4e74-9485-66f74e907f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-qjztn"
Jul 29 18:33:57.241513 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.239146    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/30397e41-d823-4e74-9485-66f74e907f8b-run\") pod \"tuned-qjztn\" (UID: \"30397e41-d823-4e74-9485-66f74e907f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-qjztn"
Jul 29 18:33:57.241513 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.239157    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-544jm\" (UniqueName: \"kubernetes.io/projected/6dad0cba-079a-4ec1-9f2b-8399f34fb80c-kube-api-access-544jm\") pod \"multus-additional-cni-plugins-4wgbf\" (UID: \"6dad0cba-079a-4ec1-9f2b-8399f34fb80c\") " pod="openshift-multus/multus-additional-cni-plugins-4wgbf"
Jul 29 18:33:57.241513 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.239182    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/44c368d9-c4c2-481e-bbca-3bfa6eae44d1-iptables-alerter-script\") pod \"iptables-alerter-b6rhb\" (UID: \"44c368d9-c4c2-481e-bbca-3bfa6eae44d1\") " pod="openshift-network-operator/iptables-alerter-b6rhb"
Jul 29 18:33:57.241513 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.239206    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz64d\" (UniqueName: \"kubernetes.io/projected/8fa7a55e-1003-4cc1-918e-655ccd91f071-kube-api-access-qz64d\") pod \"aws-ebs-csi-driver-node-gd95n\" (UID: \"8fa7a55e-1003-4cc1-918e-655ccd91f071\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd95n"
Jul 29 18:33:57.241513 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.239236    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/30397e41-d823-4e74-9485-66f74e907f8b-etc-sysctl-conf\") pod \"tuned-qjztn\" (UID: \"30397e41-d823-4e74-9485-66f74e907f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-qjztn"
Jul 29 18:33:57.241513 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.239259    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30397e41-d823-4e74-9485-66f74e907f8b-lib-modules\") pod \"tuned-qjztn\" (UID: \"30397e41-d823-4e74-9485-66f74e907f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-qjztn"
Jul 29 18:33:57.241513 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.239270    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6dad0cba-079a-4ec1-9f2b-8399f34fb80c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4wgbf\" (UID: \"6dad0cba-079a-4ec1-9f2b-8399f34fb80c\") " pod="openshift-multus/multus-additional-cni-plugins-4wgbf"
Jul 29 18:33:57.241513 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.239288    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6a62d29d-5bf6-444f-bd28-bf2d1e54ac87-konnectivity-ca\") pod \"konnectivity-agent-bp28x\" (UID: \"6a62d29d-5bf6-444f-bd28-bf2d1e54ac87\") " pod="kube-system/konnectivity-agent-bp28x"
Jul 29 18:33:57.241513 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.239318    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/30397e41-d823-4e74-9485-66f74e907f8b-etc-sysctl-d\") pod \"tuned-qjztn\" (UID: \"30397e41-d823-4e74-9485-66f74e907f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-qjztn"
Jul 29 18:33:57.241513 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.239325    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/37ec2c6c-1216-4d35-8d37-95071fddf90a-multus-socket-dir-parent\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:57.241513 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.239338    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b03a3682-95be-4852-b28a-8e19fee4cbe7-hosts-file\") pod \"node-resolver-t4wjl\" (UID: \"b03a3682-95be-4852-b28a-8e19fee4cbe7\") " pod="openshift-dns/node-resolver-t4wjl"
Jul 29 18:33:57.241513 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.239362    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6dad0cba-079a-4ec1-9f2b-8399f34fb80c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4wgbf\" (UID: \"6dad0cba-079a-4ec1-9f2b-8399f34fb80c\") " pod="openshift-multus/multus-additional-cni-plugins-4wgbf"
Jul 29 18:33:57.241513 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.239367    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m4ld\" (UniqueName: \"kubernetes.io/projected/b03a3682-95be-4852-b28a-8e19fee4cbe7-kube-api-access-6m4ld\") pod \"node-resolver-t4wjl\" (UID: \"b03a3682-95be-4852-b28a-8e19fee4cbe7\") " pod="openshift-dns/node-resolver-t4wjl"
Jul 29 18:33:57.241513 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.239420    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6dad0cba-079a-4ec1-9f2b-8399f34fb80c-cni-binary-copy\") pod \"multus-additional-cni-plugins-4wgbf\" (UID: \"6dad0cba-079a-4ec1-9f2b-8399f34fb80c\") " pod="openshift-multus/multus-additional-cni-plugins-4wgbf"
Jul 29 18:33:57.241513 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.239444    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8fa7a55e-1003-4cc1-918e-655ccd91f071-socket-dir\") pod \"aws-ebs-csi-driver-node-gd95n\" (UID: \"8fa7a55e-1003-4cc1-918e-655ccd91f071\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd95n"
Jul 29 18:33:57.241513 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.239465    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b03a3682-95be-4852-b28a-8e19fee4cbe7-hosts-file\") pod \"node-resolver-t4wjl\" (UID: \"b03a3682-95be-4852-b28a-8e19fee4cbe7\") " pod="openshift-dns/node-resolver-t4wjl"
Jul 29 18:33:57.242002 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.239477    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/30397e41-d823-4e74-9485-66f74e907f8b-etc-sysctl-conf\") pod \"tuned-qjztn\" (UID: \"30397e41-d823-4e74-9485-66f74e907f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-qjztn"
Jul 29 18:33:57.242002 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.239466    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8fa7a55e-1003-4cc1-918e-655ccd91f071-device-dir\") pod \"aws-ebs-csi-driver-node-gd95n\" (UID: \"8fa7a55e-1003-4cc1-918e-655ccd91f071\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd95n"
Jul 29 18:33:57.242002 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.239516    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2201b8bf-575c-4958-b909-29ff288ed75e-serviceca\") pod \"node-ca-ldlwt\" (UID: \"2201b8bf-575c-4958-b909-29ff288ed75e\") " pod="openshift-image-registry/node-ca-ldlwt"
Jul 29 18:33:57.242002 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.239522    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6dad0cba-079a-4ec1-9f2b-8399f34fb80c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4wgbf\" (UID: \"6dad0cba-079a-4ec1-9f2b-8399f34fb80c\") " pod="openshift-multus/multus-additional-cni-plugins-4wgbf"
Jul 29 18:33:57.242002 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.239539    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/37ec2c6c-1216-4d35-8d37-95071fddf90a-host-run-k8s-cni-cncf-io\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:57.242002 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.239568    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lglkr\" (UniqueName: \"kubernetes.io/projected/30397e41-d823-4e74-9485-66f74e907f8b-kube-api-access-lglkr\") pod \"tuned-qjztn\" (UID: \"30397e41-d823-4e74-9485-66f74e907f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-qjztn"
Jul 29 18:33:57.242002 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.239612    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/30397e41-d823-4e74-9485-66f74e907f8b-etc-sysctl-d\") pod \"tuned-qjztn\" (UID: \"30397e41-d823-4e74-9485-66f74e907f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-qjztn"
Jul 29 18:33:57.242002 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.239595    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6a62d29d-5bf6-444f-bd28-bf2d1e54ac87-agent-certs\") pod \"konnectivity-agent-bp28x\" (UID: \"6a62d29d-5bf6-444f-bd28-bf2d1e54ac87\") " pod="kube-system/konnectivity-agent-bp28x"
Jul 29 18:33:57.242002 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.239649    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/44c368d9-c4c2-481e-bbca-3bfa6eae44d1-host-slash\") pod \"iptables-alerter-b6rhb\" (UID: \"44c368d9-c4c2-481e-bbca-3bfa6eae44d1\") " pod="openshift-network-operator/iptables-alerter-b6rhb"
Jul 29 18:33:57.242002 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.239693    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37ec2c6c-1216-4d35-8d37-95071fddf90a-etc-kubernetes\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:57.242002 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.239704    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6dad0cba-079a-4ec1-9f2b-8399f34fb80c-cnibin\") pod \"multus-additional-cni-plugins-4wgbf\" (UID: \"6dad0cba-079a-4ec1-9f2b-8399f34fb80c\") " pod="openshift-multus/multus-additional-cni-plugins-4wgbf"
Jul 29 18:33:57.242002 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.239730    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/faa3a79b-75fb-4890-b169-09674f6d00db-dbus\") pod \"global-pull-secret-syncer-f5w95\" (UID: \"faa3a79b-75fb-4890-b169-09674f6d00db\") " pod="kube-system/global-pull-secret-syncer-f5w95"
Jul 29 18:33:57.242002 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.239910    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37ec2c6c-1216-4d35-8d37-95071fddf90a-multus-cni-dir\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:57.242002 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.239943    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37ec2c6c-1216-4d35-8d37-95071fddf90a-host-var-lib-kubelet\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:57.242002 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.239964    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/30397e41-d823-4e74-9485-66f74e907f8b-etc-systemd\") pod \"tuned-qjztn\" (UID: \"30397e41-d823-4e74-9485-66f74e907f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-qjztn"
Jul 29 18:33:57.242002 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.239984    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30397e41-d823-4e74-9485-66f74e907f8b-sys\") pod \"tuned-qjztn\" (UID: \"30397e41-d823-4e74-9485-66f74e907f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-qjztn"
Jul 29 18:33:57.242002 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.240009    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6dad0cba-079a-4ec1-9f2b-8399f34fb80c-system-cni-dir\") pod \"multus-additional-cni-plugins-4wgbf\" (UID: \"6dad0cba-079a-4ec1-9f2b-8399f34fb80c\") " pod="openshift-multus/multus-additional-cni-plugins-4wgbf"
Jul 29 18:33:57.242490 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.240035    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt2qp\" (UniqueName: \"kubernetes.io/projected/44c368d9-c4c2-481e-bbca-3bfa6eae44d1-kube-api-access-bt2qp\") pod \"iptables-alerter-b6rhb\" (UID: \"44c368d9-c4c2-481e-bbca-3bfa6eae44d1\") " pod="openshift-network-operator/iptables-alerter-b6rhb"
Jul 29 18:33:57.242490 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.240046    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/faa3a79b-75fb-4890-b169-09674f6d00db-dbus\") pod \"global-pull-secret-syncer-f5w95\" (UID: \"faa3a79b-75fb-4890-b169-09674f6d00db\") " pod="kube-system/global-pull-secret-syncer-f5w95"
Jul 29 18:33:57.242490 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.240057    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/44c368d9-c4c2-481e-bbca-3bfa6eae44d1-iptables-alerter-script\") pod \"iptables-alerter-b6rhb\" (UID: \"44c368d9-c4c2-481e-bbca-3bfa6eae44d1\") " pod="openshift-network-operator/iptables-alerter-b6rhb"
Jul 29 18:33:57.242490 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.240066    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/faa3a79b-75fb-4890-b169-09674f6d00db-original-pull-secret\") pod \"global-pull-secret-syncer-f5w95\" (UID: \"faa3a79b-75fb-4890-b169-09674f6d00db\") " pod="kube-system/global-pull-secret-syncer-f5w95"
Jul 29 18:33:57.242490 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.240079    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2201b8bf-575c-4958-b909-29ff288ed75e-serviceca\") pod \"node-ca-ldlwt\" (UID: \"2201b8bf-575c-4958-b909-29ff288ed75e\") " pod="openshift-image-registry/node-ca-ldlwt"
Jul 29 18:33:57.242490 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.240091    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8fa7a55e-1003-4cc1-918e-655ccd91f071-sys-fs\") pod \"aws-ebs-csi-driver-node-gd95n\" (UID: \"8fa7a55e-1003-4cc1-918e-655ccd91f071\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd95n"
Jul 29 18:33:57.242490 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.239479    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30397e41-d823-4e74-9485-66f74e907f8b-lib-modules\") pod \"tuned-qjztn\" (UID: \"30397e41-d823-4e74-9485-66f74e907f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-qjztn"
Jul 29 18:33:57.242490 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.240098    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/37ec2c6c-1216-4d35-8d37-95071fddf90a-host-run-k8s-cni-cncf-io\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:57.242490 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.240158    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6dad0cba-079a-4ec1-9f2b-8399f34fb80c-cnibin\") pod \"multus-additional-cni-plugins-4wgbf\" (UID: \"6dad0cba-079a-4ec1-9f2b-8399f34fb80c\") " pod="openshift-multus/multus-additional-cni-plugins-4wgbf"
Jul 29 18:33:57.242490 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:57.240181    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 18:33:57.242490 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.240211    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30397e41-d823-4e74-9485-66f74e907f8b-sys\") pod \"tuned-qjztn\" (UID: \"30397e41-d823-4e74-9485-66f74e907f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-qjztn"
Jul 29 18:33:57.242490 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:57.240237    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faa3a79b-75fb-4890-b169-09674f6d00db-original-pull-secret podName:faa3a79b-75fb-4890-b169-09674f6d00db nodeName:}" failed. No retries permitted until 2026-07-29 18:33:57.74022114 +0000 UTC m=+2.009990209 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/faa3a79b-75fb-4890-b169-09674f6d00db-original-pull-secret") pod "global-pull-secret-syncer-f5w95" (UID: "faa3a79b-75fb-4890-b169-09674f6d00db") : object "kube-system"/"original-pull-secret" not registered
Jul 29 18:33:57.242490 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.240256    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37ec2c6c-1216-4d35-8d37-95071fddf90a-host-var-lib-cni-bin\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:57.242490 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.240272    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8fa7a55e-1003-4cc1-918e-655ccd91f071-sys-fs\") pod \"aws-ebs-csi-driver-node-gd95n\" (UID: \"8fa7a55e-1003-4cc1-918e-655ccd91f071\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd95n"
Jul 29 18:33:57.242490 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.240273    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6a62d29d-5bf6-444f-bd28-bf2d1e54ac87-konnectivity-ca\") pod \"konnectivity-agent-bp28x\" (UID: \"6a62d29d-5bf6-444f-bd28-bf2d1e54ac87\") " pod="kube-system/konnectivity-agent-bp28x"
Jul 29 18:33:57.242490 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.240281    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/37ec2c6c-1216-4d35-8d37-95071fddf90a-hostroot\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:57.242490 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.240315    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/37ec2c6c-1216-4d35-8d37-95071fddf90a-hostroot\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:57.243126 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.240329    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/30397e41-d823-4e74-9485-66f74e907f8b-host\") pod \"tuned-qjztn\" (UID: \"30397e41-d823-4e74-9485-66f74e907f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-qjztn"
Jul 29 18:33:57.243126 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.240321    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/44c368d9-c4c2-481e-bbca-3bfa6eae44d1-host-slash\") pod \"iptables-alerter-b6rhb\" (UID: \"44c368d9-c4c2-481e-bbca-3bfa6eae44d1\") " pod="openshift-network-operator/iptables-alerter-b6rhb"
Jul 29 18:33:57.243126 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.240362    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37ec2c6c-1216-4d35-8d37-95071fddf90a-host-var-lib-kubelet\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:57.243126 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.240367    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30397e41-d823-4e74-9485-66f74e907f8b-var-lib-kubelet\") pod \"tuned-qjztn\" (UID: \"30397e41-d823-4e74-9485-66f74e907f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-qjztn"
Jul 29 18:33:57.243126 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.240398    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37ec2c6c-1216-4d35-8d37-95071fddf90a-host-var-lib-cni-bin\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:57.243126 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.240434    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30397e41-d823-4e74-9485-66f74e907f8b-var-lib-kubelet\") pod \"tuned-qjztn\" (UID: \"30397e41-d823-4e74-9485-66f74e907f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-qjztn"
Jul 29 18:33:57.243126 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.240516    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/30397e41-d823-4e74-9485-66f74e907f8b-host\") pod \"tuned-qjztn\" (UID: \"30397e41-d823-4e74-9485-66f74e907f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-qjztn"
Jul 29 18:33:57.243126 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.240519    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6dad0cba-079a-4ec1-9f2b-8399f34fb80c-system-cni-dir\") pod \"multus-additional-cni-plugins-4wgbf\" (UID: \"6dad0cba-079a-4ec1-9f2b-8399f34fb80c\") " pod="openshift-multus/multus-additional-cni-plugins-4wgbf"
Jul 29 18:33:57.243126 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.240574    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37ec2c6c-1216-4d35-8d37-95071fddf90a-multus-cni-dir\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:57.243126 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.240581    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8fa7a55e-1003-4cc1-918e-655ccd91f071-device-dir\") pod \"aws-ebs-csi-driver-node-gd95n\" (UID: \"8fa7a55e-1003-4cc1-918e-655ccd91f071\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd95n"
Jul 29 18:33:57.243126 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.240626    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/30397e41-d823-4e74-9485-66f74e907f8b-etc-systemd\") pod \"tuned-qjztn\" (UID: \"30397e41-d823-4e74-9485-66f74e907f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-qjztn"
Jul 29 18:33:57.243126 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.240627    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8fa7a55e-1003-4cc1-918e-655ccd91f071-socket-dir\") pod \"aws-ebs-csi-driver-node-gd95n\" (UID: \"8fa7a55e-1003-4cc1-918e-655ccd91f071\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd95n"
Jul 29 18:33:57.243126 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.241065    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6dad0cba-079a-4ec1-9f2b-8399f34fb80c-cni-binary-copy\") pod \"multus-additional-cni-plugins-4wgbf\" (UID: \"6dad0cba-079a-4ec1-9f2b-8399f34fb80c\") " pod="openshift-multus/multus-additional-cni-plugins-4wgbf"
Jul 29 18:33:57.243126 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.241624    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/30397e41-d823-4e74-9485-66f74e907f8b-etc-tuned\") pod \"tuned-qjztn\" (UID: \"30397e41-d823-4e74-9485-66f74e907f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-qjztn"
Jul 29 18:33:57.243126 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.241739    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30397e41-d823-4e74-9485-66f74e907f8b-tmp\") pod \"tuned-qjztn\" (UID: \"30397e41-d823-4e74-9485-66f74e907f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-qjztn"
Jul 29 18:33:57.243126 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.243077    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6a62d29d-5bf6-444f-bd28-bf2d1e54ac87-agent-certs\") pod \"konnectivity-agent-bp28x\" (UID: \"6a62d29d-5bf6-444f-bd28-bf2d1e54ac87\") " pod="kube-system/konnectivity-agent-bp28x"
Jul 29 18:33:57.288196 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.288168    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz64d\" (UniqueName: \"kubernetes.io/projected/8fa7a55e-1003-4cc1-918e-655ccd91f071-kube-api-access-qz64d\") pod \"aws-ebs-csi-driver-node-gd95n\" (UID: \"8fa7a55e-1003-4cc1-918e-655ccd91f071\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd95n"
Jul 29 18:33:57.290409 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.290369    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m4ld\" (UniqueName: \"kubernetes.io/projected/b03a3682-95be-4852-b28a-8e19fee4cbe7-kube-api-access-6m4ld\") pod \"node-resolver-t4wjl\" (UID: \"b03a3682-95be-4852-b28a-8e19fee4cbe7\") " pod="openshift-dns/node-resolver-t4wjl"
Jul 29 18:33:57.297621 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:57.297592    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 18:33:57.297621 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:57.297622    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 18:33:57.297621 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:57.297637    2559 projected.go:196] Error preparing data for projected volume kube-api-access-ztfbt for pod openshift-network-diagnostics/network-check-target-jh2dv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 18:33:57.297905 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:57.297731    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df3fd46a-27e2-4557-bc40-09a9fbaa2e8f-kube-api-access-ztfbt podName:df3fd46a-27e2-4557-bc40-09a9fbaa2e8f nodeName:}" failed. No retries permitted until 2026-07-29 18:33:57.797709068 +0000 UTC m=+2.067478147 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ztfbt" (UniqueName: "kubernetes.io/projected/df3fd46a-27e2-4557-bc40-09a9fbaa2e8f-kube-api-access-ztfbt") pod "network-check-target-jh2dv" (UID: "df3fd46a-27e2-4557-bc40-09a9fbaa2e8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 18:33:57.298321 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.298297    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn5k9\" (UniqueName: \"kubernetes.io/projected/4b86e30e-def7-4bf4-90da-de4fef47f433-kube-api-access-sn5k9\") pod \"network-metrics-daemon-9zxr5\" (UID: \"4b86e30e-def7-4bf4-90da-de4fef47f433\") " pod="openshift-multus/network-metrics-daemon-9zxr5"
Jul 29 18:33:57.298566 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.298548    2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 18:33:57.298894 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.298869    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4h9f\" (UniqueName: \"kubernetes.io/projected/2201b8bf-575c-4958-b909-29ff288ed75e-kube-api-access-v4h9f\") pod \"node-ca-ldlwt\" (UID: \"2201b8bf-575c-4958-b909-29ff288ed75e\") " pod="openshift-image-registry/node-ca-ldlwt"
Jul 29 18:33:57.299042 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.299025    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6m89\" (UniqueName: \"kubernetes.io/projected/37ec2c6c-1216-4d35-8d37-95071fddf90a-kube-api-access-t6m89\") pod \"multus-6vd4x\" (UID: \"37ec2c6c-1216-4d35-8d37-95071fddf90a\") " pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:57.299084 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.299033    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-544jm\" (UniqueName: \"kubernetes.io/projected/6dad0cba-079a-4ec1-9f2b-8399f34fb80c-kube-api-access-544jm\") pod \"multus-additional-cni-plugins-4wgbf\" (UID: \"6dad0cba-079a-4ec1-9f2b-8399f34fb80c\") " pod="openshift-multus/multus-additional-cni-plugins-4wgbf"
Jul 29 18:33:57.299491 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.299475    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt2qp\" (UniqueName: \"kubernetes.io/projected/44c368d9-c4c2-481e-bbca-3bfa6eae44d1-kube-api-access-bt2qp\") pod \"iptables-alerter-b6rhb\" (UID: \"44c368d9-c4c2-481e-bbca-3bfa6eae44d1\") " pod="openshift-network-operator/iptables-alerter-b6rhb"
Jul 29 18:33:57.301618 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.301588    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lglkr\" (UniqueName: \"kubernetes.io/projected/30397e41-d823-4e74-9485-66f74e907f8b-kube-api-access-lglkr\") pod \"tuned-qjztn\" (UID: \"30397e41-d823-4e74-9485-66f74e907f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-qjztn"
Jul 29 18:33:57.353814 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.353735    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:33:57.366241 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.366215    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bp28x"
Jul 29 18:33:57.367109 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:57.367076    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60f444e1_7e31_443c_8ec7_a4d04c64bf4b.slice/crio-0025464a336e5da2b8cd0af2237c8bbeaced86055a16fface9c96698e9349bf7 WatchSource:0}: Error finding container 0025464a336e5da2b8cd0af2237c8bbeaced86055a16fface9c96698e9349bf7: Status 404 returned error can't find the container with id 0025464a336e5da2b8cd0af2237c8bbeaced86055a16fface9c96698e9349bf7
Jul 29 18:33:57.375056 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:57.375030    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a62d29d_5bf6_444f_bd28_bf2d1e54ac87.slice/crio-007d4d04b8b4003611fa90374a2044e6d0dc758196edc62ae8e638f376ca67eb WatchSource:0}: Error finding container 007d4d04b8b4003611fa90374a2044e6d0dc758196edc62ae8e638f376ca67eb: Status 404 returned error can't find the container with id 007d4d04b8b4003611fa90374a2044e6d0dc758196edc62ae8e638f376ca67eb
Jul 29 18:33:57.381773 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.381750    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd95n"
Jul 29 18:33:57.390196 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:57.390170    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fa7a55e_1003_4cc1_918e_655ccd91f071.slice/crio-3e49b7edc456cb0704f15a7c80b394cb333f991f96393b27acbf16b17c405d5e WatchSource:0}: Error finding container 3e49b7edc456cb0704f15a7c80b394cb333f991f96393b27acbf16b17c405d5e: Status 404 returned error can't find the container with id 3e49b7edc456cb0704f15a7c80b394cb333f991f96393b27acbf16b17c405d5e
Jul 29 18:33:57.396591 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.396571    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t4wjl"
Jul 29 18:33:57.404921 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:57.404895    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb03a3682_95be_4852_b28a_8e19fee4cbe7.slice/crio-1e6202ac8604603d8d30c508bc785345c1611cb0da2d25efd88cff27fbd72668 WatchSource:0}: Error finding container 1e6202ac8604603d8d30c508bc785345c1611cb0da2d25efd88cff27fbd72668: Status 404 returned error can't find the container with id 1e6202ac8604603d8d30c508bc785345c1611cb0da2d25efd88cff27fbd72668
Jul 29 18:33:57.419714 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.419691    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ldlwt"
Jul 29 18:33:57.428645 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:57.428619    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2201b8bf_575c_4958_b909_29ff288ed75e.slice/crio-60a616307f3368ba965d8908a6eed3184b3e148dfca5a16f854c19624471a9a4 WatchSource:0}: Error finding container 60a616307f3368ba965d8908a6eed3184b3e148dfca5a16f854c19624471a9a4: Status 404 returned error can't find the container with id 60a616307f3368ba965d8908a6eed3184b3e148dfca5a16f854c19624471a9a4
Jul 29 18:33:57.445886 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.445861    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6vd4x"
Jul 29 18:33:57.452981 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.452961    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qjztn"
Jul 29 18:33:57.454341 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:57.454315    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37ec2c6c_1216_4d35_8d37_95071fddf90a.slice/crio-e933b07aa3e9ff2fd485f7a137fe35314e167a55091a9d3eab840920407e20fb WatchSource:0}: Error finding container e933b07aa3e9ff2fd485f7a137fe35314e167a55091a9d3eab840920407e20fb: Status 404 returned error can't find the container with id e933b07aa3e9ff2fd485f7a137fe35314e167a55091a9d3eab840920407e20fb
Jul 29 18:33:57.457482 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.457458    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4wgbf"
Jul 29 18:33:57.461360 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:57.461325    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30397e41_d823_4e74_9485_66f74e907f8b.slice/crio-85d637dd55c66653de9e2a7438ba2c313b9bedb12d9cfb6677fb000180c734fc WatchSource:0}: Error finding container 85d637dd55c66653de9e2a7438ba2c313b9bedb12d9cfb6677fb000180c734fc: Status 404 returned error can't find the container with id 85d637dd55c66653de9e2a7438ba2c313b9bedb12d9cfb6677fb000180c734fc
Jul 29 18:33:57.464452 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.464404    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-b6rhb"
Jul 29 18:33:57.466339 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:57.466318    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dad0cba_079a_4ec1_9f2b_8399f34fb80c.slice/crio-70bf2d89a7507853675df030db565ab5acf6d54a59bd0738830727b3afc1b1e3 WatchSource:0}: Error finding container 70bf2d89a7507853675df030db565ab5acf6d54a59bd0738830727b3afc1b1e3: Status 404 returned error can't find the container with id 70bf2d89a7507853675df030db565ab5acf6d54a59bd0738830727b3afc1b1e3
Jul 29 18:33:57.475513 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:33:57.475487    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44c368d9_c4c2_481e_bbca_3bfa6eae44d1.slice/crio-e15cf1b48a73bf2f1d4c52e9f4d8981268f8b6f04fe5573b48160ecfefa86732 WatchSource:0}: Error finding container e15cf1b48a73bf2f1d4c52e9f4d8981268f8b6f04fe5573b48160ecfefa86732: Status 404 returned error can't find the container with id e15cf1b48a73bf2f1d4c52e9f4d8981268f8b6f04fe5573b48160ecfefa86732
Jul 29 18:33:57.543705 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.543667    2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 18:33:57.745090 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.744991    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b86e30e-def7-4bf4-90da-de4fef47f433-metrics-certs\") pod \"network-metrics-daemon-9zxr5\" (UID: \"4b86e30e-def7-4bf4-90da-de4fef47f433\") " pod="openshift-multus/network-metrics-daemon-9zxr5"
Jul 29 18:33:57.745090 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.745065    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/faa3a79b-75fb-4890-b169-09674f6d00db-original-pull-secret\") pod \"global-pull-secret-syncer-f5w95\" (UID: \"faa3a79b-75fb-4890-b169-09674f6d00db\") " pod="kube-system/global-pull-secret-syncer-f5w95"
Jul 29 18:33:57.745289 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:57.745166    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 18:33:57.745289 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:57.745167    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 18:33:57.745289 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:57.745225    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faa3a79b-75fb-4890-b169-09674f6d00db-original-pull-secret podName:faa3a79b-75fb-4890-b169-09674f6d00db nodeName:}" failed. No retries permitted until 2026-07-29 18:33:58.745211631 +0000 UTC m=+3.014980718 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/faa3a79b-75fb-4890-b169-09674f6d00db-original-pull-secret") pod "global-pull-secret-syncer-f5w95" (UID: "faa3a79b-75fb-4890-b169-09674f6d00db") : object "kube-system"/"original-pull-secret" not registered
Jul 29 18:33:57.745289 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:57.745247    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b86e30e-def7-4bf4-90da-de4fef47f433-metrics-certs podName:4b86e30e-def7-4bf4-90da-de4fef47f433 nodeName:}" failed. No retries permitted until 2026-07-29 18:33:58.745236406 +0000 UTC m=+3.015005474 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b86e30e-def7-4bf4-90da-de4fef47f433-metrics-certs") pod "network-metrics-daemon-9zxr5" (UID: "4b86e30e-def7-4bf4-90da-de4fef47f433") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 18:33:57.845782 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:57.845743    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztfbt\" (UniqueName: \"kubernetes.io/projected/df3fd46a-27e2-4557-bc40-09a9fbaa2e8f-kube-api-access-ztfbt\") pod \"network-check-target-jh2dv\" (UID: \"df3fd46a-27e2-4557-bc40-09a9fbaa2e8f\") " pod="openshift-network-diagnostics/network-check-target-jh2dv"
Jul 29 18:33:57.845964 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:57.845941    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 18:33:57.845964 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:57.845958    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 18:33:57.846070 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:57.845969    2559 projected.go:196] Error preparing data for projected volume kube-api-access-ztfbt for pod openshift-network-diagnostics/network-check-target-jh2dv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 18:33:57.846070 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:57.846014    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df3fd46a-27e2-4557-bc40-09a9fbaa2e8f-kube-api-access-ztfbt podName:df3fd46a-27e2-4557-bc40-09a9fbaa2e8f nodeName:}" failed. No retries permitted until 2026-07-29 18:33:58.84600141 +0000 UTC m=+3.115770489 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ztfbt" (UniqueName: "kubernetes.io/projected/df3fd46a-27e2-4557-bc40-09a9fbaa2e8f-kube-api-access-ztfbt") pod "network-check-target-jh2dv" (UID: "df3fd46a-27e2-4557-bc40-09a9fbaa2e8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 18:33:58.102956 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:58.102850    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 18:28:57 +0000 UTC" deadline="2028-04-19 11:08:14.374216806 +0000 UTC"
Jul 29 18:33:58.102956 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:58.102915    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15112h34m16.271306086s"
Jul 29 18:33:58.155967 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:58.155861    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx" event={"ID":"60f444e1-7e31-443c-8ec7-a4d04c64bf4b","Type":"ContainerStarted","Data":"0025464a336e5da2b8cd0af2237c8bbeaced86055a16fface9c96698e9349bf7"}
Jul 29 18:33:58.159431 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:58.159372    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ldlwt" event={"ID":"2201b8bf-575c-4958-b909-29ff288ed75e","Type":"ContainerStarted","Data":"60a616307f3368ba965d8908a6eed3184b3e148dfca5a16f854c19624471a9a4"}
Jul 29 18:33:58.172408 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:58.172304    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t4wjl" event={"ID":"b03a3682-95be-4852-b28a-8e19fee4cbe7","Type":"ContainerStarted","Data":"1e6202ac8604603d8d30c508bc785345c1611cb0da2d25efd88cff27fbd72668"}
Jul 29 18:33:58.187451 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:58.187409    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bp28x" event={"ID":"6a62d29d-5bf6-444f-bd28-bf2d1e54ac87","Type":"ContainerStarted","Data":"007d4d04b8b4003611fa90374a2044e6d0dc758196edc62ae8e638f376ca67eb"}
Jul 29 18:33:58.202293 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:58.202248    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-b6rhb" event={"ID":"44c368d9-c4c2-481e-bbca-3bfa6eae44d1","Type":"ContainerStarted","Data":"e15cf1b48a73bf2f1d4c52e9f4d8981268f8b6f04fe5573b48160ecfefa86732"}
Jul 29 18:33:58.207698 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:58.207662    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wgbf" event={"ID":"6dad0cba-079a-4ec1-9f2b-8399f34fb80c","Type":"ContainerStarted","Data":"70bf2d89a7507853675df030db565ab5acf6d54a59bd0738830727b3afc1b1e3"}
Jul 29 18:33:58.210114 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:58.210080    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qjztn" event={"ID":"30397e41-d823-4e74-9485-66f74e907f8b","Type":"ContainerStarted","Data":"85d637dd55c66653de9e2a7438ba2c313b9bedb12d9cfb6677fb000180c734fc"}
Jul 29 18:33:58.218475 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:58.218345    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6vd4x" event={"ID":"37ec2c6c-1216-4d35-8d37-95071fddf90a","Type":"ContainerStarted","Data":"e933b07aa3e9ff2fd485f7a137fe35314e167a55091a9d3eab840920407e20fb"}
Jul 29 18:33:58.222481 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:58.222317    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd95n" event={"ID":"8fa7a55e-1003-4cc1-918e-655ccd91f071","Type":"ContainerStarted","Data":"3e49b7edc456cb0704f15a7c80b394cb333f991f96393b27acbf16b17c405d5e"}
Jul 29 18:33:58.235316 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:58.235279    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 18:33:58.756572 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:58.756514    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b86e30e-def7-4bf4-90da-de4fef47f433-metrics-certs\") pod \"network-metrics-daemon-9zxr5\" (UID: \"4b86e30e-def7-4bf4-90da-de4fef47f433\") " pod="openshift-multus/network-metrics-daemon-9zxr5"
Jul 29 18:33:58.756844 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:58.756607    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/faa3a79b-75fb-4890-b169-09674f6d00db-original-pull-secret\") pod \"global-pull-secret-syncer-f5w95\" (UID: \"faa3a79b-75fb-4890-b169-09674f6d00db\") " pod="kube-system/global-pull-secret-syncer-f5w95"
Jul 29 18:33:58.756844 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:58.756737    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 18:33:58.756844 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:58.756799    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faa3a79b-75fb-4890-b169-09674f6d00db-original-pull-secret podName:faa3a79b-75fb-4890-b169-09674f6d00db nodeName:}" failed. No retries permitted until 2026-07-29 18:34:00.756780384 +0000 UTC m=+5.026549457 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/faa3a79b-75fb-4890-b169-09674f6d00db-original-pull-secret") pod "global-pull-secret-syncer-f5w95" (UID: "faa3a79b-75fb-4890-b169-09674f6d00db") : object "kube-system"/"original-pull-secret" not registered
Jul 29 18:33:58.757212 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:58.757192    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 18:33:58.757283 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:58.757247    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b86e30e-def7-4bf4-90da-de4fef47f433-metrics-certs podName:4b86e30e-def7-4bf4-90da-de4fef47f433 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:00.757232053 +0000 UTC m=+5.027001122 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b86e30e-def7-4bf4-90da-de4fef47f433-metrics-certs") pod "network-metrics-daemon-9zxr5" (UID: "4b86e30e-def7-4bf4-90da-de4fef47f433") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 18:33:58.857237 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:58.857193    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztfbt\" (UniqueName: \"kubernetes.io/projected/df3fd46a-27e2-4557-bc40-09a9fbaa2e8f-kube-api-access-ztfbt\") pod \"network-check-target-jh2dv\" (UID: \"df3fd46a-27e2-4557-bc40-09a9fbaa2e8f\") " pod="openshift-network-diagnostics/network-check-target-jh2dv"
Jul 29 18:33:58.857454 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:58.857420    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 18:33:58.857454 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:58.857444    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 18:33:58.857568 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:58.857457    2559 projected.go:196] Error preparing data for projected volume kube-api-access-ztfbt for pod openshift-network-diagnostics/network-check-target-jh2dv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 18:33:58.857568 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:58.857517    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df3fd46a-27e2-4557-bc40-09a9fbaa2e8f-kube-api-access-ztfbt podName:df3fd46a-27e2-4557-bc40-09a9fbaa2e8f nodeName:}" failed. No retries permitted until 2026-07-29 18:34:00.857499055 +0000 UTC m=+5.127268139 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ztfbt" (UniqueName: "kubernetes.io/projected/df3fd46a-27e2-4557-bc40-09a9fbaa2e8f-kube-api-access-ztfbt") pod "network-check-target-jh2dv" (UID: "df3fd46a-27e2-4557-bc40-09a9fbaa2e8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 18:33:59.104111 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:59.104064    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 18:28:57 +0000 UTC" deadline="2028-03-14 17:26:23.190125295 +0000 UTC"
Jul 29 18:33:59.104111 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:59.104109    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14254h52m24.086021113s"
Jul 29 18:33:59.144102 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:59.144068    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zxr5"
Jul 29 18:33:59.144359 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:59.144310    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zxr5" podUID="4b86e30e-def7-4bf4-90da-de4fef47f433"
Jul 29 18:33:59.144854 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:59.144833    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5w95"
Jul 29 18:33:59.144969 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:59.144944    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5w95" podUID="faa3a79b-75fb-4890-b169-09674f6d00db"
Jul 29 18:33:59.145045 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:59.145009    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh2dv"
Jul 29 18:33:59.145102 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:33:59.145074    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh2dv" podUID="df3fd46a-27e2-4557-bc40-09a9fbaa2e8f"
Jul 29 18:33:59.903337 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:33:59.903294    2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 18:34:00.241241 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:00.241142    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-152.ec2.internal" event={"ID":"8923e83396bb7ee7f5fbbb29dfb80487","Type":"ContainerStarted","Data":"2d73dfdbac96dbfae564beaa8ad56b4ffdb40de2a25be19035fd4a11a5eff508"}
Jul 29 18:34:00.779369 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:00.779032    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/faa3a79b-75fb-4890-b169-09674f6d00db-original-pull-secret\") pod \"global-pull-secret-syncer-f5w95\" (UID: \"faa3a79b-75fb-4890-b169-09674f6d00db\") " pod="kube-system/global-pull-secret-syncer-f5w95"
Jul 29 18:34:00.779369 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:00.779325    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b86e30e-def7-4bf4-90da-de4fef47f433-metrics-certs\") pod \"network-metrics-daemon-9zxr5\" (UID: \"4b86e30e-def7-4bf4-90da-de4fef47f433\") " pod="openshift-multus/network-metrics-daemon-9zxr5"
Jul 29 18:34:00.779618 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:00.779512    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 18:34:00.779618 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:00.779577    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b86e30e-def7-4bf4-90da-de4fef47f433-metrics-certs podName:4b86e30e-def7-4bf4-90da-de4fef47f433 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:04.779557495 +0000 UTC m=+9.049326564 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b86e30e-def7-4bf4-90da-de4fef47f433-metrics-certs") pod "network-metrics-daemon-9zxr5" (UID: "4b86e30e-def7-4bf4-90da-de4fef47f433") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 18:34:00.780001 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:00.779980    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 18:34:00.780103 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:00.780036    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faa3a79b-75fb-4890-b169-09674f6d00db-original-pull-secret podName:faa3a79b-75fb-4890-b169-09674f6d00db nodeName:}" failed. No retries permitted until 2026-07-29 18:34:04.78002059 +0000 UTC m=+9.049789661 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/faa3a79b-75fb-4890-b169-09674f6d00db-original-pull-secret") pod "global-pull-secret-syncer-f5w95" (UID: "faa3a79b-75fb-4890-b169-09674f6d00db") : object "kube-system"/"original-pull-secret" not registered
Jul 29 18:34:00.880837 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:00.880117    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztfbt\" (UniqueName: \"kubernetes.io/projected/df3fd46a-27e2-4557-bc40-09a9fbaa2e8f-kube-api-access-ztfbt\") pod \"network-check-target-jh2dv\" (UID: \"df3fd46a-27e2-4557-bc40-09a9fbaa2e8f\") " pod="openshift-network-diagnostics/network-check-target-jh2dv"
Jul 29 18:34:00.880837 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:00.880281    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 18:34:00.880837 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:00.880304    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 18:34:00.880837 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:00.880317    2559 projected.go:196] Error preparing data for projected volume kube-api-access-ztfbt for pod openshift-network-diagnostics/network-check-target-jh2dv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 18:34:00.880837 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:00.880410    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df3fd46a-27e2-4557-bc40-09a9fbaa2e8f-kube-api-access-ztfbt podName:df3fd46a-27e2-4557-bc40-09a9fbaa2e8f nodeName:}" failed. No retries permitted until 2026-07-29 18:34:04.880373644 +0000 UTC m=+9.150142713 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ztfbt" (UniqueName: "kubernetes.io/projected/df3fd46a-27e2-4557-bc40-09a9fbaa2e8f-kube-api-access-ztfbt") pod "network-check-target-jh2dv" (UID: "df3fd46a-27e2-4557-bc40-09a9fbaa2e8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 18:34:01.144742 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:01.144703    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5w95"
Jul 29 18:34:01.144933 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:01.144860    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5w95" podUID="faa3a79b-75fb-4890-b169-09674f6d00db"
Jul 29 18:34:01.145115 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:01.145091    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh2dv"
Jul 29 18:34:01.145213 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:01.145128    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zxr5"
Jul 29 18:34:01.145213 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:01.145199    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh2dv" podUID="df3fd46a-27e2-4557-bc40-09a9fbaa2e8f"
Jul 29 18:34:01.145324 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:01.145256    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zxr5" podUID="4b86e30e-def7-4bf4-90da-de4fef47f433"
Jul 29 18:34:01.251438 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:01.250938    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-152.ec2.internal" event={"ID":"2c2c229256a0f446f156df43183000bc","Type":"ContainerStarted","Data":"b15e37d36a811f55e9cf07cbc8654e41606b80a426fc408a09224a57afaa50a2"}
Jul 29 18:34:02.256372 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:02.256328    2559 generic.go:356] "Generic (PLEG): container finished" podID="2c2c229256a0f446f156df43183000bc" containerID="b15e37d36a811f55e9cf07cbc8654e41606b80a426fc408a09224a57afaa50a2" exitCode=0
Jul 29 18:34:02.256838 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:02.256440    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-152.ec2.internal" event={"ID":"2c2c229256a0f446f156df43183000bc","Type":"ContainerDied","Data":"b15e37d36a811f55e9cf07cbc8654e41606b80a426fc408a09224a57afaa50a2"}
Jul 29 18:34:03.145456 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:03.144580    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zxr5"
Jul 29 18:34:03.145456 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:03.144727    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zxr5" podUID="4b86e30e-def7-4bf4-90da-de4fef47f433"
Jul 29 18:34:03.145456 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:03.145160    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5w95"
Jul 29 18:34:03.145456 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:03.145264    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5w95" podUID="faa3a79b-75fb-4890-b169-09674f6d00db"
Jul 29 18:34:03.145456 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:03.145332    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh2dv"
Jul 29 18:34:03.145456 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:03.145422    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh2dv" podUID="df3fd46a-27e2-4557-bc40-09a9fbaa2e8f"
Jul 29 18:34:04.814410 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:04.814322    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/faa3a79b-75fb-4890-b169-09674f6d00db-original-pull-secret\") pod \"global-pull-secret-syncer-f5w95\" (UID: \"faa3a79b-75fb-4890-b169-09674f6d00db\") " pod="kube-system/global-pull-secret-syncer-f5w95"
Jul 29 18:34:04.814410 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:04.814400    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b86e30e-def7-4bf4-90da-de4fef47f433-metrics-certs\") pod \"network-metrics-daemon-9zxr5\" (UID: \"4b86e30e-def7-4bf4-90da-de4fef47f433\") " pod="openshift-multus/network-metrics-daemon-9zxr5"
Jul 29 18:34:04.814909 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:04.814508    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 18:34:04.814909 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:04.814556    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b86e30e-def7-4bf4-90da-de4fef47f433-metrics-certs podName:4b86e30e-def7-4bf4-90da-de4fef47f433 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:12.814541887 +0000 UTC m=+17.084310952 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b86e30e-def7-4bf4-90da-de4fef47f433-metrics-certs") pod "network-metrics-daemon-9zxr5" (UID: "4b86e30e-def7-4bf4-90da-de4fef47f433") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 18:34:04.814909 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:04.814853    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 18:34:04.814909 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:04.814895    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faa3a79b-75fb-4890-b169-09674f6d00db-original-pull-secret podName:faa3a79b-75fb-4890-b169-09674f6d00db nodeName:}" failed. No retries permitted until 2026-07-29 18:34:12.814878732 +0000 UTC m=+17.084647798 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/faa3a79b-75fb-4890-b169-09674f6d00db-original-pull-secret") pod "global-pull-secret-syncer-f5w95" (UID: "faa3a79b-75fb-4890-b169-09674f6d00db") : object "kube-system"/"original-pull-secret" not registered
Jul 29 18:34:04.915684 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:04.915639    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztfbt\" (UniqueName: \"kubernetes.io/projected/df3fd46a-27e2-4557-bc40-09a9fbaa2e8f-kube-api-access-ztfbt\") pod \"network-check-target-jh2dv\" (UID: \"df3fd46a-27e2-4557-bc40-09a9fbaa2e8f\") " pod="openshift-network-diagnostics/network-check-target-jh2dv"
Jul 29 18:34:04.915937 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:04.915833    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 18:34:04.915937 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:04.915860    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 18:34:04.915937 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:04.915876    2559 projected.go:196] Error preparing data for projected volume kube-api-access-ztfbt for pod openshift-network-diagnostics/network-check-target-jh2dv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 18:34:04.915937 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:04.915936    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df3fd46a-27e2-4557-bc40-09a9fbaa2e8f-kube-api-access-ztfbt podName:df3fd46a-27e2-4557-bc40-09a9fbaa2e8f nodeName:}" failed. No retries permitted until 2026-07-29 18:34:12.915916546 +0000 UTC m=+17.185685631 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ztfbt" (UniqueName: "kubernetes.io/projected/df3fd46a-27e2-4557-bc40-09a9fbaa2e8f-kube-api-access-ztfbt") pod "network-check-target-jh2dv" (UID: "df3fd46a-27e2-4557-bc40-09a9fbaa2e8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 18:34:05.144512 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:05.144471    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zxr5"
Jul 29 18:34:05.144699 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:05.144480    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5w95"
Jul 29 18:34:05.144699 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:05.144631    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zxr5" podUID="4b86e30e-def7-4bf4-90da-de4fef47f433"
Jul 29 18:34:05.144699 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:05.144492    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh2dv"
Jul 29 18:34:05.144934 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:05.144725    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5w95" podUID="faa3a79b-75fb-4890-b169-09674f6d00db"
Jul 29 18:34:05.144934 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:05.144770    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh2dv" podUID="df3fd46a-27e2-4557-bc40-09a9fbaa2e8f"
Jul 29 18:34:07.144775 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:07.144733    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zxr5"
Jul 29 18:34:07.145229 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:07.144733    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh2dv"
Jul 29 18:34:07.145229 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:07.144888    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zxr5" podUID="4b86e30e-def7-4bf4-90da-de4fef47f433"
Jul 29 18:34:07.145229 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:07.144737    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5w95"
Jul 29 18:34:07.145229 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:07.144955    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh2dv" podUID="df3fd46a-27e2-4557-bc40-09a9fbaa2e8f"
Jul 29 18:34:07.145229 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:07.145023    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5w95" podUID="faa3a79b-75fb-4890-b169-09674f6d00db"
Jul 29 18:34:09.144582 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:09.144544    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5w95"
Jul 29 18:34:09.144582 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:09.144579    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zxr5"
Jul 29 18:34:09.145098 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:09.144578    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh2dv"
Jul 29 18:34:09.145098 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:09.144685    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5w95" podUID="faa3a79b-75fb-4890-b169-09674f6d00db"
Jul 29 18:34:09.145098 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:09.144775    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zxr5" podUID="4b86e30e-def7-4bf4-90da-de4fef47f433"
Jul 29 18:34:09.145098 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:09.144855    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh2dv" podUID="df3fd46a-27e2-4557-bc40-09a9fbaa2e8f"
Jul 29 18:34:11.144731 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:11.144692    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh2dv"
Jul 29 18:34:11.145167 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:11.144692    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5w95"
Jul 29 18:34:11.145167 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:11.144826    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh2dv" podUID="df3fd46a-27e2-4557-bc40-09a9fbaa2e8f"
Jul 29 18:34:11.145167 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:11.144692    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zxr5"
Jul 29 18:34:11.145167 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:11.144894    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5w95" podUID="faa3a79b-75fb-4890-b169-09674f6d00db"
Jul 29 18:34:11.145167 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:11.144983    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zxr5" podUID="4b86e30e-def7-4bf4-90da-de4fef47f433"
Jul 29 18:34:12.879642 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:12.879606    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/faa3a79b-75fb-4890-b169-09674f6d00db-original-pull-secret\") pod \"global-pull-secret-syncer-f5w95\" (UID: \"faa3a79b-75fb-4890-b169-09674f6d00db\") " pod="kube-system/global-pull-secret-syncer-f5w95"
Jul 29 18:34:12.880154 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:12.879666    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b86e30e-def7-4bf4-90da-de4fef47f433-metrics-certs\") pod \"network-metrics-daemon-9zxr5\" (UID: \"4b86e30e-def7-4bf4-90da-de4fef47f433\") " pod="openshift-multus/network-metrics-daemon-9zxr5"
Jul 29 18:34:12.880154 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:12.879781    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 18:34:12.880154 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:12.879781    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 18:34:12.880154 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:12.879850    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b86e30e-def7-4bf4-90da-de4fef47f433-metrics-certs podName:4b86e30e-def7-4bf4-90da-de4fef47f433 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:28.879831328 +0000 UTC m=+33.149600407 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b86e30e-def7-4bf4-90da-de4fef47f433-metrics-certs") pod "network-metrics-daemon-9zxr5" (UID: "4b86e30e-def7-4bf4-90da-de4fef47f433") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 18:34:12.880154 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:12.879871    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faa3a79b-75fb-4890-b169-09674f6d00db-original-pull-secret podName:faa3a79b-75fb-4890-b169-09674f6d00db nodeName:}" failed. No retries permitted until 2026-07-29 18:34:28.879861189 +0000 UTC m=+33.149630257 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/faa3a79b-75fb-4890-b169-09674f6d00db-original-pull-secret") pod "global-pull-secret-syncer-f5w95" (UID: "faa3a79b-75fb-4890-b169-09674f6d00db") : object "kube-system"/"original-pull-secret" not registered
Jul 29 18:34:12.980888 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:12.980845    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztfbt\" (UniqueName: \"kubernetes.io/projected/df3fd46a-27e2-4557-bc40-09a9fbaa2e8f-kube-api-access-ztfbt\") pod \"network-check-target-jh2dv\" (UID: \"df3fd46a-27e2-4557-bc40-09a9fbaa2e8f\") " pod="openshift-network-diagnostics/network-check-target-jh2dv"
Jul 29 18:34:12.981065 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:12.980981    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 18:34:12.981065 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:12.980999    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 18:34:12.981065 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:12.981008    2559 projected.go:196] Error preparing data for projected volume kube-api-access-ztfbt for pod openshift-network-diagnostics/network-check-target-jh2dv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 18:34:12.981065 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:12.981060    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df3fd46a-27e2-4557-bc40-09a9fbaa2e8f-kube-api-access-ztfbt podName:df3fd46a-27e2-4557-bc40-09a9fbaa2e8f nodeName:}" failed. No retries permitted until 2026-07-29 18:34:28.981046193 +0000 UTC m=+33.250815272 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ztfbt" (UniqueName: "kubernetes.io/projected/df3fd46a-27e2-4557-bc40-09a9fbaa2e8f-kube-api-access-ztfbt") pod "network-check-target-jh2dv" (UID: "df3fd46a-27e2-4557-bc40-09a9fbaa2e8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 18:34:13.144098 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:13.144017    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zxr5"
Jul 29 18:34:13.144098 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:13.144016    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh2dv"
Jul 29 18:34:13.144302 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:13.144151    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zxr5" podUID="4b86e30e-def7-4bf4-90da-de4fef47f433"
Jul 29 18:34:13.144302 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:13.144210    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5w95"
Jul 29 18:34:13.144436 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:13.144324    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh2dv" podUID="df3fd46a-27e2-4557-bc40-09a9fbaa2e8f"
Jul 29 18:34:13.144436 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:13.144426    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5w95" podUID="faa3a79b-75fb-4890-b169-09674f6d00db"
Jul 29 18:34:15.144678 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:15.144635    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh2dv"
Jul 29 18:34:15.145268 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:15.144640    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5w95"
Jul 29 18:34:15.145268 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:15.144755    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh2dv" podUID="df3fd46a-27e2-4557-bc40-09a9fbaa2e8f"
Jul 29 18:34:15.145268 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:15.144854    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5w95" podUID="faa3a79b-75fb-4890-b169-09674f6d00db"
Jul 29 18:34:15.145268 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:15.144650    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zxr5"
Jul 29 18:34:15.145268 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:15.144949    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zxr5" podUID="4b86e30e-def7-4bf4-90da-de4fef47f433"
Jul 29 18:34:17.144830 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:17.144790    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zxr5"
Jul 29 18:34:17.145286 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:17.144798    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh2dv"
Jul 29 18:34:17.145286 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:17.144961    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zxr5" podUID="4b86e30e-def7-4bf4-90da-de4fef47f433"
Jul 29 18:34:17.145286 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:17.144798    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5w95"
Jul 29 18:34:17.145286 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:17.145013    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh2dv" podUID="df3fd46a-27e2-4557-bc40-09a9fbaa2e8f"
Jul 29 18:34:17.145286 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:17.145089    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5w95" podUID="faa3a79b-75fb-4890-b169-09674f6d00db"
Jul 29 18:34:18.289008 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:18.288606    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bp28x" event={"ID":"6a62d29d-5bf6-444f-bd28-bf2d1e54ac87","Type":"ContainerStarted","Data":"55101a75ed01479a6c55c09ef531e44cbbb89aaba7b5700de68109bdf23c4ecf"}
Jul 29 18:34:18.290223 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:18.290200    2559 generic.go:356] "Generic (PLEG): container finished" podID="6dad0cba-079a-4ec1-9f2b-8399f34fb80c" containerID="559c464a63d5b673bcefd32e48ff3c25aec5eef8407e706d74bdced54242c456" exitCode=0
Jul 29 18:34:18.290326 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:18.290269    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wgbf" event={"ID":"6dad0cba-079a-4ec1-9f2b-8399f34fb80c","Type":"ContainerDied","Data":"559c464a63d5b673bcefd32e48ff3c25aec5eef8407e706d74bdced54242c456"}
Jul 29 18:34:18.291765 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:18.291744    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qjztn" event={"ID":"30397e41-d823-4e74-9485-66f74e907f8b","Type":"ContainerStarted","Data":"076e168440c8dd5ce1f75d7665d293fb3a29c8865d604a889c143239164882fa"}
Jul 29 18:34:18.293424 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:18.293345    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6vd4x" event={"ID":"37ec2c6c-1216-4d35-8d37-95071fddf90a","Type":"ContainerStarted","Data":"333c313deeb4f0f30402e8eb7e12e8a85635d8f0541adedad7d3dfceda711ddd"}
Jul 29 18:34:18.294797 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:18.294777    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd95n" event={"ID":"8fa7a55e-1003-4cc1-918e-655ccd91f071","Type":"ContainerStarted","Data":"a9539dda48e053d877f8a541f372833ab9c5da2ebc72cb549465f38974a31e29"}
Jul 29 18:34:18.297164 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:18.297146    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjdrx_60f444e1-7e31-443c-8ec7-a4d04c64bf4b/ovn-acl-logging/0.log"
Jul 29 18:34:18.297531 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:18.297509    2559 generic.go:356] "Generic (PLEG): container finished" podID="60f444e1-7e31-443c-8ec7-a4d04c64bf4b" containerID="98a7d6401d0c99d7073679bc3530a4dfcaf9e97c8a1a4a25f7ec9194aabad8eb" exitCode=1
Jul 29 18:34:18.297612 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:18.297579    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx" event={"ID":"60f444e1-7e31-443c-8ec7-a4d04c64bf4b","Type":"ContainerStarted","Data":"26ed0d341e8bc32aedec6a9a96df7f5962976b04187d5f2d8b9cf3891279d21b"}
Jul 29 18:34:18.297660 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:18.297650    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx" event={"ID":"60f444e1-7e31-443c-8ec7-a4d04c64bf4b","Type":"ContainerStarted","Data":"4e145c48850069c4afd5273380931a25ae66d0576e6e1e4c7e39a6048a78844b"}
Jul 29 18:34:18.297702 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:18.297680    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx" event={"ID":"60f444e1-7e31-443c-8ec7-a4d04c64bf4b","Type":"ContainerStarted","Data":"fc5100cadb96e678e8e8f4691faebe15873dca7dd8d151d8cdd692f7733a2eca"}
Jul 29 18:34:18.297736 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:18.297714    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx" event={"ID":"60f444e1-7e31-443c-8ec7-a4d04c64bf4b","Type":"ContainerDied","Data":"98a7d6401d0c99d7073679bc3530a4dfcaf9e97c8a1a4a25f7ec9194aabad8eb"}
Jul 29 18:34:18.297776 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:18.297758    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx" event={"ID":"60f444e1-7e31-443c-8ec7-a4d04c64bf4b","Type":"ContainerStarted","Data":"c2f0e1854a2ad458bf149149546f9f6a4edb04480edaca51555511c2fe4e6321"}
Jul 29 18:34:18.298915 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:18.298896    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-152.ec2.internal" event={"ID":"8923e83396bb7ee7f5fbbb29dfb80487","Type":"ContainerStarted","Data":"2573ad5a963d7edd2afd50d2d2100d3230d463e484c7e6d54beb169e3916e7c4"}
Jul 29 18:34:18.300707 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:18.300688    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-152.ec2.internal" event={"ID":"2c2c229256a0f446f156df43183000bc","Type":"ContainerStarted","Data":"e93057cf6c8d4839cdeddbed35bfddb4230d380736166febf52448b303c002d5"}
Jul 29 18:34:18.302072 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:18.302047    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ldlwt" event={"ID":"2201b8bf-575c-4958-b909-29ff288ed75e","Type":"ContainerStarted","Data":"c6ae5d7c30ace0b02de95a1bb50a1cc40d422af2f1f46567cae1cdaa74563895"}
Jul 29 18:34:18.303771 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:18.303748    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t4wjl" event={"ID":"b03a3682-95be-4852-b28a-8e19fee4cbe7","Type":"ContainerStarted","Data":"9c73074721d7b123ee8bc86edd10ca18d98101f0f37637c036d1dbbaf2cb5461"}
Jul 29 18:34:18.305218 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:18.305184    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bp28x" podStartSLOduration=2.388539911 podStartE2EDuration="22.305173343s" totalImagesPullingTime="19.916633432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:18.305153181 +0000 UTC m=+22.574922270" watchObservedRunningTime="2026-07-29 18:34:18.305173343 +0000 UTC m=+22.574942431"
Jul 29 18:34:18.338063 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:18.338006    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-152.ec2.internal" podStartSLOduration=22.337989731 podStartE2EDuration="22.337989731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:33:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:18.337694887 +0000 UTC m=+22.607463975" watchObservedRunningTime="2026-07-29 18:34:18.337989731 +0000 UTC m=+22.607758819"
Jul 29 18:34:18.364594 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:18.364541    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ldlwt" podStartSLOduration=10.969902837 podStartE2EDuration="22.364523695s" totalImagesPullingTime="11.394620858s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:18.364523228 +0000 UTC m=+22.634292315" watchObservedRunningTime="2026-07-29 18:34:18.364523695 +0000 UTC m=+22.634292801"
Jul 29 18:34:18.365202 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:18.365175    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-t4wjl" podStartSLOduration=2.5282961779999997 podStartE2EDuration="22.365165546s" totalImagesPullingTime="19.836869368s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:18.351062258 +0000 UTC m=+22.620831347" watchObservedRunningTime="2026-07-29 18:34:18.365165546 +0000 UTC m=+22.634934692"
Jul 29 18:34:18.380809 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:18.380742    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qjztn" podStartSLOduration=2.417008277 podStartE2EDuration="22.380724012s" totalImagesPullingTime="19.963715735s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:18.380365588 +0000 UTC m=+22.650134677" watchObservedRunningTime="2026-07-29 18:34:18.380724012 +0000 UTC m=+22.650493100"
Jul 29 18:34:18.439255 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:18.439203    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-152.ec2.internal" podStartSLOduration=22.439187267 podStartE2EDuration="22.439187267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:33:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:18.396971636 +0000 UTC m=+22.666740718" watchObservedRunningTime="2026-07-29 18:34:18.439187267 +0000 UTC m=+22.708956417"
Jul 29 18:34:18.439449 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:18.439428    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6vd4x" podStartSLOduration=2.4686830779999998 podStartE2EDuration="22.439423104s" totalImagesPullingTime="19.970740026s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:18.438681821 +0000 UTC m=+22.708450908" watchObservedRunningTime="2026-07-29 18:34:18.439423104 +0000 UTC m=+22.709192192"
Jul 29 18:34:19.144672 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:19.144640    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zxr5"
Jul 29 18:34:19.144854 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:19.144641    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh2dv"
Jul 29 18:34:19.144854 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:19.144775    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zxr5" podUID="4b86e30e-def7-4bf4-90da-de4fef47f433"
Jul 29 18:34:19.144854 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:19.144797    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5w95"
Jul 29 18:34:19.145030 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:19.144868    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh2dv" podUID="df3fd46a-27e2-4557-bc40-09a9fbaa2e8f"
Jul 29 18:34:19.145030 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:19.144958    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5w95" podUID="faa3a79b-75fb-4890-b169-09674f6d00db"
Jul 29 18:34:19.310396 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:19.310292    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjdrx_60f444e1-7e31-443c-8ec7-a4d04c64bf4b/ovn-acl-logging/0.log"
Jul 29 18:34:19.310825 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:19.310712    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx" event={"ID":"60f444e1-7e31-443c-8ec7-a4d04c64bf4b","Type":"ContainerStarted","Data":"e9a3d28ecf8259d71d88860b1cfe393fc4a4533b8fd474c3aaf95730aedd8558"}
Jul 29 18:34:19.312794 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:19.312680    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-b6rhb" event={"ID":"44c368d9-c4c2-481e-bbca-3bfa6eae44d1","Type":"ContainerStarted","Data":"5b5bfcea90a6200e67016467b95a5dda374f4537dad558bc9a558b57d7cf9833"}
Jul 29 18:34:19.335949 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:19.335889    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-b6rhb" podStartSLOduration=3.506259625 podStartE2EDuration="23.335868912s" totalImagesPullingTime="19.829609287s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:19.335249373 +0000 UTC m=+23.605018462" watchObservedRunningTime="2026-07-29 18:34:19.335868912 +0000 UTC m=+23.605638000"
Jul 29 18:34:19.792870 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:19.792845    2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 29 18:34:20.108920 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:20.108813    2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T18:34:19.792862831Z","UUID":"c1772ab7-ccc6-4a3a-9d0b-35882759d249","Handler":null,"Name":"","Endpoint":""}
Jul 29 18:34:20.110478 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:20.110448    2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 29 18:34:20.110599 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:20.110493    2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 29 18:34:20.317593 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:20.317551    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd95n" event={"ID":"8fa7a55e-1003-4cc1-918e-655ccd91f071","Type":"ContainerStarted","Data":"67a09e50765bbc82340bef9f31b25b9e988d4a2f800998c6427ff594360e2447"}
Jul 29 18:34:20.854692 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:20.854655    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bp28x"
Jul 29 18:34:20.855460 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:20.855426    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bp28x"
Jul 29 18:34:21.144009 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:21.143919    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5w95"
Jul 29 18:34:21.144171 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:21.144013    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh2dv"
Jul 29 18:34:21.144171 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:21.144125    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zxr5"
Jul 29 18:34:21.144171 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:21.144141    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5w95" podUID="faa3a79b-75fb-4890-b169-09674f6d00db"
Jul 29 18:34:21.144339 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:21.144269    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zxr5" podUID="4b86e30e-def7-4bf4-90da-de4fef47f433"
Jul 29 18:34:21.144418 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:21.144351    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh2dv" podUID="df3fd46a-27e2-4557-bc40-09a9fbaa2e8f"
Jul 29 18:34:21.323471 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:21.323439    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjdrx_60f444e1-7e31-443c-8ec7-a4d04c64bf4b/ovn-acl-logging/0.log"
Jul 29 18:34:21.324032 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:21.323999    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx" event={"ID":"60f444e1-7e31-443c-8ec7-a4d04c64bf4b","Type":"ContainerStarted","Data":"f4c673ad2cf24f1c082468091c64fd8af555e1bb095e5a57ecf08752c76ba9fa"}
Jul 29 18:34:21.324239 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:21.324221    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bp28x"
Jul 29 18:34:21.324873 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:21.324853    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bp28x"
Jul 29 18:34:23.144740 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:23.144426    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zxr5"
Jul 29 18:34:23.145494 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:23.144432    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5w95"
Jul 29 18:34:23.145494 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:23.144832    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zxr5" podUID="4b86e30e-def7-4bf4-90da-de4fef47f433"
Jul 29 18:34:23.145494 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:23.144464    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh2dv"
Jul 29 18:34:23.145494 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:23.144903    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5w95" podUID="faa3a79b-75fb-4890-b169-09674f6d00db"
Jul 29 18:34:23.145494 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:23.144952    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh2dv" podUID="df3fd46a-27e2-4557-bc40-09a9fbaa2e8f"
Jul 29 18:34:23.330915 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:23.330882    2559 generic.go:356] "Generic (PLEG): container finished" podID="6dad0cba-079a-4ec1-9f2b-8399f34fb80c" containerID="d1c2f72f6683eb3b436e354b7ed3f1b57c263965f0bbbcde1133865a5699314c" exitCode=0
Jul 29 18:34:23.331103 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:23.330958    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wgbf" event={"ID":"6dad0cba-079a-4ec1-9f2b-8399f34fb80c","Type":"ContainerDied","Data":"d1c2f72f6683eb3b436e354b7ed3f1b57c263965f0bbbcde1133865a5699314c"}
Jul 29 18:34:23.333205 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:23.333178    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd95n" event={"ID":"8fa7a55e-1003-4cc1-918e-655ccd91f071","Type":"ContainerStarted","Data":"c41df7b991f15c17ba88db5ceb93d0ea81cc74a7c081c899fa4747b70683839b"}
Jul 29 18:34:23.336638 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:23.336619    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjdrx_60f444e1-7e31-443c-8ec7-a4d04c64bf4b/ovn-acl-logging/0.log"
Jul 29 18:34:23.336994 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:23.336970    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx" event={"ID":"60f444e1-7e31-443c-8ec7-a4d04c64bf4b","Type":"ContainerStarted","Data":"cd54e564de5724a28c00883e65ab9b732850669b0d78fce0514047a71f18d368"}
Jul 29 18:34:23.337357 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:23.337340    2559 scope.go:117] "RemoveContainer" containerID="98a7d6401d0c99d7073679bc3530a4dfcaf9e97c8a1a4a25f7ec9194aabad8eb"
Jul 29 18:34:23.392974 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:23.392918    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gd95n" podStartSLOduration=2.685054919 podStartE2EDuration="27.392901924s" totalImagesPullingTime="24.707847005s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:33:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:23.392661726 +0000 UTC m=+27.662430814" watchObservedRunningTime="2026-07-29 18:34:23.392901924 +0000 UTC m=+27.662671009"
Jul 29 18:34:24.343037 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:24.342595    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjdrx_60f444e1-7e31-443c-8ec7-a4d04c64bf4b/ovn-acl-logging/0.log"
Jul 29 18:34:24.343288 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:24.343042    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx" event={"ID":"60f444e1-7e31-443c-8ec7-a4d04c64bf4b","Type":"ContainerStarted","Data":"8f98348e36d787dd0ce5d0aa5431db560cab4b6ceacc01367037c107e3a192cb"}
Jul 29 18:34:24.343344 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:24.343329    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:34:24.343651 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:24.343634    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:34:24.343748 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:24.343702    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:34:24.360165 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:24.360139    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:34:24.360994 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:24.360972    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:34:24.437812 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:24.437752    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx" podStartSLOduration=8.095059875 podStartE2EDuration="28.437730106s" totalImagesPullingTime="20.342670231s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:24.375432154 +0000 UTC m=+28.645201242" watchObservedRunningTime="2026-07-29 18:34:24.437730106 +0000 UTC m=+28.707499195"
Jul 29 18:34:25.052850 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:25.052797    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f5w95"]
Jul 29 18:34:25.053081 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:25.052975    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5w95"
Jul 29 18:34:25.053081 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:25.053065    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5w95" podUID="faa3a79b-75fb-4890-b169-09674f6d00db"
Jul 29 18:34:25.059424 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:25.059026    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jh2dv"]
Jul 29 18:34:25.059424 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:25.059208    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh2dv"
Jul 29 18:34:25.059424 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:25.059374    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh2dv" podUID="df3fd46a-27e2-4557-bc40-09a9fbaa2e8f"
Jul 29 18:34:25.059905 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:25.059882    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9zxr5"]
Jul 29 18:34:25.060345 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:25.060331    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zxr5"
Jul 29 18:34:25.060484 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:25.060461    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zxr5" podUID="4b86e30e-def7-4bf4-90da-de4fef47f433"
Jul 29 18:34:25.347767 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:25.347728    2559 generic.go:356] "Generic (PLEG): container finished" podID="6dad0cba-079a-4ec1-9f2b-8399f34fb80c" containerID="b796216e4be1794aeab82aee6a022c4d4afc63181198869e374863a5cdddef51" exitCode=0
Jul 29 18:34:25.348236 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:25.347817    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wgbf" event={"ID":"6dad0cba-079a-4ec1-9f2b-8399f34fb80c","Type":"ContainerDied","Data":"b796216e4be1794aeab82aee6a022c4d4afc63181198869e374863a5cdddef51"}
Jul 29 18:34:27.144138 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:27.144104    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh2dv"
Jul 29 18:34:27.144555 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:27.144192    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5w95"
Jul 29 18:34:27.144555 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:27.144216    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zxr5"
Jul 29 18:34:27.144555 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:27.144308    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh2dv" podUID="df3fd46a-27e2-4557-bc40-09a9fbaa2e8f"
Jul 29 18:34:27.144555 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:27.144398    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zxr5" podUID="4b86e30e-def7-4bf4-90da-de4fef47f433"
Jul 29 18:34:27.144555 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:27.144486    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5w95" podUID="faa3a79b-75fb-4890-b169-09674f6d00db"
Jul 29 18:34:28.357451 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:28.357242    2559 generic.go:356] "Generic (PLEG): container finished" podID="6dad0cba-079a-4ec1-9f2b-8399f34fb80c" containerID="26a9ee780bb0717d88ee1c329ea44ed0725d830915ff2698eb36032c4be3df07" exitCode=0
Jul 29 18:34:28.357879 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:28.357309    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wgbf" event={"ID":"6dad0cba-079a-4ec1-9f2b-8399f34fb80c","Type":"ContainerDied","Data":"26a9ee780bb0717d88ee1c329ea44ed0725d830915ff2698eb36032c4be3df07"}
Jul 29 18:34:28.897083 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:28.897042    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/faa3a79b-75fb-4890-b169-09674f6d00db-original-pull-secret\") pod \"global-pull-secret-syncer-f5w95\" (UID: \"faa3a79b-75fb-4890-b169-09674f6d00db\") " pod="kube-system/global-pull-secret-syncer-f5w95"
Jul 29 18:34:28.897292 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:28.897113    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b86e30e-def7-4bf4-90da-de4fef47f433-metrics-certs\") pod \"network-metrics-daemon-9zxr5\" (UID: \"4b86e30e-def7-4bf4-90da-de4fef47f433\") " pod="openshift-multus/network-metrics-daemon-9zxr5"
Jul 29 18:34:28.897292 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:28.897211    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 18:34:28.897292 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:28.897228    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 18:34:28.897292 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:28.897279    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faa3a79b-75fb-4890-b169-09674f6d00db-original-pull-secret podName:faa3a79b-75fb-4890-b169-09674f6d00db nodeName:}" failed. No retries permitted until 2026-07-29 18:35:00.897263597 +0000 UTC m=+65.167032665 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/faa3a79b-75fb-4890-b169-09674f6d00db-original-pull-secret") pod "global-pull-secret-syncer-f5w95" (UID: "faa3a79b-75fb-4890-b169-09674f6d00db") : object "kube-system"/"original-pull-secret" not registered
Jul 29 18:34:28.897292 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:28.897295    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b86e30e-def7-4bf4-90da-de4fef47f433-metrics-certs podName:4b86e30e-def7-4bf4-90da-de4fef47f433 nodeName:}" failed. No retries permitted until 2026-07-29 18:35:00.897287779 +0000 UTC m=+65.167056844 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b86e30e-def7-4bf4-90da-de4fef47f433-metrics-certs") pod "network-metrics-daemon-9zxr5" (UID: "4b86e30e-def7-4bf4-90da-de4fef47f433") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 18:34:28.998111 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:28.998071    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztfbt\" (UniqueName: \"kubernetes.io/projected/df3fd46a-27e2-4557-bc40-09a9fbaa2e8f-kube-api-access-ztfbt\") pod \"network-check-target-jh2dv\" (UID: \"df3fd46a-27e2-4557-bc40-09a9fbaa2e8f\") " pod="openshift-network-diagnostics/network-check-target-jh2dv"
Jul 29 18:34:28.998298 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:28.998209    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 18:34:28.998298 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:28.998229    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 18:34:28.998298 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:28.998242    2559 projected.go:196] Error preparing data for projected volume kube-api-access-ztfbt for pod openshift-network-diagnostics/network-check-target-jh2dv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 18:34:28.998298 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:28.998298    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df3fd46a-27e2-4557-bc40-09a9fbaa2e8f-kube-api-access-ztfbt podName:df3fd46a-27e2-4557-bc40-09a9fbaa2e8f nodeName:}" failed. No retries permitted until 2026-07-29 18:35:00.998282028 +0000 UTC m=+65.268051113 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ztfbt" (UniqueName: "kubernetes.io/projected/df3fd46a-27e2-4557-bc40-09a9fbaa2e8f-kube-api-access-ztfbt") pod "network-check-target-jh2dv" (UID: "df3fd46a-27e2-4557-bc40-09a9fbaa2e8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 18:34:29.144463 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:29.144424    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zxr5"
Jul 29 18:34:29.144463 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:29.144464    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh2dv"
Jul 29 18:34:29.144710 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:29.144424    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5w95"
Jul 29 18:34:29.144710 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:29.144569    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zxr5" podUID="4b86e30e-def7-4bf4-90da-de4fef47f433"
Jul 29 18:34:29.144710 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:29.144637    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5w95" podUID="faa3a79b-75fb-4890-b169-09674f6d00db"
Jul 29 18:34:29.144865 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:29.144705    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh2dv" podUID="df3fd46a-27e2-4557-bc40-09a9fbaa2e8f"
Jul 29 18:34:30.532591 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.532559    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-152.ec2.internal" event="NodeReady"
Jul 29 18:34:30.533036 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.532701    2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 29 18:34:30.568332 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.568289    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-j76ql"]
Jul 29 18:34:30.572971 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.572938    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6cb6fb9448-jbzkl"]
Jul 29 18:34:30.573156 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.573053    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-j76ql"
Jul 29 18:34:30.577268 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.576328    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 29 18:34:30.577268 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.576653    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 29 18:34:30.577268 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.577028    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 29 18:34:30.577268 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.577257    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 29 18:34:30.577714 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.577694    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-dbq4w\""
Jul 29 18:34:30.577976 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.577950    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-jxmfh"]
Jul 29 18:34:30.578692 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.578670    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cb6fb9448-jbzkl"
Jul 29 18:34:30.581864 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.581846    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zc9c4\""
Jul 29 18:34:30.581996 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.581890    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 29 18:34:30.582303 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.582286    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 29 18:34:30.582583 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.582561    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-ljn4r"]
Jul 29 18:34:30.583203 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.583182    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jxmfh"
Jul 29 18:34:30.583563 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.583539    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 29 18:34:30.584589 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.584569    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 29 18:34:30.586566 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.586547    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7688c5db95-gcqb5"]
Jul 29 18:34:30.586983 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.586682    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 29 18:34:30.587077 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.586750    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-ljn4r"
Jul 29 18:34:30.587929 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.587858    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 29 18:34:30.587929 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.587874    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 29 18:34:30.588123 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.587858    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-2bqxr\""
Jul 29 18:34:30.588123 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.588029    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 29 18:34:30.589233 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.589210    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 29 18:34:30.591275 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.591249    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-hb5lw"]
Jul 29 18:34:30.591828 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.591810    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7688c5db95-gcqb5"
Jul 29 18:34:30.592183 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.592165    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-4d45j\""
Jul 29 18:34:30.592413 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.592373    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 29 18:34:30.592507 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.592429    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 29 18:34:30.595549 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.595325    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-v9nsr"]
Jul 29 18:34:30.595911 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.595891    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-8qgnb\""
Jul 29 18:34:30.596019 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.595894    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 29 18:34:30.596172 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.596157    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 29 18:34:30.596231 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.596209    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-hb5lw"
Jul 29 18:34:30.596499 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.596416    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 29 18:34:30.598865 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.598845    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 29 18:34:30.599246 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.599226    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-j76ql"]
Jul 29 18:34:30.599328 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.599265    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-95vjs"]
Jul 29 18:34:30.600789 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.600762    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-v9nsr"
Jul 29 18:34:30.602339 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.601496    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 29 18:34:30.606188 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.601550    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 29 18:34:30.606460 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.601648    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 29 18:34:30.606625 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.601664    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 29 18:34:30.606772 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.601773    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-thcsx\""
Jul 29 18:34:30.607267 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.601818    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 29 18:34:30.608721 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.607918    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 29 18:34:30.608721 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.608405    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-45klb\""
Jul 29 18:34:30.608721 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.608687    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 29 18:34:30.610033 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.609296    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 29 18:34:30.610033 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.609532    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 29 18:34:30.610660 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.610635    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qhj2h"]
Jul 29 18:34:30.613620 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.613590    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-95vjs"
Jul 29 18:34:30.619346 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.617542    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 29 18:34:30.620288 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.619970    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2xgwr\""
Jul 29 18:34:30.620288 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.620271    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 29 18:34:30.622805 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.622776    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-4lqlg"]
Jul 29 18:34:30.623639 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.623618    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qhj2h"
Jul 29 18:34:30.627655 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.627635    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 29 18:34:30.627879 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.627863    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 29 18:34:30.628249 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.628231    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-84hqk\""
Jul 29 18:34:30.628342 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.628231    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 29 18:34:30.629566 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.629547    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-95ttd"]
Jul 29 18:34:30.630144 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.629677    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-4lqlg"
Jul 29 18:34:30.630565 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.629932    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 29 18:34:30.633014 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.632995    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 18:34:30.633167 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.633149    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-wvh4v\""
Jul 29 18:34:30.633442 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.633421    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 18:34:30.634694 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.634675    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nj275"]
Jul 29 18:34:30.634867 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.634849    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-95ttd"
Jul 29 18:34:30.637592 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.637574    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-8khrx\""
Jul 29 18:34:30.637781 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.637629    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 29 18:34:30.637873 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.637825    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 29 18:34:30.637873 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.637649    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 29 18:34:30.638110 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.638094    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 29 18:34:30.638811 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.638794    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d55599b76-4k8r9"]
Jul 29 18:34:30.640220 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.640197    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nj275"
Jul 29 18:34:30.642460 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.642438    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 29 18:34:30.642540 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.642467    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 29 18:34:30.642970 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.642947    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-hhd5f\""
Jul 29 18:34:30.643069 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.642977    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 29 18:34:30.643156 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.643138    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58dbb54c4d-zmrbt"]
Jul 29 18:34:30.643207 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.643158    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 29 18:34:30.643367 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.643347    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d55599b76-4k8r9"
Jul 29 18:34:30.646198 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.646175    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 29 18:34:30.646500 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.646460    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 29 18:34:30.646591 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.646568    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 29 18:34:30.646658 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.646615    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-xsp9t\""
Jul 29 18:34:30.646731 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.646716    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 29 18:34:30.647156 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.647138    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-v9nsr"]
Jul 29 18:34:30.647242 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.647175    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-ljn4r"]
Jul 29 18:34:30.647242 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.647199    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cb7489487-kkmn9"]
Jul 29 18:34:30.647347 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.647261    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58dbb54c4d-zmrbt"
Jul 29 18:34:30.649679 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.649655    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 29 18:34:30.649951 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.649931    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 29 18:34:30.650021 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.649933    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 29 18:34:30.650021 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.649992    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 29 18:34:30.650505 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.650489    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-jxmfh"]
Jul 29 18:34:30.650566 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.650518    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-hb5lw"]
Jul 29 18:34:30.650566 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.650542    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7688c5db95-gcqb5"]
Jul 29 18:34:30.650566 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.650563    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d55599b76-4k8r9"]
Jul 29 18:34:30.650678 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.650576    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6cb6fb9448-jbzkl"]
Jul 29 18:34:30.650678 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.650594    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nj275"]
Jul 29 18:34:30.650678 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.650616    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-4lqlg"]
Jul 29 18:34:30.650678 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.650633    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qhj2h"]
Jul 29 18:34:30.650678 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.650650    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cb7489487-kkmn9"]
Jul 29 18:34:30.650678 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.650667    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-95ttd"]
Jul 29 18:34:30.650937 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.650683    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-95vjs"]
Jul 29 18:34:30.650937 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.650705    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58dbb54c4d-zmrbt"]
Jul 29 18:34:30.650937 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.650779    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cb7489487-kkmn9"
Jul 29 18:34:30.655324 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.655303    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 29 18:34:30.713979 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.713948    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c284fbd5-a70f-4085-948c-6a2392502e6f-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-jxmfh\" (UID: \"c284fbd5-a70f-4085-948c-6a2392502e6f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jxmfh"
Jul 29 18:34:30.714141 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.714015    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66a293fb-92cb-4908-bf90-d97ed0d831a6-service-ca-bundle\") pod \"router-default-7688c5db95-gcqb5\" (UID: \"66a293fb-92cb-4908-bf90-d97ed0d831a6\") " pod="openshift-ingress/router-default-7688c5db95-gcqb5"
Jul 29 18:34:30.714141 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.714045    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/66a293fb-92cb-4908-bf90-d97ed0d831a6-default-certificate\") pod \"router-default-7688c5db95-gcqb5\" (UID: \"66a293fb-92cb-4908-bf90-d97ed0d831a6\") " pod="openshift-ingress/router-default-7688c5db95-gcqb5"
Jul 29 18:34:30.714141 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.714068    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7f8b4737-d4c2-4598-bd40-208015542831-trusted-ca\") pod \"console-operator-584dcb6b54-v9nsr\" (UID: \"7f8b4737-d4c2-4598-bd40-208015542831\") " pod="openshift-console-operator/console-operator-584dcb6b54-v9nsr"
Jul 29 18:34:30.714141 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.714100    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a84d718f-459d-4e2e-9c9e-d08d1500b145-serving-cert\") pod \"insights-operator-5885fdb759-j76ql\" (UID: \"a84d718f-459d-4e2e-9c9e-d08d1500b145\") " pod="openshift-insights/insights-operator-5885fdb759-j76ql"
Jul 29 18:34:30.714885 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.714436    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e281e146-0b8a-4358-a98b-7f577f4ff08b-image-registry-private-configuration\") pod \"image-registry-6cb6fb9448-jbzkl\" (UID: \"e281e146-0b8a-4358-a98b-7f577f4ff08b\") " pod="openshift-image-registry/image-registry-6cb6fb9448-jbzkl"
Jul 29 18:34:30.714885 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.714495    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7f8b4737-d4c2-4598-bd40-208015542831-config\") pod \"console-operator-584dcb6b54-v9nsr\" (UID: \"7f8b4737-d4c2-4598-bd40-208015542831\") " pod="openshift-console-operator/console-operator-584dcb6b54-v9nsr"
Jul 29 18:34:30.714885 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.714533    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/05f777ff-5b55-41e4-b440-1705c2c3842e-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nj275\" (UID: \"05f777ff-5b55-41e4-b440-1705c2c3842e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nj275"
Jul 29 18:34:30.714885 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.714704    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc87b3c5-0b72-4fa2-9f87-ca3d4d6da614-cert\") pod \"ingress-canary-qhj2h\" (UID: \"cc87b3c5-0b72-4fa2-9f87-ca3d4d6da614\") " pod="openshift-ingress-canary/ingress-canary-qhj2h"
Jul 29 18:34:30.714885 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.714737    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmbf8\" (UniqueName: \"kubernetes.io/projected/4b692565-aef0-416f-8fad-2c74bca55d4d-kube-api-access-fmbf8\") pod \"volume-data-source-validator-549c6d54cf-ljn4r\" (UID: \"4b692565-aef0-416f-8fad-2c74bca55d4d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-ljn4r"
Jul 29 18:34:30.714885 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.714787    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxqgs\" (UniqueName: \"kubernetes.io/projected/26f43ad8-a7b4-4140-9ee0-7e9d55edf035-kube-api-access-pxqgs\") pod \"cluster-samples-operator-6748489656-hb5lw\" (UID: \"26f43ad8-a7b4-4140-9ee0-7e9d55edf035\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-hb5lw"
Jul 29 18:34:30.714885 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.714827    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e281e146-0b8a-4358-a98b-7f577f4ff08b-bound-sa-token\") pod \"image-registry-6cb6fb9448-jbzkl\" (UID: \"e281e146-0b8a-4358-a98b-7f577f4ff08b\") " pod="openshift-image-registry/image-registry-6cb6fb9448-jbzkl"
Jul 29 18:34:30.714885 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.714850    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a84d718f-459d-4e2e-9c9e-d08d1500b145-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-j76ql\" (UID: \"a84d718f-459d-4e2e-9c9e-d08d1500b145\") " pod="openshift-insights/insights-operator-5885fdb759-j76ql"
Jul 29 18:34:30.714885 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.714872    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e281e146-0b8a-4358-a98b-7f577f4ff08b-registry-certificates\") pod \"image-registry-6cb6fb9448-jbzkl\" (UID: \"e281e146-0b8a-4358-a98b-7f577f4ff08b\") " pod="openshift-image-registry/image-registry-6cb6fb9448-jbzkl"
Jul 29 18:34:30.715334 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.714896    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cad930eb-2903-4c78-b4c4-3bcc190d199d-serving-cert\") pod \"service-ca-operator-5d6c75f84f-95ttd\" (UID: \"cad930eb-2903-4c78-b4c4-3bcc190d199d\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-95ttd"
Jul 29 18:34:30.715334 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.714920    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e281e146-0b8a-4358-a98b-7f577f4ff08b-ca-trust-extracted\") pod \"image-registry-6cb6fb9448-jbzkl\" (UID: \"e281e146-0b8a-4358-a98b-7f577f4ff08b\") " pod="openshift-image-registry/image-registry-6cb6fb9448-jbzkl"
Jul 29 18:34:30.715334 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.714947    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a84d718f-459d-4e2e-9c9e-d08d1500b145-snapshots\") pod \"insights-operator-5885fdb759-j76ql\" (UID: \"a84d718f-459d-4e2e-9c9e-d08d1500b145\") " pod="openshift-insights/insights-operator-5885fdb759-j76ql"
Jul 29 18:34:30.715334 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.714969    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7f8b4737-d4c2-4598-bd40-208015542831-serving-cert\") pod \"console-operator-584dcb6b54-v9nsr\" (UID: \"7f8b4737-d4c2-4598-bd40-208015542831\") " pod="openshift-console-operator/console-operator-584dcb6b54-v9nsr"
Jul 29 18:34:30.715334 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.714992    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c854cbbc-3a2c-46ca-adf3-86fc3028bd7c-config-volume\") pod \"dns-default-95vjs\" (UID: \"c854cbbc-3a2c-46ca-adf3-86fc3028bd7c\") " pod="openshift-dns/dns-default-95vjs"
Jul 29 18:34:30.715334 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.715015    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnccl\" (UniqueName: \"kubernetes.io/projected/c854cbbc-3a2c-46ca-adf3-86fc3028bd7c-kube-api-access-vnccl\") pod \"dns-default-95vjs\" (UID: \"c854cbbc-3a2c-46ca-adf3-86fc3028bd7c\") " pod="openshift-dns/dns-default-95vjs"
Jul 29 18:34:30.715334 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.715031    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e281e146-0b8a-4358-a98b-7f577f4ff08b-trusted-ca\") pod \"image-registry-6cb6fb9448-jbzkl\" (UID: \"e281e146-0b8a-4358-a98b-7f577f4ff08b\") " pod="openshift-image-registry/image-registry-6cb6fb9448-jbzkl"
Jul 29 18:34:30.715334 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.715046    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq6r7\" (UniqueName: \"kubernetes.io/projected/98400583-3b23-4bf0-be1f-54aa621542a0-kube-api-access-pq6r7\") pod \"network-check-source-7d88cb55d8-4lqlg\" (UID: \"98400583-3b23-4bf0-be1f-54aa621542a0\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-4lqlg"
Jul 29 18:34:30.715334 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.715063    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/66a293fb-92cb-4908-bf90-d97ed0d831a6-stats-auth\") pod \"router-default-7688c5db95-gcqb5\" (UID: \"66a293fb-92cb-4908-bf90-d97ed0d831a6\") " pod="openshift-ingress/router-default-7688c5db95-gcqb5"
Jul 29 18:34:30.715334 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.715097    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx2wt\" (UniqueName: \"kubernetes.io/projected/e281e146-0b8a-4358-a98b-7f577f4ff08b-kube-api-access-kx2wt\") pod \"image-registry-6cb6fb9448-jbzkl\" (UID: \"e281e146-0b8a-4358-a98b-7f577f4ff08b\") " pod="openshift-image-registry/image-registry-6cb6fb9448-jbzkl"
Jul 29 18:34:30.715334 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.715119    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kr67\" (UniqueName: \"kubernetes.io/projected/cad930eb-2903-4c78-b4c4-3bcc190d199d-kube-api-access-5kr67\") pod \"service-ca-operator-5d6c75f84f-95ttd\" (UID: \"cad930eb-2903-4c78-b4c4-3bcc190d199d\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-95ttd"
Jul 29 18:34:30.715334 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.715144    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c284fbd5-a70f-4085-948c-6a2392502e6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-jxmfh\" (UID: \"c284fbd5-a70f-4085-948c-6a2392502e6f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jxmfh"
Jul 29 18:34:30.715334 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.715172    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e281e146-0b8a-4358-a98b-7f577f4ff08b-registry-tls\") pod \"image-registry-6cb6fb9448-jbzkl\" (UID: \"e281e146-0b8a-4358-a98b-7f577f4ff08b\") " pod="openshift-image-registry/image-registry-6cb6fb9448-jbzkl"
Jul 29 18:34:30.715334 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.715202    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c7c618e4-ba24-4fb5-8c20-47368c4f9322-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7d55599b76-4k8r9\" (UID: \"c7c618e4-ba24-4fb5-8c20-47368c4f9322\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d55599b76-4k8r9"
Jul 29 18:34:30.715334 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.715266    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w449k\" (UniqueName: \"kubernetes.io/projected/cc87b3c5-0b72-4fa2-9f87-ca3d4d6da614-kube-api-access-w449k\") pod \"ingress-canary-qhj2h\" (UID: \"cc87b3c5-0b72-4fa2-9f87-ca3d4d6da614\") " pod="openshift-ingress-canary/ingress-canary-qhj2h"
Jul 29 18:34:30.716085 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.715326    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2bpr\" (UniqueName: \"kubernetes.io/projected/7f8b4737-d4c2-4598-bd40-208015542831-kube-api-access-z2bpr\") pod \"console-operator-584dcb6b54-v9nsr\" (UID: \"7f8b4737-d4c2-4598-bd40-208015542831\") " pod="openshift-console-operator/console-operator-584dcb6b54-v9nsr"
Jul 29 18:34:30.716085 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.715353    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/26f43ad8-a7b4-4140-9ee0-7e9d55edf035-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-hb5lw\" (UID: \"26f43ad8-a7b4-4140-9ee0-7e9d55edf035\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-hb5lw"
Jul 29 18:34:30.716085 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.715403    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e281e146-0b8a-4358-a98b-7f577f4ff08b-installation-pull-secrets\") pod \"image-registry-6cb6fb9448-jbzkl\" (UID: \"e281e146-0b8a-4358-a98b-7f577f4ff08b\") " pod="openshift-image-registry/image-registry-6cb6fb9448-jbzkl"
Jul 29 18:34:30.716085 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.715427    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66a293fb-92cb-4908-bf90-d97ed0d831a6-metrics-certs\") pod \"router-default-7688c5db95-gcqb5\" (UID: \"66a293fb-92cb-4908-bf90-d97ed0d831a6\") " pod="openshift-ingress/router-default-7688c5db95-gcqb5"
Jul 29 18:34:30.716085 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.715451    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99mbn\" (UniqueName: \"kubernetes.io/projected/a84d718f-459d-4e2e-9c9e-d08d1500b145-kube-api-access-99mbn\") pod \"insights-operator-5885fdb759-j76ql\" (UID: \"a84d718f-459d-4e2e-9c9e-d08d1500b145\") " pod="openshift-insights/insights-operator-5885fdb759-j76ql"
Jul 29 18:34:30.716085 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.715476    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/05f777ff-5b55-41e4-b440-1705c2c3842e-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nj275\" (UID: \"05f777ff-5b55-41e4-b440-1705c2c3842e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nj275"
Jul 29 18:34:30.716085 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.715508    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwrsq\" (UniqueName: \"kubernetes.io/projected/66a293fb-92cb-4908-bf90-d97ed0d831a6-kube-api-access-cwrsq\") pod \"router-default-7688c5db95-gcqb5\" (UID: \"66a293fb-92cb-4908-bf90-d97ed0d831a6\") " pod="openshift-ingress/router-default-7688c5db95-gcqb5"
Jul 29 18:34:30.716085 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.715531    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c854cbbc-3a2c-46ca-adf3-86fc3028bd7c-metrics-tls\") pod \"dns-default-95vjs\" (UID: \"c854cbbc-3a2c-46ca-adf3-86fc3028bd7c\") " pod="openshift-dns/dns-default-95vjs"
Jul 29 18:34:30.716085 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.715559    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a84d718f-459d-4e2e-9c9e-d08d1500b145-tmp\") pod \"insights-operator-5885fdb759-j76ql\" (UID: \"a84d718f-459d-4e2e-9c9e-d08d1500b145\") " pod="openshift-insights/insights-operator-5885fdb759-j76ql"
Jul 29 18:34:30.716085 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.715574    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c854cbbc-3a2c-46ca-adf3-86fc3028bd7c-tmp-dir\") pod \"dns-default-95vjs\" (UID: \"c854cbbc-3a2c-46ca-adf3-86fc3028bd7c\") " pod="openshift-dns/dns-default-95vjs"
Jul 29 18:34:30.716085 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.715589    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qnqg\" (UniqueName: \"kubernetes.io/projected/c284fbd5-a70f-4085-948c-6a2392502e6f-kube-api-access-6qnqg\") pod \"cluster-monitoring-operator-548b489cb8-jxmfh\" (UID: \"c284fbd5-a70f-4085-948c-6a2392502e6f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jxmfh"
Jul 29 18:34:30.716085 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.715606    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv9t6\" (UniqueName: \"kubernetes.io/projected/c7c618e4-ba24-4fb5-8c20-47368c4f9322-kube-api-access-hv9t6\") pod \"managed-serviceaccount-addon-agent-7d55599b76-4k8r9\" (UID: \"c7c618e4-ba24-4fb5-8c20-47368c4f9322\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d55599b76-4k8r9"
Jul 29 18:34:30.716085 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.715624    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cad930eb-2903-4c78-b4c4-3bcc190d199d-config\") pod \"service-ca-operator-5d6c75f84f-95ttd\" (UID: \"cad930eb-2903-4c78-b4c4-3bcc190d199d\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-95ttd"
Jul 29 18:34:30.716085 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.715648    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a84d718f-459d-4e2e-9c9e-d08d1500b145-service-ca-bundle\") pod \"insights-operator-5885fdb759-j76ql\" (UID: \"a84d718f-459d-4e2e-9c9e-d08d1500b145\") " pod="openshift-insights/insights-operator-5885fdb759-j76ql"
Jul 29 18:34:30.716085 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.715673    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbm6g\" (UniqueName: \"kubernetes.io/projected/05f777ff-5b55-41e4-b440-1705c2c3842e-kube-api-access-tbm6g\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nj275\" (UID: \"05f777ff-5b55-41e4-b440-1705c2c3842e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nj275"
Jul 29 18:34:30.816148 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.816056    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a84d718f-459d-4e2e-9c9e-d08d1500b145-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-j76ql\" (UID: \"a84d718f-459d-4e2e-9c9e-d08d1500b145\") " pod="openshift-insights/insights-operator-5885fdb759-j76ql"
Jul 29 18:34:30.816148 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.816090    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e281e146-0b8a-4358-a98b-7f577f4ff08b-registry-certificates\") pod \"image-registry-6cb6fb9448-jbzkl\" (UID: \"e281e146-0b8a-4358-a98b-7f577f4ff08b\") " pod="openshift-image-registry/image-registry-6cb6fb9448-jbzkl"
Jul 29 18:34:30.816148 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.816118    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cad930eb-2903-4c78-b4c4-3bcc190d199d-serving-cert\") pod \"service-ca-operator-5d6c75f84f-95ttd\" (UID: \"cad930eb-2903-4c78-b4c4-3bcc190d199d\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-95ttd"
Jul 29 18:34:30.816148 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.816140    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e281e146-0b8a-4358-a98b-7f577f4ff08b-ca-trust-extracted\") pod \"image-registry-6cb6fb9448-jbzkl\" (UID: \"e281e146-0b8a-4358-a98b-7f577f4ff08b\") " pod="openshift-image-registry/image-registry-6cb6fb9448-jbzkl"
Jul 29 18:34:30.817887 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.816166    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a84d718f-459d-4e2e-9c9e-d08d1500b145-snapshots\") pod \"insights-operator-5885fdb759-j76ql\" (UID: \"a84d718f-459d-4e2e-9c9e-d08d1500b145\") " pod="openshift-insights/insights-operator-5885fdb759-j76ql"
Jul 29 18:34:30.817887 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.816523    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7f8b4737-d4c2-4598-bd40-208015542831-serving-cert\") pod \"console-operator-584dcb6b54-v9nsr\" (UID: \"7f8b4737-d4c2-4598-bd40-208015542831\") " pod="openshift-console-operator/console-operator-584dcb6b54-v9nsr"
Jul 29 18:34:30.817887 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.816570    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c854cbbc-3a2c-46ca-adf3-86fc3028bd7c-config-volume\") pod \"dns-default-95vjs\" (UID: \"c854cbbc-3a2c-46ca-adf3-86fc3028bd7c\") " pod="openshift-dns/dns-default-95vjs"
Jul 29 18:34:30.817887 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.816594    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnccl\" (UniqueName: \"kubernetes.io/projected/c854cbbc-3a2c-46ca-adf3-86fc3028bd7c-kube-api-access-vnccl\") pod \"dns-default-95vjs\" (UID: \"c854cbbc-3a2c-46ca-adf3-86fc3028bd7c\") " pod="openshift-dns/dns-default-95vjs"
Jul 29 18:34:30.817887 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.816618    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e281e146-0b8a-4358-a98b-7f577f4ff08b-trusted-ca\") pod \"image-registry-6cb6fb9448-jbzkl\" (UID: \"e281e146-0b8a-4358-a98b-7f577f4ff08b\") " pod="openshift-image-registry/image-registry-6cb6fb9448-jbzkl"
Jul 29 18:34:30.817887 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.816643    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq6r7\" (UniqueName: \"kubernetes.io/projected/98400583-3b23-4bf0-be1f-54aa621542a0-kube-api-access-pq6r7\") pod \"network-check-source-7d88cb55d8-4lqlg\" (UID: \"98400583-3b23-4bf0-be1f-54aa621542a0\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-4lqlg"
Jul 29 18:34:30.817887 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.816666    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/66a293fb-92cb-4908-bf90-d97ed0d831a6-stats-auth\") pod \"router-default-7688c5db95-gcqb5\" (UID: \"66a293fb-92cb-4908-bf90-d97ed0d831a6\") " pod="openshift-ingress/router-default-7688c5db95-gcqb5"
Jul 29 18:34:30.817887 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.816697    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx2wt\" (UniqueName: \"kubernetes.io/projected/e281e146-0b8a-4358-a98b-7f577f4ff08b-kube-api-access-kx2wt\") pod \"image-registry-6cb6fb9448-jbzkl\" (UID: \"e281e146-0b8a-4358-a98b-7f577f4ff08b\") " pod="openshift-image-registry/image-registry-6cb6fb9448-jbzkl"
Jul 29 18:34:30.817887 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.816719    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kr67\" (UniqueName: \"kubernetes.io/projected/cad930eb-2903-4c78-b4c4-3bcc190d199d-kube-api-access-5kr67\") pod \"service-ca-operator-5d6c75f84f-95ttd\" (UID: \"cad930eb-2903-4c78-b4c4-3bcc190d199d\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-95ttd"
Jul 29 18:34:30.817887 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.816743    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c284fbd5-a70f-4085-948c-6a2392502e6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-jxmfh\" (UID: \"c284fbd5-a70f-4085-948c-6a2392502e6f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jxmfh"
Jul 29 18:34:30.817887 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.816754    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e281e146-0b8a-4358-a98b-7f577f4ff08b-ca-trust-extracted\") pod \"image-registry-6cb6fb9448-jbzkl\" (UID: \"e281e146-0b8a-4358-a98b-7f577f4ff08b\") " pod="openshift-image-registry/image-registry-6cb6fb9448-jbzkl"
Jul 29 18:34:30.817887 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.816775    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/542cc68a-f9f1-4d9c-8be9-1de3687ca26a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-58dbb54c4d-zmrbt\" (UID: \"542cc68a-f9f1-4d9c-8be9-1de3687ca26a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58dbb54c4d-zmrbt"
Jul 29 18:34:30.817887 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.816811    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a84d718f-459d-4e2e-9c9e-d08d1500b145-snapshots\") pod \"insights-operator-5885fdb759-j76ql\" (UID: \"a84d718f-459d-4e2e-9c9e-d08d1500b145\") " pod="openshift-insights/insights-operator-5885fdb759-j76ql"
Jul 29 18:34:30.817887 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.816826    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e281e146-0b8a-4358-a98b-7f577f4ff08b-registry-tls\") pod \"image-registry-6cb6fb9448-jbzkl\" (UID: \"e281e146-0b8a-4358-a98b-7f577f4ff08b\") " pod="openshift-image-registry/image-registry-6cb6fb9448-jbzkl"
Jul 29 18:34:30.817887 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.816870    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c7c618e4-ba24-4fb5-8c20-47368c4f9322-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7d55599b76-4k8r9\" (UID: \"c7c618e4-ba24-4fb5-8c20-47368c4f9322\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d55599b76-4k8r9"
Jul 29 18:34:30.817887 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:30.816912    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 18:34:30.818723 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.816916    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w449k\" (UniqueName: \"kubernetes.io/projected/cc87b3c5-0b72-4fa2-9f87-ca3d4d6da614-kube-api-access-w449k\") pod \"ingress-canary-qhj2h\" (UID: \"cc87b3c5-0b72-4fa2-9f87-ca3d4d6da614\") " pod="openshift-ingress-canary/ingress-canary-qhj2h"
Jul 29 18:34:30.818723 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:30.816924    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cb6fb9448-jbzkl: secret "image-registry-tls" not found
Jul 29 18:34:30.818723 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.816940    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2bpr\" (UniqueName: \"kubernetes.io/projected/7f8b4737-d4c2-4598-bd40-208015542831-kube-api-access-z2bpr\") pod \"console-operator-584dcb6b54-v9nsr\" (UID: \"7f8b4737-d4c2-4598-bd40-208015542831\") " pod="openshift-console-operator/console-operator-584dcb6b54-v9nsr"
Jul 29 18:34:30.818723 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.816962    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/26f43ad8-a7b4-4140-9ee0-7e9d55edf035-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-hb5lw\" (UID: \"26f43ad8-a7b4-4140-9ee0-7e9d55edf035\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-hb5lw"
Jul 29 18:34:30.818723 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:30.816982    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e281e146-0b8a-4358-a98b-7f577f4ff08b-registry-tls podName:e281e146-0b8a-4358-a98b-7f577f4ff08b nodeName:}" failed. No retries permitted until 2026-07-29 18:34:31.316964085 +0000 UTC m=+35.586733163 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e281e146-0b8a-4358-a98b-7f577f4ff08b-registry-tls") pod "image-registry-6cb6fb9448-jbzkl" (UID: "e281e146-0b8a-4358-a98b-7f577f4ff08b") : secret "image-registry-tls" not found
Jul 29 18:34:30.818723 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.817026    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e281e146-0b8a-4358-a98b-7f577f4ff08b-installation-pull-secrets\") pod \"image-registry-6cb6fb9448-jbzkl\" (UID: \"e281e146-0b8a-4358-a98b-7f577f4ff08b\") " pod="openshift-image-registry/image-registry-6cb6fb9448-jbzkl"
Jul 29 18:34:30.818723 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:30.817031    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 29 18:34:30.818723 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:30.817073    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26f43ad8-a7b4-4140-9ee0-7e9d55edf035-samples-operator-tls podName:26f43ad8-a7b4-4140-9ee0-7e9d55edf035 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:31.317063177 +0000 UTC m=+35.586832255 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/26f43ad8-a7b4-4140-9ee0-7e9d55edf035-samples-operator-tls") pod "cluster-samples-operator-6748489656-hb5lw" (UID: "26f43ad8-a7b4-4140-9ee0-7e9d55edf035") : secret "samples-operator-tls" not found
Jul 29 18:34:30.818723 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:30.817145    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 18:34:30.818723 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:30.817214    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c284fbd5-a70f-4085-948c-6a2392502e6f-cluster-monitoring-operator-tls podName:c284fbd5-a70f-4085-948c-6a2392502e6f nodeName:}" failed. No retries permitted until 2026-07-29 18:34:31.317201427 +0000 UTC m=+35.586970516 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c284fbd5-a70f-4085-948c-6a2392502e6f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-jxmfh" (UID: "c284fbd5-a70f-4085-948c-6a2392502e6f") : secret "cluster-monitoring-operator-tls" not found
Jul 29 18:34:30.818723 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.817299    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a84d718f-459d-4e2e-9c9e-d08d1500b145-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-j76ql\" (UID: \"a84d718f-459d-4e2e-9c9e-d08d1500b145\") " pod="openshift-insights/insights-operator-5885fdb759-j76ql"
Jul 29 18:34:30.818723 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.816822    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e281e146-0b8a-4358-a98b-7f577f4ff08b-registry-certificates\") pod \"image-registry-6cb6fb9448-jbzkl\" (UID: \"e281e146-0b8a-4358-a98b-7f577f4ff08b\") " pod="openshift-image-registry/image-registry-6cb6fb9448-jbzkl"
Jul 29 18:34:30.818723 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.817698    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c854cbbc-3a2c-46ca-adf3-86fc3028bd7c-config-volume\") pod \"dns-default-95vjs\" (UID: \"c854cbbc-3a2c-46ca-adf3-86fc3028bd7c\") " pod="openshift-dns/dns-default-95vjs"
Jul 29 18:34:30.818723 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.817777    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66a293fb-92cb-4908-bf90-d97ed0d831a6-metrics-certs\") pod \"router-default-7688c5db95-gcqb5\" (UID: \"66a293fb-92cb-4908-bf90-d97ed0d831a6\") " pod="openshift-ingress/router-default-7688c5db95-gcqb5"
Jul 29 18:34:30.818723 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.817816    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99mbn\" (UniqueName: \"kubernetes.io/projected/a84d718f-459d-4e2e-9c9e-d08d1500b145-kube-api-access-99mbn\") pod \"insights-operator-5885fdb759-j76ql\" (UID: \"a84d718f-459d-4e2e-9c9e-d08d1500b145\") " pod="openshift-insights/insights-operator-5885fdb759-j76ql"
Jul 29 18:34:30.819427 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.817846    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/05f777ff-5b55-41e4-b440-1705c2c3842e-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nj275\" (UID: \"05f777ff-5b55-41e4-b440-1705c2c3842e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nj275"
Jul 29 18:34:30.819427 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.817875    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/1b53e40e-25fd-4563-a3f8-df861a6c51f7-klusterlet-config\") pod \"klusterlet-addon-workmgr-6cb7489487-kkmn9\" (UID: \"1b53e40e-25fd-4563-a3f8-df861a6c51f7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cb7489487-kkmn9"
Jul 29 18:34:30.819427 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.817909    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwrsq\" (UniqueName: \"kubernetes.io/projected/66a293fb-92cb-4908-bf90-d97ed0d831a6-kube-api-access-cwrsq\") pod \"router-default-7688c5db95-gcqb5\" (UID: \"66a293fb-92cb-4908-bf90-d97ed0d831a6\") " pod="openshift-ingress/router-default-7688c5db95-gcqb5"
Jul 29 18:34:30.819427 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.817937    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c854cbbc-3a2c-46ca-adf3-86fc3028bd7c-metrics-tls\") pod \"dns-default-95vjs\" (UID: \"c854cbbc-3a2c-46ca-adf3-86fc3028bd7c\") " pod="openshift-dns/dns-default-95vjs"
Jul 29 18:34:30.819427 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.817958    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e281e146-0b8a-4358-a98b-7f577f4ff08b-trusted-ca\") pod \"image-registry-6cb6fb9448-jbzkl\" (UID: \"e281e146-0b8a-4358-a98b-7f577f4ff08b\") " pod="openshift-image-registry/image-registry-6cb6fb9448-jbzkl"
Jul 29 18:34:30.819427 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.817967    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a84d718f-459d-4e2e-9c9e-d08d1500b145-tmp\") pod \"insights-operator-5885fdb759-j76ql\" (UID: \"a84d718f-459d-4e2e-9c9e-d08d1500b145\") " pod="openshift-insights/insights-operator-5885fdb759-j76ql"
Jul 29 18:34:30.819427 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.817999    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c854cbbc-3a2c-46ca-adf3-86fc3028bd7c-tmp-dir\") pod \"dns-default-95vjs\" (UID: \"c854cbbc-3a2c-46ca-adf3-86fc3028bd7c\") " pod="openshift-dns/dns-default-95vjs"
Jul 29 18:34:30.819427 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.818061    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/542cc68a-f9f1-4d9c-8be9-1de3687ca26a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-58dbb54c4d-zmrbt\" (UID: \"542cc68a-f9f1-4d9c-8be9-1de3687ca26a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58dbb54c4d-zmrbt"
Jul 29 18:34:30.819427 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:30.818095    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 29 18:34:30.819427 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.818093    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qnqg\" (UniqueName: \"kubernetes.io/projected/c284fbd5-a70f-4085-948c-6a2392502e6f-kube-api-access-6qnqg\") pod \"cluster-monitoring-operator-548b489cb8-jxmfh\" (UID: \"c284fbd5-a70f-4085-948c-6a2392502e6f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jxmfh"
Jul 29 18:34:30.819427 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.818144    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgrzv\" (UniqueName: \"kubernetes.io/projected/542cc68a-f9f1-4d9c-8be9-1de3687ca26a-kube-api-access-tgrzv\") pod \"cluster-proxy-proxy-agent-58dbb54c4d-zmrbt\" (UID: \"542cc68a-f9f1-4d9c-8be9-1de3687ca26a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58dbb54c4d-zmrbt"
Jul 29 18:34:30.819427 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.818175    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv9t6\" (UniqueName: \"kubernetes.io/projected/c7c618e4-ba24-4fb5-8c20-47368c4f9322-kube-api-access-hv9t6\") pod \"managed-serviceaccount-addon-agent-7d55599b76-4k8r9\" (UID: \"c7c618e4-ba24-4fb5-8c20-47368c4f9322\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d55599b76-4k8r9"
Jul 29 18:34:30.819427 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.818202    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cad930eb-2903-4c78-b4c4-3bcc190d199d-config\") pod \"service-ca-operator-5d6c75f84f-95ttd\" (UID: \"cad930eb-2903-4c78-b4c4-3bcc190d199d\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-95ttd"
Jul 29 18:34:30.819427 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.818242    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a84d718f-459d-4e2e-9c9e-d08d1500b145-service-ca-bundle\") pod \"insights-operator-5885fdb759-j76ql\" (UID: \"a84d718f-459d-4e2e-9c9e-d08d1500b145\") " pod="openshift-insights/insights-operator-5885fdb759-j76ql"
Jul 29 18:34:30.819427 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.818266    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48xwf\" (UniqueName: \"kubernetes.io/projected/1b53e40e-25fd-4563-a3f8-df861a6c51f7-kube-api-access-48xwf\") pod \"klusterlet-addon-workmgr-6cb7489487-kkmn9\" (UID: \"1b53e40e-25fd-4563-a3f8-df861a6c51f7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cb7489487-kkmn9"
Jul 29 18:34:30.819427 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.818357    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbm6g\" (UniqueName: \"kubernetes.io/projected/05f777ff-5b55-41e4-b440-1705c2c3842e-kube-api-access-tbm6g\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nj275\" (UID: \"05f777ff-5b55-41e4-b440-1705c2c3842e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nj275"
Jul 29 18:34:30.820104 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.818395    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a84d718f-459d-4e2e-9c9e-d08d1500b145-tmp\") pod \"insights-operator-5885fdb759-j76ql\" (UID: \"a84d718f-459d-4e2e-9c9e-d08d1500b145\") " pod="openshift-insights/insights-operator-5885fdb759-j76ql"
Jul 29 18:34:30.820104 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.818420    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c284fbd5-a70f-4085-948c-6a2392502e6f-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-jxmfh\" (UID: \"c284fbd5-a70f-4085-948c-6a2392502e6f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jxmfh"
Jul 29 18:34:30.820104 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.818446    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66a293fb-92cb-4908-bf90-d97ed0d831a6-service-ca-bundle\") pod \"router-default-7688c5db95-gcqb5\" (UID: \"66a293fb-92cb-4908-bf90-d97ed0d831a6\") " pod="openshift-ingress/router-default-7688c5db95-gcqb5"
Jul 29 18:34:30.820104 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.818475    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/66a293fb-92cb-4908-bf90-d97ed0d831a6-default-certificate\") pod \"router-default-7688c5db95-gcqb5\" (UID: \"66a293fb-92cb-4908-bf90-d97ed0d831a6\") " pod="openshift-ingress/router-default-7688c5db95-gcqb5"
Jul 29 18:34:30.820104 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.818493    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7f8b4737-d4c2-4598-bd40-208015542831-trusted-ca\") pod \"console-operator-584dcb6b54-v9nsr\" (UID: \"7f8b4737-d4c2-4598-bd40-208015542831\") " pod="openshift-console-operator/console-operator-584dcb6b54-v9nsr"
Jul 29 18:34:30.820104 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.818513    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a84d718f-459d-4e2e-9c9e-d08d1500b145-serving-cert\") pod \"insights-operator-5885fdb759-j76ql\" (UID: \"a84d718f-459d-4e2e-9c9e-d08d1500b145\") " pod="openshift-insights/insights-operator-5885fdb759-j76ql"
Jul 29 18:34:30.820104 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.818532    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e281e146-0b8a-4358-a98b-7f577f4ff08b-image-registry-private-configuration\") pod \"image-registry-6cb6fb9448-jbzkl\" (UID: \"e281e146-0b8a-4358-a98b-7f577f4ff08b\") " pod="openshift-image-registry/image-registry-6cb6fb9448-jbzkl"
Jul 29 18:34:30.820104 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.818546    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7f8b4737-d4c2-4598-bd40-208015542831-config\") pod \"console-operator-584dcb6b54-v9nsr\" (UID: \"7f8b4737-d4c2-4598-bd40-208015542831\") " pod="openshift-console-operator/console-operator-584dcb6b54-v9nsr"
Jul 29 18:34:30.820104 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.818564    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/05f777ff-5b55-41e4-b440-1705c2c3842e-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nj275\" (UID: \"05f777ff-5b55-41e4-b440-1705c2c3842e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nj275"
Jul 29 18:34:30.820104 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.818593    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc87b3c5-0b72-4fa2-9f87-ca3d4d6da614-cert\") pod \"ingress-canary-qhj2h\" (UID: \"cc87b3c5-0b72-4fa2-9f87-ca3d4d6da614\") " pod="openshift-ingress-canary/ingress-canary-qhj2h"
Jul 29 18:34:30.820104 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.818612    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmbf8\" (UniqueName: \"kubernetes.io/projected/4b692565-aef0-416f-8fad-2c74bca55d4d-kube-api-access-fmbf8\") pod \"volume-data-source-validator-549c6d54cf-ljn4r\" (UID: \"4b692565-aef0-416f-8fad-2c74bca55d4d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-ljn4r"
Jul 29 18:34:30.820104 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.818615    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c854cbbc-3a2c-46ca-adf3-86fc3028bd7c-tmp-dir\") pod \"dns-default-95vjs\" (UID: \"c854cbbc-3a2c-46ca-adf3-86fc3028bd7c\") " pod="openshift-dns/dns-default-95vjs"
Jul 29 18:34:30.820104 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.818642    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/542cc68a-f9f1-4d9c-8be9-1de3687ca26a-ca\") pod \"cluster-proxy-proxy-agent-58dbb54c4d-zmrbt\" (UID: \"542cc68a-f9f1-4d9c-8be9-1de3687ca26a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58dbb54c4d-zmrbt"
Jul 29 18:34:30.820104 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.818670    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/542cc68a-f9f1-4d9c-8be9-1de3687ca26a-hub\") pod \"cluster-proxy-proxy-agent-58dbb54c4d-zmrbt\" (UID: \"542cc68a-f9f1-4d9c-8be9-1de3687ca26a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58dbb54c4d-zmrbt"
Jul 29 18:34:30.820104 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.818686    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/542cc68a-f9f1-4d9c-8be9-1de3687ca26a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-58dbb54c4d-zmrbt\" (UID: \"542cc68a-f9f1-4d9c-8be9-1de3687ca26a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58dbb54c4d-zmrbt"
Jul 29 18:34:30.820104 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.818721    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b53e40e-25fd-4563-a3f8-df861a6c51f7-tmp\") pod \"klusterlet-addon-workmgr-6cb7489487-kkmn9\" (UID: \"1b53e40e-25fd-4563-a3f8-df861a6c51f7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cb7489487-kkmn9"
Jul 29 18:34:30.820843 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.818741    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxqgs\" (UniqueName: \"kubernetes.io/projected/26f43ad8-a7b4-4140-9ee0-7e9d55edf035-kube-api-access-pxqgs\") pod \"cluster-samples-operator-6748489656-hb5lw\" (UID: \"26f43ad8-a7b4-4140-9ee0-7e9d55edf035\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-hb5lw"
Jul 29 18:34:30.820843 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.818772    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e281e146-0b8a-4358-a98b-7f577f4ff08b-bound-sa-token\") pod \"image-registry-6cb6fb9448-jbzkl\" (UID: \"e281e146-0b8a-4358-a98b-7f577f4ff08b\") " pod="openshift-image-registry/image-registry-6cb6fb9448-jbzkl"
Jul 29 18:34:30.820843 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:30.819053    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 18:34:30.820843 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:30.819081    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66a293fb-92cb-4908-bf90-d97ed0d831a6-metrics-certs podName:66a293fb-92cb-4908-bf90-d97ed0d831a6 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:31.31906344 +0000 UTC m=+35.588832510 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66a293fb-92cb-4908-bf90-d97ed0d831a6-metrics-certs") pod "router-default-7688c5db95-gcqb5" (UID: "66a293fb-92cb-4908-bf90-d97ed0d831a6") : secret "router-metrics-certs-default" not found
Jul 29 18:34:30.820843 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:30.819100    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c854cbbc-3a2c-46ca-adf3-86fc3028bd7c-metrics-tls podName:c854cbbc-3a2c-46ca-adf3-86fc3028bd7c nodeName:}" failed. No retries permitted until 2026-07-29 18:34:31.319091225 +0000 UTC m=+35.588860296 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c854cbbc-3a2c-46ca-adf3-86fc3028bd7c-metrics-tls") pod "dns-default-95vjs" (UID: "c854cbbc-3a2c-46ca-adf3-86fc3028bd7c") : secret "dns-default-metrics-tls" not found
Jul 29 18:34:30.820843 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:30.820005    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/66a293fb-92cb-4908-bf90-d97ed0d831a6-service-ca-bundle podName:66a293fb-92cb-4908-bf90-d97ed0d831a6 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:31.319989269 +0000 UTC m=+35.589758358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/66a293fb-92cb-4908-bf90-d97ed0d831a6-service-ca-bundle") pod "router-default-7688c5db95-gcqb5" (UID: "66a293fb-92cb-4908-bf90-d97ed0d831a6") : configmap references non-existent config key: service-ca.crt
Jul 29 18:34:30.820843 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.820352    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7f8b4737-d4c2-4598-bd40-208015542831-config\") pod \"console-operator-584dcb6b54-v9nsr\" (UID: \"7f8b4737-d4c2-4598-bd40-208015542831\") " pod="openshift-console-operator/console-operator-584dcb6b54-v9nsr"
Jul 29 18:34:30.820843 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.820489    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cad930eb-2903-4c78-b4c4-3bcc190d199d-config\") pod \"service-ca-operator-5d6c75f84f-95ttd\" (UID: \"cad930eb-2903-4c78-b4c4-3bcc190d199d\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-95ttd"
Jul 29 18:34:30.820843 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.820689    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c284fbd5-a70f-4085-948c-6a2392502e6f-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-jxmfh\" (UID: \"c284fbd5-a70f-4085-948c-6a2392502e6f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jxmfh"
Jul 29 18:34:30.820843 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.820821    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a84d718f-459d-4e2e-9c9e-d08d1500b145-service-ca-bundle\") pod \"insights-operator-5885fdb759-j76ql\" (UID: \"a84d718f-459d-4e2e-9c9e-d08d1500b145\") " pod="openshift-insights/insights-operator-5885fdb759-j76ql"
Jul 29 18:34:30.821974 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.821606    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7f8b4737-d4c2-4598-bd40-208015542831-trusted-ca\") pod \"console-operator-584dcb6b54-v9nsr\" (UID: \"7f8b4737-d4c2-4598-bd40-208015542831\") " pod="openshift-console-operator/console-operator-584dcb6b54-v9nsr"
Jul 29 18:34:30.822315 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.822268    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/05f777ff-5b55-41e4-b440-1705c2c3842e-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nj275\" (UID: \"05f777ff-5b55-41e4-b440-1705c2c3842e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nj275"
Jul 29 18:34:30.823171 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.823142    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e281e146-0b8a-4358-a98b-7f577f4ff08b-image-registry-private-configuration\") pod \"image-registry-6cb6fb9448-jbzkl\" (UID: \"e281e146-0b8a-4358-a98b-7f577f4ff08b\") " pod="openshift-image-registry/image-registry-6cb6fb9448-jbzkl"
Jul 29 18:34:30.823757 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.823733    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/05f777ff-5b55-41e4-b440-1705c2c3842e-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nj275\" (UID: \"05f777ff-5b55-41e4-b440-1705c2c3842e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nj275"
Jul 29 18:34:30.823864 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.823798    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc87b3c5-0b72-4fa2-9f87-ca3d4d6da614-cert\") pod \"ingress-canary-qhj2h\" (UID: \"cc87b3c5-0b72-4fa2-9f87-ca3d4d6da614\") " pod="openshift-ingress-canary/ingress-canary-qhj2h"
Jul 29 18:34:30.824115 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.824065    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7f8b4737-d4c2-4598-bd40-208015542831-serving-cert\") pod \"console-operator-584dcb6b54-v9nsr\" (UID: \"7f8b4737-d4c2-4598-bd40-208015542831\") " pod="openshift-console-operator/console-operator-584dcb6b54-v9nsr"
Jul 29 18:34:30.824261 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.824241    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/66a293fb-92cb-4908-bf90-d97ed0d831a6-stats-auth\") pod \"router-default-7688c5db95-gcqb5\" (UID: \"66a293fb-92cb-4908-bf90-d97ed0d831a6\") " pod="openshift-ingress/router-default-7688c5db95-gcqb5"
Jul 29 18:34:30.824463 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.824352    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c7c618e4-ba24-4fb5-8c20-47368c4f9322-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7d55599b76-4k8r9\" (UID: \"c7c618e4-ba24-4fb5-8c20-47368c4f9322\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d55599b76-4k8r9"
Jul 29 18:34:30.824645 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.824627    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a84d718f-459d-4e2e-9c9e-d08d1500b145-serving-cert\") pod \"insights-operator-5885fdb759-j76ql\" (UID: \"a84d718f-459d-4e2e-9c9e-d08d1500b145\") " pod="openshift-insights/insights-operator-5885fdb759-j76ql"
Jul 29 18:34:30.824828 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.824812    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cad930eb-2903-4c78-b4c4-3bcc190d199d-serving-cert\") pod \"service-ca-operator-5d6c75f84f-95ttd\" (UID: \"cad930eb-2903-4c78-b4c4-3bcc190d199d\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-95ttd"
Jul 29 18:34:30.826807 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.826777    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq6r7\" (UniqueName: \"kubernetes.io/projected/98400583-3b23-4bf0-be1f-54aa621542a0-kube-api-access-pq6r7\") pod \"network-check-source-7d88cb55d8-4lqlg\" (UID: \"98400583-3b23-4bf0-be1f-54aa621542a0\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-4lqlg"
Jul 29 18:34:30.826937 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.826916    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e281e146-0b8a-4358-a98b-7f577f4ff08b-installation-pull-secrets\") pod \"image-registry-6cb6fb9448-jbzkl\" (UID: \"e281e146-0b8a-4358-a98b-7f577f4ff08b\") " pod="openshift-image-registry/image-registry-6cb6fb9448-jbzkl"
Jul 29 18:34:30.827206 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.827184    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/66a293fb-92cb-4908-bf90-d97ed0d831a6-default-certificate\") pod \"router-default-7688c5db95-gcqb5\" (UID: \"66a293fb-92cb-4908-bf90-d97ed0d831a6\") " pod="openshift-ingress/router-default-7688c5db95-gcqb5"
Jul 29 18:34:30.828190 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.828166    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kr67\" (UniqueName: \"kubernetes.io/projected/cad930eb-2903-4c78-b4c4-3bcc190d199d-kube-api-access-5kr67\") pod \"service-ca-operator-5d6c75f84f-95ttd\" (UID: \"cad930eb-2903-4c78-b4c4-3bcc190d199d\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-95ttd"
Jul 29 18:34:30.829651 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.829595    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnccl\" (UniqueName: \"kubernetes.io/projected/c854cbbc-3a2c-46ca-adf3-86fc3028bd7c-kube-api-access-vnccl\") pod \"dns-default-95vjs\" (UID: \"c854cbbc-3a2c-46ca-adf3-86fc3028bd7c\") " pod="openshift-dns/dns-default-95vjs"
Jul 29 18:34:30.830033 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.830008    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w449k\" (UniqueName: \"kubernetes.io/projected/cc87b3c5-0b72-4fa2-9f87-ca3d4d6da614-kube-api-access-w449k\") pod \"ingress-canary-qhj2h\" (UID: \"cc87b3c5-0b72-4fa2-9f87-ca3d4d6da614\") " pod="openshift-ingress-canary/ingress-canary-qhj2h"
Jul 29 18:34:30.832120 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.832094    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx2wt\" (UniqueName: \"kubernetes.io/projected/e281e146-0b8a-4358-a98b-7f577f4ff08b-kube-api-access-kx2wt\") pod \"image-registry-6cb6fb9448-jbzkl\" (UID: \"e281e146-0b8a-4358-a98b-7f577f4ff08b\") " pod="openshift-image-registry/image-registry-6cb6fb9448-jbzkl"
Jul 29 18:34:30.833481 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.833454    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxqgs\" (UniqueName: \"kubernetes.io/projected/26f43ad8-a7b4-4140-9ee0-7e9d55edf035-kube-api-access-pxqgs\") pod \"cluster-samples-operator-6748489656-hb5lw\" (UID: \"26f43ad8-a7b4-4140-9ee0-7e9d55edf035\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-hb5lw"
Jul 29 18:34:30.833578 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.833553    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2bpr\" (UniqueName: \"kubernetes.io/projected/7f8b4737-d4c2-4598-bd40-208015542831-kube-api-access-z2bpr\") pod \"console-operator-584dcb6b54-v9nsr\" (UID: \"7f8b4737-d4c2-4598-bd40-208015542831\") " pod="openshift-console-operator/console-operator-584dcb6b54-v9nsr"
Jul 29 18:34:30.833829 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.833784    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbm6g\" (UniqueName: \"kubernetes.io/projected/05f777ff-5b55-41e4-b440-1705c2c3842e-kube-api-access-tbm6g\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nj275\" (UID: \"05f777ff-5b55-41e4-b440-1705c2c3842e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nj275"
Jul 29 18:34:30.833992 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.833969    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwrsq\" (UniqueName: \"kubernetes.io/projected/66a293fb-92cb-4908-bf90-d97ed0d831a6-kube-api-access-cwrsq\") pod \"router-default-7688c5db95-gcqb5\" (UID: \"66a293fb-92cb-4908-bf90-d97ed0d831a6\") " pod="openshift-ingress/router-default-7688c5db95-gcqb5"
Jul 29 18:34:30.834940 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.834897    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e281e146-0b8a-4358-a98b-7f577f4ff08b-bound-sa-token\") pod \"image-registry-6cb6fb9448-jbzkl\" (UID: \"e281e146-0b8a-4358-a98b-7f577f4ff08b\") " pod="openshift-image-registry/image-registry-6cb6fb9448-jbzkl"
Jul 29 18:34:30.835119 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.835097    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv9t6\" (UniqueName: \"kubernetes.io/projected/c7c618e4-ba24-4fb5-8c20-47368c4f9322-kube-api-access-hv9t6\") pod \"managed-serviceaccount-addon-agent-7d55599b76-4k8r9\" (UID: \"c7c618e4-ba24-4fb5-8c20-47368c4f9322\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d55599b76-4k8r9"
Jul 29 18:34:30.835238 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.835221    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99mbn\" (UniqueName: \"kubernetes.io/projected/a84d718f-459d-4e2e-9c9e-d08d1500b145-kube-api-access-99mbn\") pod \"insights-operator-5885fdb759-j76ql\" (UID: \"a84d718f-459d-4e2e-9c9e-d08d1500b145\") " pod="openshift-insights/insights-operator-5885fdb759-j76ql"
Jul 29 18:34:30.836017 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.835995    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmbf8\" (UniqueName: \"kubernetes.io/projected/4b692565-aef0-416f-8fad-2c74bca55d4d-kube-api-access-fmbf8\") pod \"volume-data-source-validator-549c6d54cf-ljn4r\" (UID: \"4b692565-aef0-416f-8fad-2c74bca55d4d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-ljn4r"
Jul 29 18:34:30.836620 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.836601    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qnqg\" (UniqueName: \"kubernetes.io/projected/c284fbd5-a70f-4085-948c-6a2392502e6f-kube-api-access-6qnqg\") pod \"cluster-monitoring-operator-548b489cb8-jxmfh\" (UID: \"c284fbd5-a70f-4085-948c-6a2392502e6f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jxmfh"
Jul 29 18:34:30.887551 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.887515    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-j76ql"
Jul 29 18:34:30.919945 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.919912    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/542cc68a-f9f1-4d9c-8be9-1de3687ca26a-ca\") pod \"cluster-proxy-proxy-agent-58dbb54c4d-zmrbt\" (UID: \"542cc68a-f9f1-4d9c-8be9-1de3687ca26a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58dbb54c4d-zmrbt"
Jul 29 18:34:30.919945 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.919947    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/542cc68a-f9f1-4d9c-8be9-1de3687ca26a-hub\") pod \"cluster-proxy-proxy-agent-58dbb54c4d-zmrbt\" (UID: \"542cc68a-f9f1-4d9c-8be9-1de3687ca26a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58dbb54c4d-zmrbt"
Jul 29 18:34:30.919945 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.919966    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/542cc68a-f9f1-4d9c-8be9-1de3687ca26a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-58dbb54c4d-zmrbt\" (UID: \"542cc68a-f9f1-4d9c-8be9-1de3687ca26a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58dbb54c4d-zmrbt"
Jul 29 18:34:30.920230 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.919984    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b53e40e-25fd-4563-a3f8-df861a6c51f7-tmp\") pod \"klusterlet-addon-workmgr-6cb7489487-kkmn9\" (UID: \"1b53e40e-25fd-4563-a3f8-df861a6c51f7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cb7489487-kkmn9"
Jul 29 18:34:30.920230 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.920199    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/542cc68a-f9f1-4d9c-8be9-1de3687ca26a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-58dbb54c4d-zmrbt\" (UID: \"542cc68a-f9f1-4d9c-8be9-1de3687ca26a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58dbb54c4d-zmrbt"
Jul 29 18:34:30.920305 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.920277    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/1b53e40e-25fd-4563-a3f8-df861a6c51f7-klusterlet-config\") pod \"klusterlet-addon-workmgr-6cb7489487-kkmn9\" (UID: \"1b53e40e-25fd-4563-a3f8-df861a6c51f7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cb7489487-kkmn9"
Jul 29 18:34:30.920338 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.920311    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/542cc68a-f9f1-4d9c-8be9-1de3687ca26a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-58dbb54c4d-zmrbt\" (UID: \"542cc68a-f9f1-4d9c-8be9-1de3687ca26a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58dbb54c4d-zmrbt"
Jul 29 18:34:30.920338 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.920331    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgrzv\" (UniqueName: \"kubernetes.io/projected/542cc68a-f9f1-4d9c-8be9-1de3687ca26a-kube-api-access-tgrzv\") pod \"cluster-proxy-proxy-agent-58dbb54c4d-zmrbt\" (UID: \"542cc68a-f9f1-4d9c-8be9-1de3687ca26a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58dbb54c4d-zmrbt"
Jul 29 18:34:30.920484 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.920350    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b53e40e-25fd-4563-a3f8-df861a6c51f7-tmp\") pod \"klusterlet-addon-workmgr-6cb7489487-kkmn9\" (UID: \"1b53e40e-25fd-4563-a3f8-df861a6c51f7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cb7489487-kkmn9"
Jul 29 18:34:30.920484 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.920359    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48xwf\" (UniqueName: \"kubernetes.io/projected/1b53e40e-25fd-4563-a3f8-df861a6c51f7-kube-api-access-48xwf\") pod \"klusterlet-addon-workmgr-6cb7489487-kkmn9\" (UID: \"1b53e40e-25fd-4563-a3f8-df861a6c51f7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cb7489487-kkmn9"
Jul 29 18:34:30.921353 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.921245    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/542cc68a-f9f1-4d9c-8be9-1de3687ca26a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-58dbb54c4d-zmrbt\" (UID: \"542cc68a-f9f1-4d9c-8be9-1de3687ca26a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58dbb54c4d-zmrbt"
Jul 29 18:34:30.923728 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.923676    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/542cc68a-f9f1-4d9c-8be9-1de3687ca26a-hub\") pod \"cluster-proxy-proxy-agent-58dbb54c4d-zmrbt\" (UID: \"542cc68a-f9f1-4d9c-8be9-1de3687ca26a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58dbb54c4d-zmrbt"
Jul 29 18:34:30.923728 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.923694    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/1b53e40e-25fd-4563-a3f8-df861a6c51f7-klusterlet-config\") pod \"klusterlet-addon-workmgr-6cb7489487-kkmn9\" (UID: \"1b53e40e-25fd-4563-a3f8-df861a6c51f7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cb7489487-kkmn9"
Jul 29 18:34:30.923728 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.923679    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/542cc68a-f9f1-4d9c-8be9-1de3687ca26a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-58dbb54c4d-zmrbt\" (UID: \"542cc68a-f9f1-4d9c-8be9-1de3687ca26a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58dbb54c4d-zmrbt"
Jul 29 18:34:30.923966 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.923747    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-ljn4r"
Jul 29 18:34:30.923966 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.923766    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/542cc68a-f9f1-4d9c-8be9-1de3687ca26a-ca\") pod \"cluster-proxy-proxy-agent-58dbb54c4d-zmrbt\" (UID: \"542cc68a-f9f1-4d9c-8be9-1de3687ca26a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58dbb54c4d-zmrbt"
Jul 29 18:34:30.924140 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.924121    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/542cc68a-f9f1-4d9c-8be9-1de3687ca26a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-58dbb54c4d-zmrbt\" (UID: \"542cc68a-f9f1-4d9c-8be9-1de3687ca26a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58dbb54c4d-zmrbt"
Jul 29 18:34:30.929309 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.929282    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48xwf\" (UniqueName: \"kubernetes.io/projected/1b53e40e-25fd-4563-a3f8-df861a6c51f7-kube-api-access-48xwf\") pod \"klusterlet-addon-workmgr-6cb7489487-kkmn9\" (UID: \"1b53e40e-25fd-4563-a3f8-df861a6c51f7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cb7489487-kkmn9"
Jul 29 18:34:30.929428 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.929407    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgrzv\" (UniqueName: \"kubernetes.io/projected/542cc68a-f9f1-4d9c-8be9-1de3687ca26a-kube-api-access-tgrzv\") pod \"cluster-proxy-proxy-agent-58dbb54c4d-zmrbt\" (UID: \"542cc68a-f9f1-4d9c-8be9-1de3687ca26a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58dbb54c4d-zmrbt"
Jul 29 18:34:30.951614 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.951580    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-v9nsr"
Jul 29 18:34:30.966955 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.966919    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qhj2h"
Jul 29 18:34:30.975909 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.975883    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-4lqlg"
Jul 29 18:34:30.983023 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.983000    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-95ttd"
Jul 29 18:34:30.990870 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:30.990850    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nj275"
Jul 29 18:34:31.006397 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:31.006356    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d55599b76-4k8r9"
Jul 29 18:34:31.023601 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:31.023567    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58dbb54c4d-zmrbt"
Jul 29 18:34:31.029962 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:31.029934    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cb7489487-kkmn9"
Jul 29 18:34:31.144920 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:31.144880    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh2dv"
Jul 29 18:34:31.144920 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:31.144910    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zxr5"
Jul 29 18:34:31.145160 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:31.144912    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5w95"
Jul 29 18:34:31.150512 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:31.150442    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 18:34:31.150666 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:31.150534    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hfcvr\""
Jul 29 18:34:31.150666 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:31.150589    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 18:34:31.153567 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:31.153372    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mfhf2\""
Jul 29 18:34:31.324162 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:31.324115    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c284fbd5-a70f-4085-948c-6a2392502e6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-jxmfh\" (UID: \"c284fbd5-a70f-4085-948c-6a2392502e6f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jxmfh"
Jul 29 18:34:31.324358 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:31.324174    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e281e146-0b8a-4358-a98b-7f577f4ff08b-registry-tls\") pod \"image-registry-6cb6fb9448-jbzkl\" (UID: \"e281e146-0b8a-4358-a98b-7f577f4ff08b\") " pod="openshift-image-registry/image-registry-6cb6fb9448-jbzkl"
Jul 29 18:34:31.324358 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:31.324220    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/26f43ad8-a7b4-4140-9ee0-7e9d55edf035-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-hb5lw\" (UID: \"26f43ad8-a7b4-4140-9ee0-7e9d55edf035\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-hb5lw"
Jul 29 18:34:31.324358 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:31.324250    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66a293fb-92cb-4908-bf90-d97ed0d831a6-metrics-certs\") pod \"router-default-7688c5db95-gcqb5\" (UID: \"66a293fb-92cb-4908-bf90-d97ed0d831a6\") " pod="openshift-ingress/router-default-7688c5db95-gcqb5"
Jul 29 18:34:31.324358 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:31.324281    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 18:34:31.324358 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:31.324315    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 18:34:31.324358 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:31.324328    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cb6fb9448-jbzkl: secret "image-registry-tls" not found
Jul 29 18:34:31.324358 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:31.324345    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 29 18:34:31.324358 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:31.324362    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 18:34:31.324738 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:31.324284    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c854cbbc-3a2c-46ca-adf3-86fc3028bd7c-metrics-tls\") pod \"dns-default-95vjs\" (UID: \"c854cbbc-3a2c-46ca-adf3-86fc3028bd7c\") " pod="openshift-dns/dns-default-95vjs"
Jul 29 18:34:31.324738 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:31.324363    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c284fbd5-a70f-4085-948c-6a2392502e6f-cluster-monitoring-operator-tls podName:c284fbd5-a70f-4085-948c-6a2392502e6f nodeName:}" failed. No retries permitted until 2026-07-29 18:34:32.324340761 +0000 UTC m=+36.594109849 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c284fbd5-a70f-4085-948c-6a2392502e6f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-jxmfh" (UID: "c284fbd5-a70f-4085-948c-6a2392502e6f") : secret "cluster-monitoring-operator-tls" not found
Jul 29 18:34:31.324738 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:31.324442    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e281e146-0b8a-4358-a98b-7f577f4ff08b-registry-tls podName:e281e146-0b8a-4358-a98b-7f577f4ff08b nodeName:}" failed. No retries permitted until 2026-07-29 18:34:32.324424888 +0000 UTC m=+36.594193956 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e281e146-0b8a-4358-a98b-7f577f4ff08b-registry-tls") pod "image-registry-6cb6fb9448-jbzkl" (UID: "e281e146-0b8a-4358-a98b-7f577f4ff08b") : secret "image-registry-tls" not found
Jul 29 18:34:31.324738 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:31.324445    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 29 18:34:31.324738 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:31.324460    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26f43ad8-a7b4-4140-9ee0-7e9d55edf035-samples-operator-tls podName:26f43ad8-a7b4-4140-9ee0-7e9d55edf035 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:32.324452699 +0000 UTC m=+36.594221768 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/26f43ad8-a7b4-4140-9ee0-7e9d55edf035-samples-operator-tls") pod "cluster-samples-operator-6748489656-hb5lw" (UID: "26f43ad8-a7b4-4140-9ee0-7e9d55edf035") : secret "samples-operator-tls" not found
Jul 29 18:34:31.324738 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:31.324479    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c854cbbc-3a2c-46ca-adf3-86fc3028bd7c-metrics-tls podName:c854cbbc-3a2c-46ca-adf3-86fc3028bd7c nodeName:}" failed. No retries permitted until 2026-07-29 18:34:32.324471117 +0000 UTC m=+36.594240218 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c854cbbc-3a2c-46ca-adf3-86fc3028bd7c-metrics-tls") pod "dns-default-95vjs" (UID: "c854cbbc-3a2c-46ca-adf3-86fc3028bd7c") : secret "dns-default-metrics-tls" not found
Jul 29 18:34:31.324738 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:31.324506    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66a293fb-92cb-4908-bf90-d97ed0d831a6-metrics-certs podName:66a293fb-92cb-4908-bf90-d97ed0d831a6 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:32.324490615 +0000 UTC m=+36.594259695 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66a293fb-92cb-4908-bf90-d97ed0d831a6-metrics-certs") pod "router-default-7688c5db95-gcqb5" (UID: "66a293fb-92cb-4908-bf90-d97ed0d831a6") : secret "router-metrics-certs-default" not found
Jul 29 18:34:31.324738 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:31.324596    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66a293fb-92cb-4908-bf90-d97ed0d831a6-service-ca-bundle\") pod \"router-default-7688c5db95-gcqb5\" (UID: \"66a293fb-92cb-4908-bf90-d97ed0d831a6\") " pod="openshift-ingress/router-default-7688c5db95-gcqb5"
Jul 29 18:34:31.325089 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:31.324758    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/66a293fb-92cb-4908-bf90-d97ed0d831a6-service-ca-bundle podName:66a293fb-92cb-4908-bf90-d97ed0d831a6 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:32.324747034 +0000 UTC m=+36.594516099 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/66a293fb-92cb-4908-bf90-d97ed0d831a6-service-ca-bundle") pod "router-default-7688c5db95-gcqb5" (UID: "66a293fb-92cb-4908-bf90-d97ed0d831a6") : configmap references non-existent config key: service-ca.crt
Jul 29 18:34:32.335636 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:32.335602    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/26f43ad8-a7b4-4140-9ee0-7e9d55edf035-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-hb5lw\" (UID: \"26f43ad8-a7b4-4140-9ee0-7e9d55edf035\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-hb5lw"
Jul 29 18:34:32.336363 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:32.335648    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66a293fb-92cb-4908-bf90-d97ed0d831a6-metrics-certs\") pod \"router-default-7688c5db95-gcqb5\" (UID: \"66a293fb-92cb-4908-bf90-d97ed0d831a6\") " pod="openshift-ingress/router-default-7688c5db95-gcqb5"
Jul 29 18:34:32.336363 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:32.335684    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c854cbbc-3a2c-46ca-adf3-86fc3028bd7c-metrics-tls\") pod \"dns-default-95vjs\" (UID: \"c854cbbc-3a2c-46ca-adf3-86fc3028bd7c\") " pod="openshift-dns/dns-default-95vjs"
Jul 29 18:34:32.336363 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:32.335736    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66a293fb-92cb-4908-bf90-d97ed0d831a6-service-ca-bundle\") pod \"router-default-7688c5db95-gcqb5\" (UID: \"66a293fb-92cb-4908-bf90-d97ed0d831a6\") " pod="openshift-ingress/router-default-7688c5db95-gcqb5"
Jul 29 18:34:32.336363 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:32.335764    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 29 18:34:32.336363 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:32.335821    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c284fbd5-a70f-4085-948c-6a2392502e6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-jxmfh\" (UID: \"c284fbd5-a70f-4085-948c-6a2392502e6f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jxmfh"
Jul 29 18:34:32.336363 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:32.335841    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26f43ad8-a7b4-4140-9ee0-7e9d55edf035-samples-operator-tls podName:26f43ad8-a7b4-4140-9ee0-7e9d55edf035 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:34.335819606 +0000 UTC m=+38.605588699 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/26f43ad8-a7b4-4140-9ee0-7e9d55edf035-samples-operator-tls") pod "cluster-samples-operator-6748489656-hb5lw" (UID: "26f43ad8-a7b4-4140-9ee0-7e9d55edf035") : secret "samples-operator-tls" not found
Jul 29 18:34:32.336363 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:32.335840    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 29 18:34:32.336363 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:32.335885    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66a293fb-92cb-4908-bf90-d97ed0d831a6-metrics-certs podName:66a293fb-92cb-4908-bf90-d97ed0d831a6 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:34.335873695 +0000 UTC m=+38.605642777 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66a293fb-92cb-4908-bf90-d97ed0d831a6-metrics-certs") pod "router-default-7688c5db95-gcqb5" (UID: "66a293fb-92cb-4908-bf90-d97ed0d831a6") : secret "router-metrics-certs-default" not found
Jul 29 18:34:32.336363 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:32.335882    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e281e146-0b8a-4358-a98b-7f577f4ff08b-registry-tls\") pod \"image-registry-6cb6fb9448-jbzkl\" (UID: \"e281e146-0b8a-4358-a98b-7f577f4ff08b\") " pod="openshift-image-registry/image-registry-6cb6fb9448-jbzkl"
Jul 29 18:34:32.336363 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:32.335899    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 18:34:32.336363 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:32.335949    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c284fbd5-a70f-4085-948c-6a2392502e6f-cluster-monitoring-operator-tls podName:c284fbd5-a70f-4085-948c-6a2392502e6f nodeName:}" failed. No retries permitted until 2026-07-29 18:34:34.335934197 +0000 UTC m=+38.605703269 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c284fbd5-a70f-4085-948c-6a2392502e6f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-jxmfh" (UID: "c284fbd5-a70f-4085-948c-6a2392502e6f") : secret "cluster-monitoring-operator-tls" not found
Jul 29 18:34:32.336363 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:32.335966    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/66a293fb-92cb-4908-bf90-d97ed0d831a6-service-ca-bundle podName:66a293fb-92cb-4908-bf90-d97ed0d831a6 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:34.335958615 +0000 UTC m=+38.605727680 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/66a293fb-92cb-4908-bf90-d97ed0d831a6-service-ca-bundle") pod "router-default-7688c5db95-gcqb5" (UID: "66a293fb-92cb-4908-bf90-d97ed0d831a6") : configmap references non-existent config key: service-ca.crt
Jul 29 18:34:32.336363 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:32.335969    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 18:34:32.336363 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:32.335995    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 18:34:32.336363 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:32.336006    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cb6fb9448-jbzkl: secret "image-registry-tls" not found
Jul 29 18:34:32.336363 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:32.336045    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e281e146-0b8a-4358-a98b-7f577f4ff08b-registry-tls podName:e281e146-0b8a-4358-a98b-7f577f4ff08b nodeName:}" failed. No retries permitted until 2026-07-29 18:34:34.336035184 +0000 UTC m=+38.605804252 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e281e146-0b8a-4358-a98b-7f577f4ff08b-registry-tls") pod "image-registry-6cb6fb9448-jbzkl" (UID: "e281e146-0b8a-4358-a98b-7f577f4ff08b") : secret "image-registry-tls" not found
Jul 29 18:34:32.336906 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:32.336060    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c854cbbc-3a2c-46ca-adf3-86fc3028bd7c-metrics-tls podName:c854cbbc-3a2c-46ca-adf3-86fc3028bd7c nodeName:}" failed. No retries permitted until 2026-07-29 18:34:34.336051543 +0000 UTC m=+38.605820612 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c854cbbc-3a2c-46ca-adf3-86fc3028bd7c-metrics-tls") pod "dns-default-95vjs" (UID: "c854cbbc-3a2c-46ca-adf3-86fc3028bd7c") : secret "dns-default-metrics-tls" not found
Jul 29 18:34:32.574297 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:32.573915    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-j76ql"]
Jul 29 18:34:32.596147 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:32.596119    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-rxng5"]
Jul 29 18:34:32.606510 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:32.603253    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-rxng5"
Jul 29 18:34:32.606510 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:32.603946    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-4lqlg"]
Jul 29 18:34:32.608334 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:32.607313    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-pvn72\""
Jul 29 18:34:32.608334 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:32.607456    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 29 18:34:32.608334 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:32.607568    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 29 18:34:32.616308 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:32.616242    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-rxng5"]
Jul 29 18:34:32.641425 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:34:32.641371    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98400583_3b23_4bf0_be1f_54aa621542a0.slice/crio-de3a82a9b6f1ee78fb0f395daa1888154a639239125f22b51d424d21d8530f42 WatchSource:0}: Error finding container de3a82a9b6f1ee78fb0f395daa1888154a639239125f22b51d424d21d8530f42: Status 404 returned error can't find the container with id de3a82a9b6f1ee78fb0f395daa1888154a639239125f22b51d424d21d8530f42
Jul 29 18:34:32.735590 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:32.735520    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d55599b76-4k8r9"]
Jul 29 18:34:32.742649 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:32.742620    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b2274e5f-4130-477a-900f-915070daeb14-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-rxng5\" (UID: \"b2274e5f-4130-477a-900f-915070daeb14\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-rxng5"
Jul 29 18:34:32.742817 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:32.742676    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2274e5f-4130-477a-900f-915070daeb14-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-rxng5\" (UID: \"b2274e5f-4130-477a-900f-915070daeb14\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-rxng5"
Jul 29 18:34:32.743845 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:32.743766    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qhj2h"]
Jul 29 18:34:32.746517 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:32.746492    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-95ttd"]
Jul 29 18:34:32.746877 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:32.746826    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bp28x_6a62d29d-5bf6-444f-bd28-bf2d1e54ac87/konnectivity-agent/0.log" limit=1024
Jul 29 18:34:32.754020 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:32.753994    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cb7489487-kkmn9"]
Jul 29 18:34:32.756470 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:32.756371    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-ljn4r"]
Jul 29 18:34:32.760880 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:32.760856    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-v9nsr"]
Jul 29 18:34:32.775287 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:32.775257    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nj275"]
Jul 29 18:34:32.780880 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:32.780854    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58dbb54c4d-zmrbt"]
Jul 29 18:34:32.841936 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:32.841906    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bp28x_6a62d29d-5bf6-444f-bd28-bf2d1e54ac87/konnectivity-agent/0.log" limit=1024
Jul 29 18:34:32.843912 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:32.843893    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2274e5f-4130-477a-900f-915070daeb14-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-rxng5\" (UID: \"b2274e5f-4130-477a-900f-915070daeb14\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-rxng5"
Jul 29 18:34:32.844045 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:32.844032    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b2274e5f-4130-477a-900f-915070daeb14-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-rxng5\" (UID: \"b2274e5f-4130-477a-900f-915070daeb14\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-rxng5"
Jul 29 18:34:32.844100 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:32.844062    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 18:34:32.844156 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:32.844142    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2274e5f-4130-477a-900f-915070daeb14-networking-console-plugin-cert podName:b2274e5f-4130-477a-900f-915070daeb14 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:33.344119788 +0000 UTC m=+37.613888853 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b2274e5f-4130-477a-900f-915070daeb14-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-rxng5" (UID: "b2274e5f-4130-477a-900f-915070daeb14") : secret "networking-console-plugin-cert" not found
Jul 29 18:34:32.844651 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:32.844633    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b2274e5f-4130-477a-900f-915070daeb14-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-rxng5\" (UID: \"b2274e5f-4130-477a-900f-915070daeb14\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-rxng5"
Jul 29 18:34:32.942878 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:32.942839    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bp28x_6a62d29d-5bf6-444f-bd28-bf2d1e54ac87/konnectivity-agent/0.log" limit=1024
Jul 29 18:34:33.350257 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:33.349452    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2274e5f-4130-477a-900f-915070daeb14-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-rxng5\" (UID: \"b2274e5f-4130-477a-900f-915070daeb14\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-rxng5"
Jul 29 18:34:33.350257 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:33.349816    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 18:34:33.350257 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:33.349885    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2274e5f-4130-477a-900f-915070daeb14-networking-console-plugin-cert podName:b2274e5f-4130-477a-900f-915070daeb14 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:34.349865746 +0000 UTC m=+38.619634818 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b2274e5f-4130-477a-900f-915070daeb14-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-rxng5" (UID: "b2274e5f-4130-477a-900f-915070daeb14") : secret "networking-console-plugin-cert" not found
Jul 29 18:34:33.391609 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:33.391570    2559 generic.go:356] "Generic (PLEG): container finished" podID="6dad0cba-079a-4ec1-9f2b-8399f34fb80c" containerID="0a58b1c0c8a19f234c8f54acb9e775e7b9f843983677762296cf699e4fa1ac60" exitCode=0
Jul 29 18:34:33.391787 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:33.391714    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wgbf" event={"ID":"6dad0cba-079a-4ec1-9f2b-8399f34fb80c","Type":"ContainerDied","Data":"0a58b1c0c8a19f234c8f54acb9e775e7b9f843983677762296cf699e4fa1ac60"}
Jul 29 18:34:33.397628 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:33.397585    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-ljn4r" event={"ID":"4b692565-aef0-416f-8fad-2c74bca55d4d","Type":"ContainerStarted","Data":"7dc300f21c19f594b49cd7a57615fa7d085f5aef0c1a587411baaf41b380ca8a"}
Jul 29 18:34:33.399757 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:33.399685    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cb7489487-kkmn9" event={"ID":"1b53e40e-25fd-4563-a3f8-df861a6c51f7","Type":"ContainerStarted","Data":"b853f7e82dea9bff7588b0a61bce98f91c0990ca77f9cff94842a9a65c35b0c8"}
Jul 29 18:34:33.405351 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:33.405268    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qhj2h" event={"ID":"cc87b3c5-0b72-4fa2-9f87-ca3d4d6da614","Type":"ContainerStarted","Data":"fc95d3e42ac73f40d559055ae69ba26871b1f7e1005d5d2eabb3db66553dd0ff"}
Jul 29 18:34:33.412272 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:33.412193    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-j76ql" event={"ID":"a84d718f-459d-4e2e-9c9e-d08d1500b145","Type":"ContainerStarted","Data":"ba2f56e594b717ef9f352452b4aa5f1204ffd4c2758d67e69f08e76a918e243c"}
Jul 29 18:34:33.416407 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:33.416348    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-4lqlg" event={"ID":"98400583-3b23-4bf0-be1f-54aa621542a0","Type":"ContainerStarted","Data":"de3a82a9b6f1ee78fb0f395daa1888154a639239125f22b51d424d21d8530f42"}
Jul 29 18:34:33.420699 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:33.420620    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d55599b76-4k8r9" event={"ID":"c7c618e4-ba24-4fb5-8c20-47368c4f9322","Type":"ContainerStarted","Data":"a05b2b92e35ba38462379616633d229a79f6bc1c810e4bc81638534868ababd9"}
Jul 29 18:34:33.422557 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:33.422484    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-95ttd" event={"ID":"cad930eb-2903-4c78-b4c4-3bcc190d199d","Type":"ContainerStarted","Data":"b6fe4854d39617922abe2b32cfeb29cc32969f8117d05b49a0a902f9ccd696d2"}
Jul 29 18:34:33.424287 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:33.424208    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nj275" event={"ID":"05f777ff-5b55-41e4-b440-1705c2c3842e","Type":"ContainerStarted","Data":"c8ad72cbcb2b114409745c09fd1cc4a7784d5c142a2651cbc1ad7f690843fcb0"}
Jul 29 18:34:33.432505 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:33.432418    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58dbb54c4d-zmrbt" event={"ID":"542cc68a-f9f1-4d9c-8be9-1de3687ca26a","Type":"ContainerStarted","Data":"bc4cde2b019f6289a26f6210147eb69c42afbeec760cfd9e9f786a0e0dda61da"}
Jul 29 18:34:33.433987 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:33.433911    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-v9nsr" event={"ID":"7f8b4737-d4c2-4598-bd40-208015542831","Type":"ContainerStarted","Data":"c004bf1e4fdc274eeaa893fab0d41af985cb0e7b1a102f0eb3c19e62e7109fc2"}
Jul 29 18:34:34.362624 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:34.362321    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c854cbbc-3a2c-46ca-adf3-86fc3028bd7c-metrics-tls\") pod \"dns-default-95vjs\" (UID: \"c854cbbc-3a2c-46ca-adf3-86fc3028bd7c\") " pod="openshift-dns/dns-default-95vjs"
Jul 29 18:34:34.363057 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:34.362715    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66a293fb-92cb-4908-bf90-d97ed0d831a6-service-ca-bundle\") pod \"router-default-7688c5db95-gcqb5\" (UID: \"66a293fb-92cb-4908-bf90-d97ed0d831a6\") " pod="openshift-ingress/router-default-7688c5db95-gcqb5"
Jul 29 18:34:34.363057 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:34.362790    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2274e5f-4130-477a-900f-915070daeb14-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-rxng5\" (UID: \"b2274e5f-4130-477a-900f-915070daeb14\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-rxng5"
Jul 29 18:34:34.363057 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:34.362862    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c284fbd5-a70f-4085-948c-6a2392502e6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-jxmfh\" (UID: \"c284fbd5-a70f-4085-948c-6a2392502e6f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jxmfh"
Jul 29 18:34:34.363057 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:34.362894    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e281e146-0b8a-4358-a98b-7f577f4ff08b-registry-tls\") pod \"image-registry-6cb6fb9448-jbzkl\" (UID: \"e281e146-0b8a-4358-a98b-7f577f4ff08b\") " pod="openshift-image-registry/image-registry-6cb6fb9448-jbzkl"
Jul 29 18:34:34.363057 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:34.362935    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/26f43ad8-a7b4-4140-9ee0-7e9d55edf035-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-hb5lw\" (UID: \"26f43ad8-a7b4-4140-9ee0-7e9d55edf035\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-hb5lw"
Jul 29 18:34:34.363057 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:34.362966    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66a293fb-92cb-4908-bf90-d97ed0d831a6-metrics-certs\") pod \"router-default-7688c5db95-gcqb5\" (UID: \"66a293fb-92cb-4908-bf90-d97ed0d831a6\") " pod="openshift-ingress/router-default-7688c5db95-gcqb5"
Jul 29 18:34:34.363352 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:34.363107    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 29 18:34:34.363352 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:34.363171    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66a293fb-92cb-4908-bf90-d97ed0d831a6-metrics-certs podName:66a293fb-92cb-4908-bf90-d97ed0d831a6 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:38.36315328 +0000 UTC m=+42.632922347 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66a293fb-92cb-4908-bf90-d97ed0d831a6-metrics-certs") pod "router-default-7688c5db95-gcqb5" (UID: "66a293fb-92cb-4908-bf90-d97ed0d831a6") : secret "router-metrics-certs-default" not found
Jul 29 18:34:34.363624 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:34.363602    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 18:34:34.363702 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:34.363662    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c854cbbc-3a2c-46ca-adf3-86fc3028bd7c-metrics-tls podName:c854cbbc-3a2c-46ca-adf3-86fc3028bd7c nodeName:}" failed. No retries permitted until 2026-07-29 18:34:38.363647493 +0000 UTC m=+42.633416560 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c854cbbc-3a2c-46ca-adf3-86fc3028bd7c-metrics-tls") pod "dns-default-95vjs" (UID: "c854cbbc-3a2c-46ca-adf3-86fc3028bd7c") : secret "dns-default-metrics-tls" not found
Jul 29 18:34:34.363759 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:34.363727    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/66a293fb-92cb-4908-bf90-d97ed0d831a6-service-ca-bundle podName:66a293fb-92cb-4908-bf90-d97ed0d831a6 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:38.363717515 +0000 UTC m=+42.633486585 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/66a293fb-92cb-4908-bf90-d97ed0d831a6-service-ca-bundle") pod "router-default-7688c5db95-gcqb5" (UID: "66a293fb-92cb-4908-bf90-d97ed0d831a6") : configmap references non-existent config key: service-ca.crt
Jul 29 18:34:34.363813 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:34.363790    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 18:34:34.363858 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:34.363821    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2274e5f-4130-477a-900f-915070daeb14-networking-console-plugin-cert podName:b2274e5f-4130-477a-900f-915070daeb14 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:36.363811841 +0000 UTC m=+40.633580920 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b2274e5f-4130-477a-900f-915070daeb14-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-rxng5" (UID: "b2274e5f-4130-477a-900f-915070daeb14") : secret "networking-console-plugin-cert" not found
Jul 29 18:34:34.363910 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:34.363885    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 18:34:34.363951 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:34.363915    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c284fbd5-a70f-4085-948c-6a2392502e6f-cluster-monitoring-operator-tls podName:c284fbd5-a70f-4085-948c-6a2392502e6f nodeName:}" failed. No retries permitted until 2026-07-29 18:34:38.363904819 +0000 UTC m=+42.633673899 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c284fbd5-a70f-4085-948c-6a2392502e6f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-jxmfh" (UID: "c284fbd5-a70f-4085-948c-6a2392502e6f") : secret "cluster-monitoring-operator-tls" not found
Jul 29 18:34:34.364002 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:34.363973    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 18:34:34.364002 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:34.363984    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cb6fb9448-jbzkl: secret "image-registry-tls" not found
Jul 29 18:34:34.364090 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:34.364016    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e281e146-0b8a-4358-a98b-7f577f4ff08b-registry-tls podName:e281e146-0b8a-4358-a98b-7f577f4ff08b nodeName:}" failed. No retries permitted until 2026-07-29 18:34:38.364004722 +0000 UTC m=+42.633773801 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e281e146-0b8a-4358-a98b-7f577f4ff08b-registry-tls") pod "image-registry-6cb6fb9448-jbzkl" (UID: "e281e146-0b8a-4358-a98b-7f577f4ff08b") : secret "image-registry-tls" not found
Jul 29 18:34:34.364090 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:34.364073    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 29 18:34:34.364187 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:34.364103    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26f43ad8-a7b4-4140-9ee0-7e9d55edf035-samples-operator-tls podName:26f43ad8-a7b4-4140-9ee0-7e9d55edf035 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:38.364093658 +0000 UTC m=+42.633862741 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/26f43ad8-a7b4-4140-9ee0-7e9d55edf035-samples-operator-tls") pod "cluster-samples-operator-6748489656-hb5lw" (UID: "26f43ad8-a7b4-4140-9ee0-7e9d55edf035") : secret "samples-operator-tls" not found
Jul 29 18:34:34.482403 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:34.482350    2559 generic.go:356] "Generic (PLEG): container finished" podID="6dad0cba-079a-4ec1-9f2b-8399f34fb80c" containerID="c00741d82288176421ea986698f551620476f01e729600179edc37d8e16ee733" exitCode=0
Jul 29 18:34:34.482580 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:34.482432    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wgbf" event={"ID":"6dad0cba-079a-4ec1-9f2b-8399f34fb80c","Type":"ContainerDied","Data":"c00741d82288176421ea986698f551620476f01e729600179edc37d8e16ee733"}
Jul 29 18:34:35.329951 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:35.329686    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bp28x_6a62d29d-5bf6-444f-bd28-bf2d1e54ac87/konnectivity-agent/0.log" limit=1024
Jul 29 18:34:35.426853 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:35.426760    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bp28x_6a62d29d-5bf6-444f-bd28-bf2d1e54ac87/konnectivity-agent/0.log" limit=1024
Jul 29 18:34:35.521465 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:35.520500    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wgbf" event={"ID":"6dad0cba-079a-4ec1-9f2b-8399f34fb80c","Type":"ContainerStarted","Data":"9b16986d96bcd7ca8cd1fdd5f86f2aaaa5b8b8066b063396c6e3ccaf539c3c17"}
Jul 29 18:34:35.670963 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:35.670780    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bp28x_6a62d29d-5bf6-444f-bd28-bf2d1e54ac87/konnectivity-agent/0.log" limit=1024
Jul 29 18:34:35.739085 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:35.739050    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bp28x_6a62d29d-5bf6-444f-bd28-bf2d1e54ac87/konnectivity-agent/0.log" limit=1024
Jul 29 18:34:36.390090 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:36.390043    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2274e5f-4130-477a-900f-915070daeb14-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-rxng5\" (UID: \"b2274e5f-4130-477a-900f-915070daeb14\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-rxng5"
Jul 29 18:34:36.390246 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:36.390214    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 18:34:36.390296 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:36.390280    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2274e5f-4130-477a-900f-915070daeb14-networking-console-plugin-cert podName:b2274e5f-4130-477a-900f-915070daeb14 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:40.390260851 +0000 UTC m=+44.660029946 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b2274e5f-4130-477a-900f-915070daeb14-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-rxng5" (UID: "b2274e5f-4130-477a-900f-915070daeb14") : secret "networking-console-plugin-cert" not found
Jul 29 18:34:37.478984 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:37.478948    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bp28x_6a62d29d-5bf6-444f-bd28-bf2d1e54ac87/konnectivity-agent/0.log" limit=1024
Jul 29 18:34:37.533251 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:37.533216    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bp28x_6a62d29d-5bf6-444f-bd28-bf2d1e54ac87/konnectivity-agent/0.log" limit=1024
Jul 29 18:34:38.412616 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:38.412571    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c284fbd5-a70f-4085-948c-6a2392502e6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-jxmfh\" (UID: \"c284fbd5-a70f-4085-948c-6a2392502e6f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jxmfh"
Jul 29 18:34:38.412616 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:38.412630    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e281e146-0b8a-4358-a98b-7f577f4ff08b-registry-tls\") pod \"image-registry-6cb6fb9448-jbzkl\" (UID: \"e281e146-0b8a-4358-a98b-7f577f4ff08b\") " pod="openshift-image-registry/image-registry-6cb6fb9448-jbzkl"
Jul 29 18:34:38.412891 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:38.412733    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 18:34:38.412891 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:38.412750    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cb6fb9448-jbzkl: secret "image-registry-tls" not found
Jul 29 18:34:38.412891 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:38.412766    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 18:34:38.412891 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:38.412826    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e281e146-0b8a-4358-a98b-7f577f4ff08b-registry-tls podName:e281e146-0b8a-4358-a98b-7f577f4ff08b nodeName:}" failed. No retries permitted until 2026-07-29 18:34:46.412806739 +0000 UTC m=+50.682575820 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e281e146-0b8a-4358-a98b-7f577f4ff08b-registry-tls") pod "image-registry-6cb6fb9448-jbzkl" (UID: "e281e146-0b8a-4358-a98b-7f577f4ff08b") : secret "image-registry-tls" not found
Jul 29 18:34:38.412891 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:38.412852    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c284fbd5-a70f-4085-948c-6a2392502e6f-cluster-monitoring-operator-tls podName:c284fbd5-a70f-4085-948c-6a2392502e6f nodeName:}" failed. No retries permitted until 2026-07-29 18:34:46.412832791 +0000 UTC m=+50.682601872 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c284fbd5-a70f-4085-948c-6a2392502e6f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-jxmfh" (UID: "c284fbd5-a70f-4085-948c-6a2392502e6f") : secret "cluster-monitoring-operator-tls" not found
Jul 29 18:34:38.413092 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:38.412907    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/26f43ad8-a7b4-4140-9ee0-7e9d55edf035-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-hb5lw\" (UID: \"26f43ad8-a7b4-4140-9ee0-7e9d55edf035\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-hb5lw"
Jul 29 18:34:38.413092 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:38.412944    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66a293fb-92cb-4908-bf90-d97ed0d831a6-metrics-certs\") pod \"router-default-7688c5db95-gcqb5\" (UID: \"66a293fb-92cb-4908-bf90-d97ed0d831a6\") " pod="openshift-ingress/router-default-7688c5db95-gcqb5"
Jul 29 18:34:38.413092 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:38.412986    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c854cbbc-3a2c-46ca-adf3-86fc3028bd7c-metrics-tls\") pod \"dns-default-95vjs\" (UID: \"c854cbbc-3a2c-46ca-adf3-86fc3028bd7c\") " pod="openshift-dns/dns-default-95vjs"
Jul 29 18:34:38.413092 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:38.413050    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66a293fb-92cb-4908-bf90-d97ed0d831a6-service-ca-bundle\") pod \"router-default-7688c5db95-gcqb5\" (UID: \"66a293fb-92cb-4908-bf90-d97ed0d831a6\") " pod="openshift-ingress/router-default-7688c5db95-gcqb5"
Jul 29 18:34:38.413092 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:38.413070    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 29 18:34:38.413245 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:38.413128    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 18:34:38.413245 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:38.413070    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 29 18:34:38.413245 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:38.413130    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26f43ad8-a7b4-4140-9ee0-7e9d55edf035-samples-operator-tls podName:26f43ad8-a7b4-4140-9ee0-7e9d55edf035 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:46.413112378 +0000 UTC m=+50.682881465 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/26f43ad8-a7b4-4140-9ee0-7e9d55edf035-samples-operator-tls") pod "cluster-samples-operator-6748489656-hb5lw" (UID: "26f43ad8-a7b4-4140-9ee0-7e9d55edf035") : secret "samples-operator-tls" not found
Jul 29 18:34:38.413245 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:38.413184    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/66a293fb-92cb-4908-bf90-d97ed0d831a6-service-ca-bundle podName:66a293fb-92cb-4908-bf90-d97ed0d831a6 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:46.413172575 +0000 UTC m=+50.682941646 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/66a293fb-92cb-4908-bf90-d97ed0d831a6-service-ca-bundle") pod "router-default-7688c5db95-gcqb5" (UID: "66a293fb-92cb-4908-bf90-d97ed0d831a6") : configmap references non-existent config key: service-ca.crt
Jul 29 18:34:38.413401 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:38.413249    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c854cbbc-3a2c-46ca-adf3-86fc3028bd7c-metrics-tls podName:c854cbbc-3a2c-46ca-adf3-86fc3028bd7c nodeName:}" failed. No retries permitted until 2026-07-29 18:34:46.413222532 +0000 UTC m=+50.682991600 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c854cbbc-3a2c-46ca-adf3-86fc3028bd7c-metrics-tls") pod "dns-default-95vjs" (UID: "c854cbbc-3a2c-46ca-adf3-86fc3028bd7c") : secret "dns-default-metrics-tls" not found
Jul 29 18:34:38.413401 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:38.413267    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66a293fb-92cb-4908-bf90-d97ed0d831a6-metrics-certs podName:66a293fb-92cb-4908-bf90-d97ed0d831a6 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:46.413260432 +0000 UTC m=+50.683029497 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66a293fb-92cb-4908-bf90-d97ed0d831a6-metrics-certs") pod "router-default-7688c5db95-gcqb5" (UID: "66a293fb-92cb-4908-bf90-d97ed0d831a6") : secret "router-metrics-certs-default" not found
Jul 29 18:34:40.433543 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:40.433504    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2274e5f-4130-477a-900f-915070daeb14-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-rxng5\" (UID: \"b2274e5f-4130-477a-900f-915070daeb14\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-rxng5"
Jul 29 18:34:40.434053 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:40.433687    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 18:34:40.434053 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:40.433773    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2274e5f-4130-477a-900f-915070daeb14-networking-console-plugin-cert podName:b2274e5f-4130-477a-900f-915070daeb14 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:48.433751862 +0000 UTC m=+52.703520949 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b2274e5f-4130-477a-900f-915070daeb14-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-rxng5" (UID: "b2274e5f-4130-477a-900f-915070daeb14") : secret "networking-console-plugin-cert" not found
Jul 29 18:34:41.977114 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:41.977069    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bp28x_6a62d29d-5bf6-444f-bd28-bf2d1e54ac87/konnectivity-agent/0.log" limit=1024
Jul 29 18:34:42.031864 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:42.031826    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bp28x_6a62d29d-5bf6-444f-bd28-bf2d1e54ac87/konnectivity-agent/0.log" limit=1024
Jul 29 18:34:43.056306 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:43.056262    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bp28x_6a62d29d-5bf6-444f-bd28-bf2d1e54ac87/konnectivity-agent/0.log" limit=1024
Jul 29 18:34:43.157535 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:43.157494    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bp28x_6a62d29d-5bf6-444f-bd28-bf2d1e54ac87/konnectivity-agent/0.log" limit=1024
Jul 29 18:34:45.014456 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:45.014416    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bp28x_6a62d29d-5bf6-444f-bd28-bf2d1e54ac87/konnectivity-agent/0.log" limit=1024
Jul 29 18:34:45.106556 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:45.106517    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bp28x_6a62d29d-5bf6-444f-bd28-bf2d1e54ac87/konnectivity-agent/0.log" limit=1024
Jul 29 18:34:45.947306 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:45.947267    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bp28x_6a62d29d-5bf6-444f-bd28-bf2d1e54ac87/konnectivity-agent/0.log" limit=1024
Jul 29 18:34:46.497208 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:46.497175    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c284fbd5-a70f-4085-948c-6a2392502e6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-jxmfh\" (UID: \"c284fbd5-a70f-4085-948c-6a2392502e6f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jxmfh"
Jul 29 18:34:46.497804 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:46.497708    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e281e146-0b8a-4358-a98b-7f577f4ff08b-registry-tls\") pod \"image-registry-6cb6fb9448-jbzkl\" (UID: \"e281e146-0b8a-4358-a98b-7f577f4ff08b\") " pod="openshift-image-registry/image-registry-6cb6fb9448-jbzkl"
Jul 29 18:34:46.497804 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:46.497761    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/26f43ad8-a7b4-4140-9ee0-7e9d55edf035-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-hb5lw\" (UID: \"26f43ad8-a7b4-4140-9ee0-7e9d55edf035\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-hb5lw"
Jul 29 18:34:46.498250 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:46.497467    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 18:34:46.498250 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:46.498091    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c284fbd5-a70f-4085-948c-6a2392502e6f-cluster-monitoring-operator-tls podName:c284fbd5-a70f-4085-948c-6a2392502e6f nodeName:}" failed. No retries permitted until 2026-07-29 18:35:02.498071713 +0000 UTC m=+66.767840785 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c284fbd5-a70f-4085-948c-6a2392502e6f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-jxmfh" (UID: "c284fbd5-a70f-4085-948c-6a2392502e6f") : secret "cluster-monitoring-operator-tls" not found
Jul 29 18:34:46.498250 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:46.498026    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 18:34:46.498250 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:46.498111    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cb6fb9448-jbzkl: secret "image-registry-tls" not found
Jul 29 18:34:46.498250 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:46.498140    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e281e146-0b8a-4358-a98b-7f577f4ff08b-registry-tls podName:e281e146-0b8a-4358-a98b-7f577f4ff08b nodeName:}" failed. No retries permitted until 2026-07-29 18:35:02.498130975 +0000 UTC m=+66.767900046 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e281e146-0b8a-4358-a98b-7f577f4ff08b-registry-tls") pod "image-registry-6cb6fb9448-jbzkl" (UID: "e281e146-0b8a-4358-a98b-7f577f4ff08b") : secret "image-registry-tls" not found
Jul 29 18:34:46.498250 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:46.498197    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 29 18:34:46.498250 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:46.498225    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26f43ad8-a7b4-4140-9ee0-7e9d55edf035-samples-operator-tls podName:26f43ad8-a7b4-4140-9ee0-7e9d55edf035 nodeName:}" failed. No retries permitted until 2026-07-29 18:35:02.49821584 +0000 UTC m=+66.767984910 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/26f43ad8-a7b4-4140-9ee0-7e9d55edf035-samples-operator-tls") pod "cluster-samples-operator-6748489656-hb5lw" (UID: "26f43ad8-a7b4-4140-9ee0-7e9d55edf035") : secret "samples-operator-tls" not found
Jul 29 18:34:46.498781 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:46.498722    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 29 18:34:46.498781 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:46.498765    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66a293fb-92cb-4908-bf90-d97ed0d831a6-metrics-certs podName:66a293fb-92cb-4908-bf90-d97ed0d831a6 nodeName:}" failed. No retries permitted until 2026-07-29 18:35:02.498751591 +0000 UTC m=+66.768520662 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66a293fb-92cb-4908-bf90-d97ed0d831a6-metrics-certs") pod "router-default-7688c5db95-gcqb5" (UID: "66a293fb-92cb-4908-bf90-d97ed0d831a6") : secret "router-metrics-certs-default" not found
Jul 29 18:34:46.499318 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:46.497793    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66a293fb-92cb-4908-bf90-d97ed0d831a6-metrics-certs\") pod \"router-default-7688c5db95-gcqb5\" (UID: \"66a293fb-92cb-4908-bf90-d97ed0d831a6\") " pod="openshift-ingress/router-default-7688c5db95-gcqb5"
Jul 29 18:34:46.499318 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:46.498955    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c854cbbc-3a2c-46ca-adf3-86fc3028bd7c-metrics-tls\") pod \"dns-default-95vjs\" (UID: \"c854cbbc-3a2c-46ca-adf3-86fc3028bd7c\") " pod="openshift-dns/dns-default-95vjs"
Jul 29 18:34:46.499318 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:46.499027    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66a293fb-92cb-4908-bf90-d97ed0d831a6-service-ca-bundle\") pod \"router-default-7688c5db95-gcqb5\" (UID: \"66a293fb-92cb-4908-bf90-d97ed0d831a6\") " pod="openshift-ingress/router-default-7688c5db95-gcqb5"
Jul 29 18:34:46.499318 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:46.499206    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/66a293fb-92cb-4908-bf90-d97ed0d831a6-service-ca-bundle podName:66a293fb-92cb-4908-bf90-d97ed0d831a6 nodeName:}" failed. No retries permitted until 2026-07-29 18:35:02.49919465 +0000 UTC m=+66.768963720 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/66a293fb-92cb-4908-bf90-d97ed0d831a6-service-ca-bundle") pod "router-default-7688c5db95-gcqb5" (UID: "66a293fb-92cb-4908-bf90-d97ed0d831a6") : configmap references non-existent config key: service-ca.crt
Jul 29 18:34:46.499318 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:46.499278    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 18:34:46.499318 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:46.499304    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c854cbbc-3a2c-46ca-adf3-86fc3028bd7c-metrics-tls podName:c854cbbc-3a2c-46ca-adf3-86fc3028bd7c nodeName:}" failed. No retries permitted until 2026-07-29 18:35:02.499295064 +0000 UTC m=+66.769064133 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c854cbbc-3a2c-46ca-adf3-86fc3028bd7c-metrics-tls") pod "dns-default-95vjs" (UID: "c854cbbc-3a2c-46ca-adf3-86fc3028bd7c") : secret "dns-default-metrics-tls" not found
Jul 29 18:34:46.572405 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:46.568866    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d55599b76-4k8r9" event={"ID":"c7c618e4-ba24-4fb5-8c20-47368c4f9322","Type":"ContainerStarted","Data":"c9d265001919bdfa81b7eb0a8aa100c76cfe025756f51f0981e29bd2e35e07c6"}
Jul 29 18:34:46.572405 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:46.571175    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-95ttd" event={"ID":"cad930eb-2903-4c78-b4c4-3bcc190d199d","Type":"ContainerStarted","Data":"1647e407180f226f83d92072917ac903aee86a66fe3256562199e1dcee781e1a"}
Jul 29 18:34:46.578404 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:46.575026    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nj275" event={"ID":"05f777ff-5b55-41e4-b440-1705c2c3842e","Type":"ContainerStarted","Data":"bdc4eef31622ba6b28f484d52a061c3479b51e041c5934b90b167d37259d2891"}
Jul 29 18:34:46.578404 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:46.577059    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58dbb54c4d-zmrbt" event={"ID":"542cc68a-f9f1-4d9c-8be9-1de3687ca26a","Type":"ContainerStarted","Data":"dbd6708e7042df98dc40c1260e88263a645b12370b782ef636670288083f0119"}
Jul 29 18:34:46.579415 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:46.578727    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-v9nsr_7f8b4737-d4c2-4598-bd40-208015542831/console-operator/0.log"
Jul 29 18:34:46.579415 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:46.578763    2559 generic.go:356] "Generic (PLEG): container finished" podID="7f8b4737-d4c2-4598-bd40-208015542831" containerID="8ad5a606b9ea3b4217c67814014b8f345c31e9f972729fbfacd1b37c0e3ea1db" exitCode=255
Jul 29 18:34:46.579415 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:46.579085    2559 scope.go:117] "RemoveContainer" containerID="8ad5a606b9ea3b4217c67814014b8f345c31e9f972729fbfacd1b37c0e3ea1db"
Jul 29 18:34:46.579415 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:46.579327    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-v9nsr" event={"ID":"7f8b4737-d4c2-4598-bd40-208015542831","Type":"ContainerDied","Data":"8ad5a606b9ea3b4217c67814014b8f345c31e9f972729fbfacd1b37c0e3ea1db"}
Jul 29 18:34:46.583961 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:46.583889    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-ljn4r" event={"ID":"4b692565-aef0-416f-8fad-2c74bca55d4d","Type":"ContainerStarted","Data":"109279223e582651826357ba20b3207941eeed8fc465a307bd617b2ffec18782"}
Jul 29 18:34:46.586292 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:46.586144    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4wgbf" podStartSLOduration=19.236747683 podStartE2EDuration="50.586124992s" totalImagesPullingTime="31.349377309s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:33:56 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:35.547604294 +0000 UTC m=+39.817373381" watchObservedRunningTime="2026-07-29 18:34:46.586124992 +0000 UTC m=+50.855894080"
Jul 29 18:34:46.594098 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:46.589617    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cb7489487-kkmn9" event={"ID":"1b53e40e-25fd-4563-a3f8-df861a6c51f7","Type":"ContainerStarted","Data":"fa0c1dea65819db72a6e4947c51a88280f26194e13dabb65724fc241c52ce8e3"}
Jul 29 18:34:46.594098 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:46.590536    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cb7489487-kkmn9"
Jul 29 18:34:46.594098 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:46.592035    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qhj2h" event={"ID":"cc87b3c5-0b72-4fa2-9f87-ca3d4d6da614","Type":"ContainerStarted","Data":"159a4ed944283987038e94e19118bfb708522c3006a80a4c973e2d0732fff6de"}
Jul 29 18:34:46.594098 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:46.592732    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cb7489487-kkmn9"
Jul 29 18:34:46.596677 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:46.596557    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d55599b76-4k8r9" podStartSLOduration=32.255192933000004 podStartE2EDuration="45.596544509s" totalImagesPullingTime="13.341351576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:34:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:46.587514357 +0000 UTC m=+50.857283444" watchObservedRunningTime="2026-07-29 18:34:46.596544509 +0000 UTC m=+50.866313595"
Jul 29 18:34:46.603783 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:46.603252    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-j76ql" event={"ID":"a84d718f-459d-4e2e-9c9e-d08d1500b145","Type":"ContainerStarted","Data":"b3a5b8fc0938ad47f5979f658d7ec1781b3975361eee90984b83d12b131d813e"}
Jul 29 18:34:46.609789 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:46.609297    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-4lqlg" event={"ID":"98400583-3b23-4bf0-be1f-54aa621542a0","Type":"ContainerStarted","Data":"28455404c7dcd7863247aba085c94b73f3afdb94ded2a79f2e370de199bb3ee2"}
Jul 29 18:34:46.627043 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:46.626987    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-ljn4r" podStartSLOduration=37.295377145 podStartE2EDuration="48.626967748s" totalImagesPullingTime="11.331590603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:33:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:46.626658457 +0000 UTC m=+50.896427544" watchObservedRunningTime="2026-07-29 18:34:46.626967748 +0000 UTC m=+50.896736837"
Jul 29 18:34:46.646309 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:46.645830    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nj275" podStartSLOduration=30.371289797 podStartE2EDuration="43.645805299s" totalImagesPullingTime="13.274515502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:34:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:46.64463169 +0000 UTC m=+50.914400790" watchObservedRunningTime="2026-07-29 18:34:46.645805299 +0000 UTC m=+50.915574379"
Jul 29 18:34:46.666006 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:46.664712    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-95ttd" podStartSLOduration=27.388772562 podStartE2EDuration="40.664691685s" totalImagesPullingTime="13.275919123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:34:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:46.664040724 +0000 UTC m=+50.933809812" watchObservedRunningTime="2026-07-29 18:34:46.664691685 +0000 UTC m=+50.934460770"
Jul 29 18:34:46.687036 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:46.686961    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cb7489487-kkmn9" podStartSLOduration=32.333685689 podStartE2EDuration="45.686940194s" totalImagesPullingTime="13.353254505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:34:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:46.685836896 +0000 UTC m=+50.955605985" watchObservedRunningTime="2026-07-29 18:34:46.686940194 +0000 UTC m=+50.956709282"
Jul 29 18:34:46.717182 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:46.713735    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qhj2h" podStartSLOduration=3.371153647 podStartE2EDuration="16.713712401s" totalImagesPullingTime="13.342558754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:46.711223101 +0000 UTC m=+50.980992188" watchObservedRunningTime="2026-07-29 18:34:46.713712401 +0000 UTC m=+50.983481487"
Jul 29 18:34:46.719079 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:46.718249    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bp28x_6a62d29d-5bf6-444f-bd28-bf2d1e54ac87/konnectivity-agent/0.log" limit=1024
Jul 29 18:34:46.728066 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:46.728004    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-4lqlg" podStartSLOduration=27.24843058 podStartE2EDuration="40.727984308s" totalImagesPullingTime="13.479553728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:34:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:46.727854831 +0000 UTC m=+50.997623919" watchObservedRunningTime="2026-07-29 18:34:46.727984308 +0000 UTC m=+50.997753377"
Jul 29 18:34:46.765603 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:46.759706    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-j76ql" podStartSLOduration=36.211092744 podStartE2EDuration="49.759684147s" totalImagesPullingTime="13.548591403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:33:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:46.753430085 +0000 UTC m=+51.023199172" watchObservedRunningTime="2026-07-29 18:34:46.759684147 +0000 UTC m=+51.029453239"
Jul 29 18:34:46.826135 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:46.826091    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bp28x_6a62d29d-5bf6-444f-bd28-bf2d1e54ac87/konnectivity-agent/0.log" limit=1024
Jul 29 18:34:47.616091 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:47.616050    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58dbb54c4d-zmrbt" event={"ID":"542cc68a-f9f1-4d9c-8be9-1de3687ca26a","Type":"ContainerStarted","Data":"65154b16dcf8fa1d8fc1028f219ee2121991d15676cd635fbe1152598be5fde6"}
Jul 29 18:34:47.616553 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:47.616108    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58dbb54c4d-zmrbt" event={"ID":"542cc68a-f9f1-4d9c-8be9-1de3687ca26a","Type":"ContainerStarted","Data":"c85271eecc74ba266f05af8706eb26fe96e5e1208dd98d462563eb361e12d616"}
Jul 29 18:34:47.630236 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:47.630209    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-v9nsr_7f8b4737-d4c2-4598-bd40-208015542831/console-operator/1.log"
Jul 29 18:34:47.630717 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:47.630700    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-v9nsr_7f8b4737-d4c2-4598-bd40-208015542831/console-operator/0.log"
Jul 29 18:34:47.630825 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:47.630740    2559 generic.go:356] "Generic (PLEG): container finished" podID="7f8b4737-d4c2-4598-bd40-208015542831" containerID="126c4356a4bb1b565afc72e63d7edeff663cabae3115ba399c8253e06bc08656" exitCode=255
Jul 29 18:34:47.630952 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:47.630929    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-v9nsr" event={"ID":"7f8b4737-d4c2-4598-bd40-208015542831","Type":"ContainerDied","Data":"126c4356a4bb1b565afc72e63d7edeff663cabae3115ba399c8253e06bc08656"}
Jul 29 18:34:47.631129 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:47.631102    2559 scope.go:117] "RemoveContainer" containerID="8ad5a606b9ea3b4217c67814014b8f345c31e9f972729fbfacd1b37c0e3ea1db"
Jul 29 18:34:47.631547 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:47.631530    2559 scope.go:117] "RemoveContainer" containerID="126c4356a4bb1b565afc72e63d7edeff663cabae3115ba399c8253e06bc08656"
Jul 29 18:34:47.632209 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:47.632186    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-v9nsr_openshift-console-operator(7f8b4737-d4c2-4598-bd40-208015542831)\"" pod="openshift-console-operator/console-operator-584dcb6b54-v9nsr" podUID="7f8b4737-d4c2-4598-bd40-208015542831"
Jul 29 18:34:47.643276 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:47.643224    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58dbb54c4d-zmrbt" podStartSLOduration=33.303773676 podStartE2EDuration="46.643210376s" totalImagesPullingTime="13.3394367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:34:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:47.641891021 +0000 UTC m=+51.911660109" watchObservedRunningTime="2026-07-29 18:34:47.643210376 +0000 UTC m=+51.912979461"
Jul 29 18:34:47.812336 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:47.812303    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wpg7b"]
Jul 29 18:34:47.836642 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:47.836447    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wpg7b"]
Jul 29 18:34:47.836878 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:47.836604    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wpg7b"
Jul 29 18:34:47.841175 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:47.840620    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9grlj\""
Jul 29 18:34:47.841175 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:47.840658    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 29 18:34:47.841175 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:47.840772    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 29 18:34:47.847896 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:47.847856    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bp28x_6a62d29d-5bf6-444f-bd28-bf2d1e54ac87/konnectivity-agent/0.log" limit=1024
Jul 29 18:34:47.917328 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:47.917228    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c9d825b-217d-499b-b8a4-b0dbf396b283-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wpg7b\" (UID: \"6c9d825b-217d-499b-b8a4-b0dbf396b283\") " pod="openshift-insights/insights-runtime-extractor-wpg7b"
Jul 29 18:34:47.917499 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:47.917372    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c9d825b-217d-499b-b8a4-b0dbf396b283-data-volume\") pod \"insights-runtime-extractor-wpg7b\" (UID: \"6c9d825b-217d-499b-b8a4-b0dbf396b283\") " pod="openshift-insights/insights-runtime-extractor-wpg7b"
Jul 29 18:34:47.917558 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:47.917530    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c9d825b-217d-499b-b8a4-b0dbf396b283-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wpg7b\" (UID: \"6c9d825b-217d-499b-b8a4-b0dbf396b283\") " pod="openshift-insights/insights-runtime-extractor-wpg7b"
Jul 29 18:34:47.917594 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:47.917572    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94g26\" (UniqueName: \"kubernetes.io/projected/6c9d825b-217d-499b-b8a4-b0dbf396b283-kube-api-access-94g26\") pod \"insights-runtime-extractor-wpg7b\" (UID: \"6c9d825b-217d-499b-b8a4-b0dbf396b283\") " pod="openshift-insights/insights-runtime-extractor-wpg7b"
Jul 29 18:34:47.917671 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:47.917655    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6c9d825b-217d-499b-b8a4-b0dbf396b283-crio-socket\") pod \"insights-runtime-extractor-wpg7b\" (UID: \"6c9d825b-217d-499b-b8a4-b0dbf396b283\") " pod="openshift-insights/insights-runtime-extractor-wpg7b"
Jul 29 18:34:47.919514 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:47.919485    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bp28x_6a62d29d-5bf6-444f-bd28-bf2d1e54ac87/konnectivity-agent/0.log" limit=1024
Jul 29 18:34:48.018440 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:48.018368    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c9d825b-217d-499b-b8a4-b0dbf396b283-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wpg7b\" (UID: \"6c9d825b-217d-499b-b8a4-b0dbf396b283\") " pod="openshift-insights/insights-runtime-extractor-wpg7b"
Jul 29 18:34:48.018715 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:48.018489    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c9d825b-217d-499b-b8a4-b0dbf396b283-data-volume\") pod \"insights-runtime-extractor-wpg7b\" (UID: \"6c9d825b-217d-499b-b8a4-b0dbf396b283\") " pod="openshift-insights/insights-runtime-extractor-wpg7b"
Jul 29 18:34:48.018715 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:48.018543    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 18:34:48.018715 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:48.018564    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c9d825b-217d-499b-b8a4-b0dbf396b283-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wpg7b\" (UID: \"6c9d825b-217d-499b-b8a4-b0dbf396b283\") " pod="openshift-insights/insights-runtime-extractor-wpg7b"
Jul 29 18:34:48.018715 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:48.018594    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94g26\" (UniqueName: \"kubernetes.io/projected/6c9d825b-217d-499b-b8a4-b0dbf396b283-kube-api-access-94g26\") pod \"insights-runtime-extractor-wpg7b\" (UID: \"6c9d825b-217d-499b-b8a4-b0dbf396b283\") " pod="openshift-insights/insights-runtime-extractor-wpg7b"
Jul 29 18:34:48.018715 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:48.018619    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c9d825b-217d-499b-b8a4-b0dbf396b283-insights-runtime-extractor-tls podName:6c9d825b-217d-499b-b8a4-b0dbf396b283 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:48.518599044 +0000 UTC m=+52.788368124 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c9d825b-217d-499b-b8a4-b0dbf396b283-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wpg7b" (UID: "6c9d825b-217d-499b-b8a4-b0dbf396b283") : secret "insights-runtime-extractor-tls" not found
Jul 29 18:34:48.018715 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:48.018703    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 18:34:48.018715 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:48.018707    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6c9d825b-217d-499b-b8a4-b0dbf396b283-crio-socket\") pod \"insights-runtime-extractor-wpg7b\" (UID: \"6c9d825b-217d-499b-b8a4-b0dbf396b283\") " pod="openshift-insights/insights-runtime-extractor-wpg7b"
Jul 29 18:34:48.019032 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:48.018756    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c9d825b-217d-499b-b8a4-b0dbf396b283-kube-rbac-proxy-cm podName:6c9d825b-217d-499b-b8a4-b0dbf396b283 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:48.518747488 +0000 UTC m=+52.788516603 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6c9d825b-217d-499b-b8a4-b0dbf396b283-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wpg7b" (UID: "6c9d825b-217d-499b-b8a4-b0dbf396b283") : configmap "kube-rbac-proxy" not found
Jul 29 18:34:48.019032 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:48.018798    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6c9d825b-217d-499b-b8a4-b0dbf396b283-crio-socket\") pod \"insights-runtime-extractor-wpg7b\" (UID: \"6c9d825b-217d-499b-b8a4-b0dbf396b283\") " pod="openshift-insights/insights-runtime-extractor-wpg7b"
Jul 29 18:34:48.019032 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:48.018872    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c9d825b-217d-499b-b8a4-b0dbf396b283-data-volume\") pod \"insights-runtime-extractor-wpg7b\" (UID: \"6c9d825b-217d-499b-b8a4-b0dbf396b283\") " pod="openshift-insights/insights-runtime-extractor-wpg7b"
Jul 29 18:34:48.029549 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:48.029522    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94g26\" (UniqueName: \"kubernetes.io/projected/6c9d825b-217d-499b-b8a4-b0dbf396b283-kube-api-access-94g26\") pod \"insights-runtime-extractor-wpg7b\" (UID: \"6c9d825b-217d-499b-b8a4-b0dbf396b283\") " pod="openshift-insights/insights-runtime-extractor-wpg7b"
Jul 29 18:34:48.208279 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:48.208183    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bp28x_6a62d29d-5bf6-444f-bd28-bf2d1e54ac87/konnectivity-agent/0.log" limit=1024
Jul 29 18:34:48.313360 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:48.313326    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-j5wbh"]
Jul 29 18:34:48.314445 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:48.314409    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bp28x_6a62d29d-5bf6-444f-bd28-bf2d1e54ac87/konnectivity-agent/0.log" limit=1024
Jul 29 18:34:48.336151 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:48.336123    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-j5wbh"]
Jul 29 18:34:48.336296 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:48.336240    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-j5wbh"
Jul 29 18:34:48.339355 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:48.339324    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-wnhjj\""
Jul 29 18:34:48.339501 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:48.339356    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 29 18:34:48.339501 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:48.339356    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 29 18:34:48.408230 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:48.408201    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bp28x_6a62d29d-5bf6-444f-bd28-bf2d1e54ac87/konnectivity-agent/0.log" limit=1024
Jul 29 18:34:48.422459 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:48.422412    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn9n5\" (UniqueName: \"kubernetes.io/projected/32e5eb71-45e9-4586-81ea-f1d56935a083-kube-api-access-wn9n5\") pod \"migrator-6865c9967-j5wbh\" (UID: \"32e5eb71-45e9-4586-81ea-f1d56935a083\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-j5wbh"
Jul 29 18:34:48.509663 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:48.509570    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bp28x_6a62d29d-5bf6-444f-bd28-bf2d1e54ac87/konnectivity-agent/0.log" limit=1024
Jul 29 18:34:48.523605 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:48.523579    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c9d825b-217d-499b-b8a4-b0dbf396b283-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wpg7b\" (UID: \"6c9d825b-217d-499b-b8a4-b0dbf396b283\") " pod="openshift-insights/insights-runtime-extractor-wpg7b"
Jul 29 18:34:48.523769 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:48.523645    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2274e5f-4130-477a-900f-915070daeb14-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-rxng5\" (UID: \"b2274e5f-4130-477a-900f-915070daeb14\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-rxng5"
Jul 29 18:34:48.523769 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:48.523673    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c9d825b-217d-499b-b8a4-b0dbf396b283-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wpg7b\" (UID: \"6c9d825b-217d-499b-b8a4-b0dbf396b283\") " pod="openshift-insights/insights-runtime-extractor-wpg7b"
Jul 29 18:34:48.523769 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:48.523692    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn9n5\" (UniqueName: \"kubernetes.io/projected/32e5eb71-45e9-4586-81ea-f1d56935a083-kube-api-access-wn9n5\") pod \"migrator-6865c9967-j5wbh\" (UID: \"32e5eb71-45e9-4586-81ea-f1d56935a083\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-j5wbh"
Jul 29 18:34:48.523769 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:48.523733    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 18:34:48.523897 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:48.523784    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 18:34:48.523897 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:48.523787    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 18:34:48.523897 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:48.523802    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c9d825b-217d-499b-b8a4-b0dbf396b283-insights-runtime-extractor-tls podName:6c9d825b-217d-499b-b8a4-b0dbf396b283 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:49.523784339 +0000 UTC m=+53.793553405 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c9d825b-217d-499b-b8a4-b0dbf396b283-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wpg7b" (UID: "6c9d825b-217d-499b-b8a4-b0dbf396b283") : secret "insights-runtime-extractor-tls" not found
Jul 29 18:34:48.523897 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:48.523837    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2274e5f-4130-477a-900f-915070daeb14-networking-console-plugin-cert podName:b2274e5f-4130-477a-900f-915070daeb14 nodeName:}" failed. No retries permitted until 2026-07-29 18:35:04.523819527 +0000 UTC m=+68.793588595 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b2274e5f-4130-477a-900f-915070daeb14-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-rxng5" (UID: "b2274e5f-4130-477a-900f-915070daeb14") : secret "networking-console-plugin-cert" not found
Jul 29 18:34:48.524034 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:48.523912    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c9d825b-217d-499b-b8a4-b0dbf396b283-kube-rbac-proxy-cm podName:6c9d825b-217d-499b-b8a4-b0dbf396b283 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:49.523903977 +0000 UTC m=+53.793673042 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6c9d825b-217d-499b-b8a4-b0dbf396b283-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wpg7b" (UID: "6c9d825b-217d-499b-b8a4-b0dbf396b283") : configmap "kube-rbac-proxy" not found
Jul 29 18:34:48.535658 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:48.535629    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn9n5\" (UniqueName: \"kubernetes.io/projected/32e5eb71-45e9-4586-81ea-f1d56935a083-kube-api-access-wn9n5\") pod \"migrator-6865c9967-j5wbh\" (UID: \"32e5eb71-45e9-4586-81ea-f1d56935a083\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-j5wbh"
Jul 29 18:34:48.622633 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:48.622600    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bp28x_6a62d29d-5bf6-444f-bd28-bf2d1e54ac87/konnectivity-agent/0.log" limit=1024
Jul 29 18:34:48.636631 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:48.636606    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-v9nsr_7f8b4737-d4c2-4598-bd40-208015542831/console-operator/1.log"
Jul 29 18:34:48.636994 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:48.636978    2559 scope.go:117] "RemoveContainer" containerID="126c4356a4bb1b565afc72e63d7edeff663cabae3115ba399c8253e06bc08656"
Jul 29 18:34:48.637203 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:48.637180    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-v9nsr_openshift-console-operator(7f8b4737-d4c2-4598-bd40-208015542831)\"" pod="openshift-console-operator/console-operator-584dcb6b54-v9nsr" podUID="7f8b4737-d4c2-4598-bd40-208015542831"
Jul 29 18:34:48.645754 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:48.645731    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-j5wbh"
Jul 29 18:34:48.800369 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:48.800285    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-j5wbh"]
Jul 29 18:34:48.803978 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:34:48.803949    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32e5eb71_45e9_4586_81ea_f1d56935a083.slice/crio-abd741a110556d100e8f671400c30c44c421154d149c23539b9dfafde5b80bb3 WatchSource:0}: Error finding container abd741a110556d100e8f671400c30c44c421154d149c23539b9dfafde5b80bb3: Status 404 returned error can't find the container with id abd741a110556d100e8f671400c30c44c421154d149c23539b9dfafde5b80bb3
Jul 29 18:34:48.996742 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:48.996711    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t4wjl_b03a3682-95be-4852-b28a-8e19fee4cbe7/dns-node-resolver/0.log"
Jul 29 18:34:49.534233 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:49.534194    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c9d825b-217d-499b-b8a4-b0dbf396b283-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wpg7b\" (UID: \"6c9d825b-217d-499b-b8a4-b0dbf396b283\") " pod="openshift-insights/insights-runtime-extractor-wpg7b"
Jul 29 18:34:49.534436 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:49.534332    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 18:34:49.534436 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:49.534399    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c9d825b-217d-499b-b8a4-b0dbf396b283-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wpg7b\" (UID: \"6c9d825b-217d-499b-b8a4-b0dbf396b283\") " pod="openshift-insights/insights-runtime-extractor-wpg7b"
Jul 29 18:34:49.534436 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:49.534432    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c9d825b-217d-499b-b8a4-b0dbf396b283-kube-rbac-proxy-cm podName:6c9d825b-217d-499b-b8a4-b0dbf396b283 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:51.534408877 +0000 UTC m=+55.804177955 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6c9d825b-217d-499b-b8a4-b0dbf396b283-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wpg7b" (UID: "6c9d825b-217d-499b-b8a4-b0dbf396b283") : configmap "kube-rbac-proxy" not found
Jul 29 18:34:49.534585 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:49.534495    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 18:34:49.534585 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:49.534547    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c9d825b-217d-499b-b8a4-b0dbf396b283-insights-runtime-extractor-tls podName:6c9d825b-217d-499b-b8a4-b0dbf396b283 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:51.534530283 +0000 UTC m=+55.804299355 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c9d825b-217d-499b-b8a4-b0dbf396b283-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wpg7b" (UID: "6c9d825b-217d-499b-b8a4-b0dbf396b283") : secret "insights-runtime-extractor-tls" not found
Jul 29 18:34:49.642167 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:49.642131    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-j5wbh" event={"ID":"32e5eb71-45e9-4586-81ea-f1d56935a083","Type":"ContainerStarted","Data":"abd741a110556d100e8f671400c30c44c421154d149c23539b9dfafde5b80bb3"}
Jul 29 18:34:49.998074 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:49.998049    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ldlwt_2201b8bf-575c-4958-b909-29ff288ed75e/node-ca/0.log"
Jul 29 18:34:50.513155 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:50.513120    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-bxh5q"]
Jul 29 18:34:50.516356 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:50.516338    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-bxh5q"
Jul 29 18:34:50.518964 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:50.518940    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 29 18:34:50.519415 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:50.519357    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 29 18:34:50.519539 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:50.519459    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 29 18:34:50.520013 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:50.519995    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 29 18:34:50.520091 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:50.520062    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-fb5kg\""
Jul 29 18:34:50.523566 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:50.523537    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-bxh5q"]
Jul 29 18:34:50.597071 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:50.597039    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qhj2h_cc87b3c5-0b72-4fa2-9f87-ca3d4d6da614/serve-healthcheck-canary/0.log"
Jul 29 18:34:50.643503 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:50.643474    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6173aea3-6dd2-41fd-818e-1bf54b3930f8-signing-key\") pod \"service-ca-658d7b7c78-bxh5q\" (UID: \"6173aea3-6dd2-41fd-818e-1bf54b3930f8\") " pod="openshift-service-ca/service-ca-658d7b7c78-bxh5q"
Jul 29 18:34:50.643915 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:50.643623    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6173aea3-6dd2-41fd-818e-1bf54b3930f8-signing-cabundle\") pod \"service-ca-658d7b7c78-bxh5q\" (UID: \"6173aea3-6dd2-41fd-818e-1bf54b3930f8\") " pod="openshift-service-ca/service-ca-658d7b7c78-bxh5q"
Jul 29 18:34:50.643915 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:50.643676    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfg2w\" (UniqueName: \"kubernetes.io/projected/6173aea3-6dd2-41fd-818e-1bf54b3930f8-kube-api-access-sfg2w\") pod \"service-ca-658d7b7c78-bxh5q\" (UID: \"6173aea3-6dd2-41fd-818e-1bf54b3930f8\") " pod="openshift-service-ca/service-ca-658d7b7c78-bxh5q"
Jul 29 18:34:50.647795 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:50.647767    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-j5wbh" event={"ID":"32e5eb71-45e9-4586-81ea-f1d56935a083","Type":"ContainerStarted","Data":"685b56a1de8ba2b9912fd7652a763c1e67ae4ddebaeeb4406da7781f3362b685"}
Jul 29 18:34:50.647937 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:50.647816    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-j5wbh" event={"ID":"32e5eb71-45e9-4586-81ea-f1d56935a083","Type":"ContainerStarted","Data":"b11518931fe21a0c42f2d91d4a437469dab9a54b7b9d27c98a8b1a654f2f2b05"}
Jul 29 18:34:50.664528 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:50.664476    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-j5wbh" podStartSLOduration=1.428458586 podStartE2EDuration="2.664457393s" totalImagesPullingTime="1.235998807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:34:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:50.663023591 +0000 UTC m=+54.932792679" watchObservedRunningTime="2026-07-29 18:34:50.664457393 +0000 UTC m=+54.934226533"
Jul 29 18:34:50.744453 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:50.744408    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6173aea3-6dd2-41fd-818e-1bf54b3930f8-signing-key\") pod \"service-ca-658d7b7c78-bxh5q\" (UID: \"6173aea3-6dd2-41fd-818e-1bf54b3930f8\") " pod="openshift-service-ca/service-ca-658d7b7c78-bxh5q"
Jul 29 18:34:50.744641 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:50.744542    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6173aea3-6dd2-41fd-818e-1bf54b3930f8-signing-cabundle\") pod \"service-ca-658d7b7c78-bxh5q\" (UID: \"6173aea3-6dd2-41fd-818e-1bf54b3930f8\") " pod="openshift-service-ca/service-ca-658d7b7c78-bxh5q"
Jul 29 18:34:50.744641 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:50.744568    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfg2w\" (UniqueName: \"kubernetes.io/projected/6173aea3-6dd2-41fd-818e-1bf54b3930f8-kube-api-access-sfg2w\") pod \"service-ca-658d7b7c78-bxh5q\" (UID: \"6173aea3-6dd2-41fd-818e-1bf54b3930f8\") " pod="openshift-service-ca/service-ca-658d7b7c78-bxh5q"
Jul 29 18:34:50.745507 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:50.745482    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6173aea3-6dd2-41fd-818e-1bf54b3930f8-signing-cabundle\") pod \"service-ca-658d7b7c78-bxh5q\" (UID: \"6173aea3-6dd2-41fd-818e-1bf54b3930f8\") " pod="openshift-service-ca/service-ca-658d7b7c78-bxh5q"
Jul 29 18:34:50.747106 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:50.747071    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6173aea3-6dd2-41fd-818e-1bf54b3930f8-signing-key\") pod \"service-ca-658d7b7c78-bxh5q\" (UID: \"6173aea3-6dd2-41fd-818e-1bf54b3930f8\") " pod="openshift-service-ca/service-ca-658d7b7c78-bxh5q"
Jul 29 18:34:50.755975 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:50.755947    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfg2w\" (UniqueName: \"kubernetes.io/projected/6173aea3-6dd2-41fd-818e-1bf54b3930f8-kube-api-access-sfg2w\") pod \"service-ca-658d7b7c78-bxh5q\" (UID: \"6173aea3-6dd2-41fd-818e-1bf54b3930f8\") " pod="openshift-service-ca/service-ca-658d7b7c78-bxh5q"
Jul 29 18:34:50.826104 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:50.826080    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-bxh5q"
Jul 29 18:34:50.952077 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:50.952040    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-v9nsr"
Jul 29 18:34:50.952241 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:50.952110    2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-v9nsr"
Jul 29 18:34:50.952525 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:50.952509    2559 scope.go:117] "RemoveContainer" containerID="126c4356a4bb1b565afc72e63d7edeff663cabae3115ba399c8253e06bc08656"
Jul 29 18:34:50.952717 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:50.952694    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-v9nsr_openshift-console-operator(7f8b4737-d4c2-4598-bd40-208015542831)\"" pod="openshift-console-operator/console-operator-584dcb6b54-v9nsr" podUID="7f8b4737-d4c2-4598-bd40-208015542831"
Jul 29 18:34:50.957362 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:50.957343    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-bxh5q"]
Jul 29 18:34:50.959766 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:34:50.959741    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6173aea3_6dd2_41fd_818e_1bf54b3930f8.slice/crio-46438c0cd45089bc512a2b39e6cee025d5243913cec6c62400ff5178f86f6b5e WatchSource:0}: Error finding container 46438c0cd45089bc512a2b39e6cee025d5243913cec6c62400ff5178f86f6b5e: Status 404 returned error can't find the container with id 46438c0cd45089bc512a2b39e6cee025d5243913cec6c62400ff5178f86f6b5e
Jul 29 18:34:51.551805 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:51.551765    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c9d825b-217d-499b-b8a4-b0dbf396b283-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wpg7b\" (UID: \"6c9d825b-217d-499b-b8a4-b0dbf396b283\") " pod="openshift-insights/insights-runtime-extractor-wpg7b"
Jul 29 18:34:51.551987 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:51.551880    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c9d825b-217d-499b-b8a4-b0dbf396b283-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wpg7b\" (UID: \"6c9d825b-217d-499b-b8a4-b0dbf396b283\") " pod="openshift-insights/insights-runtime-extractor-wpg7b"
Jul 29 18:34:51.551987 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:51.551917    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 18:34:51.552066 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:51.551999    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 18:34:51.552066 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:51.552009    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c9d825b-217d-499b-b8a4-b0dbf396b283-kube-rbac-proxy-cm podName:6c9d825b-217d-499b-b8a4-b0dbf396b283 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:55.551979454 +0000 UTC m=+59.821748541 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6c9d825b-217d-499b-b8a4-b0dbf396b283-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wpg7b" (UID: "6c9d825b-217d-499b-b8a4-b0dbf396b283") : configmap "kube-rbac-proxy" not found
Jul 29 18:34:51.552066 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:51.552038    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c9d825b-217d-499b-b8a4-b0dbf396b283-insights-runtime-extractor-tls podName:6c9d825b-217d-499b-b8a4-b0dbf396b283 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:55.552027267 +0000 UTC m=+59.821796346 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c9d825b-217d-499b-b8a4-b0dbf396b283-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wpg7b" (UID: "6c9d825b-217d-499b-b8a4-b0dbf396b283") : secret "insights-runtime-extractor-tls" not found
Jul 29 18:34:51.655276 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:51.655238    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-bxh5q" event={"ID":"6173aea3-6dd2-41fd-818e-1bf54b3930f8","Type":"ContainerStarted","Data":"94fdab909445547ddc1cc2afb478d6bd3cd40515721f8fbc6b05577435e47862"}
Jul 29 18:34:51.655276 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:51.655287    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-bxh5q" event={"ID":"6173aea3-6dd2-41fd-818e-1bf54b3930f8","Type":"ContainerStarted","Data":"46438c0cd45089bc512a2b39e6cee025d5243913cec6c62400ff5178f86f6b5e"}
Jul 29 18:34:51.678034 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:51.677973    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-bxh5q" podStartSLOduration=1.677952603 podStartE2EDuration="1.677952603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:34:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:51.677269928 +0000 UTC m=+55.947039017" watchObservedRunningTime="2026-07-29 18:34:51.677952603 +0000 UTC m=+55.947721689"
Jul 29 18:34:55.595159 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:55.595118    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c9d825b-217d-499b-b8a4-b0dbf396b283-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wpg7b\" (UID: \"6c9d825b-217d-499b-b8a4-b0dbf396b283\") " pod="openshift-insights/insights-runtime-extractor-wpg7b"
Jul 29 18:34:55.595657 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:55.595223    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c9d825b-217d-499b-b8a4-b0dbf396b283-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wpg7b\" (UID: \"6c9d825b-217d-499b-b8a4-b0dbf396b283\") " pod="openshift-insights/insights-runtime-extractor-wpg7b"
Jul 29 18:34:55.595657 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:55.595272    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 18:34:55.595657 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:55.595359    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 18:34:55.595657 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:55.595391    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c9d825b-217d-499b-b8a4-b0dbf396b283-insights-runtime-extractor-tls podName:6c9d825b-217d-499b-b8a4-b0dbf396b283 nodeName:}" failed. No retries permitted until 2026-07-29 18:35:03.595352 +0000 UTC m=+67.865121085 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c9d825b-217d-499b-b8a4-b0dbf396b283-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wpg7b" (UID: "6c9d825b-217d-499b-b8a4-b0dbf396b283") : secret "insights-runtime-extractor-tls" not found
Jul 29 18:34:55.595657 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:34:55.595428    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c9d825b-217d-499b-b8a4-b0dbf396b283-kube-rbac-proxy-cm podName:6c9d825b-217d-499b-b8a4-b0dbf396b283 nodeName:}" failed. No retries permitted until 2026-07-29 18:35:03.595418164 +0000 UTC m=+67.865187230 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6c9d825b-217d-499b-b8a4-b0dbf396b283-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wpg7b" (UID: "6c9d825b-217d-499b-b8a4-b0dbf396b283") : configmap "kube-rbac-proxy" not found
Jul 29 18:34:56.368230 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:34:56.368197    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jjdrx"
Jul 29 18:35:00.946887 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:00.946844    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b86e30e-def7-4bf4-90da-de4fef47f433-metrics-certs\") pod \"network-metrics-daemon-9zxr5\" (UID: \"4b86e30e-def7-4bf4-90da-de4fef47f433\") " pod="openshift-multus/network-metrics-daemon-9zxr5"
Jul 29 18:35:00.947284 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:00.947086    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/faa3a79b-75fb-4890-b169-09674f6d00db-original-pull-secret\") pod \"global-pull-secret-syncer-f5w95\" (UID: \"faa3a79b-75fb-4890-b169-09674f6d00db\") " pod="kube-system/global-pull-secret-syncer-f5w95"
Jul 29 18:35:00.949848 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:00.949825    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 18:35:00.950480 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:00.950465    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 18:35:00.960030 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:00.959999    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b86e30e-def7-4bf4-90da-de4fef47f433-metrics-certs\") pod \"network-metrics-daemon-9zxr5\" (UID: \"4b86e30e-def7-4bf4-90da-de4fef47f433\") " pod="openshift-multus/network-metrics-daemon-9zxr5"
Jul 29 18:35:00.960030 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:00.960024    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/faa3a79b-75fb-4890-b169-09674f6d00db-original-pull-secret\") pod \"global-pull-secret-syncer-f5w95\" (UID: \"faa3a79b-75fb-4890-b169-09674f6d00db\") " pod="kube-system/global-pull-secret-syncer-f5w95"
Jul 29 18:35:01.048170 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:01.048124    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztfbt\" (UniqueName: \"kubernetes.io/projected/df3fd46a-27e2-4557-bc40-09a9fbaa2e8f-kube-api-access-ztfbt\") pod \"network-check-target-jh2dv\" (UID: \"df3fd46a-27e2-4557-bc40-09a9fbaa2e8f\") " pod="openshift-network-diagnostics/network-check-target-jh2dv"
Jul 29 18:35:01.050863 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:01.050837    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztfbt\" (UniqueName: \"kubernetes.io/projected/df3fd46a-27e2-4557-bc40-09a9fbaa2e8f-kube-api-access-ztfbt\") pod \"network-check-target-jh2dv\" (UID: \"df3fd46a-27e2-4557-bc40-09a9fbaa2e8f\") " pod="openshift-network-diagnostics/network-check-target-jh2dv"
Jul 29 18:35:01.159302 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:01.159250    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hfcvr\""
Jul 29 18:35:01.164575 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:01.164557    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mfhf2\""
Jul 29 18:35:01.167429 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:01.167407    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh2dv"
Jul 29 18:35:01.168582 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:01.168565    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5w95"
Jul 29 18:35:01.172568 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:01.172552    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zxr5"
Jul 29 18:35:01.349144 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:01.349114    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jh2dv"]
Jul 29 18:35:01.353197 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:35:01.351241    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf3fd46a_27e2_4557_bc40_09a9fbaa2e8f.slice/crio-19e583f30bf09b68a73c8bf7285ea783f1d2287ad5b01a4a31017f4f02d2f6b6 WatchSource:0}: Error finding container 19e583f30bf09b68a73c8bf7285ea783f1d2287ad5b01a4a31017f4f02d2f6b6: Status 404 returned error can't find the container with id 19e583f30bf09b68a73c8bf7285ea783f1d2287ad5b01a4a31017f4f02d2f6b6
Jul 29 18:35:01.371004 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:01.370888    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9zxr5"]
Jul 29 18:35:01.381559 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:35:01.381528    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b86e30e_def7_4bf4_90da_de4fef47f433.slice/crio-d98846a92e4ef784dcce602b6182f7ece807d08593aa541379fabe8f1a35ea89 WatchSource:0}: Error finding container d98846a92e4ef784dcce602b6182f7ece807d08593aa541379fabe8f1a35ea89: Status 404 returned error can't find the container with id d98846a92e4ef784dcce602b6182f7ece807d08593aa541379fabe8f1a35ea89
Jul 29 18:35:01.571663 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:01.571576    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f5w95"]
Jul 29 18:35:01.574501 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:35:01.574471    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaa3a79b_75fb_4890_b169_09674f6d00db.slice/crio-256b2d5a516955b457fa1bf5765f18f794371afb43405553d57df376b0affe70 WatchSource:0}: Error finding container 256b2d5a516955b457fa1bf5765f18f794371afb43405553d57df376b0affe70: Status 404 returned error can't find the container with id 256b2d5a516955b457fa1bf5765f18f794371afb43405553d57df376b0affe70
Jul 29 18:35:01.690897 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:01.690860    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9zxr5" event={"ID":"4b86e30e-def7-4bf4-90da-de4fef47f433","Type":"ContainerStarted","Data":"d98846a92e4ef784dcce602b6182f7ece807d08593aa541379fabe8f1a35ea89"}
Jul 29 18:35:01.692343 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:01.692313    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jh2dv" event={"ID":"df3fd46a-27e2-4557-bc40-09a9fbaa2e8f","Type":"ContainerStarted","Data":"f6d21ed8271e704506c26c688aca78ecec3e96e1c1fecf042285fb8abf77cd44"}
Jul 29 18:35:01.692514 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:01.692363    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jh2dv"
Jul 29 18:35:01.692514 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:01.692398    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jh2dv" event={"ID":"df3fd46a-27e2-4557-bc40-09a9fbaa2e8f","Type":"ContainerStarted","Data":"19e583f30bf09b68a73c8bf7285ea783f1d2287ad5b01a4a31017f4f02d2f6b6"}
Jul 29 18:35:01.693373 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:01.693356    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f5w95" event={"ID":"faa3a79b-75fb-4890-b169-09674f6d00db","Type":"ContainerStarted","Data":"256b2d5a516955b457fa1bf5765f18f794371afb43405553d57df376b0affe70"}
Jul 29 18:35:01.711872 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:01.711817    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jh2dv" podStartSLOduration=65.711796749 podStartE2EDuration="1m5.711796749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:33:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:01.711724247 +0000 UTC m=+65.981493333" watchObservedRunningTime="2026-07-29 18:35:01.711796749 +0000 UTC m=+65.981565836"
Jul 29 18:35:02.563847 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:02.562939    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66a293fb-92cb-4908-bf90-d97ed0d831a6-service-ca-bundle\") pod \"router-default-7688c5db95-gcqb5\" (UID: \"66a293fb-92cb-4908-bf90-d97ed0d831a6\") " pod="openshift-ingress/router-default-7688c5db95-gcqb5"
Jul 29 18:35:02.563847 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:02.563084    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c284fbd5-a70f-4085-948c-6a2392502e6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-jxmfh\" (UID: \"c284fbd5-a70f-4085-948c-6a2392502e6f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jxmfh"
Jul 29 18:35:02.563847 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:02.563123    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e281e146-0b8a-4358-a98b-7f577f4ff08b-registry-tls\") pod \"image-registry-6cb6fb9448-jbzkl\" (UID: \"e281e146-0b8a-4358-a98b-7f577f4ff08b\") " pod="openshift-image-registry/image-registry-6cb6fb9448-jbzkl"
Jul 29 18:35:02.563847 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:02.563160    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/26f43ad8-a7b4-4140-9ee0-7e9d55edf035-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-hb5lw\" (UID: \"26f43ad8-a7b4-4140-9ee0-7e9d55edf035\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-hb5lw"
Jul 29 18:35:02.563847 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:02.563190    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66a293fb-92cb-4908-bf90-d97ed0d831a6-metrics-certs\") pod \"router-default-7688c5db95-gcqb5\" (UID: \"66a293fb-92cb-4908-bf90-d97ed0d831a6\") " pod="openshift-ingress/router-default-7688c5db95-gcqb5"
Jul 29 18:35:02.563847 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:02.563233    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c854cbbc-3a2c-46ca-adf3-86fc3028bd7c-metrics-tls\") pod \"dns-default-95vjs\" (UID: \"c854cbbc-3a2c-46ca-adf3-86fc3028bd7c\") " pod="openshift-dns/dns-default-95vjs"
Jul 29 18:35:02.563847 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:02.563777    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66a293fb-92cb-4908-bf90-d97ed0d831a6-service-ca-bundle\") pod \"router-default-7688c5db95-gcqb5\" (UID: \"66a293fb-92cb-4908-bf90-d97ed0d831a6\") " pod="openshift-ingress/router-default-7688c5db95-gcqb5"
Jul 29 18:35:02.567317 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:02.567255    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66a293fb-92cb-4908-bf90-d97ed0d831a6-metrics-certs\") pod \"router-default-7688c5db95-gcqb5\" (UID: \"66a293fb-92cb-4908-bf90-d97ed0d831a6\") " pod="openshift-ingress/router-default-7688c5db95-gcqb5"
Jul 29 18:35:02.569113 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:02.569083    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c284fbd5-a70f-4085-948c-6a2392502e6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-jxmfh\" (UID: \"c284fbd5-a70f-4085-948c-6a2392502e6f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jxmfh"
Jul 29 18:35:02.569240 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:02.569121    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e281e146-0b8a-4358-a98b-7f577f4ff08b-registry-tls\") pod \"image-registry-6cb6fb9448-jbzkl\" (UID: \"e281e146-0b8a-4358-a98b-7f577f4ff08b\") " pod="openshift-image-registry/image-registry-6cb6fb9448-jbzkl"
Jul 29 18:35:02.569614 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:02.569572    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c854cbbc-3a2c-46ca-adf3-86fc3028bd7c-metrics-tls\") pod \"dns-default-95vjs\" (UID: \"c854cbbc-3a2c-46ca-adf3-86fc3028bd7c\") " pod="openshift-dns/dns-default-95vjs"
Jul 29 18:35:02.571108 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:02.571087    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/26f43ad8-a7b4-4140-9ee0-7e9d55edf035-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-hb5lw\" (UID: \"26f43ad8-a7b4-4140-9ee0-7e9d55edf035\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-hb5lw"
Jul 29 18:35:02.699689 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:02.699661    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zc9c4\""
Jul 29 18:35:02.707827 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:02.707800    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cb6fb9448-jbzkl"
Jul 29 18:35:02.713773 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:02.713732    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-2bqxr\""
Jul 29 18:35:02.722075 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:02.722051    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jxmfh"
Jul 29 18:35:02.738183 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:02.738154    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-8qgnb\""
Jul 29 18:35:02.746178 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:02.746151    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7688c5db95-gcqb5"
Jul 29 18:35:02.748662 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:02.748627    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-thcsx\""
Jul 29 18:35:02.757130 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:02.757088    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-hb5lw"
Jul 29 18:35:02.760872 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:02.760848    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2xgwr\""
Jul 29 18:35:02.769421 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:02.769374    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-95vjs"
Jul 29 18:35:02.885901 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:02.885861    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6cb6fb9448-jbzkl"]
Jul 29 18:35:02.938196 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:02.938157    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-jxmfh"]
Jul 29 18:35:02.997590 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:35:02.997553    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode281e146_0b8a_4358_a98b_7f577f4ff08b.slice/crio-71e1be586e0b48aaff7a2dcecd301b120385c857244a370821541f4728d9bfb8 WatchSource:0}: Error finding container 71e1be586e0b48aaff7a2dcecd301b120385c857244a370821541f4728d9bfb8: Status 404 returned error can't find the container with id 71e1be586e0b48aaff7a2dcecd301b120385c857244a370821541f4728d9bfb8
Jul 29 18:35:02.998222 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:35:02.998197    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc284fbd5_a70f_4085_948c_6a2392502e6f.slice/crio-8dfb7283ebca62d7ac2d16fc56ff746dd2636a17bba1bb948bacac761bcfc7bb WatchSource:0}: Error finding container 8dfb7283ebca62d7ac2d16fc56ff746dd2636a17bba1bb948bacac761bcfc7bb: Status 404 returned error can't find the container with id 8dfb7283ebca62d7ac2d16fc56ff746dd2636a17bba1bb948bacac761bcfc7bb
Jul 29 18:35:03.145390 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:03.145347    2559 scope.go:117] "RemoveContainer" containerID="126c4356a4bb1b565afc72e63d7edeff663cabae3115ba399c8253e06bc08656"
Jul 29 18:35:03.206790 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:03.206709    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-hb5lw"]
Jul 29 18:35:03.239302 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:03.239275    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-95vjs"]
Jul 29 18:35:03.243299 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:35:03.243268    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc854cbbc_3a2c_46ca_adf3_86fc3028bd7c.slice/crio-5fffb2c1a851b725320a7e6331c385e61d85c5d086599fa2dfa1762296629a35 WatchSource:0}: Error finding container 5fffb2c1a851b725320a7e6331c385e61d85c5d086599fa2dfa1762296629a35: Status 404 returned error can't find the container with id 5fffb2c1a851b725320a7e6331c385e61d85c5d086599fa2dfa1762296629a35
Jul 29 18:35:03.251321 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:03.251259    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7688c5db95-gcqb5"]
Jul 29 18:35:03.254298 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:35:03.254265    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66a293fb_92cb_4908_bf90_d97ed0d831a6.slice/crio-a7b8ac24ac053abce11c7c467f82937893d715d9154ffeec91a268ddf8af574d WatchSource:0}: Error finding container a7b8ac24ac053abce11c7c467f82937893d715d9154ffeec91a268ddf8af574d: Status 404 returned error can't find the container with id a7b8ac24ac053abce11c7c467f82937893d715d9154ffeec91a268ddf8af574d
Jul 29 18:35:03.675330 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:03.675016    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c9d825b-217d-499b-b8a4-b0dbf396b283-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wpg7b\" (UID: \"6c9d825b-217d-499b-b8a4-b0dbf396b283\") " pod="openshift-insights/insights-runtime-extractor-wpg7b"
Jul 29 18:35:03.675330 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:03.675144    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c9d825b-217d-499b-b8a4-b0dbf396b283-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wpg7b\" (UID: \"6c9d825b-217d-499b-b8a4-b0dbf396b283\") " pod="openshift-insights/insights-runtime-extractor-wpg7b"
Jul 29 18:35:03.676454 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:03.676428    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c9d825b-217d-499b-b8a4-b0dbf396b283-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wpg7b\" (UID: \"6c9d825b-217d-499b-b8a4-b0dbf396b283\") " pod="openshift-insights/insights-runtime-extractor-wpg7b"
Jul 29 18:35:03.677875 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:03.677851    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c9d825b-217d-499b-b8a4-b0dbf396b283-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wpg7b\" (UID: \"6c9d825b-217d-499b-b8a4-b0dbf396b283\") " pod="openshift-insights/insights-runtime-extractor-wpg7b"
Jul 29 18:35:03.704301 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:03.704258    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-95vjs" event={"ID":"c854cbbc-3a2c-46ca-adf3-86fc3028bd7c","Type":"ContainerStarted","Data":"5fffb2c1a851b725320a7e6331c385e61d85c5d086599fa2dfa1762296629a35"}
Jul 29 18:35:03.707241 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:03.706764    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cb6fb9448-jbzkl" event={"ID":"e281e146-0b8a-4358-a98b-7f577f4ff08b","Type":"ContainerStarted","Data":"17e5ae82c534891dcf6c418f9c5c7e6fcef4a6fdc83d3364aa6ac9e7a73c72e4"}
Jul 29 18:35:03.707241 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:03.706826    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cb6fb9448-jbzkl" event={"ID":"e281e146-0b8a-4358-a98b-7f577f4ff08b","Type":"ContainerStarted","Data":"71e1be586e0b48aaff7a2dcecd301b120385c857244a370821541f4728d9bfb8"}
Jul 29 18:35:03.707241 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:03.706922    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6cb6fb9448-jbzkl"
Jul 29 18:35:03.709189 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:03.709162    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-v9nsr_7f8b4737-d4c2-4598-bd40-208015542831/console-operator/1.log"
Jul 29 18:35:03.709327 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:03.709275    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-v9nsr" event={"ID":"7f8b4737-d4c2-4598-bd40-208015542831","Type":"ContainerStarted","Data":"c3a2a2cb29ef1c3747796ea3ca8a5cc4de4bdb0d84f4ebc3dffa26488f8b0281"}
Jul 29 18:35:03.709862 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:03.709813    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-v9nsr"
Jul 29 18:35:03.712162 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:03.712129    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7688c5db95-gcqb5" event={"ID":"66a293fb-92cb-4908-bf90-d97ed0d831a6","Type":"ContainerStarted","Data":"c21d3a27a13abccc64083f8b139b5023b68bb558976d396342f7f13b8c178c9e"}
Jul 29 18:35:03.712270 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:03.712185    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7688c5db95-gcqb5" event={"ID":"66a293fb-92cb-4908-bf90-d97ed0d831a6","Type":"ContainerStarted","Data":"a7b8ac24ac053abce11c7c467f82937893d715d9154ffeec91a268ddf8af574d"}
Jul 29 18:35:03.713729 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:03.713705    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jxmfh" event={"ID":"c284fbd5-a70f-4085-948c-6a2392502e6f","Type":"ContainerStarted","Data":"8dfb7283ebca62d7ac2d16fc56ff746dd2636a17bba1bb948bacac761bcfc7bb"}
Jul 29 18:35:03.715935 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:03.715906    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9zxr5" event={"ID":"4b86e30e-def7-4bf4-90da-de4fef47f433","Type":"ContainerStarted","Data":"fe072b41f32d8b44644309c1f6d2af5ec584b1c0ead027c7d3f3a6702cf6b6a5"}
Jul 29 18:35:03.716040 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:03.715958    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9zxr5" event={"ID":"4b86e30e-def7-4bf4-90da-de4fef47f433","Type":"ContainerStarted","Data":"5030a30e5f945419dbb119802785877d8f35b0ea19ce76678ae1dd0f3c52c4eb"}
Jul 29 18:35:03.720508 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:03.720481    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-hb5lw" event={"ID":"26f43ad8-a7b4-4140-9ee0-7e9d55edf035","Type":"ContainerStarted","Data":"9e714af8d3fa263cf5cee3cec059a1fcfb254f1133834bd4bfa8f99c33a2dbb3"}
Jul 29 18:35:03.727596 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:03.727547    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6cb6fb9448-jbzkl" podStartSLOduration=67.727524635 podStartE2EDuration="1m7.727524635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:33:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:03.726483217 +0000 UTC m=+67.996252306" watchObservedRunningTime="2026-07-29 18:35:03.727524635 +0000 UTC m=+67.997293722"
Jul 29 18:35:03.748035 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:03.747998    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7688c5db95-gcqb5"
Jul 29 18:35:03.748593 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:03.748532    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7688c5db95-gcqb5" podStartSLOduration=66.748514992 podStartE2EDuration="1m6.748514992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:33:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:03.745915485 +0000 UTC m=+68.015684573" watchObservedRunningTime="2026-07-29 18:35:03.748514992 +0000 UTC m=+68.018284081"
Jul 29 18:35:03.756522 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:03.756492    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7688c5db95-gcqb5"
Jul 29 18:35:03.757463 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:03.756502    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9grlj\""
Jul 29 18:35:03.762809 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:03.762753    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wpg7b"
Jul 29 18:35:03.786552 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:03.786490    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-v9nsr" podStartSLOduration=51.546904349 podStartE2EDuration="1m4.786467869s" totalImagesPullingTime="13.23956352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:03.766065435 +0000 UTC m=+68.035834521" watchObservedRunningTime="2026-07-29 18:35:03.786467869 +0000 UTC m=+68.056236958"
Jul 29 18:35:03.805986 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:03.803975    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9zxr5" podStartSLOduration=66.158674698 podStartE2EDuration="1m7.803950667s" totalImagesPullingTime="1.645275969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:03.785818139 +0000 UTC m=+68.055587218" watchObservedRunningTime="2026-07-29 18:35:03.803950667 +0000 UTC m=+68.073719752"
Jul 29 18:35:03.956070 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:03.956038    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wpg7b"]
Jul 29 18:35:03.963939 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:35:03.963903    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c9d825b_217d_499b_b8a4_b0dbf396b283.slice/crio-deaf4848e7cad664bdf2c0a6eed81c926c613f78076ef1b52b6257672e52d575 WatchSource:0}: Error finding container deaf4848e7cad664bdf2c0a6eed81c926c613f78076ef1b52b6257672e52d575: Status 404 returned error can't find the container with id deaf4848e7cad664bdf2c0a6eed81c926c613f78076ef1b52b6257672e52d575
Jul 29 18:35:04.406027 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:04.406001    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-v9nsr"
Jul 29 18:35:04.584474 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:04.584326    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2274e5f-4130-477a-900f-915070daeb14-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-rxng5\" (UID: \"b2274e5f-4130-477a-900f-915070daeb14\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-rxng5"
Jul 29 18:35:04.588985 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:04.588955    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2274e5f-4130-477a-900f-915070daeb14-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-rxng5\" (UID: \"b2274e5f-4130-477a-900f-915070daeb14\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-rxng5"
Jul 29 18:35:04.731588 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:04.730354    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wpg7b" event={"ID":"6c9d825b-217d-499b-b8a4-b0dbf396b283","Type":"ContainerStarted","Data":"deaf4848e7cad664bdf2c0a6eed81c926c613f78076ef1b52b6257672e52d575"}
Jul 29 18:35:04.731588 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:04.730443    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7688c5db95-gcqb5"
Jul 29 18:35:04.732870 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:04.732636    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7688c5db95-gcqb5"
Jul 29 18:35:04.743289 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:04.743257    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-pvn72\""
Jul 29 18:35:04.751242 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:04.751211    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-rxng5"
Jul 29 18:35:11.580019 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:11.579983    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-rxng5"]
Jul 29 18:35:11.866731 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:35:11.866705    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2274e5f_4130_477a_900f_915070daeb14.slice/crio-1c0bbddd1a986038304d7277e16c8733a7482540ac91563d43d42d7a9e4e3515 WatchSource:0}: Error finding container 1c0bbddd1a986038304d7277e16c8733a7482540ac91563d43d42d7a9e4e3515: Status 404 returned error can't find the container with id 1c0bbddd1a986038304d7277e16c8733a7482540ac91563d43d42d7a9e4e3515
Jul 29 18:35:12.764455 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:12.764416    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wpg7b" event={"ID":"6c9d825b-217d-499b-b8a4-b0dbf396b283","Type":"ContainerStarted","Data":"7a826cd5a758f1978c15a992816697610fcc41ca2ddc410248145f34a33733aa"}
Jul 29 18:35:12.766641 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:12.766611    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-95vjs" event={"ID":"c854cbbc-3a2c-46ca-adf3-86fc3028bd7c","Type":"ContainerStarted","Data":"6d961c4e767579eb097e6b0e86da6409149f96e13639bf7e757811b0ba45146e"}
Jul 29 18:35:12.766783 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:12.766674    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-95vjs"
Jul 29 18:35:12.766783 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:12.766700    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-95vjs" event={"ID":"c854cbbc-3a2c-46ca-adf3-86fc3028bd7c","Type":"ContainerStarted","Data":"6b700fd6332efc5e0907e094cf24ff5b9a2a8b57ac6a48c6c36eedceb3bfa1a2"}
Jul 29 18:35:12.770444 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:12.768663    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jxmfh" event={"ID":"c284fbd5-a70f-4085-948c-6a2392502e6f","Type":"ContainerStarted","Data":"bd4ea87bffea4879909e401c1fb382c4fd30d92521bb3b11b861a8fb615aec8c"}
Jul 29 18:35:12.771833 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:12.771802    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-hb5lw" event={"ID":"26f43ad8-a7b4-4140-9ee0-7e9d55edf035","Type":"ContainerStarted","Data":"d5e7cc3e8604cebff2e632a2586be338ce928920c44be3feb28fa24afe4bb3d1"}
Jul 29 18:35:12.771977 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:12.771867    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-hb5lw" event={"ID":"26f43ad8-a7b4-4140-9ee0-7e9d55edf035","Type":"ContainerStarted","Data":"3e0b10a349f0f72fb717b116c218c736e4b3c7fa16ef42762d05fa38358f6a21"}
Jul 29 18:35:12.773760 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:12.773724    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f5w95" event={"ID":"faa3a79b-75fb-4890-b169-09674f6d00db","Type":"ContainerStarted","Data":"fbfda6cd92262e261a2fc8d5b20a7ab38a68ed8b610f877fc42d4662d7060129"}
Jul 29 18:35:12.775407 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:12.775361    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-rxng5" event={"ID":"b2274e5f-4130-477a-900f-915070daeb14","Type":"ContainerStarted","Data":"1c0bbddd1a986038304d7277e16c8733a7482540ac91563d43d42d7a9e4e3515"}
Jul 29 18:35:12.814169 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:12.814137    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-6vgm5"]
Jul 29 18:35:12.820005 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:12.819952    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-95vjs" podStartSLOduration=34.200983816 podStartE2EDuration="42.819932827s" totalImagesPullingTime="8.618949011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:12.819427738 +0000 UTC m=+77.089196825" watchObservedRunningTime="2026-07-29 18:35:12.819932827 +0000 UTC m=+77.089701913"
Jul 29 18:35:12.829073 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:12.829041    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-6vgm5"
Jul 29 18:35:12.831603 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:12.831370    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-7v5zb\""
Jul 29 18:35:12.831905 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:12.831889    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 29 18:35:12.832565 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:12.832485    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-6vgm5"]
Jul 29 18:35:12.847267 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:12.847202    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-f5w95" podStartSLOduration=66.528063551 podStartE2EDuration="1m16.847180222s" totalImagesPullingTime="10.319116671s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:12.846646553 +0000 UTC m=+77.116415640" watchObservedRunningTime="2026-07-29 18:35:12.847180222 +0000 UTC m=+77.116949305"
Jul 29 18:35:12.878454 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:12.878391    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jxmfh" podStartSLOduration=67.032392697 podStartE2EDuration="1m15.878356657s" totalImagesPullingTime="8.84596396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:33:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:12.876347268 +0000 UTC m=+77.146116355" watchObservedRunningTime="2026-07-29 18:35:12.878356657 +0000 UTC m=+77.148125743"
Jul 29 18:35:12.903682 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:12.903644    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-24vj5"]
Jul 29 18:35:12.917915 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:12.917883    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-24vj5"
Jul 29 18:35:12.924278 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:12.922444    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 29 18:35:12.924278 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:12.923252    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-hb5lw" podStartSLOduration=66.294612892 podStartE2EDuration="1m14.92323327s" totalImagesPullingTime="8.628620378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:33:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:12.91103543 +0000 UTC m=+77.180804517" watchObservedRunningTime="2026-07-29 18:35:12.92323327 +0000 UTC m=+77.193002359"
Jul 29 18:35:12.924278 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:12.923850    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-9v7fk\""
Jul 29 18:35:12.927325 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:12.926834    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 29 18:35:12.928112 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:12.928084    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-24vj5"]
Jul 29 18:35:12.965209 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:12.965171    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c307b2b5-1ce6-4f9f-9358-393654020bc9-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-6vgm5\" (UID: \"c307b2b5-1ce6-4f9f-9358-393654020bc9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-6vgm5"
Jul 29 18:35:13.066038 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:13.065942    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c307b2b5-1ce6-4f9f-9358-393654020bc9-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-6vgm5\" (UID: \"c307b2b5-1ce6-4f9f-9358-393654020bc9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-6vgm5"
Jul 29 18:35:13.066038 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:13.066015    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cwr4\" (UniqueName: \"kubernetes.io/projected/26035b26-9c55-49cf-a2e9-48634f8c5cd5-kube-api-access-7cwr4\") pod \"downloads-7c97dc4555-24vj5\" (UID: \"26035b26-9c55-49cf-a2e9-48634f8c5cd5\") " pod="openshift-console/downloads-7c97dc4555-24vj5"
Jul 29 18:35:13.069183 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:13.069151    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c307b2b5-1ce6-4f9f-9358-393654020bc9-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-6vgm5\" (UID: \"c307b2b5-1ce6-4f9f-9358-393654020bc9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-6vgm5"
Jul 29 18:35:13.142470 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:13.142433    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-6vgm5"
Jul 29 18:35:13.166495 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:13.166460    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cwr4\" (UniqueName: \"kubernetes.io/projected/26035b26-9c55-49cf-a2e9-48634f8c5cd5-kube-api-access-7cwr4\") pod \"downloads-7c97dc4555-24vj5\" (UID: \"26035b26-9c55-49cf-a2e9-48634f8c5cd5\") " pod="openshift-console/downloads-7c97dc4555-24vj5"
Jul 29 18:35:13.183936 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:13.183903    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cwr4\" (UniqueName: \"kubernetes.io/projected/26035b26-9c55-49cf-a2e9-48634f8c5cd5-kube-api-access-7cwr4\") pod \"downloads-7c97dc4555-24vj5\" (UID: \"26035b26-9c55-49cf-a2e9-48634f8c5cd5\") " pod="openshift-console/downloads-7c97dc4555-24vj5"
Jul 29 18:35:13.231550 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:13.231518    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-24vj5"
Jul 29 18:35:13.341711 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:13.340159    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-6vgm5"]
Jul 29 18:35:13.345451 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:35:13.345412    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc307b2b5_1ce6_4f9f_9358_393654020bc9.slice/crio-85fe0daf6e1be2228253b5d03bbc1df89e6d5e9499acc5a5bfd9afdb3949480b WatchSource:0}: Error finding container 85fe0daf6e1be2228253b5d03bbc1df89e6d5e9499acc5a5bfd9afdb3949480b: Status 404 returned error can't find the container with id 85fe0daf6e1be2228253b5d03bbc1df89e6d5e9499acc5a5bfd9afdb3949480b
Jul 29 18:35:13.362532 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:13.362502    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7fd9d847f4-pzwjk"]
Jul 29 18:35:13.378469 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:13.376450    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fd9d847f4-pzwjk"]
Jul 29 18:35:13.378469 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:13.376499    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fd9d847f4-pzwjk"
Jul 29 18:35:13.382252 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:13.381743    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 29 18:35:13.382252 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:13.381760    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 29 18:35:13.382252 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:13.381955    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9hd9l\""
Jul 29 18:35:13.382252 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:13.382142    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 29 18:35:13.383170 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:13.383094    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 29 18:35:13.383354 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:13.383339    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 29 18:35:13.435327 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:13.435259    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-24vj5"]
Jul 29 18:35:13.572176 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:13.571881    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5540cda-33a8-446e-a74b-4c67ebd287bd-service-ca\") pod \"console-7fd9d847f4-pzwjk\" (UID: \"b5540cda-33a8-446e-a74b-4c67ebd287bd\") " pod="openshift-console/console-7fd9d847f4-pzwjk"
Jul 29 18:35:13.572176 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:13.571935    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45xfr\" (UniqueName: \"kubernetes.io/projected/b5540cda-33a8-446e-a74b-4c67ebd287bd-kube-api-access-45xfr\") pod \"console-7fd9d847f4-pzwjk\" (UID: \"b5540cda-33a8-446e-a74b-4c67ebd287bd\") " pod="openshift-console/console-7fd9d847f4-pzwjk"
Jul 29 18:35:13.572176 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:13.571972    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5540cda-33a8-446e-a74b-4c67ebd287bd-console-serving-cert\") pod \"console-7fd9d847f4-pzwjk\" (UID: \"b5540cda-33a8-446e-a74b-4c67ebd287bd\") " pod="openshift-console/console-7fd9d847f4-pzwjk"
Jul 29 18:35:13.572176 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:13.571994    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5540cda-33a8-446e-a74b-4c67ebd287bd-console-config\") pod \"console-7fd9d847f4-pzwjk\" (UID: \"b5540cda-33a8-446e-a74b-4c67ebd287bd\") " pod="openshift-console/console-7fd9d847f4-pzwjk"
Jul 29 18:35:13.572176 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:13.572019    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5540cda-33a8-446e-a74b-4c67ebd287bd-console-oauth-config\") pod \"console-7fd9d847f4-pzwjk\" (UID: \"b5540cda-33a8-446e-a74b-4c67ebd287bd\") " pod="openshift-console/console-7fd9d847f4-pzwjk"
Jul 29 18:35:13.572176 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:13.572069    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5540cda-33a8-446e-a74b-4c67ebd287bd-oauth-serving-cert\") pod \"console-7fd9d847f4-pzwjk\" (UID: \"b5540cda-33a8-446e-a74b-4c67ebd287bd\") " pod="openshift-console/console-7fd9d847f4-pzwjk"
Jul 29 18:35:13.673450 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:13.673342    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5540cda-33a8-446e-a74b-4c67ebd287bd-service-ca\") pod \"console-7fd9d847f4-pzwjk\" (UID: \"b5540cda-33a8-446e-a74b-4c67ebd287bd\") " pod="openshift-console/console-7fd9d847f4-pzwjk"
Jul 29 18:35:13.673450 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:13.673425    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45xfr\" (UniqueName: \"kubernetes.io/projected/b5540cda-33a8-446e-a74b-4c67ebd287bd-kube-api-access-45xfr\") pod \"console-7fd9d847f4-pzwjk\" (UID: \"b5540cda-33a8-446e-a74b-4c67ebd287bd\") " pod="openshift-console/console-7fd9d847f4-pzwjk"
Jul 29 18:35:13.673676 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:13.673459    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5540cda-33a8-446e-a74b-4c67ebd287bd-console-serving-cert\") pod \"console-7fd9d847f4-pzwjk\" (UID: \"b5540cda-33a8-446e-a74b-4c67ebd287bd\") " pod="openshift-console/console-7fd9d847f4-pzwjk"
Jul 29 18:35:13.673676 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:13.673483    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5540cda-33a8-446e-a74b-4c67ebd287bd-console-config\") pod \"console-7fd9d847f4-pzwjk\" (UID: \"b5540cda-33a8-446e-a74b-4c67ebd287bd\") " pod="openshift-console/console-7fd9d847f4-pzwjk"
Jul 29 18:35:13.673676 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:13.673514    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5540cda-33a8-446e-a74b-4c67ebd287bd-console-oauth-config\") pod \"console-7fd9d847f4-pzwjk\" (UID: \"b5540cda-33a8-446e-a74b-4c67ebd287bd\") " pod="openshift-console/console-7fd9d847f4-pzwjk"
Jul 29 18:35:13.673676 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:13.673563    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5540cda-33a8-446e-a74b-4c67ebd287bd-oauth-serving-cert\") pod \"console-7fd9d847f4-pzwjk\" (UID: \"b5540cda-33a8-446e-a74b-4c67ebd287bd\") " pod="openshift-console/console-7fd9d847f4-pzwjk"
Jul 29 18:35:13.674195 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:13.674162    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5540cda-33a8-446e-a74b-4c67ebd287bd-service-ca\") pod \"console-7fd9d847f4-pzwjk\" (UID: \"b5540cda-33a8-446e-a74b-4c67ebd287bd\") " pod="openshift-console/console-7fd9d847f4-pzwjk"
Jul 29 18:35:13.674325 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:13.674208    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5540cda-33a8-446e-a74b-4c67ebd287bd-oauth-serving-cert\") pod \"console-7fd9d847f4-pzwjk\" (UID: \"b5540cda-33a8-446e-a74b-4c67ebd287bd\") " pod="openshift-console/console-7fd9d847f4-pzwjk"
Jul 29 18:35:13.674325 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:13.674249    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5540cda-33a8-446e-a74b-4c67ebd287bd-console-config\") pod \"console-7fd9d847f4-pzwjk\" (UID: \"b5540cda-33a8-446e-a74b-4c67ebd287bd\") " pod="openshift-console/console-7fd9d847f4-pzwjk"
Jul 29 18:35:13.677516 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:13.677489    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5540cda-33a8-446e-a74b-4c67ebd287bd-console-oauth-config\") pod \"console-7fd9d847f4-pzwjk\" (UID: \"b5540cda-33a8-446e-a74b-4c67ebd287bd\") " pod="openshift-console/console-7fd9d847f4-pzwjk"
Jul 29 18:35:13.677619 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:13.677517    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5540cda-33a8-446e-a74b-4c67ebd287bd-console-serving-cert\") pod \"console-7fd9d847f4-pzwjk\" (UID: \"b5540cda-33a8-446e-a74b-4c67ebd287bd\") " pod="openshift-console/console-7fd9d847f4-pzwjk"
Jul 29 18:35:13.687043 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:13.686969    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45xfr\" (UniqueName: \"kubernetes.io/projected/b5540cda-33a8-446e-a74b-4c67ebd287bd-kube-api-access-45xfr\") pod \"console-7fd9d847f4-pzwjk\" (UID: \"b5540cda-33a8-446e-a74b-4c67ebd287bd\") " pod="openshift-console/console-7fd9d847f4-pzwjk"
Jul 29 18:35:13.690612 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:13.690583    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fd9d847f4-pzwjk"
Jul 29 18:35:13.790041 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:13.790006    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-6vgm5" event={"ID":"c307b2b5-1ce6-4f9f-9358-393654020bc9","Type":"ContainerStarted","Data":"85fe0daf6e1be2228253b5d03bbc1df89e6d5e9499acc5a5bfd9afdb3949480b"}
Jul 29 18:35:14.062117 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:14.062085    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fd9d847f4-pzwjk"]
Jul 29 18:35:14.065062 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:35:14.065029    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5540cda_33a8_446e_a74b_4c67ebd287bd.slice/crio-07bf6190b6bfc1f0563fbd0d012df935208a17eca97ebf1ff544857f50b2b09f WatchSource:0}: Error finding container 07bf6190b6bfc1f0563fbd0d012df935208a17eca97ebf1ff544857f50b2b09f: Status 404 returned error can't find the container with id 07bf6190b6bfc1f0563fbd0d012df935208a17eca97ebf1ff544857f50b2b09f
Jul 29 18:35:14.795862 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:14.795813    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-24vj5" event={"ID":"26035b26-9c55-49cf-a2e9-48634f8c5cd5","Type":"ContainerStarted","Data":"ac5d19b49ab6d10b9944910ef74433acb9c5c3a296f0a49214049e1b5fc29e77"}
Jul 29 18:35:14.797319 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:14.797284    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fd9d847f4-pzwjk" event={"ID":"b5540cda-33a8-446e-a74b-4c67ebd287bd","Type":"ContainerStarted","Data":"07bf6190b6bfc1f0563fbd0d012df935208a17eca97ebf1ff544857f50b2b09f"}
Jul 29 18:35:18.819623 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:18.819574    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-rxng5" event={"ID":"b2274e5f-4130-477a-900f-915070daeb14","Type":"ContainerStarted","Data":"6bf1c0980dac03bc57c5d4e1959c4a1842c4728ac596bf649e731852de44405e"}
Jul 29 18:35:18.823261 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:18.822510    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wpg7b" event={"ID":"6c9d825b-217d-499b-b8a4-b0dbf396b283","Type":"ContainerStarted","Data":"179f54a3ae3380f3eebf5fd20ea0e591644e5039a9f58fb3012c2bb2b3cfbb55"}
Jul 29 18:35:18.836303 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:18.836244    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-rxng5" podStartSLOduration=41.168959638 podStartE2EDuration="46.836227031s" totalImagesPullingTime="5.667267393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:34:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:18.834259773 +0000 UTC m=+83.104028861" watchObservedRunningTime="2026-07-29 18:35:18.836227031 +0000 UTC m=+83.105996120"
Jul 29 18:35:20.833979 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:20.833930    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-6vgm5" event={"ID":"c307b2b5-1ce6-4f9f-9358-393654020bc9","Type":"ContainerStarted","Data":"f56fb0db3a50116b7e68279d4d8b142848f46166e15959f5654bbcd0a673f312"}
Jul 29 18:35:20.834637 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:20.834176    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-6vgm5"
Jul 29 18:35:20.836588 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:20.836554    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fd9d847f4-pzwjk" event={"ID":"b5540cda-33a8-446e-a74b-4c67ebd287bd","Type":"ContainerStarted","Data":"357ab095fdeb0736d7f0e28ab422c7dba3186b07d6dedf93a8c83d2447053958"}
Jul 29 18:35:20.840758 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:20.840710    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-6vgm5"
Jul 29 18:35:20.854299 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:20.854233    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-6vgm5" podStartSLOduration=2.384068067 podStartE2EDuration="8.85421216s" totalImagesPullingTime="6.470144093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:35:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:20.852804074 +0000 UTC m=+85.122573163" watchObservedRunningTime="2026-07-29 18:35:20.85421216 +0000 UTC m=+85.123981249"
Jul 29 18:35:20.870347 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:20.870291    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7fd9d847f4-pzwjk" podStartSLOduration=2.104578513 podStartE2EDuration="7.870268997s" totalImagesPullingTime="5.765690484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:35:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:20.869213043 +0000 UTC m=+85.138982131" watchObservedRunningTime="2026-07-29 18:35:20.870268997 +0000 UTC m=+85.140038085"
Jul 29 18:35:22.716024 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:22.715979    2559 patch_prober.go:28] interesting pod/image-registry-6cb6fb9448-jbzkl container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 29 18:35:22.716504 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:22.716088    2559 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6cb6fb9448-jbzkl" podUID="e281e146-0b8a-4358-a98b-7f577f4ff08b" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 18:35:22.793771 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:22.793738    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-95vjs"
Jul 29 18:35:23.690915 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:23.690827    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7fd9d847f4-pzwjk"
Jul 29 18:35:23.690915 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:23.690913    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7fd9d847f4-pzwjk"
Jul 29 18:35:23.692643 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:23.692611    2559 patch_prober.go:28] interesting pod/console-7fd9d847f4-pzwjk container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.27:8443/health\": dial tcp 10.132.0.27:8443: connect: connection refused" start-of-body=
Jul 29 18:35:23.692779 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:23.692666    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-7fd9d847f4-pzwjk" podUID="b5540cda-33a8-446e-a74b-4c67ebd287bd" containerName="console" probeResult="failure" output="Get \"https://10.132.0.27:8443/health\": dial tcp 10.132.0.27:8443: connect: connection refused"
Jul 29 18:35:23.854312 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:23.854279    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wpg7b" event={"ID":"6c9d825b-217d-499b-b8a4-b0dbf396b283","Type":"ContainerStarted","Data":"4ecb4ebb6433cd6d1cc79916bbaceaad9e7be215ee8d648532127bf7eace055e"}
Jul 29 18:35:23.874497 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:23.874440    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wpg7b" podStartSLOduration=17.680590711 podStartE2EDuration="36.874422028s" totalImagesPullingTime="19.193831317s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:23.872804448 +0000 UTC m=+88.142573772" watchObservedRunningTime="2026-07-29 18:35:23.874422028 +0000 UTC m=+88.144191115"
Jul 29 18:35:24.736116 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:24.736084    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6cb6fb9448-jbzkl"
Jul 29 18:35:29.170746 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:29.170703    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-v62d6"]
Jul 29 18:35:29.201113 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:29.201077    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-v62d6"
Jul 29 18:35:29.205162 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:29.205136    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 29 18:35:29.205409 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:29.205368    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 29 18:35:29.205867 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:29.205846    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 29 18:35:29.206296 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:29.206139    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 29 18:35:29.207345 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:29.207328    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-sn2rv\""
Jul 29 18:35:29.329047 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:29.329011    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1d18cddd-00d7-4d19-a31d-574900bef72b-root\") pod \"node-exporter-v62d6\" (UID: \"1d18cddd-00d7-4d19-a31d-574900bef72b\") " pod="openshift-monitoring/node-exporter-v62d6"
Jul 29 18:35:29.329241 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:29.329148    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1d18cddd-00d7-4d19-a31d-574900bef72b-node-exporter-textfile\") pod \"node-exporter-v62d6\" (UID: \"1d18cddd-00d7-4d19-a31d-574900bef72b\") " pod="openshift-monitoring/node-exporter-v62d6"
Jul 29 18:35:29.329241 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:29.329182    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1d18cddd-00d7-4d19-a31d-574900bef72b-node-exporter-wtmp\") pod \"node-exporter-v62d6\" (UID: \"1d18cddd-00d7-4d19-a31d-574900bef72b\") " pod="openshift-monitoring/node-exporter-v62d6"
Jul 29 18:35:29.329241 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:29.329217    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1d18cddd-00d7-4d19-a31d-574900bef72b-node-exporter-accelerators-collector-config\") pod \"node-exporter-v62d6\" (UID: \"1d18cddd-00d7-4d19-a31d-574900bef72b\") " pod="openshift-monitoring/node-exporter-v62d6"
Jul 29 18:35:29.329435 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:29.329248    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hwjh\" (UniqueName: \"kubernetes.io/projected/1d18cddd-00d7-4d19-a31d-574900bef72b-kube-api-access-8hwjh\") pod \"node-exporter-v62d6\" (UID: \"1d18cddd-00d7-4d19-a31d-574900bef72b\") " pod="openshift-monitoring/node-exporter-v62d6"
Jul 29 18:35:29.329435 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:29.329314    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1d18cddd-00d7-4d19-a31d-574900bef72b-node-exporter-tls\") pod \"node-exporter-v62d6\" (UID: \"1d18cddd-00d7-4d19-a31d-574900bef72b\") " pod="openshift-monitoring/node-exporter-v62d6"
Jul 29 18:35:29.329435 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:29.329357    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d18cddd-00d7-4d19-a31d-574900bef72b-sys\") pod \"node-exporter-v62d6\" (UID: \"1d18cddd-00d7-4d19-a31d-574900bef72b\") " pod="openshift-monitoring/node-exporter-v62d6"
Jul 29 18:35:29.329435 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:29.329417    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d18cddd-00d7-4d19-a31d-574900bef72b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v62d6\" (UID: \"1d18cddd-00d7-4d19-a31d-574900bef72b\") " pod="openshift-monitoring/node-exporter-v62d6"
Jul 29 18:35:29.329616 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:29.329455    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d18cddd-00d7-4d19-a31d-574900bef72b-metrics-client-ca\") pod \"node-exporter-v62d6\" (UID: \"1d18cddd-00d7-4d19-a31d-574900bef72b\") " pod="openshift-monitoring/node-exporter-v62d6"
Jul 29 18:35:29.430777 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:29.430684    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1d18cddd-00d7-4d19-a31d-574900bef72b-root\") pod \"node-exporter-v62d6\" (UID: \"1d18cddd-00d7-4d19-a31d-574900bef72b\") " pod="openshift-monitoring/node-exporter-v62d6"
Jul 29 18:35:29.430777 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:29.430766    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1d18cddd-00d7-4d19-a31d-574900bef72b-node-exporter-textfile\") pod \"node-exporter-v62d6\" (UID: \"1d18cddd-00d7-4d19-a31d-574900bef72b\") " pod="openshift-monitoring/node-exporter-v62d6"
Jul 29 18:35:29.431001 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:29.430791    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1d18cddd-00d7-4d19-a31d-574900bef72b-node-exporter-wtmp\") pod \"node-exporter-v62d6\" (UID: \"1d18cddd-00d7-4d19-a31d-574900bef72b\") " pod="openshift-monitoring/node-exporter-v62d6"
Jul 29 18:35:29.431001 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:29.430820    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1d18cddd-00d7-4d19-a31d-574900bef72b-node-exporter-accelerators-collector-config\") pod \"node-exporter-v62d6\" (UID: \"1d18cddd-00d7-4d19-a31d-574900bef72b\") " pod="openshift-monitoring/node-exporter-v62d6"
Jul 29 18:35:29.431001 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:29.430848    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hwjh\" (UniqueName: \"kubernetes.io/projected/1d18cddd-00d7-4d19-a31d-574900bef72b-kube-api-access-8hwjh\") pod \"node-exporter-v62d6\" (UID: \"1d18cddd-00d7-4d19-a31d-574900bef72b\") " pod="openshift-monitoring/node-exporter-v62d6"
Jul 29 18:35:29.431001 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:29.430883    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1d18cddd-00d7-4d19-a31d-574900bef72b-node-exporter-tls\") pod \"node-exporter-v62d6\" (UID: \"1d18cddd-00d7-4d19-a31d-574900bef72b\") " pod="openshift-monitoring/node-exporter-v62d6"
Jul 29 18:35:29.431001 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:29.430910    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d18cddd-00d7-4d19-a31d-574900bef72b-sys\") pod \"node-exporter-v62d6\" (UID: \"1d18cddd-00d7-4d19-a31d-574900bef72b\") " pod="openshift-monitoring/node-exporter-v62d6"
Jul 29 18:35:29.431001 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:29.430944    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d18cddd-00d7-4d19-a31d-574900bef72b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v62d6\" (UID: \"1d18cddd-00d7-4d19-a31d-574900bef72b\") " pod="openshift-monitoring/node-exporter-v62d6"
Jul 29 18:35:29.431001 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:29.430980    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d18cddd-00d7-4d19-a31d-574900bef72b-metrics-client-ca\") pod \"node-exporter-v62d6\" (UID: \"1d18cddd-00d7-4d19-a31d-574900bef72b\") " pod="openshift-monitoring/node-exporter-v62d6"
Jul 29 18:35:29.431661 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:29.431639    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d18cddd-00d7-4d19-a31d-574900bef72b-metrics-client-ca\") pod \"node-exporter-v62d6\" (UID: \"1d18cddd-00d7-4d19-a31d-574900bef72b\") " pod="openshift-monitoring/node-exporter-v62d6"
Jul 29 18:35:29.431779 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:29.431719    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1d18cddd-00d7-4d19-a31d-574900bef72b-root\") pod \"node-exporter-v62d6\" (UID: \"1d18cddd-00d7-4d19-a31d-574900bef72b\") " pod="openshift-monitoring/node-exporter-v62d6"
Jul 29 18:35:29.431956 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:29.431936    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1d18cddd-00d7-4d19-a31d-574900bef72b-node-exporter-textfile\") pod \"node-exporter-v62d6\" (UID: \"1d18cddd-00d7-4d19-a31d-574900bef72b\") " pod="openshift-monitoring/node-exporter-v62d6"
Jul 29 18:35:29.432077 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:29.432062    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1d18cddd-00d7-4d19-a31d-574900bef72b-node-exporter-wtmp\") pod \"node-exporter-v62d6\" (UID: \"1d18cddd-00d7-4d19-a31d-574900bef72b\") " pod="openshift-monitoring/node-exporter-v62d6"
Jul 29 18:35:29.432507 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:29.432488    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1d18cddd-00d7-4d19-a31d-574900bef72b-node-exporter-accelerators-collector-config\") pod \"node-exporter-v62d6\" (UID: \"1d18cddd-00d7-4d19-a31d-574900bef72b\") " pod="openshift-monitoring/node-exporter-v62d6"
Jul 29 18:35:29.432911 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:35:29.432892    2559 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 29 18:35:29.432982 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:35:29.432952    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d18cddd-00d7-4d19-a31d-574900bef72b-node-exporter-tls podName:1d18cddd-00d7-4d19-a31d-574900bef72b nodeName:}" failed. No retries permitted until 2026-07-29 18:35:29.932935587 +0000 UTC m=+94.202704657 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/1d18cddd-00d7-4d19-a31d-574900bef72b-node-exporter-tls") pod "node-exporter-v62d6" (UID: "1d18cddd-00d7-4d19-a31d-574900bef72b") : secret "node-exporter-tls" not found
Jul 29 18:35:29.433155 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:29.433137    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d18cddd-00d7-4d19-a31d-574900bef72b-sys\") pod \"node-exporter-v62d6\" (UID: \"1d18cddd-00d7-4d19-a31d-574900bef72b\") " pod="openshift-monitoring/node-exporter-v62d6"
Jul 29 18:35:29.435883 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:29.435858    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d18cddd-00d7-4d19-a31d-574900bef72b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v62d6\" (UID: \"1d18cddd-00d7-4d19-a31d-574900bef72b\") " pod="openshift-monitoring/node-exporter-v62d6"
Jul 29 18:35:29.445344 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:29.445317    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hwjh\" (UniqueName: \"kubernetes.io/projected/1d18cddd-00d7-4d19-a31d-574900bef72b-kube-api-access-8hwjh\") pod \"node-exporter-v62d6\" (UID: \"1d18cddd-00d7-4d19-a31d-574900bef72b\") " pod="openshift-monitoring/node-exporter-v62d6"
Jul 29 18:35:29.936052 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:29.936002    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1d18cddd-00d7-4d19-a31d-574900bef72b-node-exporter-tls\") pod \"node-exporter-v62d6\" (UID: \"1d18cddd-00d7-4d19-a31d-574900bef72b\") " pod="openshift-monitoring/node-exporter-v62d6"
Jul 29 18:35:29.938937 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:29.938898    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1d18cddd-00d7-4d19-a31d-574900bef72b-node-exporter-tls\") pod \"node-exporter-v62d6\" (UID: \"1d18cddd-00d7-4d19-a31d-574900bef72b\") " pod="openshift-monitoring/node-exporter-v62d6"
Jul 29 18:35:30.114916 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:30.114875    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-v62d6"
Jul 29 18:35:31.798417 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:31.798360    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fd9d847f4-pzwjk"]
Jul 29 18:35:32.701556 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:32.701527    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jh2dv"
Jul 29 18:35:32.910903 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:35:32.910868    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d18cddd_00d7_4d19_a31d_574900bef72b.slice/crio-d406d74f2554d81938681b0c15f8b1022a17da3989c3091e4e5e8b105ffd5b04 WatchSource:0}: Error finding container d406d74f2554d81938681b0c15f8b1022a17da3989c3091e4e5e8b105ffd5b04: Status 404 returned error can't find the container with id d406d74f2554d81938681b0c15f8b1022a17da3989c3091e4e5e8b105ffd5b04
Jul 29 18:35:33.903651 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:33.903615    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v62d6" event={"ID":"1d18cddd-00d7-4d19-a31d-574900bef72b","Type":"ContainerStarted","Data":"d406d74f2554d81938681b0c15f8b1022a17da3989c3091e4e5e8b105ffd5b04"}
Jul 29 18:35:33.905570 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:33.905535    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-24vj5" event={"ID":"26035b26-9c55-49cf-a2e9-48634f8c5cd5","Type":"ContainerStarted","Data":"3a58339870e4f4a6f72124a5f3eaae4a30bbf579b56a73b091aa83b05fb4c8d9"}
Jul 29 18:35:33.905763 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:33.905742    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-24vj5"
Jul 29 18:35:33.917812 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:33.917781    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-24vj5"
Jul 29 18:35:33.928967 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:33.928832    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-24vj5" podStartSLOduration=2.571171115 podStartE2EDuration="21.928809142s" totalImagesPullingTime="19.357638027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:35:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:33.926364166 +0000 UTC m=+98.196133265" watchObservedRunningTime="2026-07-29 18:35:33.928809142 +0000 UTC m=+98.198578230"
Jul 29 18:35:34.912896 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:34.912804    2559 generic.go:356] "Generic (PLEG): container finished" podID="1d18cddd-00d7-4d19-a31d-574900bef72b" containerID="47c705bc15bc1a115ec277d98a1c35ba0b8544624e4259fb49710bbe46974ddd" exitCode=0
Jul 29 18:35:34.913066 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:34.912902    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v62d6" event={"ID":"1d18cddd-00d7-4d19-a31d-574900bef72b","Type":"ContainerDied","Data":"47c705bc15bc1a115ec277d98a1c35ba0b8544624e4259fb49710bbe46974ddd"}
Jul 29 18:35:35.920901 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:35.920845    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v62d6" event={"ID":"1d18cddd-00d7-4d19-a31d-574900bef72b","Type":"ContainerStarted","Data":"c4f5b1da4c9a4f1b9b80445e2f880348e57443abb0b2ee6a4c07643f953138ea"}
Jul 29 18:35:35.921354 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:35.920969    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v62d6" event={"ID":"1d18cddd-00d7-4d19-a31d-574900bef72b","Type":"ContainerStarted","Data":"76d2209fbf705aa841e31380f9c4e3fd91ea6e413278e361f877154701e872d8"}
Jul 29 18:35:35.943001 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:35.942927    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-v62d6" podStartSLOduration=5.364650606 podStartE2EDuration="6.942911775s" totalImagesPullingTime="1.463324669s" totalInitContainerRuntime="114.9365ms" isStatefulPod=true podCreationTimestamp="2026-07-29 18:35:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:35.940993769 +0000 UTC m=+100.210762860" watchObservedRunningTime="2026-07-29 18:35:35.942911775 +0000 UTC m=+100.212680917"
Jul 29 18:35:36.086032 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:36.085998    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6cb6fb9448-jbzkl"]
Jul 29 18:35:52.998046 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:52.998011    2559 generic.go:356] "Generic (PLEG): container finished" podID="05f777ff-5b55-41e4-b440-1705c2c3842e" containerID="bdc4eef31622ba6b28f484d52a061c3479b51e041c5934b90b167d37259d2891" exitCode=0
Jul 29 18:35:52.998500 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:52.998088    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nj275" event={"ID":"05f777ff-5b55-41e4-b440-1705c2c3842e","Type":"ContainerDied","Data":"bdc4eef31622ba6b28f484d52a061c3479b51e041c5934b90b167d37259d2891"}
Jul 29 18:35:52.998500 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:52.998461    2559 scope.go:117] "RemoveContainer" containerID="bdc4eef31622ba6b28f484d52a061c3479b51e041c5934b90b167d37259d2891"
Jul 29 18:35:54.004509 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:54.004475    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nj275" event={"ID":"05f777ff-5b55-41e4-b440-1705c2c3842e","Type":"ContainerStarted","Data":"01d9163ce481b1dac7f906ab7967835763920e8192f8bec8fb819f5ba9e0ac25"}
Jul 29 18:35:56.823469 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:56.823403    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7fd9d847f4-pzwjk" podUID="b5540cda-33a8-446e-a74b-4c67ebd287bd" containerName="console" containerID="cri-o://357ab095fdeb0736d7f0e28ab422c7dba3186b07d6dedf93a8c83d2447053958" gracePeriod=15
Jul 29 18:35:57.020725 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:57.020695    2559 generic.go:356] "Generic (PLEG): container finished" podID="b5540cda-33a8-446e-a74b-4c67ebd287bd" containerID="357ab095fdeb0736d7f0e28ab422c7dba3186b07d6dedf93a8c83d2447053958" exitCode=0
Jul 29 18:35:57.020879 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:57.020752    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fd9d847f4-pzwjk" event={"ID":"b5540cda-33a8-446e-a74b-4c67ebd287bd","Type":"ContainerDied","Data":"357ab095fdeb0736d7f0e28ab422c7dba3186b07d6dedf93a8c83d2447053958"}
Jul 29 18:35:57.114718 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:57.114695    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fd9d847f4-pzwjk"
Jul 29 18:35:57.181426 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:57.181391    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5540cda-33a8-446e-a74b-4c67ebd287bd-console-serving-cert\") pod \"b5540cda-33a8-446e-a74b-4c67ebd287bd\" (UID: \"b5540cda-33a8-446e-a74b-4c67ebd287bd\") "
Jul 29 18:35:57.181426 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:57.181421    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5540cda-33a8-446e-a74b-4c67ebd287bd-oauth-serving-cert\") pod \"b5540cda-33a8-446e-a74b-4c67ebd287bd\" (UID: \"b5540cda-33a8-446e-a74b-4c67ebd287bd\") "
Jul 29 18:35:57.181426 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:57.181439    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5540cda-33a8-446e-a74b-4c67ebd287bd-service-ca\") pod \"b5540cda-33a8-446e-a74b-4c67ebd287bd\" (UID: \"b5540cda-33a8-446e-a74b-4c67ebd287bd\") "
Jul 29 18:35:57.181712 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:57.181460    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45xfr\" (UniqueName: \"kubernetes.io/projected/b5540cda-33a8-446e-a74b-4c67ebd287bd-kube-api-access-45xfr\") pod \"b5540cda-33a8-446e-a74b-4c67ebd287bd\" (UID: \"b5540cda-33a8-446e-a74b-4c67ebd287bd\") "
Jul 29 18:35:57.181712 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:57.181496    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5540cda-33a8-446e-a74b-4c67ebd287bd-console-oauth-config\") pod \"b5540cda-33a8-446e-a74b-4c67ebd287bd\" (UID: \"b5540cda-33a8-446e-a74b-4c67ebd287bd\") "
Jul 29 18:35:57.181712 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:57.181630    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5540cda-33a8-446e-a74b-4c67ebd287bd-console-config\") pod \"b5540cda-33a8-446e-a74b-4c67ebd287bd\" (UID: \"b5540cda-33a8-446e-a74b-4c67ebd287bd\") "
Jul 29 18:35:57.181914 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:57.181874    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5540cda-33a8-446e-a74b-4c67ebd287bd-service-ca" (OuterVolumeSpecName: "service-ca") pod "b5540cda-33a8-446e-a74b-4c67ebd287bd" (UID: "b5540cda-33a8-446e-a74b-4c67ebd287bd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 18:35:57.181987 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:57.181880    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5540cda-33a8-446e-a74b-4c67ebd287bd-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b5540cda-33a8-446e-a74b-4c67ebd287bd" (UID: "b5540cda-33a8-446e-a74b-4c67ebd287bd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 18:35:57.181987 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:57.181943    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5540cda-33a8-446e-a74b-4c67ebd287bd-console-config" (OuterVolumeSpecName: "console-config") pod "b5540cda-33a8-446e-a74b-4c67ebd287bd" (UID: "b5540cda-33a8-446e-a74b-4c67ebd287bd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 18:35:57.183954 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:57.183917    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5540cda-33a8-446e-a74b-4c67ebd287bd-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b5540cda-33a8-446e-a74b-4c67ebd287bd" (UID: "b5540cda-33a8-446e-a74b-4c67ebd287bd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 18:35:57.184062 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:57.183962    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5540cda-33a8-446e-a74b-4c67ebd287bd-kube-api-access-45xfr" (OuterVolumeSpecName: "kube-api-access-45xfr") pod "b5540cda-33a8-446e-a74b-4c67ebd287bd" (UID: "b5540cda-33a8-446e-a74b-4c67ebd287bd"). InnerVolumeSpecName "kube-api-access-45xfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 18:35:57.184062 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:57.183962    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5540cda-33a8-446e-a74b-4c67ebd287bd-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b5540cda-33a8-446e-a74b-4c67ebd287bd" (UID: "b5540cda-33a8-446e-a74b-4c67ebd287bd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 18:35:57.282725 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:57.282685    2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5540cda-33a8-446e-a74b-4c67ebd287bd-console-config\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 18:35:57.282725 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:57.282720    2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5540cda-33a8-446e-a74b-4c67ebd287bd-console-serving-cert\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 18:35:57.282725 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:57.282731    2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5540cda-33a8-446e-a74b-4c67ebd287bd-oauth-serving-cert\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 18:35:57.282957 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:57.282740    2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5540cda-33a8-446e-a74b-4c67ebd287bd-service-ca\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 18:35:57.282957 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:57.282748    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45xfr\" (UniqueName: \"kubernetes.io/projected/b5540cda-33a8-446e-a74b-4c67ebd287bd-kube-api-access-45xfr\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 18:35:57.282957 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:57.282757    2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5540cda-33a8-446e-a74b-4c67ebd287bd-console-oauth-config\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 18:35:58.027099 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:58.027068    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fd9d847f4-pzwjk"
Jul 29 18:35:58.027099 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:58.027088    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fd9d847f4-pzwjk" event={"ID":"b5540cda-33a8-446e-a74b-4c67ebd287bd","Type":"ContainerDied","Data":"07bf6190b6bfc1f0563fbd0d012df935208a17eca97ebf1ff544857f50b2b09f"}
Jul 29 18:35:58.027677 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:58.027158    2559 scope.go:117] "RemoveContainer" containerID="357ab095fdeb0736d7f0e28ab422c7dba3186b07d6dedf93a8c83d2447053958"
Jul 29 18:35:58.028724 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:58.028699    2559 generic.go:356] "Generic (PLEG): container finished" podID="a84d718f-459d-4e2e-9c9e-d08d1500b145" containerID="b3a5b8fc0938ad47f5979f658d7ec1781b3975361eee90984b83d12b131d813e" exitCode=0
Jul 29 18:35:58.028788 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:58.028746    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-j76ql" event={"ID":"a84d718f-459d-4e2e-9c9e-d08d1500b145","Type":"ContainerDied","Data":"b3a5b8fc0938ad47f5979f658d7ec1781b3975361eee90984b83d12b131d813e"}
Jul 29 18:35:58.029348 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:58.029330    2559 scope.go:117] "RemoveContainer" containerID="b3a5b8fc0938ad47f5979f658d7ec1781b3975361eee90984b83d12b131d813e"
Jul 29 18:35:58.065323 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:58.065292    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fd9d847f4-pzwjk"]
Jul 29 18:35:58.069538 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:58.069513    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7fd9d847f4-pzwjk"]
Jul 29 18:35:58.151414 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:58.151362    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5540cda-33a8-446e-a74b-4c67ebd287bd" path="/var/lib/kubelet/pods/b5540cda-33a8-446e-a74b-4c67ebd287bd/volumes"
Jul 29 18:35:59.035822 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:35:59.035791    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-j76ql" event={"ID":"a84d718f-459d-4e2e-9c9e-d08d1500b145","Type":"ContainerStarted","Data":"0477d798fa5eb2fb07dfedca345073d7a9ff9f82710397c27348d5776362f622"}
Jul 29 18:36:01.115614 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:01.115527    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6cb6fb9448-jbzkl" podUID="e281e146-0b8a-4358-a98b-7f577f4ff08b" containerName="registry" containerID="cri-o://17e5ae82c534891dcf6c418f9c5c7e6fcef4a6fdc83d3364aa6ac9e7a73c72e4" gracePeriod=30
Jul 29 18:36:01.409563 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:01.409534    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cb6fb9448-jbzkl"
Jul 29 18:36:01.414865 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:01.414816    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e281e146-0b8a-4358-a98b-7f577f4ff08b-registry-certificates\") pod \"e281e146-0b8a-4358-a98b-7f577f4ff08b\" (UID: \"e281e146-0b8a-4358-a98b-7f577f4ff08b\") "
Jul 29 18:36:01.414865 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:01.414870    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e281e146-0b8a-4358-a98b-7f577f4ff08b-trusted-ca\") pod \"e281e146-0b8a-4358-a98b-7f577f4ff08b\" (UID: \"e281e146-0b8a-4358-a98b-7f577f4ff08b\") "
Jul 29 18:36:01.415048 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:01.414912    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e281e146-0b8a-4358-a98b-7f577f4ff08b-installation-pull-secrets\") pod \"e281e146-0b8a-4358-a98b-7f577f4ff08b\" (UID: \"e281e146-0b8a-4358-a98b-7f577f4ff08b\") "
Jul 29 18:36:01.415048 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:01.414938    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e281e146-0b8a-4358-a98b-7f577f4ff08b-bound-sa-token\") pod \"e281e146-0b8a-4358-a98b-7f577f4ff08b\" (UID: \"e281e146-0b8a-4358-a98b-7f577f4ff08b\") "
Jul 29 18:36:01.415048 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:01.414982    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e281e146-0b8a-4358-a98b-7f577f4ff08b-image-registry-private-configuration\") pod \"e281e146-0b8a-4358-a98b-7f577f4ff08b\" (UID: \"e281e146-0b8a-4358-a98b-7f577f4ff08b\") "
Jul 29 18:36:01.415048 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:01.415004    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e281e146-0b8a-4358-a98b-7f577f4ff08b-registry-tls\") pod \"e281e146-0b8a-4358-a98b-7f577f4ff08b\" (UID: \"e281e146-0b8a-4358-a98b-7f577f4ff08b\") "
Jul 29 18:36:01.415048 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:01.415037    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e281e146-0b8a-4358-a98b-7f577f4ff08b-ca-trust-extracted\") pod \"e281e146-0b8a-4358-a98b-7f577f4ff08b\" (UID: \"e281e146-0b8a-4358-a98b-7f577f4ff08b\") "
Jul 29 18:36:01.415296 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:01.415067    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kx2wt\" (UniqueName: \"kubernetes.io/projected/e281e146-0b8a-4358-a98b-7f577f4ff08b-kube-api-access-kx2wt\") pod \"e281e146-0b8a-4358-a98b-7f577f4ff08b\" (UID: \"e281e146-0b8a-4358-a98b-7f577f4ff08b\") "
Jul 29 18:36:01.415296 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:01.415262    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e281e146-0b8a-4358-a98b-7f577f4ff08b-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "e281e146-0b8a-4358-a98b-7f577f4ff08b" (UID: "e281e146-0b8a-4358-a98b-7f577f4ff08b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 18:36:01.415470 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:01.415442    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e281e146-0b8a-4358-a98b-7f577f4ff08b-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "e281e146-0b8a-4358-a98b-7f577f4ff08b" (UID: "e281e146-0b8a-4358-a98b-7f577f4ff08b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 18:36:01.417750 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:01.417726    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e281e146-0b8a-4358-a98b-7f577f4ff08b-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "e281e146-0b8a-4358-a98b-7f577f4ff08b" (UID: "e281e146-0b8a-4358-a98b-7f577f4ff08b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 18:36:01.418087 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:01.418054    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e281e146-0b8a-4358-a98b-7f577f4ff08b-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "e281e146-0b8a-4358-a98b-7f577f4ff08b" (UID: "e281e146-0b8a-4358-a98b-7f577f4ff08b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 18:36:01.418169 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:01.418095    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e281e146-0b8a-4358-a98b-7f577f4ff08b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e281e146-0b8a-4358-a98b-7f577f4ff08b" (UID: "e281e146-0b8a-4358-a98b-7f577f4ff08b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 18:36:01.418169 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:01.418124    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e281e146-0b8a-4358-a98b-7f577f4ff08b-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "e281e146-0b8a-4358-a98b-7f577f4ff08b" (UID: "e281e146-0b8a-4358-a98b-7f577f4ff08b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 18:36:01.418236 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:01.418212    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e281e146-0b8a-4358-a98b-7f577f4ff08b-kube-api-access-kx2wt" (OuterVolumeSpecName: "kube-api-access-kx2wt") pod "e281e146-0b8a-4358-a98b-7f577f4ff08b" (UID: "e281e146-0b8a-4358-a98b-7f577f4ff08b"). InnerVolumeSpecName "kube-api-access-kx2wt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 18:36:01.423472 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:01.423442    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e281e146-0b8a-4358-a98b-7f577f4ff08b-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "e281e146-0b8a-4358-a98b-7f577f4ff08b" (UID: "e281e146-0b8a-4358-a98b-7f577f4ff08b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:36:01.516445 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:01.516372    2559 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e281e146-0b8a-4358-a98b-7f577f4ff08b-image-registry-private-configuration\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 18:36:01.516445 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:01.516439    2559 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e281e146-0b8a-4358-a98b-7f577f4ff08b-registry-tls\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 18:36:01.516445 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:01.516452    2559 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e281e146-0b8a-4358-a98b-7f577f4ff08b-ca-trust-extracted\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 18:36:01.516445 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:01.516461    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx2wt\" (UniqueName: \"kubernetes.io/projected/e281e146-0b8a-4358-a98b-7f577f4ff08b-kube-api-access-kx2wt\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 18:36:01.516713 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:01.516469    2559 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e281e146-0b8a-4358-a98b-7f577f4ff08b-registry-certificates\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 18:36:01.516713 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:01.516477    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e281e146-0b8a-4358-a98b-7f577f4ff08b-trusted-ca\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 18:36:01.516713 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:01.516486    2559 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e281e146-0b8a-4358-a98b-7f577f4ff08b-installation-pull-secrets\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 18:36:01.516713 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:01.516521    2559 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e281e146-0b8a-4358-a98b-7f577f4ff08b-bound-sa-token\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 18:36:02.050674 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:02.050639    2559 generic.go:356] "Generic (PLEG): container finished" podID="e281e146-0b8a-4358-a98b-7f577f4ff08b" containerID="17e5ae82c534891dcf6c418f9c5c7e6fcef4a6fdc83d3364aa6ac9e7a73c72e4" exitCode=0
Jul 29 18:36:02.050869 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:02.050685    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cb6fb9448-jbzkl"
Jul 29 18:36:02.050869 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:02.050729    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cb6fb9448-jbzkl" event={"ID":"e281e146-0b8a-4358-a98b-7f577f4ff08b","Type":"ContainerDied","Data":"17e5ae82c534891dcf6c418f9c5c7e6fcef4a6fdc83d3364aa6ac9e7a73c72e4"}
Jul 29 18:36:02.050869 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:02.050788    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cb6fb9448-jbzkl" event={"ID":"e281e146-0b8a-4358-a98b-7f577f4ff08b","Type":"ContainerDied","Data":"71e1be586e0b48aaff7a2dcecd301b120385c857244a370821541f4728d9bfb8"}
Jul 29 18:36:02.050869 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:02.050815    2559 scope.go:117] "RemoveContainer" containerID="17e5ae82c534891dcf6c418f9c5c7e6fcef4a6fdc83d3364aa6ac9e7a73c72e4"
Jul 29 18:36:02.059792 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:02.059771    2559 scope.go:117] "RemoveContainer" containerID="17e5ae82c534891dcf6c418f9c5c7e6fcef4a6fdc83d3364aa6ac9e7a73c72e4"
Jul 29 18:36:02.060112 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:36:02.060091    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17e5ae82c534891dcf6c418f9c5c7e6fcef4a6fdc83d3364aa6ac9e7a73c72e4\": container with ID starting with 17e5ae82c534891dcf6c418f9c5c7e6fcef4a6fdc83d3364aa6ac9e7a73c72e4 not found: ID does not exist" containerID="17e5ae82c534891dcf6c418f9c5c7e6fcef4a6fdc83d3364aa6ac9e7a73c72e4"
Jul 29 18:36:02.060183 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:02.060127    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17e5ae82c534891dcf6c418f9c5c7e6fcef4a6fdc83d3364aa6ac9e7a73c72e4"} err="failed to get container status \"17e5ae82c534891dcf6c418f9c5c7e6fcef4a6fdc83d3364aa6ac9e7a73c72e4\": rpc error: code = NotFound desc = could not find container \"17e5ae82c534891dcf6c418f9c5c7e6fcef4a6fdc83d3364aa6ac9e7a73c72e4\": container with ID starting with 17e5ae82c534891dcf6c418f9c5c7e6fcef4a6fdc83d3364aa6ac9e7a73c72e4 not found: ID does not exist"
Jul 29 18:36:02.071989 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:02.071954    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6cb6fb9448-jbzkl"]
Jul 29 18:36:02.076400 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:02.076349    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6cb6fb9448-jbzkl"]
Jul 29 18:36:02.149344 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:02.149310    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e281e146-0b8a-4358-a98b-7f577f4ff08b" path="/var/lib/kubelet/pods/e281e146-0b8a-4358-a98b-7f577f4ff08b/volumes"
Jul 29 18:36:18.129448 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:18.129412    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-95ttd" event={"ID":"cad930eb-2903-4c78-b4c4-3bcc190d199d","Type":"ContainerDied","Data":"1647e407180f226f83d92072917ac903aee86a66fe3256562199e1dcee781e1a"}
Jul 29 18:36:18.129864 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:18.129414    2559 generic.go:356] "Generic (PLEG): container finished" podID="cad930eb-2903-4c78-b4c4-3bcc190d199d" containerID="1647e407180f226f83d92072917ac903aee86a66fe3256562199e1dcee781e1a" exitCode=0
Jul 29 18:36:18.129864 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:18.129753    2559 scope.go:117] "RemoveContainer" containerID="1647e407180f226f83d92072917ac903aee86a66fe3256562199e1dcee781e1a"
Jul 29 18:36:19.135795 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:19.135759    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-95ttd" event={"ID":"cad930eb-2903-4c78-b4c4-3bcc190d199d","Type":"ContainerStarted","Data":"d4a43e7cf02c5ec2008a51dd3ff5d4aba1ccafab2106fdeaea5f031301f4146e"}
Jul 29 18:36:21.025867 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:21.025823    2559 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58dbb54c4d-zmrbt" podUID="542cc68a-f9f1-4d9c-8be9-1de3687ca26a" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 29 18:36:31.025427 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:31.025365    2559 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58dbb54c4d-zmrbt" podUID="542cc68a-f9f1-4d9c-8be9-1de3687ca26a" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 29 18:36:41.025372 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:41.025328    2559 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58dbb54c4d-zmrbt" podUID="542cc68a-f9f1-4d9c-8be9-1de3687ca26a" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 29 18:36:41.025848 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:41.025435    2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58dbb54c4d-zmrbt"
Jul 29 18:36:41.025912 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:41.025894    2559 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"65154b16dcf8fa1d8fc1028f219ee2121991d15676cd635fbe1152598be5fde6"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58dbb54c4d-zmrbt" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 29 18:36:41.025950 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:41.025931    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58dbb54c4d-zmrbt" podUID="542cc68a-f9f1-4d9c-8be9-1de3687ca26a" containerName="service-proxy" containerID="cri-o://65154b16dcf8fa1d8fc1028f219ee2121991d15676cd635fbe1152598be5fde6" gracePeriod=30
Jul 29 18:36:41.230828 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:41.230792    2559 generic.go:356] "Generic (PLEG): container finished" podID="542cc68a-f9f1-4d9c-8be9-1de3687ca26a" containerID="65154b16dcf8fa1d8fc1028f219ee2121991d15676cd635fbe1152598be5fde6" exitCode=2
Jul 29 18:36:41.230995 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:41.230863    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58dbb54c4d-zmrbt" event={"ID":"542cc68a-f9f1-4d9c-8be9-1de3687ca26a","Type":"ContainerDied","Data":"65154b16dcf8fa1d8fc1028f219ee2121991d15676cd635fbe1152598be5fde6"}
Jul 29 18:36:41.230995 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:36:41.230913    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58dbb54c4d-zmrbt" event={"ID":"542cc68a-f9f1-4d9c-8be9-1de3687ca26a","Type":"ContainerStarted","Data":"ccf444c5cbc8bc1767d6ae304cfe80c08cd38ae4e0068bca79ed397ea4aaaa63"}
Jul 29 18:38:56.039078 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:38:56.039047    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-v9nsr_7f8b4737-d4c2-4598-bd40-208015542831/console-operator/1.log"
Jul 29 18:38:56.044894 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:38:56.044866    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-v9nsr_7f8b4737-d4c2-4598-bd40-208015542831/console-operator/1.log"
Jul 29 18:38:56.047211 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:38:56.047185    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjdrx_60f444e1-7e31-443c-8ec7-a4d04c64bf4b/ovn-acl-logging/0.log"
Jul 29 18:38:56.050244 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:38:56.050219    2559 kubelet.go:1648] "Image garbage collection succeeded"
Jul 29 18:38:56.056918 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:38:56.056894    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjdrx_60f444e1-7e31-443c-8ec7-a4d04c64bf4b/ovn-acl-logging/0.log"
Jul 29 18:40:18.010926 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:18.010889    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-qn29c"]
Jul 29 18:40:18.011446 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:18.011185    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5540cda-33a8-446e-a74b-4c67ebd287bd" containerName="console"
Jul 29 18:40:18.011446 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:18.011195    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5540cda-33a8-446e-a74b-4c67ebd287bd" containerName="console"
Jul 29 18:40:18.011446 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:18.011216    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e281e146-0b8a-4358-a98b-7f577f4ff08b" containerName="registry"
Jul 29 18:40:18.011446 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:18.011222    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e281e146-0b8a-4358-a98b-7f577f4ff08b" containerName="registry"
Jul 29 18:40:18.011446 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:18.011272    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5540cda-33a8-446e-a74b-4c67ebd287bd" containerName="console"
Jul 29 18:40:18.011446 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:18.011281    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e281e146-0b8a-4358-a98b-7f577f4ff08b" containerName="registry"
Jul 29 18:40:18.014120 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:18.014104    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-qn29c"
Jul 29 18:40:18.016940 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:18.016908    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-b6vgb\""
Jul 29 18:40:18.017582 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:18.017563    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 29 18:40:18.017687 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:18.017671    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 29 18:40:18.018218 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:18.018188    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 29 18:40:18.018325 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:18.018225    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 29 18:40:18.022678 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:18.022655    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e820933-bb87-4c58-b7aa-0cb939217b29-tmp\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-qn29c\" (UID: \"9e820933-bb87-4c58-b7aa-0cb939217b29\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-qn29c"
Jul 29 18:40:18.022813 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:18.022796    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9e820933-bb87-4c58-b7aa-0cb939217b29-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-qn29c\" (UID: \"9e820933-bb87-4c58-b7aa-0cb939217b29\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-qn29c"
Jul 29 18:40:18.022853 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:18.022834    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbsk5\" (UniqueName: \"kubernetes.io/projected/9e820933-bb87-4c58-b7aa-0cb939217b29-kube-api-access-zbsk5\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-qn29c\" (UID: \"9e820933-bb87-4c58-b7aa-0cb939217b29\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-qn29c"
Jul 29 18:40:18.022893 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:18.022879    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9e820933-bb87-4c58-b7aa-0cb939217b29-webhook-cert\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-qn29c\" (UID: \"9e820933-bb87-4c58-b7aa-0cb939217b29\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-qn29c"
Jul 29 18:40:18.022936 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:18.022908    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9e820933-bb87-4c58-b7aa-0cb939217b29-manifests\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-qn29c\" (UID: \"9e820933-bb87-4c58-b7aa-0cb939217b29\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-qn29c"
Jul 29 18:40:18.037356 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:18.034849    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-qn29c"]
Jul 29 18:40:18.123574 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:18.123536    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9e820933-bb87-4c58-b7aa-0cb939217b29-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-qn29c\" (UID: \"9e820933-bb87-4c58-b7aa-0cb939217b29\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-qn29c"
Jul 29 18:40:18.123574 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:18.123574    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbsk5\" (UniqueName: \"kubernetes.io/projected/9e820933-bb87-4c58-b7aa-0cb939217b29-kube-api-access-zbsk5\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-qn29c\" (UID: \"9e820933-bb87-4c58-b7aa-0cb939217b29\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-qn29c"
Jul 29 18:40:18.123834 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:18.123596    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9e820933-bb87-4c58-b7aa-0cb939217b29-webhook-cert\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-qn29c\" (UID: \"9e820933-bb87-4c58-b7aa-0cb939217b29\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-qn29c"
Jul 29 18:40:18.123834 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:18.123612    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9e820933-bb87-4c58-b7aa-0cb939217b29-manifests\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-qn29c\" (UID: \"9e820933-bb87-4c58-b7aa-0cb939217b29\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-qn29c"
Jul 29 18:40:18.123834 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:18.123799    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e820933-bb87-4c58-b7aa-0cb939217b29-tmp\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-qn29c\" (UID: \"9e820933-bb87-4c58-b7aa-0cb939217b29\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-qn29c"
Jul 29 18:40:18.124137 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:18.124113    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9e820933-bb87-4c58-b7aa-0cb939217b29-manifests\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-qn29c\" (UID: \"9e820933-bb87-4c58-b7aa-0cb939217b29\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-qn29c"
Jul 29 18:40:18.126150 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:18.126126    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e820933-bb87-4c58-b7aa-0cb939217b29-tmp\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-qn29c\" (UID: \"9e820933-bb87-4c58-b7aa-0cb939217b29\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-qn29c"
Jul 29 18:40:18.126319 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:18.126195    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9e820933-bb87-4c58-b7aa-0cb939217b29-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-qn29c\" (UID: \"9e820933-bb87-4c58-b7aa-0cb939217b29\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-qn29c"
Jul 29 18:40:18.126319 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:18.126258    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9e820933-bb87-4c58-b7aa-0cb939217b29-webhook-cert\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-qn29c\" (UID: \"9e820933-bb87-4c58-b7aa-0cb939217b29\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-qn29c"
Jul 29 18:40:18.159096 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:18.159067    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbsk5\" (UniqueName: \"kubernetes.io/projected/9e820933-bb87-4c58-b7aa-0cb939217b29-kube-api-access-zbsk5\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-qn29c\" (UID: \"9e820933-bb87-4c58-b7aa-0cb939217b29\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-qn29c"
Jul 29 18:40:18.324774 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:18.324683    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-qn29c"
Jul 29 18:40:18.494075 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:40:18.494044    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e820933_bb87_4c58_b7aa_0cb939217b29.slice/crio-597c5fca82e68ec5c73485756629f239eb1503948b01a4891b78393612993faa WatchSource:0}: Error finding container 597c5fca82e68ec5c73485756629f239eb1503948b01a4891b78393612993faa: Status 404 returned error can't find the container with id 597c5fca82e68ec5c73485756629f239eb1503948b01a4891b78393612993faa
Jul 29 18:40:18.495812 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:18.495793    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 18:40:18.503636 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:18.503604    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-qn29c"]
Jul 29 18:40:19.101171 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:19.101134    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-qn29c" event={"ID":"9e820933-bb87-4c58-b7aa-0cb939217b29","Type":"ContainerStarted","Data":"597c5fca82e68ec5c73485756629f239eb1503948b01a4891b78393612993faa"}
Jul 29 18:40:22.118397 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:22.118343    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-qn29c" event={"ID":"9e820933-bb87-4c58-b7aa-0cb939217b29","Type":"ContainerStarted","Data":"4d4c86e8ed93c6bc7b70530af78012fe087858a884d4c4857d754084ca2ca651"}
Jul 29 18:40:23.123705 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:23.123670    2559 generic.go:356] "Generic (PLEG): container finished" podID="9e820933-bb87-4c58-b7aa-0cb939217b29" containerID="4d4c86e8ed93c6bc7b70530af78012fe087858a884d4c4857d754084ca2ca651" exitCode=0
Jul 29 18:40:23.124095 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:23.123755    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-qn29c" event={"ID":"9e820933-bb87-4c58-b7aa-0cb939217b29","Type":"ContainerDied","Data":"4d4c86e8ed93c6bc7b70530af78012fe087858a884d4c4857d754084ca2ca651"}
Jul 29 18:40:24.130272 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:24.130232    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-qn29c" event={"ID":"9e820933-bb87-4c58-b7aa-0cb939217b29","Type":"ContainerStarted","Data":"5affbedd7018240b568729481cd03e9d3dfcfc0190abdc1a70a7a0302c27517d"}
Jul 29 18:40:24.130784 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:24.130341    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-qn29c"
Jul 29 18:40:24.154223 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:24.154154    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-qn29c" podStartSLOduration=2.5514293009999998 podStartE2EDuration="7.15413763s" totalImagesPullingTime="3.525084191s" totalInitContainerRuntime="1.077624138s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:40:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:40:24.153035497 +0000 UTC m=+388.422804582" watchObservedRunningTime="2026-07-29 18:40:24.15413763 +0000 UTC m=+388.423906708"
Jul 29 18:40:35.137569 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:35.137540    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-qn29c"
Jul 29 18:40:38.117853 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:38.117818    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b7494b68d-8mvm4"]
Jul 29 18:40:38.126111 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:38.126085    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5b7494b68d-8mvm4"
Jul 29 18:40:38.128952 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:38.128925    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 29 18:40:38.129130 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:38.128925    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 29 18:40:38.129901 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:38.129880    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-5c2bs\""
Jul 29 18:40:38.130399 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:38.130366    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 29 18:40:38.130592 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:38.130416    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 29 18:40:38.132634 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:38.132608    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b7494b68d-8mvm4"]
Jul 29 18:40:38.136517 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:38.136490    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 29 18:40:38.201551 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:38.201516    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ebb787a-5d11-4a08-89dc-335737c2c9d5-cert\") pod \"lws-controller-manager-5b7494b68d-8mvm4\" (UID: \"2ebb787a-5d11-4a08-89dc-335737c2c9d5\") " pod="openshift-lws-operator/lws-controller-manager-5b7494b68d-8mvm4"
Jul 29 18:40:38.201767 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:38.201594    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgr95\" (UniqueName: \"kubernetes.io/projected/2ebb787a-5d11-4a08-89dc-335737c2c9d5-kube-api-access-lgr95\") pod \"lws-controller-manager-5b7494b68d-8mvm4\" (UID: \"2ebb787a-5d11-4a08-89dc-335737c2c9d5\") " pod="openshift-lws-operator/lws-controller-manager-5b7494b68d-8mvm4"
Jul 29 18:40:38.201767 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:38.201639    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2ebb787a-5d11-4a08-89dc-335737c2c9d5-manager-config\") pod \"lws-controller-manager-5b7494b68d-8mvm4\" (UID: \"2ebb787a-5d11-4a08-89dc-335737c2c9d5\") " pod="openshift-lws-operator/lws-controller-manager-5b7494b68d-8mvm4"
Jul 29 18:40:38.201767 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:38.201659    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2ebb787a-5d11-4a08-89dc-335737c2c9d5-metrics-cert\") pod \"lws-controller-manager-5b7494b68d-8mvm4\" (UID: \"2ebb787a-5d11-4a08-89dc-335737c2c9d5\") " pod="openshift-lws-operator/lws-controller-manager-5b7494b68d-8mvm4"
Jul 29 18:40:38.302824 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:38.302784    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ebb787a-5d11-4a08-89dc-335737c2c9d5-cert\") pod \"lws-controller-manager-5b7494b68d-8mvm4\" (UID: \"2ebb787a-5d11-4a08-89dc-335737c2c9d5\") " pod="openshift-lws-operator/lws-controller-manager-5b7494b68d-8mvm4"
Jul 29 18:40:38.303012 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:38.302864    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgr95\" (UniqueName: \"kubernetes.io/projected/2ebb787a-5d11-4a08-89dc-335737c2c9d5-kube-api-access-lgr95\") pod \"lws-controller-manager-5b7494b68d-8mvm4\" (UID: \"2ebb787a-5d11-4a08-89dc-335737c2c9d5\") " pod="openshift-lws-operator/lws-controller-manager-5b7494b68d-8mvm4"
Jul 29 18:40:38.303012 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:38.302909    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2ebb787a-5d11-4a08-89dc-335737c2c9d5-manager-config\") pod \"lws-controller-manager-5b7494b68d-8mvm4\" (UID: \"2ebb787a-5d11-4a08-89dc-335737c2c9d5\") " pod="openshift-lws-operator/lws-controller-manager-5b7494b68d-8mvm4"
Jul 29 18:40:38.303012 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:38.302928    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2ebb787a-5d11-4a08-89dc-335737c2c9d5-metrics-cert\") pod \"lws-controller-manager-5b7494b68d-8mvm4\" (UID: \"2ebb787a-5d11-4a08-89dc-335737c2c9d5\") " pod="openshift-lws-operator/lws-controller-manager-5b7494b68d-8mvm4"
Jul 29 18:40:38.303722 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:38.303634    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2ebb787a-5d11-4a08-89dc-335737c2c9d5-manager-config\") pod \"lws-controller-manager-5b7494b68d-8mvm4\" (UID: \"2ebb787a-5d11-4a08-89dc-335737c2c9d5\") " pod="openshift-lws-operator/lws-controller-manager-5b7494b68d-8mvm4"
Jul 29 18:40:38.305695 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:38.305673    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ebb787a-5d11-4a08-89dc-335737c2c9d5-cert\") pod \"lws-controller-manager-5b7494b68d-8mvm4\" (UID: \"2ebb787a-5d11-4a08-89dc-335737c2c9d5\") " pod="openshift-lws-operator/lws-controller-manager-5b7494b68d-8mvm4"
Jul 29 18:40:38.305786 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:38.305673    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2ebb787a-5d11-4a08-89dc-335737c2c9d5-metrics-cert\") pod \"lws-controller-manager-5b7494b68d-8mvm4\" (UID: \"2ebb787a-5d11-4a08-89dc-335737c2c9d5\") " pod="openshift-lws-operator/lws-controller-manager-5b7494b68d-8mvm4"
Jul 29 18:40:38.312724 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:38.312699    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgr95\" (UniqueName: \"kubernetes.io/projected/2ebb787a-5d11-4a08-89dc-335737c2c9d5-kube-api-access-lgr95\") pod \"lws-controller-manager-5b7494b68d-8mvm4\" (UID: \"2ebb787a-5d11-4a08-89dc-335737c2c9d5\") " pod="openshift-lws-operator/lws-controller-manager-5b7494b68d-8mvm4"
Jul 29 18:40:38.438646 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:38.438550    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5b7494b68d-8mvm4"
Jul 29 18:40:38.592358 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:38.592322    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b7494b68d-8mvm4"]
Jul 29 18:40:38.595443 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:40:38.595414    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ebb787a_5d11_4a08_89dc_335737c2c9d5.slice/crio-ad420bc969990d4a842236986569d4c312a5c49adfbc2e64af3167cb22aa4343 WatchSource:0}: Error finding container ad420bc969990d4a842236986569d4c312a5c49adfbc2e64af3167cb22aa4343: Status 404 returned error can't find the container with id ad420bc969990d4a842236986569d4c312a5c49adfbc2e64af3167cb22aa4343
Jul 29 18:40:39.196768 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:39.196681    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5b7494b68d-8mvm4" event={"ID":"2ebb787a-5d11-4a08-89dc-335737c2c9d5","Type":"ContainerStarted","Data":"ad420bc969990d4a842236986569d4c312a5c49adfbc2e64af3167cb22aa4343"}
Jul 29 18:40:41.208171 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:41.208068    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5b7494b68d-8mvm4" event={"ID":"2ebb787a-5d11-4a08-89dc-335737c2c9d5","Type":"ContainerStarted","Data":"47a6672002bf3673a9c03254da7ee52e36f5e346adc38a88e69b3c767b1de153"}
Jul 29 18:40:41.208540 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:41.208167    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5b7494b68d-8mvm4"
Jul 29 18:40:41.225628 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:41.225577    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5b7494b68d-8mvm4" podStartSLOduration=0.9043069 podStartE2EDuration="3.22555487s" totalImagesPullingTime="2.32124797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:40:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:40:41.224328977 +0000 UTC m=+405.494098065" watchObservedRunningTime="2026-07-29 18:40:41.22555487 +0000 UTC m=+405.495323953"
Jul 29 18:40:48.340270 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:48.340227    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-b4kv2"]
Jul 29 18:40:48.343628 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:48.343608    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-b4kv2"
Jul 29 18:40:48.346901 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:48.346877    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 29 18:40:48.347037 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:48.346915    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-vv9gs\""
Jul 29 18:40:48.355921 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:48.355896    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-7gxpk"]
Jul 29 18:40:48.359244 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:48.359219    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7gxpk"
Jul 29 18:40:48.359413 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:48.359266    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-b4kv2"]
Jul 29 18:40:48.361617 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:48.361595    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 29 18:40:48.361721 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:48.361638    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-xn6dh\""
Jul 29 18:40:48.373780 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:48.373750    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-7gxpk"]
Jul 29 18:40:48.493411 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:48.493338    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b524f44-0875-4793-80e3-5886ee9bf148-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-7gxpk\" (UID: \"2b524f44-0875-4793-80e3-5886ee9bf148\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7gxpk"
Jul 29 18:40:48.493565 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:48.493419    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnks2\" (UniqueName: \"kubernetes.io/projected/2b524f44-0875-4793-80e3-5886ee9bf148-kube-api-access-cnks2\") pod \"llmisvc-controller-manager-777c6bd7d5-7gxpk\" (UID: \"2b524f44-0875-4793-80e3-5886ee9bf148\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7gxpk"
Jul 29 18:40:48.493565 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:48.493462    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lwjn\" (UniqueName: \"kubernetes.io/projected/7fbb1c34-1fcb-4b7b-bd0f-b2c2e0502bba-kube-api-access-6lwjn\") pod \"kserve-controller-manager-d6f4b65b8-b4kv2\" (UID: \"7fbb1c34-1fcb-4b7b-bd0f-b2c2e0502bba\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-b4kv2"
Jul 29 18:40:48.493565 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:48.493540    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7fbb1c34-1fcb-4b7b-bd0f-b2c2e0502bba-cert\") pod \"kserve-controller-manager-d6f4b65b8-b4kv2\" (UID: \"7fbb1c34-1fcb-4b7b-bd0f-b2c2e0502bba\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-b4kv2"
Jul 29 18:40:48.594611 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:48.594526    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b524f44-0875-4793-80e3-5886ee9bf148-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-7gxpk\" (UID: \"2b524f44-0875-4793-80e3-5886ee9bf148\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7gxpk"
Jul 29 18:40:48.594611 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:48.594568    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnks2\" (UniqueName: \"kubernetes.io/projected/2b524f44-0875-4793-80e3-5886ee9bf148-kube-api-access-cnks2\") pod \"llmisvc-controller-manager-777c6bd7d5-7gxpk\" (UID: \"2b524f44-0875-4793-80e3-5886ee9bf148\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7gxpk"
Jul 29 18:40:48.594611 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:48.594604    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lwjn\" (UniqueName: \"kubernetes.io/projected/7fbb1c34-1fcb-4b7b-bd0f-b2c2e0502bba-kube-api-access-6lwjn\") pod \"kserve-controller-manager-d6f4b65b8-b4kv2\" (UID: \"7fbb1c34-1fcb-4b7b-bd0f-b2c2e0502bba\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-b4kv2"
Jul 29 18:40:48.594903 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:48.594756    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7fbb1c34-1fcb-4b7b-bd0f-b2c2e0502bba-cert\") pod \"kserve-controller-manager-d6f4b65b8-b4kv2\" (UID: \"7fbb1c34-1fcb-4b7b-bd0f-b2c2e0502bba\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-b4kv2"
Jul 29 18:40:48.597325 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:48.597291    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b524f44-0875-4793-80e3-5886ee9bf148-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-7gxpk\" (UID: \"2b524f44-0875-4793-80e3-5886ee9bf148\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7gxpk"
Jul 29 18:40:48.597477 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:48.597343    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7fbb1c34-1fcb-4b7b-bd0f-b2c2e0502bba-cert\") pod \"kserve-controller-manager-d6f4b65b8-b4kv2\" (UID: \"7fbb1c34-1fcb-4b7b-bd0f-b2c2e0502bba\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-b4kv2"
Jul 29 18:40:48.606718 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:48.606689    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnks2\" (UniqueName: \"kubernetes.io/projected/2b524f44-0875-4793-80e3-5886ee9bf148-kube-api-access-cnks2\") pod \"llmisvc-controller-manager-777c6bd7d5-7gxpk\" (UID: \"2b524f44-0875-4793-80e3-5886ee9bf148\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7gxpk"
Jul 29 18:40:48.610647 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:48.610619    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lwjn\" (UniqueName: \"kubernetes.io/projected/7fbb1c34-1fcb-4b7b-bd0f-b2c2e0502bba-kube-api-access-6lwjn\") pod \"kserve-controller-manager-d6f4b65b8-b4kv2\" (UID: \"7fbb1c34-1fcb-4b7b-bd0f-b2c2e0502bba\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-b4kv2"
Jul 29 18:40:48.655141 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:48.655107    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-b4kv2"
Jul 29 18:40:48.669449 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:48.669414    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7gxpk"
Jul 29 18:40:48.806984 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:48.806952    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-b4kv2"]
Jul 29 18:40:48.809513 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:40:48.809473    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fbb1c34_1fcb_4b7b_bd0f_b2c2e0502bba.slice/crio-172ddb4b3832432c84eeca6e66751ec7604089ec64b448d74e55abfbcc91cf7e WatchSource:0}: Error finding container 172ddb4b3832432c84eeca6e66751ec7604089ec64b448d74e55abfbcc91cf7e: Status 404 returned error can't find the container with id 172ddb4b3832432c84eeca6e66751ec7604089ec64b448d74e55abfbcc91cf7e
Jul 29 18:40:48.833281 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:48.833247    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-7gxpk"]
Jul 29 18:40:48.836773 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:40:48.836741    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2b524f44_0875_4793_80e3_5886ee9bf148.slice/crio-dfaa9c57621731f33ad806a45751d46df495a6a757b8c040bb76470ceb452b37 WatchSource:0}: Error finding container dfaa9c57621731f33ad806a45751d46df495a6a757b8c040bb76470ceb452b37: Status 404 returned error can't find the container with id dfaa9c57621731f33ad806a45751d46df495a6a757b8c040bb76470ceb452b37
Jul 29 18:40:49.251450 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:49.251409    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-b4kv2" event={"ID":"7fbb1c34-1fcb-4b7b-bd0f-b2c2e0502bba","Type":"ContainerStarted","Data":"172ddb4b3832432c84eeca6e66751ec7604089ec64b448d74e55abfbcc91cf7e"}
Jul 29 18:40:49.252641 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:49.252616    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7gxpk" event={"ID":"2b524f44-0875-4793-80e3-5886ee9bf148","Type":"ContainerStarted","Data":"dfaa9c57621731f33ad806a45751d46df495a6a757b8c040bb76470ceb452b37"}
Jul 29 18:40:52.215286 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:52.215199    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5b7494b68d-8mvm4"
Jul 29 18:40:52.268741 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:52.268694    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-b4kv2" event={"ID":"7fbb1c34-1fcb-4b7b-bd0f-b2c2e0502bba","Type":"ContainerStarted","Data":"26f3e4e61fb2fdb1781b8e23332c4a99e88ee40bca5ac7467508097eeddcd24e"}
Jul 29 18:40:52.268955 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:52.268774    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-b4kv2"
Jul 29 18:40:52.270261 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:52.270225    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7gxpk" event={"ID":"2b524f44-0875-4793-80e3-5886ee9bf148","Type":"ContainerStarted","Data":"fb395566b8aeabcaf3b9eef905061b9170f1af072bd10473b68507cc877856e0"}
Jul 29 18:40:52.270416 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:52.270287    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7gxpk"
Jul 29 18:40:52.295437 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:52.295362    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-b4kv2" podStartSLOduration=1.139968898 podStartE2EDuration="4.295347223s" totalImagesPullingTime="3.155378325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:40:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:40:52.294015319 +0000 UTC m=+416.563784407" watchObservedRunningTime="2026-07-29 18:40:52.295347223 +0000 UTC m=+416.565116312"
Jul 29 18:40:52.312103 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:40:52.312049    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7gxpk" podStartSLOduration=1.179423525 podStartE2EDuration="4.312032559s" totalImagesPullingTime="3.132609034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:40:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:40:52.310883488 +0000 UTC m=+416.580652575" watchObservedRunningTime="2026-07-29 18:40:52.312032559 +0000 UTC m=+416.581801683"
Jul 29 18:41:23.277312 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:41:23.277232    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7gxpk"
Jul 29 18:41:23.280227 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:41:23.280207    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-b4kv2"
Jul 29 18:42:17.163410 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:17.163357    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x94kh"]
Jul 29 18:42:17.166430 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:17.166410    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x94kh"
Jul 29 18:42:17.169744 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:17.169717    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-4757h\""
Jul 29 18:42:17.169893 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:17.169727    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 29 18:42:17.170594 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:17.170578    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 29 18:42:17.187549 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:17.187520    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x94kh"]
Jul 29 18:42:17.281408 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:17.281349    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmxx9\" (UniqueName: \"kubernetes.io/projected/e2ab87d5-b21f-45c2-8cd1-b501665e368f-kube-api-access-nmxx9\") pod \"limitador-operator-controller-manager-859d7b78bd-x94kh\" (UID: \"e2ab87d5-b21f-45c2-8cd1-b501665e368f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x94kh"
Jul 29 18:42:17.382454 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:17.382418    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmxx9\" (UniqueName: \"kubernetes.io/projected/e2ab87d5-b21f-45c2-8cd1-b501665e368f-kube-api-access-nmxx9\") pod \"limitador-operator-controller-manager-859d7b78bd-x94kh\" (UID: \"e2ab87d5-b21f-45c2-8cd1-b501665e368f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x94kh"
Jul 29 18:42:17.393003 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:17.392969    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmxx9\" (UniqueName: \"kubernetes.io/projected/e2ab87d5-b21f-45c2-8cd1-b501665e368f-kube-api-access-nmxx9\") pod \"limitador-operator-controller-manager-859d7b78bd-x94kh\" (UID: \"e2ab87d5-b21f-45c2-8cd1-b501665e368f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x94kh"
Jul 29 18:42:17.477069 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:17.476973    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x94kh"
Jul 29 18:42:17.635831 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:17.635695    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x94kh"]
Jul 29 18:42:17.638968 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:42:17.638937    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2ab87d5_b21f_45c2_8cd1_b501665e368f.slice/crio-2b5bb7284cd01c1f49d6994e3c29fd86bbe928f583340f1323d326186f0bbc66 WatchSource:0}: Error finding container 2b5bb7284cd01c1f49d6994e3c29fd86bbe928f583340f1323d326186f0bbc66: Status 404 returned error can't find the container with id 2b5bb7284cd01c1f49d6994e3c29fd86bbe928f583340f1323d326186f0bbc66
Jul 29 18:42:17.654213 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:17.654176    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x94kh" event={"ID":"e2ab87d5-b21f-45c2-8cd1-b501665e368f","Type":"ContainerStarted","Data":"2b5bb7284cd01c1f49d6994e3c29fd86bbe928f583340f1323d326186f0bbc66"}
Jul 29 18:42:20.671662 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:20.671622    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x94kh" event={"ID":"e2ab87d5-b21f-45c2-8cd1-b501665e368f","Type":"ContainerStarted","Data":"3b7dfdba662ec82f3ea26600d9956ae46847cba6269f8b93ad925170a29574c8"}
Jul 29 18:42:20.672063 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:20.671712    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x94kh"
Jul 29 18:42:20.691088 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:20.691034    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x94kh" podStartSLOduration=1.500161514 podStartE2EDuration="3.69101595s" totalImagesPullingTime="2.190854436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:42:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:42:20.689481734 +0000 UTC m=+504.959250821" watchObservedRunningTime="2026-07-29 18:42:20.69101595 +0000 UTC m=+504.960785036"
Jul 29 18:42:31.678943 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:31.678907    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x94kh"
Jul 29 18:42:32.759100 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:32.759059    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x94kh"]
Jul 29 18:42:32.759555 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:32.759318    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x94kh" podUID="e2ab87d5-b21f-45c2-8cd1-b501665e368f" containerName="manager" containerID="cri-o://3b7dfdba662ec82f3ea26600d9956ae46847cba6269f8b93ad925170a29574c8" gracePeriod=2
Jul 29 18:42:32.767495 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:32.767452    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x94kh"]
Jul 29 18:42:32.800784 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:32.800749    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lhmrl"]
Jul 29 18:42:32.801113 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:32.801099    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2ab87d5-b21f-45c2-8cd1-b501665e368f" containerName="manager"
Jul 29 18:42:32.801163 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:32.801115    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2ab87d5-b21f-45c2-8cd1-b501665e368f" containerName="manager"
Jul 29 18:42:32.801204 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:32.801196    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2ab87d5-b21f-45c2-8cd1-b501665e368f" containerName="manager"
Jul 29 18:42:32.804281 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:32.804257    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lhmrl"
Jul 29 18:42:32.830139 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:42:32.830090    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-x94kh\" is forbidden: User \"system:node:ip-10-0-138-152.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-152.ec2.internal' and this object" podUID="e2ab87d5-b21f-45c2-8cd1-b501665e368f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x94kh"
Jul 29 18:42:32.831239 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:32.831212    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lhmrl"]
Jul 29 18:42:32.922690 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:32.922651    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m2cf\" (UniqueName: \"kubernetes.io/projected/bf735020-ded2-4908-ae14-05c16e3c760b-kube-api-access-5m2cf\") pod \"limitador-operator-controller-manager-859d7b78bd-lhmrl\" (UID: \"bf735020-ded2-4908-ae14-05c16e3c760b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lhmrl"
Jul 29 18:42:33.008692 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:33.008660    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x94kh"
Jul 29 18:42:33.011373 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:42:33.011323    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-x94kh\" is forbidden: User \"system:node:ip-10-0-138-152.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-152.ec2.internal' and this object" podUID="e2ab87d5-b21f-45c2-8cd1-b501665e368f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x94kh"
Jul 29 18:42:33.023108 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:33.023081    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m2cf\" (UniqueName: \"kubernetes.io/projected/bf735020-ded2-4908-ae14-05c16e3c760b-kube-api-access-5m2cf\") pod \"limitador-operator-controller-manager-859d7b78bd-lhmrl\" (UID: \"bf735020-ded2-4908-ae14-05c16e3c760b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lhmrl"
Jul 29 18:42:33.036256 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:33.036221    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m2cf\" (UniqueName: \"kubernetes.io/projected/bf735020-ded2-4908-ae14-05c16e3c760b-kube-api-access-5m2cf\") pod \"limitador-operator-controller-manager-859d7b78bd-lhmrl\" (UID: \"bf735020-ded2-4908-ae14-05c16e3c760b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lhmrl"
Jul 29 18:42:33.124267 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:33.124225    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmxx9\" (UniqueName: \"kubernetes.io/projected/e2ab87d5-b21f-45c2-8cd1-b501665e368f-kube-api-access-nmxx9\") pod \"e2ab87d5-b21f-45c2-8cd1-b501665e368f\" (UID: \"e2ab87d5-b21f-45c2-8cd1-b501665e368f\") "
Jul 29 18:42:33.126665 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:33.126632    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2ab87d5-b21f-45c2-8cd1-b501665e368f-kube-api-access-nmxx9" (OuterVolumeSpecName: "kube-api-access-nmxx9") pod "e2ab87d5-b21f-45c2-8cd1-b501665e368f" (UID: "e2ab87d5-b21f-45c2-8cd1-b501665e368f"). InnerVolumeSpecName "kube-api-access-nmxx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 18:42:33.143806 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:33.143773    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lhmrl"
Jul 29 18:42:33.225100 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:33.225065    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmxx9\" (UniqueName: \"kubernetes.io/projected/e2ab87d5-b21f-45c2-8cd1-b501665e368f-kube-api-access-nmxx9\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 18:42:33.290559 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:33.290534    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lhmrl"]
Jul 29 18:42:33.293200 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:42:33.293168    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf735020_ded2_4908_ae14_05c16e3c760b.slice/crio-d815417d7c739dba3114aa0f91d6e7a8f5ad7f50eece0bfc190235c67a27be28 WatchSource:0}: Error finding container d815417d7c739dba3114aa0f91d6e7a8f5ad7f50eece0bfc190235c67a27be28: Status 404 returned error can't find the container with id d815417d7c739dba3114aa0f91d6e7a8f5ad7f50eece0bfc190235c67a27be28
Jul 29 18:42:33.733262 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:33.733226    2559 generic.go:356] "Generic (PLEG): container finished" podID="e2ab87d5-b21f-45c2-8cd1-b501665e368f" containerID="3b7dfdba662ec82f3ea26600d9956ae46847cba6269f8b93ad925170a29574c8" exitCode=0
Jul 29 18:42:33.733468 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:33.733276    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x94kh"
Jul 29 18:42:33.733468 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:33.733320    2559 scope.go:117] "RemoveContainer" containerID="3b7dfdba662ec82f3ea26600d9956ae46847cba6269f8b93ad925170a29574c8"
Jul 29 18:42:33.735097 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:33.735062    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lhmrl" event={"ID":"bf735020-ded2-4908-ae14-05c16e3c760b","Type":"ContainerStarted","Data":"717423ca9fbd02adf19d8d4961c73a07f6cf62cc8b88bd2356e6fdd98ea5da22"}
Jul 29 18:42:33.735201 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:33.735119    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lhmrl" event={"ID":"bf735020-ded2-4908-ae14-05c16e3c760b","Type":"ContainerStarted","Data":"d815417d7c739dba3114aa0f91d6e7a8f5ad7f50eece0bfc190235c67a27be28"}
Jul 29 18:42:33.735201 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:33.735169    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lhmrl"
Jul 29 18:42:33.735950 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:42:33.735929    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-x94kh\" is forbidden: User \"system:node:ip-10-0-138-152.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-152.ec2.internal' and this object" podUID="e2ab87d5-b21f-45c2-8cd1-b501665e368f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x94kh"
Jul 29 18:42:33.742712 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:33.742692    2559 scope.go:117] "RemoveContainer" containerID="3b7dfdba662ec82f3ea26600d9956ae46847cba6269f8b93ad925170a29574c8"
Jul 29 18:42:33.742960 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:42:33.742942    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b7dfdba662ec82f3ea26600d9956ae46847cba6269f8b93ad925170a29574c8\": container with ID starting with 3b7dfdba662ec82f3ea26600d9956ae46847cba6269f8b93ad925170a29574c8 not found: ID does not exist" containerID="3b7dfdba662ec82f3ea26600d9956ae46847cba6269f8b93ad925170a29574c8"
Jul 29 18:42:33.743005 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:33.742970    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b7dfdba662ec82f3ea26600d9956ae46847cba6269f8b93ad925170a29574c8"} err="failed to get container status \"3b7dfdba662ec82f3ea26600d9956ae46847cba6269f8b93ad925170a29574c8\": rpc error: code = NotFound desc = could not find container \"3b7dfdba662ec82f3ea26600d9956ae46847cba6269f8b93ad925170a29574c8\": container with ID starting with 3b7dfdba662ec82f3ea26600d9956ae46847cba6269f8b93ad925170a29574c8 not found: ID does not exist"
Jul 29 18:42:33.757604 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:33.757552    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lhmrl" podStartSLOduration=1.757537448 podStartE2EDuration="1.757537448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:42:33.755374675 +0000 UTC m=+518.025143762" watchObservedRunningTime="2026-07-29 18:42:33.757537448 +0000 UTC m=+518.027306533"
Jul 29 18:42:33.758198 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:42:33.758175    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-x94kh\" is forbidden: User \"system:node:ip-10-0-138-152.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-152.ec2.internal' and this object" podUID="e2ab87d5-b21f-45c2-8cd1-b501665e368f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x94kh"
Jul 29 18:42:33.760477 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:42:33.760458    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-x94kh\" is forbidden: User \"system:node:ip-10-0-138-152.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-152.ec2.internal' and this object" podUID="e2ab87d5-b21f-45c2-8cd1-b501665e368f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x94kh"
Jul 29 18:42:34.150258 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:34.150225    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2ab87d5-b21f-45c2-8cd1-b501665e368f" path="/var/lib/kubelet/pods/e2ab87d5-b21f-45c2-8cd1-b501665e368f/volumes"
Jul 29 18:42:44.743490 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:42:44.743454    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lhmrl"
Jul 29 18:43:03.351702 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:43:03.351657    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:43:03.356588 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:43:03.356562    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-268m5"
Jul 29 18:43:03.359003 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:43:03.358981    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 29 18:43:03.359144 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:43:03.358981    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-bnlth\""
Jul 29 18:43:03.363927 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:43:03.363899    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:43:03.385595 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:43:03.385556    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/463080e3-ad6e-4f21-a9b4-c30b44f452c6-config-file\") pod \"limitador-limitador-54d6d988d5-268m5\" (UID: \"463080e3-ad6e-4f21-a9b4-c30b44f452c6\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-268m5"
Jul 29 18:43:03.385595 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:43:03.385595    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khxtj\" (UniqueName: \"kubernetes.io/projected/463080e3-ad6e-4f21-a9b4-c30b44f452c6-kube-api-access-khxtj\") pod \"limitador-limitador-54d6d988d5-268m5\" (UID: \"463080e3-ad6e-4f21-a9b4-c30b44f452c6\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-268m5"
Jul 29 18:43:03.391165 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:43:03.391135    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:43:03.486913 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:43:03.486873    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/463080e3-ad6e-4f21-a9b4-c30b44f452c6-config-file\") pod \"limitador-limitador-54d6d988d5-268m5\" (UID: \"463080e3-ad6e-4f21-a9b4-c30b44f452c6\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-268m5"
Jul 29 18:43:03.486913 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:43:03.486914    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khxtj\" (UniqueName: \"kubernetes.io/projected/463080e3-ad6e-4f21-a9b4-c30b44f452c6-kube-api-access-khxtj\") pod \"limitador-limitador-54d6d988d5-268m5\" (UID: \"463080e3-ad6e-4f21-a9b4-c30b44f452c6\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-268m5"
Jul 29 18:43:03.487539 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:43:03.487520    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/463080e3-ad6e-4f21-a9b4-c30b44f452c6-config-file\") pod \"limitador-limitador-54d6d988d5-268m5\" (UID: \"463080e3-ad6e-4f21-a9b4-c30b44f452c6\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-268m5"
Jul 29 18:43:03.495121 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:43:03.495096    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khxtj\" (UniqueName: \"kubernetes.io/projected/463080e3-ad6e-4f21-a9b4-c30b44f452c6-kube-api-access-khxtj\") pod \"limitador-limitador-54d6d988d5-268m5\" (UID: \"463080e3-ad6e-4f21-a9b4-c30b44f452c6\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-268m5"
Jul 29 18:43:03.668672 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:43:03.668585    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-268m5"
Jul 29 18:43:03.817250 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:43:03.817226    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:43:03.820098 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:43:03.820070    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod463080e3_ad6e_4f21_a9b4_c30b44f452c6.slice/crio-fe826519af04f9052e1b065f4af4bcbd7b4ef6753175a7486c25e1985162ba33 WatchSource:0}: Error finding container fe826519af04f9052e1b065f4af4bcbd7b4ef6753175a7486c25e1985162ba33: Status 404 returned error can't find the container with id fe826519af04f9052e1b065f4af4bcbd7b4ef6753175a7486c25e1985162ba33
Jul 29 18:43:03.871816 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:43:03.871774    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-268m5" event={"ID":"463080e3-ad6e-4f21-a9b4-c30b44f452c6","Type":"ContainerStarted","Data":"fe826519af04f9052e1b065f4af4bcbd7b4ef6753175a7486c25e1985162ba33"}
Jul 29 18:43:06.890497 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:43:06.890459    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-268m5" event={"ID":"463080e3-ad6e-4f21-a9b4-c30b44f452c6","Type":"ContainerStarted","Data":"ac0dcc112c740d09577fe59667d09ba57e45884a4015c937f48b4f1b9639657d"}
Jul 29 18:43:06.890902 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:43:06.890610    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-268m5"
Jul 29 18:43:06.907079 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:43:06.907020    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-268m5" podStartSLOduration=1.19319963 podStartE2EDuration="3.906998635s" totalImagesPullingTime="2.713799005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:43:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:43:06.90616896 +0000 UTC m=+551.175938048" watchObservedRunningTime="2026-07-29 18:43:06.906998635 +0000 UTC m=+551.176767722"
Jul 29 18:43:17.895515 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:43:17.895485    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-268m5"
Jul 29 18:43:56.069249 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:43:56.069218    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-v9nsr_7f8b4737-d4c2-4598-bd40-208015542831/console-operator/1.log"
Jul 29 18:43:56.077073 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:43:56.077044    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjdrx_60f444e1-7e31-443c-8ec7-a4d04c64bf4b/ovn-acl-logging/0.log"
Jul 29 18:43:56.077234 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:43:56.077164    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-v9nsr_7f8b4737-d4c2-4598-bd40-208015542831/console-operator/1.log"
Jul 29 18:43:56.084796 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:43:56.084770    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjdrx_60f444e1-7e31-443c-8ec7-a4d04c64bf4b/ovn-acl-logging/0.log"
Jul 29 18:44:15.974076 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:44:15.974034    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:44:34.533569 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:44:34.533531    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:44:53.039267 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:44:53.039232    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:45:07.937979 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:45:07.937941    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:45:17.533087 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:45:17.533048    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:45:38.034814 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:45:38.034773    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:45:51.842494 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:45:51.842406    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:46:56.041246 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:46:56.041207    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:47:06.634758 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:47:06.634718    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:47:17.066726 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:47:17.066638    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:48:56.099001 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:48:56.098975    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-v9nsr_7f8b4737-d4c2-4598-bd40-208015542831/console-operator/1.log"
Jul 29 18:48:56.104715 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:48:56.104690    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-v9nsr_7f8b4737-d4c2-4598-bd40-208015542831/console-operator/1.log"
Jul 29 18:48:56.106761 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:48:56.106738    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjdrx_60f444e1-7e31-443c-8ec7-a4d04c64bf4b/ovn-acl-logging/0.log"
Jul 29 18:48:56.112162 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:48:56.112138    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjdrx_60f444e1-7e31-443c-8ec7-a4d04c64bf4b/ovn-acl-logging/0.log"
Jul 29 18:48:59.647093 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:48:59.647046    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:49:08.644093 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:49:08.644054    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:49:22.933579 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:49:22.933537    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:49:33.332592 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:49:33.332556    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:49:42.137854 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:49:42.137814    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:49:50.239488 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:49:50.239442    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:49:58.543002 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:49:58.542963    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:50:08.248144 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:50:08.248108    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:50:18.441661 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:50:18.441578    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:50:29.343052 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:50:29.343007    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:50:39.847007 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:50:39.846967    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:50:49.940147 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:50:49.940106    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:51:12.544246 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:51:12.544204    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:51:26.251570 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:51:26.251534    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:51:49.739314 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:51:49.739230    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:52:00.072246 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:52:00.072197    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:52:09.944880 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:52:09.944842    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:52:27.151840 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:52:27.151801    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:52:38.156082 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:52:38.156051    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:52:48.346403 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:52:48.346348    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:52:57.743123 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:52:57.743081    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:53:06.951679 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:53:06.951639    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:53:56.127956 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:53:56.127927    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-v9nsr_7f8b4737-d4c2-4598-bd40-208015542831/console-operator/1.log"
Jul 29 18:53:56.130284 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:53:56.130264    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-v9nsr_7f8b4737-d4c2-4598-bd40-208015542831/console-operator/1.log"
Jul 29 18:53:56.135537 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:53:56.135514    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjdrx_60f444e1-7e31-443c-8ec7-a4d04c64bf4b/ovn-acl-logging/0.log"
Jul 29 18:53:56.137824 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:53:56.137796    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjdrx_60f444e1-7e31-443c-8ec7-a4d04c64bf4b/ovn-acl-logging/0.log"
Jul 29 18:55:04.955428 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:55:04.954095    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:55:16.047463 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:55:16.047422    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:55:32.450323 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:55:32.450279    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:55:45.972879 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:55:45.972836    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:55:58.051565 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:55:58.051529    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:56:06.336829 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:56:06.336789    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:56:17.850233 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:56:17.850144    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:56:28.739748 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:56:28.739700    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:56:37.540882 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:56:37.540846    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:56:48.540212 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:56:48.540172    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:57:00.040003 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:00.039961    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:57:07.650974 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:07.650933    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:57:26.342977 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:26.342937    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:57:37.033328 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:37.033283    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:57:41.183421 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:41.183363    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9"]
Jul 29 18:57:41.186851 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:41.186831    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9"
Jul 29 18:57:41.189557 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:41.189529    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 29 18:57:41.189708 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:41.189564    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 29 18:57:41.190618 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:41.190601    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-90b41ea4-kserve-self-signed-certs\""
Jul 29 18:57:41.190618 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:41.190615    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-479pt\""
Jul 29 18:57:41.210126 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:41.210096    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9"]
Jul 29 18:57:41.306024 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:41.305977    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e21b25a-73c8-4964-8a0d-03903a1de771-tmp-dir\") pod \"e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9\" (UID: \"4e21b25a-73c8-4964-8a0d-03903a1de771\") " pod="llm/e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9"
Jul 29 18:57:41.306230 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:41.306039    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4p9v\" (UniqueName: \"kubernetes.io/projected/4e21b25a-73c8-4964-8a0d-03903a1de771-kube-api-access-v4p9v\") pod \"e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9\" (UID: \"4e21b25a-73c8-4964-8a0d-03903a1de771\") " pod="llm/e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9"
Jul 29 18:57:41.306230 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:41.306080    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e21b25a-73c8-4964-8a0d-03903a1de771-model-cache\") pod \"e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9\" (UID: \"4e21b25a-73c8-4964-8a0d-03903a1de771\") " pod="llm/e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9"
Jul 29 18:57:41.306230 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:41.306104    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e21b25a-73c8-4964-8a0d-03903a1de771-home\") pod \"e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9\" (UID: \"4e21b25a-73c8-4964-8a0d-03903a1de771\") " pod="llm/e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9"
Jul 29 18:57:41.306230 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:41.306182    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e21b25a-73c8-4964-8a0d-03903a1de771-dshm\") pod \"e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9\" (UID: \"4e21b25a-73c8-4964-8a0d-03903a1de771\") " pod="llm/e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9"
Jul 29 18:57:41.306422 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:41.306242    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e21b25a-73c8-4964-8a0d-03903a1de771-tls-certs\") pod \"e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9\" (UID: \"4e21b25a-73c8-4964-8a0d-03903a1de771\") " pod="llm/e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9"
Jul 29 18:57:41.406995 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:41.406956    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e21b25a-73c8-4964-8a0d-03903a1de771-dshm\") pod \"e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9\" (UID: \"4e21b25a-73c8-4964-8a0d-03903a1de771\") " pod="llm/e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9"
Jul 29 18:57:41.407190 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:41.407015    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e21b25a-73c8-4964-8a0d-03903a1de771-tls-certs\") pod \"e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9\" (UID: \"4e21b25a-73c8-4964-8a0d-03903a1de771\") " pod="llm/e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9"
Jul 29 18:57:41.407190 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:41.407052    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e21b25a-73c8-4964-8a0d-03903a1de771-tmp-dir\") pod \"e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9\" (UID: \"4e21b25a-73c8-4964-8a0d-03903a1de771\") " pod="llm/e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9"
Jul 29 18:57:41.407190 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:41.407077    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4p9v\" (UniqueName: \"kubernetes.io/projected/4e21b25a-73c8-4964-8a0d-03903a1de771-kube-api-access-v4p9v\") pod \"e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9\" (UID: \"4e21b25a-73c8-4964-8a0d-03903a1de771\") " pod="llm/e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9"
Jul 29 18:57:41.407190 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:41.407095    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e21b25a-73c8-4964-8a0d-03903a1de771-model-cache\") pod \"e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9\" (UID: \"4e21b25a-73c8-4964-8a0d-03903a1de771\") " pod="llm/e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9"
Jul 29 18:57:41.407190 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:41.407113    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e21b25a-73c8-4964-8a0d-03903a1de771-home\") pod \"e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9\" (UID: \"4e21b25a-73c8-4964-8a0d-03903a1de771\") " pod="llm/e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9"
Jul 29 18:57:41.407542 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:41.407518    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e21b25a-73c8-4964-8a0d-03903a1de771-tmp-dir\") pod \"e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9\" (UID: \"4e21b25a-73c8-4964-8a0d-03903a1de771\") " pod="llm/e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9"
Jul 29 18:57:41.407594 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:41.407548    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e21b25a-73c8-4964-8a0d-03903a1de771-model-cache\") pod \"e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9\" (UID: \"4e21b25a-73c8-4964-8a0d-03903a1de771\") " pod="llm/e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9"
Jul 29 18:57:41.407594 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:41.407568    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e21b25a-73c8-4964-8a0d-03903a1de771-home\") pod \"e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9\" (UID: \"4e21b25a-73c8-4964-8a0d-03903a1de771\") " pod="llm/e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9"
Jul 29 18:57:41.409369 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:41.409348    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e21b25a-73c8-4964-8a0d-03903a1de771-dshm\") pod \"e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9\" (UID: \"4e21b25a-73c8-4964-8a0d-03903a1de771\") " pod="llm/e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9"
Jul 29 18:57:41.409806 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:41.409787    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e21b25a-73c8-4964-8a0d-03903a1de771-tls-certs\") pod \"e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9\" (UID: \"4e21b25a-73c8-4964-8a0d-03903a1de771\") " pod="llm/e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9"
Jul 29 18:57:41.415251 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:41.415221    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4p9v\" (UniqueName: \"kubernetes.io/projected/4e21b25a-73c8-4964-8a0d-03903a1de771-kube-api-access-v4p9v\") pod \"e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9\" (UID: \"4e21b25a-73c8-4964-8a0d-03903a1de771\") " pod="llm/e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9"
Jul 29 18:57:41.496430 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:41.496313    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9"
Jul 29 18:57:41.640225 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:41.640197    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9"]
Jul 29 18:57:41.643092 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:57:41.643053    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e21b25a_73c8_4964_8a0d_03903a1de771.slice/crio-7e8871b8c70ca117ab779e75f1a34c1eed81cdc27bee7e7a06462a9c75a62968 WatchSource:0}: Error finding container 7e8871b8c70ca117ab779e75f1a34c1eed81cdc27bee7e7a06462a9c75a62968: Status 404 returned error can't find the container with id 7e8871b8c70ca117ab779e75f1a34c1eed81cdc27bee7e7a06462a9c75a62968
Jul 29 18:57:41.645158 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:41.645136    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 18:57:41.938274 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:41.938208    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9" event={"ID":"4e21b25a-73c8-4964-8a0d-03903a1de771","Type":"ContainerStarted","Data":"7e8871b8c70ca117ab779e75f1a34c1eed81cdc27bee7e7a06462a9c75a62968"}
Jul 29 18:57:42.944281 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:42.944242    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9" event={"ID":"4e21b25a-73c8-4964-8a0d-03903a1de771","Type":"ContainerStarted","Data":"2a02691befa62b4e4b3727e44b4f39dcb9aae6ce493826f26cf1df8537a237b0"}
Jul 29 18:57:42.961275 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:42.961168    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9" podStartSLOduration=0.900395538 podStartE2EDuration="1.961149109s" totalImagesPullingTime="1.060753571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:57:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:57:42.960348944 +0000 UTC m=+1427.230118031" watchObservedRunningTime="2026-07-29 18:57:42.961149109 +0000 UTC m=+1427.230918188"
Jul 29 18:57:50.965755 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:50.965665    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d"]
Jul 29 18:57:50.969549 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:50.969523    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d"
Jul 29 18:57:50.972163 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:50.972141    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-90b41ea4-kserve-self-signed-certs\""
Jul 29 18:57:50.978167 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:50.978137    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d"]
Jul 29 18:57:51.097270 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:51.097222    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4trxv\" (UniqueName: \"kubernetes.io/projected/76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d-kube-api-access-4trxv\") pod \"e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d\" (UID: \"76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d\") " pod="llm/e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d"
Jul 29 18:57:51.097504 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:51.097280    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d-tls-certs\") pod \"e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d\" (UID: \"76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d\") " pod="llm/e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d"
Jul 29 18:57:51.097504 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:51.097356    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d-home\") pod \"e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d\" (UID: \"76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d\") " pod="llm/e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d"
Jul 29 18:57:51.097504 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:51.097433    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d-model-cache\") pod \"e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d\" (UID: \"76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d\") " pod="llm/e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d"
Jul 29 18:57:51.097504 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:51.097459    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d-tmp-dir\") pod \"e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d\" (UID: \"76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d\") " pod="llm/e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d"
Jul 29 18:57:51.097649 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:51.097537    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d-dshm\") pod \"e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d\" (UID: \"76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d\") " pod="llm/e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d"
Jul 29 18:57:51.198288 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:51.198247    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d-tls-certs\") pod \"e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d\" (UID: \"76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d\") " pod="llm/e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d"
Jul 29 18:57:51.198503 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:51.198297    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d-home\") pod \"e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d\" (UID: \"76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d\") " pod="llm/e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d"
Jul 29 18:57:51.198503 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:51.198317    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d-model-cache\") pod \"e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d\" (UID: \"76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d\") " pod="llm/e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d"
Jul 29 18:57:51.198503 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:51.198331    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d-tmp-dir\") pod \"e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d\" (UID: \"76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d\") " pod="llm/e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d"
Jul 29 18:57:51.198503 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:51.198373    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d-dshm\") pod \"e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d\" (UID: \"76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d\") " pod="llm/e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d"
Jul 29 18:57:51.198503 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:51.198446    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4trxv\" (UniqueName: \"kubernetes.io/projected/76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d-kube-api-access-4trxv\") pod \"e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d\" (UID: \"76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d\") " pod="llm/e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d"
Jul 29 18:57:51.198769 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:51.198737    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d-home\") pod \"e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d\" (UID: \"76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d\") " pod="llm/e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d"
Jul 29 18:57:51.198926 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:51.198786    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d-model-cache\") pod \"e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d\" (UID: \"76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d\") " pod="llm/e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d"
Jul 29 18:57:51.198926 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:51.198804    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d-tmp-dir\") pod \"e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d\" (UID: \"76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d\") " pod="llm/e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d"
Jul 29 18:57:51.200774 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:51.200757    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d-dshm\") pod \"e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d\" (UID: \"76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d\") " pod="llm/e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d"
Jul 29 18:57:51.201071 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:51.201055    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d-tls-certs\") pod \"e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d\" (UID: \"76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d\") " pod="llm/e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d"
Jul 29 18:57:51.206552 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:51.206532    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4trxv\" (UniqueName: \"kubernetes.io/projected/76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d-kube-api-access-4trxv\") pod \"e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d\" (UID: \"76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d\") " pod="llm/e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d"
Jul 29 18:57:51.279649 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:51.279541    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d"
Jul 29 18:57:51.425621 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:51.425584    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d"]
Jul 29 18:57:51.429644 ip-10-0-138-152 kubenswrapper[2559]: W0729 18:57:51.429607    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76ea1ca2_bb00_4ffb_9d47_8540f19b5c2d.slice/crio-3d1e79f2bddc57f9708974544c5932fe46a581358f8cf946696014ae6eda5728 WatchSource:0}: Error finding container 3d1e79f2bddc57f9708974544c5932fe46a581358f8cf946696014ae6eda5728: Status 404 returned error can't find the container with id 3d1e79f2bddc57f9708974544c5932fe46a581358f8cf946696014ae6eda5728
Jul 29 18:57:51.496970 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:51.496927    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9"
Jul 29 18:57:51.497101 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:51.496997    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9"
Jul 29 18:57:51.510531 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:51.510503    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9"
Jul 29 18:57:51.990822 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:51.990784    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d" event={"ID":"76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d","Type":"ContainerStarted","Data":"1a27497ddecb5cffc375653e0bb79b48b7516a832665836408b0e8851b4dc079"}
Jul 29 18:57:51.991254 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:51.990838    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d" event={"ID":"76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d","Type":"ContainerStarted","Data":"3d1e79f2bddc57f9708974544c5932fe46a581358f8cf946696014ae6eda5728"}
Jul 29 18:57:52.002155 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:52.002130    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9"
Jul 29 18:57:52.008139 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:52.008089    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d" podStartSLOduration=2.008072601 podStartE2EDuration="2.008072601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:57:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:57:52.006754072 +0000 UTC m=+1436.276523162" watchObservedRunningTime="2026-07-29 18:57:52.008072601 +0000 UTC m=+1436.277841688"
Jul 29 18:57:53.645583 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:57:53.645543    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:58:01.280400 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:01.280320    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d"
Jul 29 18:58:01.280400 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:01.280423    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d"
Jul 29 18:58:01.293358 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:01.293326    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d"
Jul 29 18:58:02.049893 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:02.049857    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d"
Jul 29 18:58:24.488704 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:24.488660    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9"]
Jul 29 18:58:25.391228 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:25.391183    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d"]
Jul 29 18:58:28.634931 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:28.634894    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:58:39.509077 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:39.509025    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9" podUID="4e21b25a-73c8-4964-8a0d-03903a1de771" containerName="main" containerID="cri-o://2a02691befa62b4e4b3727e44b4f39dcb9aae6ce493826f26cf1df8537a237b0" gracePeriod=45
Jul 29 18:58:39.774399 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:39.774310    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9"
Jul 29 18:58:39.919680 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:39.919647    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e21b25a-73c8-4964-8a0d-03903a1de771-tls-certs\") pod \"4e21b25a-73c8-4964-8a0d-03903a1de771\" (UID: \"4e21b25a-73c8-4964-8a0d-03903a1de771\") "
Jul 29 18:58:39.919680 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:39.919693    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4p9v\" (UniqueName: \"kubernetes.io/projected/4e21b25a-73c8-4964-8a0d-03903a1de771-kube-api-access-v4p9v\") pod \"4e21b25a-73c8-4964-8a0d-03903a1de771\" (UID: \"4e21b25a-73c8-4964-8a0d-03903a1de771\") "
Jul 29 18:58:39.919943 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:39.919751    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e21b25a-73c8-4964-8a0d-03903a1de771-dshm\") pod \"4e21b25a-73c8-4964-8a0d-03903a1de771\" (UID: \"4e21b25a-73c8-4964-8a0d-03903a1de771\") "
Jul 29 18:58:39.919943 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:39.919771    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e21b25a-73c8-4964-8a0d-03903a1de771-tmp-dir\") pod \"4e21b25a-73c8-4964-8a0d-03903a1de771\" (UID: \"4e21b25a-73c8-4964-8a0d-03903a1de771\") "
Jul 29 18:58:39.919943 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:39.919787    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e21b25a-73c8-4964-8a0d-03903a1de771-home\") pod \"4e21b25a-73c8-4964-8a0d-03903a1de771\" (UID: \"4e21b25a-73c8-4964-8a0d-03903a1de771\") "
Jul 29 18:58:39.919943 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:39.919861    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e21b25a-73c8-4964-8a0d-03903a1de771-model-cache\") pod \"4e21b25a-73c8-4964-8a0d-03903a1de771\" (UID: \"4e21b25a-73c8-4964-8a0d-03903a1de771\") "
Jul 29 18:58:39.920144 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:39.920083    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e21b25a-73c8-4964-8a0d-03903a1de771-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4e21b25a-73c8-4964-8a0d-03903a1de771" (UID: "4e21b25a-73c8-4964-8a0d-03903a1de771"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:58:39.920144 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:39.920151    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e21b25a-73c8-4964-8a0d-03903a1de771-home" (OuterVolumeSpecName: "home") pod "4e21b25a-73c8-4964-8a0d-03903a1de771" (UID: "4e21b25a-73c8-4964-8a0d-03903a1de771"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:58:39.920144 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:39.920181    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e21b25a-73c8-4964-8a0d-03903a1de771-model-cache" (OuterVolumeSpecName: "model-cache") pod "4e21b25a-73c8-4964-8a0d-03903a1de771" (UID: "4e21b25a-73c8-4964-8a0d-03903a1de771"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:58:39.922366 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:39.922336    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e21b25a-73c8-4964-8a0d-03903a1de771-dshm" (OuterVolumeSpecName: "dshm") pod "4e21b25a-73c8-4964-8a0d-03903a1de771" (UID: "4e21b25a-73c8-4964-8a0d-03903a1de771"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:58:39.922366 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:39.922353    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e21b25a-73c8-4964-8a0d-03903a1de771-kube-api-access-v4p9v" (OuterVolumeSpecName: "kube-api-access-v4p9v") pod "4e21b25a-73c8-4964-8a0d-03903a1de771" (UID: "4e21b25a-73c8-4964-8a0d-03903a1de771"). InnerVolumeSpecName "kube-api-access-v4p9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 18:58:39.922366 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:39.922347    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e21b25a-73c8-4964-8a0d-03903a1de771-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4e21b25a-73c8-4964-8a0d-03903a1de771" (UID: "4e21b25a-73c8-4964-8a0d-03903a1de771"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 18:58:40.021204 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:40.021168    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e21b25a-73c8-4964-8a0d-03903a1de771-model-cache\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 18:58:40.021204 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:40.021198    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e21b25a-73c8-4964-8a0d-03903a1de771-tls-certs\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 18:58:40.021204 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:40.021208    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4p9v\" (UniqueName: \"kubernetes.io/projected/4e21b25a-73c8-4964-8a0d-03903a1de771-kube-api-access-v4p9v\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 18:58:40.021459 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:40.021218    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e21b25a-73c8-4964-8a0d-03903a1de771-dshm\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 18:58:40.021459 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:40.021244    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e21b25a-73c8-4964-8a0d-03903a1de771-tmp-dir\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 18:58:40.021459 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:40.021253    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e21b25a-73c8-4964-8a0d-03903a1de771-home\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 18:58:40.223820 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:40.223781    2559 generic.go:356] "Generic (PLEG): container finished" podID="4e21b25a-73c8-4964-8a0d-03903a1de771" containerID="2a02691befa62b4e4b3727e44b4f39dcb9aae6ce493826f26cf1df8537a237b0" exitCode=0
Jul 29 18:58:40.224019 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:40.223833    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9"
Jul 29 18:58:40.224019 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:40.223865    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9" event={"ID":"4e21b25a-73c8-4964-8a0d-03903a1de771","Type":"ContainerDied","Data":"2a02691befa62b4e4b3727e44b4f39dcb9aae6ce493826f26cf1df8537a237b0"}
Jul 29 18:58:40.224019 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:40.223917    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9" event={"ID":"4e21b25a-73c8-4964-8a0d-03903a1de771","Type":"ContainerDied","Data":"7e8871b8c70ca117ab779e75f1a34c1eed81cdc27bee7e7a06462a9c75a62968"}
Jul 29 18:58:40.224019 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:40.223942    2559 scope.go:117] "RemoveContainer" containerID="2a02691befa62b4e4b3727e44b4f39dcb9aae6ce493826f26cf1df8537a237b0"
Jul 29 18:58:40.233457 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:40.233439    2559 scope.go:117] "RemoveContainer" containerID="2a02691befa62b4e4b3727e44b4f39dcb9aae6ce493826f26cf1df8537a237b0"
Jul 29 18:58:40.233768 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:58:40.233749    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a02691befa62b4e4b3727e44b4f39dcb9aae6ce493826f26cf1df8537a237b0\": container with ID starting with 2a02691befa62b4e4b3727e44b4f39dcb9aae6ce493826f26cf1df8537a237b0 not found: ID does not exist" containerID="2a02691befa62b4e4b3727e44b4f39dcb9aae6ce493826f26cf1df8537a237b0"
Jul 29 18:58:40.233818 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:40.233778    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a02691befa62b4e4b3727e44b4f39dcb9aae6ce493826f26cf1df8537a237b0"} err="failed to get container status \"2a02691befa62b4e4b3727e44b4f39dcb9aae6ce493826f26cf1df8537a237b0\": rpc error: code = NotFound desc = could not find container \"2a02691befa62b4e4b3727e44b4f39dcb9aae6ce493826f26cf1df8537a237b0\": container with ID starting with 2a02691befa62b4e4b3727e44b4f39dcb9aae6ce493826f26cf1df8537a237b0 not found: ID does not exist"
Jul 29 18:58:40.239596 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:40.239569    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9"]
Jul 29 18:58:40.243233 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:40.243207    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-90b41ea4-kserve-6747585f95-k5vv9"]
Jul 29 18:58:40.411402 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:40.411276    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d" podUID="76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d" containerName="main" containerID="cri-o://1a27497ddecb5cffc375653e0bb79b48b7516a832665836408b0e8851b4dc079" gracePeriod=45
Jul 29 18:58:40.672053 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:40.671992    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d"
Jul 29 18:58:40.826304 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:40.826264    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d-tmp-dir\") pod \"76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d\" (UID: \"76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d\") "
Jul 29 18:58:40.826304 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:40.826313    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d-model-cache\") pod \"76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d\" (UID: \"76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d\") "
Jul 29 18:58:40.826584 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:40.826334    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d-tls-certs\") pod \"76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d\" (UID: \"76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d\") "
Jul 29 18:58:40.826584 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:40.826370    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d-home\") pod \"76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d\" (UID: \"76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d\") "
Jul 29 18:58:40.826584 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:40.826440    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4trxv\" (UniqueName: \"kubernetes.io/projected/76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d-kube-api-access-4trxv\") pod \"76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d\" (UID: \"76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d\") "
Jul 29 18:58:40.826584 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:40.826482    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d-dshm\") pod \"76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d\" (UID: \"76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d\") "
Jul 29 18:58:40.826785 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:40.826622    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d" (UID: "76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:58:40.826785 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:40.826634    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d-model-cache" (OuterVolumeSpecName: "model-cache") pod "76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d" (UID: "76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:58:40.826785 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:40.826703    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d-home" (OuterVolumeSpecName: "home") pod "76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d" (UID: "76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:58:40.826785 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:40.826782    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d-tmp-dir\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 18:58:40.826926 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:40.826799    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d-model-cache\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 18:58:40.826926 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:40.826813    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d-home\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 18:58:40.828786 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:40.828762    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d-dshm" (OuterVolumeSpecName: "dshm") pod "76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d" (UID: "76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 18:58:40.828916 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:40.828811    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d" (UID: "76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 18:58:40.828916 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:40.828821    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d-kube-api-access-4trxv" (OuterVolumeSpecName: "kube-api-access-4trxv") pod "76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d" (UID: "76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d"). InnerVolumeSpecName "kube-api-access-4trxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 18:58:40.927639 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:40.927545    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d-tls-certs\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 18:58:40.927639 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:40.927581    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4trxv\" (UniqueName: \"kubernetes.io/projected/76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d-kube-api-access-4trxv\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 18:58:40.927639 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:40.927590    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d-dshm\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 18:58:41.230973 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:41.230882    2559 generic.go:356] "Generic (PLEG): container finished" podID="76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d" containerID="1a27497ddecb5cffc375653e0bb79b48b7516a832665836408b0e8851b4dc079" exitCode=0
Jul 29 18:58:41.230973 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:41.230926    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d"
Jul 29 18:58:41.231189 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:41.230969    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d" event={"ID":"76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d","Type":"ContainerDied","Data":"1a27497ddecb5cffc375653e0bb79b48b7516a832665836408b0e8851b4dc079"}
Jul 29 18:58:41.231189 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:41.231013    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d" event={"ID":"76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d","Type":"ContainerDied","Data":"3d1e79f2bddc57f9708974544c5932fe46a581358f8cf946696014ae6eda5728"}
Jul 29 18:58:41.231189 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:41.231030    2559 scope.go:117] "RemoveContainer" containerID="1a27497ddecb5cffc375653e0bb79b48b7516a832665836408b0e8851b4dc079"
Jul 29 18:58:41.239850 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:41.239836    2559 scope.go:117] "RemoveContainer" containerID="1a27497ddecb5cffc375653e0bb79b48b7516a832665836408b0e8851b4dc079"
Jul 29 18:58:41.240101 ip-10-0-138-152 kubenswrapper[2559]: E0729 18:58:41.240084    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a27497ddecb5cffc375653e0bb79b48b7516a832665836408b0e8851b4dc079\": container with ID starting with 1a27497ddecb5cffc375653e0bb79b48b7516a832665836408b0e8851b4dc079 not found: ID does not exist" containerID="1a27497ddecb5cffc375653e0bb79b48b7516a832665836408b0e8851b4dc079"
Jul 29 18:58:41.240146 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:41.240112    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a27497ddecb5cffc375653e0bb79b48b7516a832665836408b0e8851b4dc079"} err="failed to get container status \"1a27497ddecb5cffc375653e0bb79b48b7516a832665836408b0e8851b4dc079\": rpc error: code = NotFound desc = could not find container \"1a27497ddecb5cffc375653e0bb79b48b7516a832665836408b0e8851b4dc079\": container with ID starting with 1a27497ddecb5cffc375653e0bb79b48b7516a832665836408b0e8851b4dc079 not found: ID does not exist"
Jul 29 18:58:41.250885 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:41.250855    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d"]
Jul 29 18:58:41.255732 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:41.255707    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-90b41ea4-kserve-5fc5f449ff-4bw7d"]
Jul 29 18:58:42.150229 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:42.150196    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e21b25a-73c8-4964-8a0d-03903a1de771" path="/var/lib/kubelet/pods/4e21b25a-73c8-4964-8a0d-03903a1de771/volumes"
Jul 29 18:58:42.150627 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:42.150587    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d" path="/var/lib/kubelet/pods/76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d/volumes"
Jul 29 18:58:42.328324 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:42.328289    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:58:50.444277 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:50.444233    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:58:56.158069 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:56.158030    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-v9nsr_7f8b4737-d4c2-4598-bd40-208015542831/console-operator/1.log"
Jul 29 18:58:56.158572 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:56.158079    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-v9nsr_7f8b4737-d4c2-4598-bd40-208015542831/console-operator/1.log"
Jul 29 18:58:56.165810 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:56.165788    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjdrx_60f444e1-7e31-443c-8ec7-a4d04c64bf4b/ovn-acl-logging/0.log"
Jul 29 18:58:56.166188 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:58:56.166167    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjdrx_60f444e1-7e31-443c-8ec7-a4d04c64bf4b/ovn-acl-logging/0.log"
Jul 29 18:59:02.435893 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:59:02.435851    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:59:12.336236 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:59:12.336197    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:59:21.438832 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:59:21.438747    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:59:31.542931 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:59:31.542893    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:59:40.648987 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:59:40.648950    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 18:59:50.237456 ip-10-0-138-152 kubenswrapper[2559]: I0729 18:59:50.237423    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:00:00.143123 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:00:00.143083    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755860-8rw6q"]
Jul 29 19:00:00.143655 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:00:00.143633    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e21b25a-73c8-4964-8a0d-03903a1de771" containerName="main"
Jul 29 19:00:00.143724 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:00:00.143658    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e21b25a-73c8-4964-8a0d-03903a1de771" containerName="main"
Jul 29 19:00:00.143724 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:00:00.143687    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d" containerName="main"
Jul 29 19:00:00.143724 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:00:00.143697    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d" containerName="main"
Jul 29 19:00:00.143922 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:00:00.143785    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="76ea1ca2-bb00-4ffb-9d47-8540f19b5c2d" containerName="main"
Jul 29 19:00:00.143922 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:00:00.143802    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e21b25a-73c8-4964-8a0d-03903a1de771" containerName="main"
Jul 29 19:00:00.146899 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:00:00.146872    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755860-8rw6q"
Jul 29 19:00:00.149448 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:00:00.149420    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vgb9z\""
Jul 29 19:00:00.154445 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:00:00.154419    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755860-8rw6q"]
Jul 29 19:00:00.158683 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:00:00.158662    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755860-8rw6q"
Jul 29 19:00:00.305891 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:00:00.305851    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755860-8rw6q"]
Jul 29 19:00:00.308472 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:00:00.308441    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d3fdb5f_b06c_40a6_84a2_9f47bbd0e735.slice/crio-75f4b1dd784e44c8ffd705529d71bfc3289ae623e70b836b3150fd25a4c52ecd WatchSource:0}: Error finding container 75f4b1dd784e44c8ffd705529d71bfc3289ae623e70b836b3150fd25a4c52ecd: Status 404 returned error can't find the container with id 75f4b1dd784e44c8ffd705529d71bfc3289ae623e70b836b3150fd25a4c52ecd
Jul 29 19:00:00.604251 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:00:00.604213    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755860-8rw6q" event={"ID":"3d3fdb5f-b06c-40a6-84a2-9f47bbd0e735","Type":"ContainerStarted","Data":"75f4b1dd784e44c8ffd705529d71bfc3289ae623e70b836b3150fd25a4c52ecd"}
Jul 29 19:00:03.246413 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:00:03.246353    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:00:04.626900 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:00:04.626853    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755860-8rw6q" event={"ID":"3d3fdb5f-b06c-40a6-84a2-9f47bbd0e735","Type":"ContainerStarted","Data":"642c00bdc972c77733a70c213917be0b6e28c2e5053b98a1f008146b0fcf00cf"}
Jul 29 19:00:04.642754 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:00:04.642691    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755860-8rw6q" podStartSLOduration=1.028085858 podStartE2EDuration="4.64267071s" totalImagesPullingTime="3.614584852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:00:04.64223476 +0000 UTC m=+1568.912003916" watchObservedRunningTime="2026-07-29 19:00:04.64267071 +0000 UTC m=+1568.912439796"
Jul 29 19:00:05.633671 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:00:05.633633    2559 generic.go:356] "Generic (PLEG): container finished" podID="3d3fdb5f-b06c-40a6-84a2-9f47bbd0e735" containerID="642c00bdc972c77733a70c213917be0b6e28c2e5053b98a1f008146b0fcf00cf" exitCode=0
Jul 29 19:00:05.635989 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:00:05.633714    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755860-8rw6q" event={"ID":"3d3fdb5f-b06c-40a6-84a2-9f47bbd0e735","Type":"ContainerDied","Data":"642c00bdc972c77733a70c213917be0b6e28c2e5053b98a1f008146b0fcf00cf"}
Jul 29 19:00:06.773464 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:00:06.773438    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755860-8rw6q"
Jul 29 19:00:07.644653 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:00:07.644624    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755860-8rw6q"
Jul 29 19:00:07.644653 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:00:07.644631    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755860-8rw6q" event={"ID":"3d3fdb5f-b06c-40a6-84a2-9f47bbd0e735","Type":"ContainerDied","Data":"75f4b1dd784e44c8ffd705529d71bfc3289ae623e70b836b3150fd25a4c52ecd"}
Jul 29 19:00:07.644873 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:00:07.644675    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75f4b1dd784e44c8ffd705529d71bfc3289ae623e70b836b3150fd25a4c52ecd"
Jul 29 19:00:12.459089 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:00:12.459051    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:00:21.946689 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:00:21.946647    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:00:34.454586 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:00:34.454541    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:00:43.869862 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:00:43.869820    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:00:55.263144 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:00:55.263110    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:01:07.301999 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:01:07.301961    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:01:18.376037 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:01:18.375999    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:01:29.668168 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:01:29.668128    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:01:37.980078 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:01:37.980039    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:01:51.472592 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:01:51.472553    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:02:01.347531 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:01.347489    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:02:11.035414 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:11.035357    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:02:15.789746 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:15.789708    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e24ce08-f45d6b488-dmg62"]
Jul 29 19:02:15.790137 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:15.790121    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d3fdb5f-b06c-40a6-84a2-9f47bbd0e735" containerName="cleanup"
Jul 29 19:02:15.790137 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:15.790134    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d3fdb5f-b06c-40a6-84a2-9f47bbd0e735" containerName="cleanup"
Jul 29 19:02:15.790213 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:15.790201    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d3fdb5f-b06c-40a6-84a2-9f47bbd0e735" containerName="cleanup"
Jul 29 19:02:15.793339 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:15.793306    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e24ce08-f45d6b488-dmg62"
Jul 29 19:02:15.796220 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:15.796194    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0e24ce08\""
Jul 29 19:02:15.796359 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:15.796194    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0e24ce08-dockercfg-9zd2g\""
Jul 29 19:02:15.802040 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:15.801989    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e24ce08-f45d6b488-dmg62"]
Jul 29 19:02:15.898714 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:15.898681    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn7vh\" (UniqueName: \"kubernetes.io/projected/8b6f9e15-a288-4104-b3b6-c710b9bc4b0b-kube-api-access-kn7vh\") pod \"payload-pre-processing-e2e-ait-0e24ce08-f45d6b488-dmg62\" (UID: \"8b6f9e15-a288-4104-b3b6-c710b9bc4b0b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e24ce08-f45d6b488-dmg62"
Jul 29 19:02:15.898714 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:15.898725    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b6f9e15-a288-4104-b3b6-c710b9bc4b0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0e24ce08-f45d6b488-dmg62\" (UID: \"8b6f9e15-a288-4104-b3b6-c710b9bc4b0b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e24ce08-f45d6b488-dmg62"
Jul 29 19:02:16.000230 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:16.000185    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn7vh\" (UniqueName: \"kubernetes.io/projected/8b6f9e15-a288-4104-b3b6-c710b9bc4b0b-kube-api-access-kn7vh\") pod \"payload-pre-processing-e2e-ait-0e24ce08-f45d6b488-dmg62\" (UID: \"8b6f9e15-a288-4104-b3b6-c710b9bc4b0b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e24ce08-f45d6b488-dmg62"
Jul 29 19:02:16.000478 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:16.000235    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b6f9e15-a288-4104-b3b6-c710b9bc4b0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0e24ce08-f45d6b488-dmg62\" (UID: \"8b6f9e15-a288-4104-b3b6-c710b9bc4b0b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e24ce08-f45d6b488-dmg62"
Jul 29 19:02:16.001059 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:16.001027    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b6f9e15-a288-4104-b3b6-c710b9bc4b0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0e24ce08-f45d6b488-dmg62\" (UID: \"8b6f9e15-a288-4104-b3b6-c710b9bc4b0b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e24ce08-f45d6b488-dmg62"
Jul 29 19:02:16.008975 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:16.008944    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn7vh\" (UniqueName: \"kubernetes.io/projected/8b6f9e15-a288-4104-b3b6-c710b9bc4b0b-kube-api-access-kn7vh\") pod \"payload-pre-processing-e2e-ait-0e24ce08-f45d6b488-dmg62\" (UID: \"8b6f9e15-a288-4104-b3b6-c710b9bc4b0b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e24ce08-f45d6b488-dmg62"
Jul 29 19:02:16.105487 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:16.105446    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e24ce08-f45d6b488-dmg62"
Jul 29 19:02:16.251236 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:16.251199    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e24ce08-f45d6b488-dmg62"]
Jul 29 19:02:16.254593 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:02:16.254565    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b6f9e15_a288_4104_b3b6_c710b9bc4b0b.slice/crio-724176bc69e0133b3be38d9c9fd7819d3e3be61d2334f390b6793ef71aaffc7b WatchSource:0}: Error finding container 724176bc69e0133b3be38d9c9fd7819d3e3be61d2334f390b6793ef71aaffc7b: Status 404 returned error can't find the container with id 724176bc69e0133b3be38d9c9fd7819d3e3be61d2334f390b6793ef71aaffc7b
Jul 29 19:02:17.256109 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:17.256029    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e24ce08-f45d6b488-dmg62" event={"ID":"8b6f9e15-a288-4104-b3b6-c710b9bc4b0b","Type":"ContainerStarted","Data":"724176bc69e0133b3be38d9c9fd7819d3e3be61d2334f390b6793ef71aaffc7b"}
Jul 29 19:02:20.273579 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:20.273538    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e24ce08-f45d6b488-dmg62" event={"ID":"8b6f9e15-a288-4104-b3b6-c710b9bc4b0b","Type":"ContainerStarted","Data":"e4ec79a0cf538d75819702f8d63293c5a45d0675c784448dc95e64b494790794"}
Jul 29 19:02:20.273969 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:20.273643    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e24ce08-f45d6b488-dmg62"
Jul 29 19:02:20.289637 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:20.289523    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e24ce08-f45d6b488-dmg62" podStartSLOduration=1.5190234980000001 podStartE2EDuration="5.28950304s" totalImagesPullingTime="3.770479542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:02:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:02:20.288812104 +0000 UTC m=+1704.558581193" watchObservedRunningTime="2026-07-29 19:02:20.28950304 +0000 UTC m=+1704.559272129"
Jul 29 19:02:22.443583 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:22.443544    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:02:31.281261 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:31.281211    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e24ce08-f45d6b488-dmg62"
Jul 29 19:02:40.568850 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:40.568811    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e24ce08-p6v66"]
Jul 29 19:02:40.572289 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:40.572270    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e24ce08-p6v66"
Jul 29 19:02:40.574870 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:40.574843    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vgb9z\""
Jul 29 19:02:40.574998 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:40.574843    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 19:02:40.581123 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:40.581095    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e24ce08-p6v66"]
Jul 29 19:02:40.605911 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:40.605877    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e382020-89be-4daf-9bb8-67ff552c8d99-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e24ce08-p6v66\" (UID: \"4e382020-89be-4daf-9bb8-67ff552c8d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e24ce08-p6v66"
Jul 29 19:02:40.706890 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:40.706844    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e382020-89be-4daf-9bb8-67ff552c8d99-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e24ce08-p6v66\" (UID: \"4e382020-89be-4daf-9bb8-67ff552c8d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e24ce08-p6v66"
Jul 29 19:02:40.707566 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:40.707529    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e382020-89be-4daf-9bb8-67ff552c8d99-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e24ce08-p6v66\" (UID: \"4e382020-89be-4daf-9bb8-67ff552c8d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e24ce08-p6v66"
Jul 29 19:02:40.884807 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:40.884764    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e24ce08-p6v66"
Jul 29 19:02:41.024142 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:41.023938    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e24ce08-p6v66"]
Jul 29 19:02:41.027092 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:02:41.027064    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e382020_89be_4daf_9bb8_67ff552c8d99.slice/crio-79b28eddf8e4b5b6b53788c52798174bcc7daa3aacce062e5d364c386903e028 WatchSource:0}: Error finding container 79b28eddf8e4b5b6b53788c52798174bcc7daa3aacce062e5d364c386903e028: Status 404 returned error can't find the container with id 79b28eddf8e4b5b6b53788c52798174bcc7daa3aacce062e5d364c386903e028
Jul 29 19:02:41.374829 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:41.374793    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e24ce08-p6v66" event={"ID":"4e382020-89be-4daf-9bb8-67ff552c8d99","Type":"ContainerStarted","Data":"6d0b44aa2c19b473b19ae00afc340cdd5eb1b8a1b0a5fcf92364834c84b08fd6"}
Jul 29 19:02:41.375003 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:41.374852    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e24ce08-p6v66" event={"ID":"4e382020-89be-4daf-9bb8-67ff552c8d99","Type":"ContainerStarted","Data":"79b28eddf8e4b5b6b53788c52798174bcc7daa3aacce062e5d364c386903e028"}
Jul 29 19:02:41.392536 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:41.392486    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e24ce08-p6v66" podStartSLOduration=1.392468328 podStartE2EDuration="1.392468328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:02:41.391493928 +0000 UTC m=+1725.661263020" watchObservedRunningTime="2026-07-29 19:02:41.392468328 +0000 UTC m=+1725.662237457"
Jul 29 19:02:42.381487 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:42.381451    2559 generic.go:356] "Generic (PLEG): container finished" podID="4e382020-89be-4daf-9bb8-67ff552c8d99" containerID="6d0b44aa2c19b473b19ae00afc340cdd5eb1b8a1b0a5fcf92364834c84b08fd6" exitCode=0
Jul 29 19:02:42.381883 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:42.381544    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e24ce08-p6v66" event={"ID":"4e382020-89be-4daf-9bb8-67ff552c8d99","Type":"ContainerDied","Data":"6d0b44aa2c19b473b19ae00afc340cdd5eb1b8a1b0a5fcf92364834c84b08fd6"}
Jul 29 19:02:43.526324 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:43.526293    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e24ce08-p6v66"
Jul 29 19:02:43.633671 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:43.633632    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e382020-89be-4daf-9bb8-67ff552c8d99-maas-api-service-ca\") pod \"4e382020-89be-4daf-9bb8-67ff552c8d99\" (UID: \"4e382020-89be-4daf-9bb8-67ff552c8d99\") "
Jul 29 19:02:43.633999 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:43.633976    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e382020-89be-4daf-9bb8-67ff552c8d99-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4e382020-89be-4daf-9bb8-67ff552c8d99" (UID: "4e382020-89be-4daf-9bb8-67ff552c8d99"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:02:43.735225 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:43.735138    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e382020-89be-4daf-9bb8-67ff552c8d99-maas-api-service-ca\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:02:44.392105 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:44.392076    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e24ce08-p6v66"
Jul 29 19:02:44.392105 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:44.392097    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e24ce08-p6v66" event={"ID":"4e382020-89be-4daf-9bb8-67ff552c8d99","Type":"ContainerDied","Data":"79b28eddf8e4b5b6b53788c52798174bcc7daa3aacce062e5d364c386903e028"}
Jul 29 19:02:44.392311 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:44.392139    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79b28eddf8e4b5b6b53788c52798174bcc7daa3aacce062e5d364c386903e028"
Jul 29 19:02:45.430257 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:45.430220    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e24ce08-f45d6b488-dmg62"]
Jul 29 19:02:45.430683 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:45.430556    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e24ce08-f45d6b488-dmg62" podUID="8b6f9e15-a288-4104-b3b6-c710b9bc4b0b" containerName="payload-pre-processing" containerID="cri-o://e4ec79a0cf538d75819702f8d63293c5a45d0675c784448dc95e64b494790794" gracePeriod=30
Jul 29 19:02:45.559533 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:45.559496    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e24ce08-p6v66"]
Jul 29 19:02:45.562790 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:45.562759    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e24ce08-p6v66"]
Jul 29 19:02:45.691070 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:45.691003    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e24ce08-f45d6b488-dmg62"
Jul 29 19:02:45.752765 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:45.752729    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b6f9e15-a288-4104-b3b6-c710b9bc4b0b-plugins-config-volume\") pod \"8b6f9e15-a288-4104-b3b6-c710b9bc4b0b\" (UID: \"8b6f9e15-a288-4104-b3b6-c710b9bc4b0b\") "
Jul 29 19:02:45.752971 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:45.752825    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kn7vh\" (UniqueName: \"kubernetes.io/projected/8b6f9e15-a288-4104-b3b6-c710b9bc4b0b-kube-api-access-kn7vh\") pod \"8b6f9e15-a288-4104-b3b6-c710b9bc4b0b\" (UID: \"8b6f9e15-a288-4104-b3b6-c710b9bc4b0b\") "
Jul 29 19:02:45.753149 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:45.753126    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b6f9e15-a288-4104-b3b6-c710b9bc4b0b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8b6f9e15-a288-4104-b3b6-c710b9bc4b0b" (UID: "8b6f9e15-a288-4104-b3b6-c710b9bc4b0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:02:45.755235 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:45.755214    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b6f9e15-a288-4104-b3b6-c710b9bc4b0b-kube-api-access-kn7vh" (OuterVolumeSpecName: "kube-api-access-kn7vh") pod "8b6f9e15-a288-4104-b3b6-c710b9bc4b0b" (UID: "8b6f9e15-a288-4104-b3b6-c710b9bc4b0b"). InnerVolumeSpecName "kube-api-access-kn7vh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:02:45.853838 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:45.853790    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b6f9e15-a288-4104-b3b6-c710b9bc4b0b-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:02:45.853838 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:45.853829    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn7vh\" (UniqueName: \"kubernetes.io/projected/8b6f9e15-a288-4104-b3b6-c710b9bc4b0b-kube-api-access-kn7vh\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:02:46.150768 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:46.150731    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e382020-89be-4daf-9bb8-67ff552c8d99" path="/var/lib/kubelet/pods/4e382020-89be-4daf-9bb8-67ff552c8d99/volumes"
Jul 29 19:02:46.403200 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:46.403111    2559 generic.go:356] "Generic (PLEG): container finished" podID="8b6f9e15-a288-4104-b3b6-c710b9bc4b0b" containerID="e4ec79a0cf538d75819702f8d63293c5a45d0675c784448dc95e64b494790794" exitCode=0
Jul 29 19:02:46.403200 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:46.403161    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e24ce08-f45d6b488-dmg62"
Jul 29 19:02:46.403200 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:46.403176    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e24ce08-f45d6b488-dmg62" event={"ID":"8b6f9e15-a288-4104-b3b6-c710b9bc4b0b","Type":"ContainerDied","Data":"e4ec79a0cf538d75819702f8d63293c5a45d0675c784448dc95e64b494790794"}
Jul 29 19:02:46.403487 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:46.403229    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e24ce08-f45d6b488-dmg62" event={"ID":"8b6f9e15-a288-4104-b3b6-c710b9bc4b0b","Type":"ContainerDied","Data":"724176bc69e0133b3be38d9c9fd7819d3e3be61d2334f390b6793ef71aaffc7b"}
Jul 29 19:02:46.403487 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:46.403259    2559 scope.go:117] "RemoveContainer" containerID="e4ec79a0cf538d75819702f8d63293c5a45d0675c784448dc95e64b494790794"
Jul 29 19:02:46.412482 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:46.412459    2559 scope.go:117] "RemoveContainer" containerID="e4ec79a0cf538d75819702f8d63293c5a45d0675c784448dc95e64b494790794"
Jul 29 19:02:46.412798 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:02:46.412773    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4ec79a0cf538d75819702f8d63293c5a45d0675c784448dc95e64b494790794\": container with ID starting with e4ec79a0cf538d75819702f8d63293c5a45d0675c784448dc95e64b494790794 not found: ID does not exist" containerID="e4ec79a0cf538d75819702f8d63293c5a45d0675c784448dc95e64b494790794"
Jul 29 19:02:46.412874 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:46.412806    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4ec79a0cf538d75819702f8d63293c5a45d0675c784448dc95e64b494790794"} err="failed to get container status \"e4ec79a0cf538d75819702f8d63293c5a45d0675c784448dc95e64b494790794\": rpc error: code = NotFound desc = could not find container \"e4ec79a0cf538d75819702f8d63293c5a45d0675c784448dc95e64b494790794\": container with ID starting with e4ec79a0cf538d75819702f8d63293c5a45d0675c784448dc95e64b494790794 not found: ID does not exist"
Jul 29 19:02:46.420162 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:46.420131    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e24ce08-f45d6b488-dmg62"]
Jul 29 19:02:46.428812 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:46.428786    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e24ce08-f45d6b488-dmg62"]
Jul 29 19:02:48.150592 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:48.150563    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b6f9e15-a288-4104-b3b6-c710b9bc4b0b" path="/var/lib/kubelet/pods/8b6f9e15-a288-4104-b3b6-c710b9bc4b0b/volumes"
Jul 29 19:02:48.334716 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:48.334675    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:02:49.927167 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:49.927128    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-03561f0f-768f55dd65-fvmlx"]
Jul 29 19:02:49.927593 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:49.927489    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e382020-89be-4daf-9bb8-67ff552c8d99" containerName="revoke-keys"
Jul 29 19:02:49.927593 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:49.927501    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e382020-89be-4daf-9bb8-67ff552c8d99" containerName="revoke-keys"
Jul 29 19:02:49.927593 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:49.927522    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b6f9e15-a288-4104-b3b6-c710b9bc4b0b" containerName="payload-pre-processing"
Jul 29 19:02:49.927593 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:49.927528    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b6f9e15-a288-4104-b3b6-c710b9bc4b0b" containerName="payload-pre-processing"
Jul 29 19:02:49.927593 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:49.927583    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e382020-89be-4daf-9bb8-67ff552c8d99" containerName="revoke-keys"
Jul 29 19:02:49.927593 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:49.927591    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b6f9e15-a288-4104-b3b6-c710b9bc4b0b" containerName="payload-pre-processing"
Jul 29 19:02:49.931909 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:49.931885    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-03561f0f-768f55dd65-fvmlx"
Jul 29 19:02:49.934194 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:49.934171    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-03561f0f-dockercfg-wcbbd\""
Jul 29 19:02:49.934335 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:49.934302    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-03561f0f\""
Jul 29 19:02:49.941218 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:49.941191    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-03561f0f-768f55dd65-fvmlx"]
Jul 29 19:02:49.965002 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:49.964968    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-03561f0f-94b749f56-ptppj"]
Jul 29 19:02:49.968343 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:49.968325    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-03561f0f-94b749f56-ptppj"
Jul 29 19:02:49.978932 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:49.978907    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-03561f0f-94b749f56-ptppj"]
Jul 29 19:02:49.990293 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:49.990259    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac091bb2-424c-48eb-9434-42e99235f7ce-plugins-config-volume\") pod \"payload-processing-e2e-migrate-03561f0f-768f55dd65-fvmlx\" (UID: \"ac091bb2-424c-48eb-9434-42e99235f7ce\") " pod="openshift-ingress/payload-processing-e2e-migrate-03561f0f-768f55dd65-fvmlx"
Jul 29 19:02:49.990502 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:49.990413    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4rc7\" (UniqueName: \"kubernetes.io/projected/ac091bb2-424c-48eb-9434-42e99235f7ce-kube-api-access-p4rc7\") pod \"payload-processing-e2e-migrate-03561f0f-768f55dd65-fvmlx\" (UID: \"ac091bb2-424c-48eb-9434-42e99235f7ce\") " pod="openshift-ingress/payload-processing-e2e-migrate-03561f0f-768f55dd65-fvmlx"
Jul 29 19:02:50.091038 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:50.090999    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f97506fa-1781-446d-aea2-49425566897f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-03561f0f-94b749f56-ptppj\" (UID: \"f97506fa-1781-446d-aea2-49425566897f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-03561f0f-94b749f56-ptppj"
Jul 29 19:02:50.091239 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:50.091048    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac091bb2-424c-48eb-9434-42e99235f7ce-plugins-config-volume\") pod \"payload-processing-e2e-migrate-03561f0f-768f55dd65-fvmlx\" (UID: \"ac091bb2-424c-48eb-9434-42e99235f7ce\") " pod="openshift-ingress/payload-processing-e2e-migrate-03561f0f-768f55dd65-fvmlx"
Jul 29 19:02:50.091239 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:50.091102    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzflj\" (UniqueName: \"kubernetes.io/projected/f97506fa-1781-446d-aea2-49425566897f-kube-api-access-jzflj\") pod \"payload-pre-processing-e2e-migrate-03561f0f-94b749f56-ptppj\" (UID: \"f97506fa-1781-446d-aea2-49425566897f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-03561f0f-94b749f56-ptppj"
Jul 29 19:02:50.091239 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:50.091166    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4rc7\" (UniqueName: \"kubernetes.io/projected/ac091bb2-424c-48eb-9434-42e99235f7ce-kube-api-access-p4rc7\") pod \"payload-processing-e2e-migrate-03561f0f-768f55dd65-fvmlx\" (UID: \"ac091bb2-424c-48eb-9434-42e99235f7ce\") " pod="openshift-ingress/payload-processing-e2e-migrate-03561f0f-768f55dd65-fvmlx"
Jul 29 19:02:50.091786 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:50.091765    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac091bb2-424c-48eb-9434-42e99235f7ce-plugins-config-volume\") pod \"payload-processing-e2e-migrate-03561f0f-768f55dd65-fvmlx\" (UID: \"ac091bb2-424c-48eb-9434-42e99235f7ce\") " pod="openshift-ingress/payload-processing-e2e-migrate-03561f0f-768f55dd65-fvmlx"
Jul 29 19:02:50.099497 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:50.099475    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4rc7\" (UniqueName: \"kubernetes.io/projected/ac091bb2-424c-48eb-9434-42e99235f7ce-kube-api-access-p4rc7\") pod \"payload-processing-e2e-migrate-03561f0f-768f55dd65-fvmlx\" (UID: \"ac091bb2-424c-48eb-9434-42e99235f7ce\") " pod="openshift-ingress/payload-processing-e2e-migrate-03561f0f-768f55dd65-fvmlx"
Jul 29 19:02:50.192532 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:50.192425    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f97506fa-1781-446d-aea2-49425566897f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-03561f0f-94b749f56-ptppj\" (UID: \"f97506fa-1781-446d-aea2-49425566897f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-03561f0f-94b749f56-ptppj"
Jul 29 19:02:50.192532 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:50.192487    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzflj\" (UniqueName: \"kubernetes.io/projected/f97506fa-1781-446d-aea2-49425566897f-kube-api-access-jzflj\") pod \"payload-pre-processing-e2e-migrate-03561f0f-94b749f56-ptppj\" (UID: \"f97506fa-1781-446d-aea2-49425566897f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-03561f0f-94b749f56-ptppj"
Jul 29 19:02:50.193176 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:50.193156    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f97506fa-1781-446d-aea2-49425566897f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-03561f0f-94b749f56-ptppj\" (UID: \"f97506fa-1781-446d-aea2-49425566897f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-03561f0f-94b749f56-ptppj"
Jul 29 19:02:50.207683 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:50.207649    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzflj\" (UniqueName: \"kubernetes.io/projected/f97506fa-1781-446d-aea2-49425566897f-kube-api-access-jzflj\") pod \"payload-pre-processing-e2e-migrate-03561f0f-94b749f56-ptppj\" (UID: \"f97506fa-1781-446d-aea2-49425566897f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-03561f0f-94b749f56-ptppj"
Jul 29 19:02:50.244455 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:50.244413    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-03561f0f-768f55dd65-fvmlx"
Jul 29 19:02:50.280716 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:50.280678    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-03561f0f-94b749f56-ptppj"
Jul 29 19:02:50.397440 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:50.397399    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-03561f0f-768f55dd65-fvmlx"]
Jul 29 19:02:50.405608 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:02:50.405527    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac091bb2_424c_48eb_9434_42e99235f7ce.slice/crio-1c30dd76833bfcdfd714535b9d1f2ade4324598b43bced17d2e998a84a0eb181 WatchSource:0}: Error finding container 1c30dd76833bfcdfd714535b9d1f2ade4324598b43bced17d2e998a84a0eb181: Status 404 returned error can't find the container with id 1c30dd76833bfcdfd714535b9d1f2ade4324598b43bced17d2e998a84a0eb181
Jul 29 19:02:50.426345 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:50.426310    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-03561f0f-768f55dd65-fvmlx" event={"ID":"ac091bb2-424c-48eb-9434-42e99235f7ce","Type":"ContainerStarted","Data":"1c30dd76833bfcdfd714535b9d1f2ade4324598b43bced17d2e998a84a0eb181"}
Jul 29 19:02:50.441326 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:50.441287    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-03561f0f-94b749f56-ptppj"]
Jul 29 19:02:50.447771 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:02:50.447737    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf97506fa_1781_446d_aea2_49425566897f.slice/crio-ba2a31802c0ec677c2639f12483ac4a8f0a9e9eb1539e052ceb72d179ef44d41 WatchSource:0}: Error finding container ba2a31802c0ec677c2639f12483ac4a8f0a9e9eb1539e052ceb72d179ef44d41: Status 404 returned error can't find the container with id ba2a31802c0ec677c2639f12483ac4a8f0a9e9eb1539e052ceb72d179ef44d41
Jul 29 19:02:51.434105 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:51.434068    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-03561f0f-94b749f56-ptppj" event={"ID":"f97506fa-1781-446d-aea2-49425566897f","Type":"ContainerStarted","Data":"1e724781167301790328ffe556d09fe360ce584309eedad7be048fdcb47a1eba"}
Jul 29 19:02:51.434657 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:51.434156    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-03561f0f-94b749f56-ptppj" event={"ID":"f97506fa-1781-446d-aea2-49425566897f","Type":"ContainerStarted","Data":"ba2a31802c0ec677c2639f12483ac4a8f0a9e9eb1539e052ceb72d179ef44d41"}
Jul 29 19:02:51.434657 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:51.434207    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-03561f0f-94b749f56-ptppj"
Jul 29 19:02:51.435790 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:51.435762    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-03561f0f-768f55dd65-fvmlx" event={"ID":"ac091bb2-424c-48eb-9434-42e99235f7ce","Type":"ContainerStarted","Data":"ae3d1388657a1461f6954f615602180d3e7ccc59b7140c19b92635bde35fe0ce"}
Jul 29 19:02:51.435946 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:51.435811    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-03561f0f-768f55dd65-fvmlx"
Jul 29 19:02:51.450881 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:51.450828    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-03561f0f-94b749f56-ptppj" podStartSLOduration=2.450811944 podStartE2EDuration="2.450811944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:02:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:02:51.450109833 +0000 UTC m=+1735.719878918" watchObservedRunningTime="2026-07-29 19:02:51.450811944 +0000 UTC m=+1735.720581029"
Jul 29 19:02:51.466839 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:51.466771    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-03561f0f-768f55dd65-fvmlx" podStartSLOduration=2.466750887 podStartE2EDuration="2.466750887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:02:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:02:51.465948777 +0000 UTC m=+1735.735717864" watchObservedRunningTime="2026-07-29 19:02:51.466750887 +0000 UTC m=+1735.736519976"
Jul 29 19:02:59.546005 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:02:59.545966    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:03:02.442732 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:02.442699    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-03561f0f-94b749f56-ptppj"
Jul 29 19:03:02.443136 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:02.442775    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-03561f0f-768f55dd65-fvmlx"
Jul 29 19:03:14.186545 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:14.186501    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-03561f0f-27zr4"]
Jul 29 19:03:14.191697 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:14.191673    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-03561f0f-27zr4"
Jul 29 19:03:14.194590 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:14.194558    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vgb9z\""
Jul 29 19:03:14.194590 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:14.194559    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 19:03:14.200262 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:14.200239    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-03561f0f-27zr4"]
Jul 29 19:03:14.315320 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:14.315267    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5678e8a8-87e3-49c7-a578-27e9d6cd3c8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-03561f0f-27zr4\" (UID: \"5678e8a8-87e3-49c7-a578-27e9d6cd3c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-03561f0f-27zr4"
Jul 29 19:03:14.416013 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:14.415971    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5678e8a8-87e3-49c7-a578-27e9d6cd3c8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-03561f0f-27zr4\" (UID: \"5678e8a8-87e3-49c7-a578-27e9d6cd3c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-03561f0f-27zr4"
Jul 29 19:03:14.416667 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:14.416647    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5678e8a8-87e3-49c7-a578-27e9d6cd3c8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-03561f0f-27zr4\" (UID: \"5678e8a8-87e3-49c7-a578-27e9d6cd3c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-03561f0f-27zr4"
Jul 29 19:03:14.503290 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:14.503198    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-03561f0f-27zr4"
Jul 29 19:03:14.654233 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:14.654206    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-03561f0f-27zr4"]
Jul 29 19:03:14.657109 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:03:14.657075    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5678e8a8_87e3_49c7_a578_27e9d6cd3c8e.slice/crio-38c5954dde085c087dcf5dba5ebe1d7db97f34fe18c6839eeee42ca8217400d9 WatchSource:0}: Error finding container 38c5954dde085c087dcf5dba5ebe1d7db97f34fe18c6839eeee42ca8217400d9: Status 404 returned error can't find the container with id 38c5954dde085c087dcf5dba5ebe1d7db97f34fe18c6839eeee42ca8217400d9
Jul 29 19:03:15.556793 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:15.556757    2559 generic.go:356] "Generic (PLEG): container finished" podID="5678e8a8-87e3-49c7-a578-27e9d6cd3c8e" containerID="791abd3d75dce515eceaf7a672cc680c627e139494a7776ede0aa2f787538a05" exitCode=0
Jul 29 19:03:15.557181 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:15.556827    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-03561f0f-27zr4" event={"ID":"5678e8a8-87e3-49c7-a578-27e9d6cd3c8e","Type":"ContainerDied","Data":"791abd3d75dce515eceaf7a672cc680c627e139494a7776ede0aa2f787538a05"}
Jul 29 19:03:15.557181 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:15.556903    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-03561f0f-27zr4" event={"ID":"5678e8a8-87e3-49c7-a578-27e9d6cd3c8e","Type":"ContainerStarted","Data":"38c5954dde085c087dcf5dba5ebe1d7db97f34fe18c6839eeee42ca8217400d9"}
Jul 29 19:03:16.702123 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:16.702096    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-03561f0f-27zr4"
Jul 29 19:03:16.837917 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:16.837886    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5678e8a8-87e3-49c7-a578-27e9d6cd3c8e-maas-api-service-ca\") pod \"5678e8a8-87e3-49c7-a578-27e9d6cd3c8e\" (UID: \"5678e8a8-87e3-49c7-a578-27e9d6cd3c8e\") "
Jul 29 19:03:16.838279 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:16.838256    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5678e8a8-87e3-49c7-a578-27e9d6cd3c8e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5678e8a8-87e3-49c7-a578-27e9d6cd3c8e" (UID: "5678e8a8-87e3-49c7-a578-27e9d6cd3c8e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:03:16.939255 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:16.939217    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5678e8a8-87e3-49c7-a578-27e9d6cd3c8e-maas-api-service-ca\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:03:17.567498 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:17.567458    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-03561f0f-27zr4" event={"ID":"5678e8a8-87e3-49c7-a578-27e9d6cd3c8e","Type":"ContainerDied","Data":"38c5954dde085c087dcf5dba5ebe1d7db97f34fe18c6839eeee42ca8217400d9"}
Jul 29 19:03:17.567498 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:17.567496    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-03561f0f-27zr4"
Jul 29 19:03:17.567795 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:17.567521    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38c5954dde085c087dcf5dba5ebe1d7db97f34fe18c6839eeee42ca8217400d9"
Jul 29 19:03:20.507089 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:20.507055    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-03561f0f-768f55dd65-fvmlx"]
Jul 29 19:03:20.507604 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:20.507300    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-03561f0f-768f55dd65-fvmlx" podUID="ac091bb2-424c-48eb-9434-42e99235f7ce" containerName="payload-processing" containerID="cri-o://ae3d1388657a1461f6954f615602180d3e7ccc59b7140c19b92635bde35fe0ce" gracePeriod=30
Jul 29 19:03:20.520953 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:20.520908    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-03561f0f-94b749f56-ptppj"]
Jul 29 19:03:20.521187 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:20.521161    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-03561f0f-94b749f56-ptppj" podUID="f97506fa-1781-446d-aea2-49425566897f" containerName="payload-pre-processing" containerID="cri-o://1e724781167301790328ffe556d09fe360ce584309eedad7be048fdcb47a1eba" gracePeriod=30
Jul 29 19:03:20.647231 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:20.647194    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-03561f0f-27zr4"]
Jul 29 19:03:20.649677 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:20.649647    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-03561f0f-27zr4"]
Jul 29 19:03:20.807045 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:20.807016    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-03561f0f-768f55dd65-fvmlx"
Jul 29 19:03:20.815702 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:20.815676    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-03561f0f-94b749f56-ptppj"
Jul 29 19:03:20.875031 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:20.874993    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac091bb2-424c-48eb-9434-42e99235f7ce-plugins-config-volume\") pod \"ac091bb2-424c-48eb-9434-42e99235f7ce\" (UID: \"ac091bb2-424c-48eb-9434-42e99235f7ce\") "
Jul 29 19:03:20.875031 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:20.875040    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4rc7\" (UniqueName: \"kubernetes.io/projected/ac091bb2-424c-48eb-9434-42e99235f7ce-kube-api-access-p4rc7\") pod \"ac091bb2-424c-48eb-9434-42e99235f7ce\" (UID: \"ac091bb2-424c-48eb-9434-42e99235f7ce\") "
Jul 29 19:03:20.875262 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:20.875059    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzflj\" (UniqueName: \"kubernetes.io/projected/f97506fa-1781-446d-aea2-49425566897f-kube-api-access-jzflj\") pod \"f97506fa-1781-446d-aea2-49425566897f\" (UID: \"f97506fa-1781-446d-aea2-49425566897f\") "
Jul 29 19:03:20.875316 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:20.875268    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f97506fa-1781-446d-aea2-49425566897f-plugins-config-volume\") pod \"f97506fa-1781-446d-aea2-49425566897f\" (UID: \"f97506fa-1781-446d-aea2-49425566897f\") "
Jul 29 19:03:20.875460 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:20.875436    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac091bb2-424c-48eb-9434-42e99235f7ce-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ac091bb2-424c-48eb-9434-42e99235f7ce" (UID: "ac091bb2-424c-48eb-9434-42e99235f7ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:03:20.875664 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:20.875643    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f97506fa-1781-446d-aea2-49425566897f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f97506fa-1781-446d-aea2-49425566897f" (UID: "f97506fa-1781-446d-aea2-49425566897f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:03:20.875717 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:20.875705    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac091bb2-424c-48eb-9434-42e99235f7ce-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:03:20.875762 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:20.875723    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f97506fa-1781-446d-aea2-49425566897f-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:03:20.877330 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:20.877303    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac091bb2-424c-48eb-9434-42e99235f7ce-kube-api-access-p4rc7" (OuterVolumeSpecName: "kube-api-access-p4rc7") pod "ac091bb2-424c-48eb-9434-42e99235f7ce" (UID: "ac091bb2-424c-48eb-9434-42e99235f7ce"). InnerVolumeSpecName "kube-api-access-p4rc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:03:20.877468 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:20.877411    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f97506fa-1781-446d-aea2-49425566897f-kube-api-access-jzflj" (OuterVolumeSpecName: "kube-api-access-jzflj") pod "f97506fa-1781-446d-aea2-49425566897f" (UID: "f97506fa-1781-446d-aea2-49425566897f"). InnerVolumeSpecName "kube-api-access-jzflj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:03:20.976983 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:20.976934    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4rc7\" (UniqueName: \"kubernetes.io/projected/ac091bb2-424c-48eb-9434-42e99235f7ce-kube-api-access-p4rc7\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:03:20.976983 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:20.976978    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzflj\" (UniqueName: \"kubernetes.io/projected/f97506fa-1781-446d-aea2-49425566897f-kube-api-access-jzflj\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:03:21.588130 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:21.588095    2559 generic.go:356] "Generic (PLEG): container finished" podID="f97506fa-1781-446d-aea2-49425566897f" containerID="1e724781167301790328ffe556d09fe360ce584309eedad7be048fdcb47a1eba" exitCode=0
Jul 29 19:03:21.588605 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:21.588172    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-03561f0f-94b749f56-ptppj"
Jul 29 19:03:21.588605 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:21.588177    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-03561f0f-94b749f56-ptppj" event={"ID":"f97506fa-1781-446d-aea2-49425566897f","Type":"ContainerDied","Data":"1e724781167301790328ffe556d09fe360ce584309eedad7be048fdcb47a1eba"}
Jul 29 19:03:21.588605 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:21.588246    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-03561f0f-94b749f56-ptppj" event={"ID":"f97506fa-1781-446d-aea2-49425566897f","Type":"ContainerDied","Data":"ba2a31802c0ec677c2639f12483ac4a8f0a9e9eb1539e052ceb72d179ef44d41"}
Jul 29 19:03:21.588605 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:21.588280    2559 scope.go:117] "RemoveContainer" containerID="1e724781167301790328ffe556d09fe360ce584309eedad7be048fdcb47a1eba"
Jul 29 19:03:21.589616 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:21.589586    2559 generic.go:356] "Generic (PLEG): container finished" podID="ac091bb2-424c-48eb-9434-42e99235f7ce" containerID="ae3d1388657a1461f6954f615602180d3e7ccc59b7140c19b92635bde35fe0ce" exitCode=0
Jul 29 19:03:21.589684 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:21.589614    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-03561f0f-768f55dd65-fvmlx" event={"ID":"ac091bb2-424c-48eb-9434-42e99235f7ce","Type":"ContainerDied","Data":"ae3d1388657a1461f6954f615602180d3e7ccc59b7140c19b92635bde35fe0ce"}
Jul 29 19:03:21.589684 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:21.589660    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-03561f0f-768f55dd65-fvmlx"
Jul 29 19:03:21.589781 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:21.589690    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-03561f0f-768f55dd65-fvmlx" event={"ID":"ac091bb2-424c-48eb-9434-42e99235f7ce","Type":"ContainerDied","Data":"1c30dd76833bfcdfd714535b9d1f2ade4324598b43bced17d2e998a84a0eb181"}
Jul 29 19:03:21.598500 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:21.598480    2559 scope.go:117] "RemoveContainer" containerID="1e724781167301790328ffe556d09fe360ce584309eedad7be048fdcb47a1eba"
Jul 29 19:03:21.598876 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:03:21.598850    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e724781167301790328ffe556d09fe360ce584309eedad7be048fdcb47a1eba\": container with ID starting with 1e724781167301790328ffe556d09fe360ce584309eedad7be048fdcb47a1eba not found: ID does not exist" containerID="1e724781167301790328ffe556d09fe360ce584309eedad7be048fdcb47a1eba"
Jul 29 19:03:21.598992 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:21.598884    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e724781167301790328ffe556d09fe360ce584309eedad7be048fdcb47a1eba"} err="failed to get container status \"1e724781167301790328ffe556d09fe360ce584309eedad7be048fdcb47a1eba\": rpc error: code = NotFound desc = could not find container \"1e724781167301790328ffe556d09fe360ce584309eedad7be048fdcb47a1eba\": container with ID starting with 1e724781167301790328ffe556d09fe360ce584309eedad7be048fdcb47a1eba not found: ID does not exist"
Jul 29 19:03:21.598992 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:21.598908    2559 scope.go:117] "RemoveContainer" containerID="ae3d1388657a1461f6954f615602180d3e7ccc59b7140c19b92635bde35fe0ce"
Jul 29 19:03:21.608891 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:21.608865    2559 scope.go:117] "RemoveContainer" containerID="ae3d1388657a1461f6954f615602180d3e7ccc59b7140c19b92635bde35fe0ce"
Jul 29 19:03:21.610244 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:03:21.609643    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae3d1388657a1461f6954f615602180d3e7ccc59b7140c19b92635bde35fe0ce\": container with ID starting with ae3d1388657a1461f6954f615602180d3e7ccc59b7140c19b92635bde35fe0ce not found: ID does not exist" containerID="ae3d1388657a1461f6954f615602180d3e7ccc59b7140c19b92635bde35fe0ce"
Jul 29 19:03:21.610374 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:21.610255    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae3d1388657a1461f6954f615602180d3e7ccc59b7140c19b92635bde35fe0ce"} err="failed to get container status \"ae3d1388657a1461f6954f615602180d3e7ccc59b7140c19b92635bde35fe0ce\": rpc error: code = NotFound desc = could not find container \"ae3d1388657a1461f6954f615602180d3e7ccc59b7140c19b92635bde35fe0ce\": container with ID starting with ae3d1388657a1461f6954f615602180d3e7ccc59b7140c19b92635bde35fe0ce not found: ID does not exist"
Jul 29 19:03:21.614862 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:03:21.614834    2559 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf97506fa_1781_446d_aea2_49425566897f.slice/crio-ba2a31802c0ec677c2639f12483ac4a8f0a9e9eb1539e052ceb72d179ef44d41\": RecentStats: unable to find data in memory cache]"
Jul 29 19:03:21.626722 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:21.626690    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-03561f0f-768f55dd65-fvmlx"]
Jul 29 19:03:21.641891 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:21.641860    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-03561f0f-768f55dd65-fvmlx"]
Jul 29 19:03:21.661368 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:21.661337    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-03561f0f-94b749f56-ptppj"]
Jul 29 19:03:21.667849 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:21.667819    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-03561f0f-94b749f56-ptppj"]
Jul 29 19:03:22.150823 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:22.150793    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5678e8a8-87e3-49c7-a578-27e9d6cd3c8e" path="/var/lib/kubelet/pods/5678e8a8-87e3-49c7-a578-27e9d6cd3c8e/volumes"
Jul 29 19:03:22.151134 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:22.151122    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac091bb2-424c-48eb-9434-42e99235f7ce" path="/var/lib/kubelet/pods/ac091bb2-424c-48eb-9434-42e99235f7ce/volumes"
Jul 29 19:03:22.151433 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:22.151421    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f97506fa-1781-446d-aea2-49425566897f" path="/var/lib/kubelet/pods/f97506fa-1781-446d-aea2-49425566897f/volumes"
Jul 29 19:03:24.485083 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:24.485047    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:03:25.002715 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:25.002680    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1b5315ce-7d74c7966d-mm8zm"]
Jul 29 19:03:25.003064 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:25.003050    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5678e8a8-87e3-49c7-a578-27e9d6cd3c8e" containerName="revoke-keys"
Jul 29 19:03:25.003107 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:25.003067    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5678e8a8-87e3-49c7-a578-27e9d6cd3c8e" containerName="revoke-keys"
Jul 29 19:03:25.003107 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:25.003088    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac091bb2-424c-48eb-9434-42e99235f7ce" containerName="payload-processing"
Jul 29 19:03:25.003107 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:25.003093    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac091bb2-424c-48eb-9434-42e99235f7ce" containerName="payload-processing"
Jul 29 19:03:25.003197 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:25.003114    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f97506fa-1781-446d-aea2-49425566897f" containerName="payload-pre-processing"
Jul 29 19:03:25.003197 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:25.003122    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f97506fa-1781-446d-aea2-49425566897f" containerName="payload-pre-processing"
Jul 29 19:03:25.003197 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:25.003177    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac091bb2-424c-48eb-9434-42e99235f7ce" containerName="payload-processing"
Jul 29 19:03:25.003197 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:25.003187    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f97506fa-1781-446d-aea2-49425566897f" containerName="payload-pre-processing"
Jul 29 19:03:25.003197 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:25.003197    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5678e8a8-87e3-49c7-a578-27e9d6cd3c8e" containerName="revoke-keys"
Jul 29 19:03:25.007708 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:25.007690    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b5315ce-7d74c7966d-mm8zm"
Jul 29 19:03:25.010529 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:25.010498    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1b5315ce-dockercfg-fxm9g\""
Jul 29 19:03:25.010672 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:25.010532    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1b5315ce\""
Jul 29 19:03:25.017148 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:25.017122    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1b5315ce-7d74c7966d-mm8zm"]
Jul 29 19:03:25.035642 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:25.035609    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1b5315ce-766469487d-gndw2"]
Jul 29 19:03:25.038987 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:25.038962    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1b5315ce-766469487d-gndw2"
Jul 29 19:03:25.049638 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:25.049614    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1b5315ce-766469487d-gndw2"]
Jul 29 19:03:25.110955 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:25.110920    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnwqq\" (UniqueName: \"kubernetes.io/projected/a4f0d68c-86ae-44e5-8581-d87ac05130a6-kube-api-access-dnwqq\") pod \"payload-processing-e2e-ait-1b5315ce-766469487d-gndw2\" (UID: \"a4f0d68c-86ae-44e5-8581-d87ac05130a6\") " pod="openshift-ingress/payload-processing-e2e-ait-1b5315ce-766469487d-gndw2"
Jul 29 19:03:25.111140 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:25.110965    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94144a46-30b5-4e53-97a5-1dfe08cd973b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1b5315ce-7d74c7966d-mm8zm\" (UID: \"94144a46-30b5-4e53-97a5-1dfe08cd973b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1b5315ce-7d74c7966d-mm8zm"
Jul 29 19:03:25.111140 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:25.111078    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flrqt\" (UniqueName: \"kubernetes.io/projected/94144a46-30b5-4e53-97a5-1dfe08cd973b-kube-api-access-flrqt\") pod \"payload-pre-processing-e2e-ait-1b5315ce-7d74c7966d-mm8zm\" (UID: \"94144a46-30b5-4e53-97a5-1dfe08cd973b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1b5315ce-7d74c7966d-mm8zm"
Jul 29 19:03:25.111140 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:25.111115    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4f0d68c-86ae-44e5-8581-d87ac05130a6-plugins-config-volume\") pod \"payload-processing-e2e-ait-1b5315ce-766469487d-gndw2\" (UID: \"a4f0d68c-86ae-44e5-8581-d87ac05130a6\") " pod="openshift-ingress/payload-processing-e2e-ait-1b5315ce-766469487d-gndw2"
Jul 29 19:03:25.212068 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:25.212026    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4f0d68c-86ae-44e5-8581-d87ac05130a6-plugins-config-volume\") pod \"payload-processing-e2e-ait-1b5315ce-766469487d-gndw2\" (UID: \"a4f0d68c-86ae-44e5-8581-d87ac05130a6\") " pod="openshift-ingress/payload-processing-e2e-ait-1b5315ce-766469487d-gndw2"
Jul 29 19:03:25.212244 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:25.212099    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnwqq\" (UniqueName: \"kubernetes.io/projected/a4f0d68c-86ae-44e5-8581-d87ac05130a6-kube-api-access-dnwqq\") pod \"payload-processing-e2e-ait-1b5315ce-766469487d-gndw2\" (UID: \"a4f0d68c-86ae-44e5-8581-d87ac05130a6\") " pod="openshift-ingress/payload-processing-e2e-ait-1b5315ce-766469487d-gndw2"
Jul 29 19:03:25.212244 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:25.212134    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94144a46-30b5-4e53-97a5-1dfe08cd973b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1b5315ce-7d74c7966d-mm8zm\" (UID: \"94144a46-30b5-4e53-97a5-1dfe08cd973b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1b5315ce-7d74c7966d-mm8zm"
Jul 29 19:03:25.212244 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:25.212203    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flrqt\" (UniqueName: \"kubernetes.io/projected/94144a46-30b5-4e53-97a5-1dfe08cd973b-kube-api-access-flrqt\") pod \"payload-pre-processing-e2e-ait-1b5315ce-7d74c7966d-mm8zm\" (UID: \"94144a46-30b5-4e53-97a5-1dfe08cd973b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1b5315ce-7d74c7966d-mm8zm"
Jul 29 19:03:25.212915 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:25.212887    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94144a46-30b5-4e53-97a5-1dfe08cd973b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1b5315ce-7d74c7966d-mm8zm\" (UID: \"94144a46-30b5-4e53-97a5-1dfe08cd973b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1b5315ce-7d74c7966d-mm8zm"
Jul 29 19:03:25.212953 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:25.212905    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4f0d68c-86ae-44e5-8581-d87ac05130a6-plugins-config-volume\") pod \"payload-processing-e2e-ait-1b5315ce-766469487d-gndw2\" (UID: \"a4f0d68c-86ae-44e5-8581-d87ac05130a6\") " pod="openshift-ingress/payload-processing-e2e-ait-1b5315ce-766469487d-gndw2"
Jul 29 19:03:25.221003 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:25.220973    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnwqq\" (UniqueName: \"kubernetes.io/projected/a4f0d68c-86ae-44e5-8581-d87ac05130a6-kube-api-access-dnwqq\") pod \"payload-processing-e2e-ait-1b5315ce-766469487d-gndw2\" (UID: \"a4f0d68c-86ae-44e5-8581-d87ac05130a6\") " pod="openshift-ingress/payload-processing-e2e-ait-1b5315ce-766469487d-gndw2"
Jul 29 19:03:25.221456 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:25.221435    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flrqt\" (UniqueName: \"kubernetes.io/projected/94144a46-30b5-4e53-97a5-1dfe08cd973b-kube-api-access-flrqt\") pod \"payload-pre-processing-e2e-ait-1b5315ce-7d74c7966d-mm8zm\" (UID: \"94144a46-30b5-4e53-97a5-1dfe08cd973b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1b5315ce-7d74c7966d-mm8zm"
Jul 29 19:03:25.318796 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:25.318691    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b5315ce-7d74c7966d-mm8zm"
Jul 29 19:03:25.350282 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:25.350248    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1b5315ce-766469487d-gndw2"
Jul 29 19:03:25.479976 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:25.479943    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1b5315ce-7d74c7966d-mm8zm"]
Jul 29 19:03:25.482817 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:03:25.482784    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94144a46_30b5_4e53_97a5_1dfe08cd973b.slice/crio-528a903ad2b47c8e1a83cf4877a6835d75cb5bb6c8882c8c42645a6db05e6b2d WatchSource:0}: Error finding container 528a903ad2b47c8e1a83cf4877a6835d75cb5bb6c8882c8c42645a6db05e6b2d: Status 404 returned error can't find the container with id 528a903ad2b47c8e1a83cf4877a6835d75cb5bb6c8882c8c42645a6db05e6b2d
Jul 29 19:03:25.512043 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:25.512013    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1b5315ce-766469487d-gndw2"]
Jul 29 19:03:25.514553 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:03:25.514524    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4f0d68c_86ae_44e5_8581_d87ac05130a6.slice/crio-34ecfe102cc4f867c0e35432f209baf993ffd1c66e419b4a7c6d418ae766344b WatchSource:0}: Error finding container 34ecfe102cc4f867c0e35432f209baf993ffd1c66e419b4a7c6d418ae766344b: Status 404 returned error can't find the container with id 34ecfe102cc4f867c0e35432f209baf993ffd1c66e419b4a7c6d418ae766344b
Jul 29 19:03:25.615686 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:25.615650    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1b5315ce-766469487d-gndw2" event={"ID":"a4f0d68c-86ae-44e5-8581-d87ac05130a6","Type":"ContainerStarted","Data":"bfea59549659f99795953af8937e0264997c16ee652488c0f4efe44e0ff6455a"}
Jul 29 19:03:25.615866 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:25.615704    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1b5315ce-766469487d-gndw2" event={"ID":"a4f0d68c-86ae-44e5-8581-d87ac05130a6","Type":"ContainerStarted","Data":"34ecfe102cc4f867c0e35432f209baf993ffd1c66e419b4a7c6d418ae766344b"}
Jul 29 19:03:25.615866 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:25.615803    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-1b5315ce-766469487d-gndw2"
Jul 29 19:03:25.617404 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:25.617358    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b5315ce-7d74c7966d-mm8zm" event={"ID":"94144a46-30b5-4e53-97a5-1dfe08cd973b","Type":"ContainerStarted","Data":"ec7d43ab0e15404cb1533045e08a2c9926a248b245c4af1e9eb6d915da9becd9"}
Jul 29 19:03:25.617564 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:25.617434    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b5315ce-7d74c7966d-mm8zm" event={"ID":"94144a46-30b5-4e53-97a5-1dfe08cd973b","Type":"ContainerStarted","Data":"528a903ad2b47c8e1a83cf4877a6835d75cb5bb6c8882c8c42645a6db05e6b2d"}
Jul 29 19:03:25.617564 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:25.617526    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b5315ce-7d74c7966d-mm8zm"
Jul 29 19:03:25.633480 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:25.633422    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-1b5315ce-766469487d-gndw2" podStartSLOduration=0.633402425 podStartE2EDuration="633.402425ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:03:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:03:25.631614052 +0000 UTC m=+1769.901383140" watchObservedRunningTime="2026-07-29 19:03:25.633402425 +0000 UTC m=+1769.903171511"
Jul 29 19:03:25.649541 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:25.649492    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b5315ce-7d74c7966d-mm8zm" podStartSLOduration=1.649475798 podStartE2EDuration="1.649475798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:03:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:03:25.647708023 +0000 UTC m=+1769.917477108" watchObservedRunningTime="2026-07-29 19:03:25.649475798 +0000 UTC m=+1769.919244884"
Jul 29 19:03:35.842496 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:35.842458    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:03:36.627736 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:36.627709    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b5315ce-7d74c7966d-mm8zm"
Jul 29 19:03:36.627964 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:36.627781    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-1b5315ce-766469487d-gndw2"
Jul 29 19:03:50.107650 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:50.107568    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b5315ce-v4jq6"]
Jul 29 19:03:50.111245 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:50.111221    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b5315ce-v4jq6"
Jul 29 19:03:50.113720 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:50.113702    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 19:03:50.113833 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:50.113698    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vgb9z\""
Jul 29 19:03:50.117808 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:50.117776    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b5315ce-v4jq6"]
Jul 29 19:03:50.237013 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:50.236974    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf5f0152-31e8-4de1-a8fa-248dbf7efcb5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1b5315ce-v4jq6\" (UID: \"bf5f0152-31e8-4de1-a8fa-248dbf7efcb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b5315ce-v4jq6"
Jul 29 19:03:50.338184 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:50.338140    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf5f0152-31e8-4de1-a8fa-248dbf7efcb5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1b5315ce-v4jq6\" (UID: \"bf5f0152-31e8-4de1-a8fa-248dbf7efcb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b5315ce-v4jq6"
Jul 29 19:03:50.338817 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:50.338795    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf5f0152-31e8-4de1-a8fa-248dbf7efcb5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1b5315ce-v4jq6\" (UID: \"bf5f0152-31e8-4de1-a8fa-248dbf7efcb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b5315ce-v4jq6"
Jul 29 19:03:50.424311 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:50.424222    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b5315ce-v4jq6"
Jul 29 19:03:50.565906 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:50.565863    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b5315ce-v4jq6"]
Jul 29 19:03:50.568670 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:03:50.568642    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf5f0152_31e8_4de1_a8fa_248dbf7efcb5.slice/crio-abc61455fbc2c8e65738287457eabc8e492229cac2d5b3a9a19143c643944fbc WatchSource:0}: Error finding container abc61455fbc2c8e65738287457eabc8e492229cac2d5b3a9a19143c643944fbc: Status 404 returned error can't find the container with id abc61455fbc2c8e65738287457eabc8e492229cac2d5b3a9a19143c643944fbc
Jul 29 19:03:50.742622 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:50.742586    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b5315ce-v4jq6" event={"ID":"bf5f0152-31e8-4de1-a8fa-248dbf7efcb5","Type":"ContainerStarted","Data":"abc61455fbc2c8e65738287457eabc8e492229cac2d5b3a9a19143c643944fbc"}
Jul 29 19:03:51.749460 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:51.749423    2559 generic.go:356] "Generic (PLEG): container finished" podID="bf5f0152-31e8-4de1-a8fa-248dbf7efcb5" containerID="e28908aa6bffcb2ae48a4ac2425b5bc8705e89aa8ecf0d74bea976248aae5587" exitCode=0
Jul 29 19:03:51.749849 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:51.749512    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b5315ce-v4jq6" event={"ID":"bf5f0152-31e8-4de1-a8fa-248dbf7efcb5","Type":"ContainerDied","Data":"e28908aa6bffcb2ae48a4ac2425b5bc8705e89aa8ecf0d74bea976248aae5587"}
Jul 29 19:03:52.895190 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:52.895162    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b5315ce-v4jq6"
Jul 29 19:03:53.067899 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:53.067797    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf5f0152-31e8-4de1-a8fa-248dbf7efcb5-maas-api-service-ca\") pod \"bf5f0152-31e8-4de1-a8fa-248dbf7efcb5\" (UID: \"bf5f0152-31e8-4de1-a8fa-248dbf7efcb5\") "
Jul 29 19:03:53.068156 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:53.068135    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf5f0152-31e8-4de1-a8fa-248dbf7efcb5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bf5f0152-31e8-4de1-a8fa-248dbf7efcb5" (UID: "bf5f0152-31e8-4de1-a8fa-248dbf7efcb5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:03:53.169163 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:53.169125    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf5f0152-31e8-4de1-a8fa-248dbf7efcb5-maas-api-service-ca\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:03:53.761098 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:53.761066    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b5315ce-v4jq6"
Jul 29 19:03:53.761273 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:53.761108    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b5315ce-v4jq6" event={"ID":"bf5f0152-31e8-4de1-a8fa-248dbf7efcb5","Type":"ContainerDied","Data":"abc61455fbc2c8e65738287457eabc8e492229cac2d5b3a9a19143c643944fbc"}
Jul 29 19:03:53.761273 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:53.761156    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abc61455fbc2c8e65738287457eabc8e492229cac2d5b3a9a19143c643944fbc"
Jul 29 19:03:54.985148 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:54.985112    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1b5315ce-766469487d-gndw2"]
Jul 29 19:03:54.985647 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:54.985348    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-1b5315ce-766469487d-gndw2" podUID="a4f0d68c-86ae-44e5-8581-d87ac05130a6" containerName="payload-processing" containerID="cri-o://bfea59549659f99795953af8937e0264997c16ee652488c0f4efe44e0ff6455a" gracePeriod=30
Jul 29 19:03:54.993710 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:54.993677    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1b5315ce-7d74c7966d-mm8zm"]
Jul 29 19:03:54.994166 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:54.994135    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b5315ce-7d74c7966d-mm8zm" podUID="94144a46-30b5-4e53-97a5-1dfe08cd973b" containerName="payload-pre-processing" containerID="cri-o://ec7d43ab0e15404cb1533045e08a2c9926a248b245c4af1e9eb6d915da9becd9" gracePeriod=30
Jul 29 19:03:55.129693 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:55.129658    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b5315ce-v4jq6"]
Jul 29 19:03:55.132518 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:55.132486    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b5315ce-v4jq6"]
Jul 29 19:03:55.277972 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:55.277945    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b5315ce-7d74c7966d-mm8zm"
Jul 29 19:03:55.280961 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:55.280940    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1b5315ce-766469487d-gndw2"
Jul 29 19:03:55.388061 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:55.388025    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnwqq\" (UniqueName: \"kubernetes.io/projected/a4f0d68c-86ae-44e5-8581-d87ac05130a6-kube-api-access-dnwqq\") pod \"a4f0d68c-86ae-44e5-8581-d87ac05130a6\" (UID: \"a4f0d68c-86ae-44e5-8581-d87ac05130a6\") "
Jul 29 19:03:55.388273 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:55.388085    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4f0d68c-86ae-44e5-8581-d87ac05130a6-plugins-config-volume\") pod \"a4f0d68c-86ae-44e5-8581-d87ac05130a6\" (UID: \"a4f0d68c-86ae-44e5-8581-d87ac05130a6\") "
Jul 29 19:03:55.388273 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:55.388116    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-flrqt\" (UniqueName: \"kubernetes.io/projected/94144a46-30b5-4e53-97a5-1dfe08cd973b-kube-api-access-flrqt\") pod \"94144a46-30b5-4e53-97a5-1dfe08cd973b\" (UID: \"94144a46-30b5-4e53-97a5-1dfe08cd973b\") "
Jul 29 19:03:55.388273 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:55.388243    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94144a46-30b5-4e53-97a5-1dfe08cd973b-plugins-config-volume\") pod \"94144a46-30b5-4e53-97a5-1dfe08cd973b\" (UID: \"94144a46-30b5-4e53-97a5-1dfe08cd973b\") "
Jul 29 19:03:55.388562 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:55.388535    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4f0d68c-86ae-44e5-8581-d87ac05130a6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a4f0d68c-86ae-44e5-8581-d87ac05130a6" (UID: "a4f0d68c-86ae-44e5-8581-d87ac05130a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:03:55.388610 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:55.388599    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94144a46-30b5-4e53-97a5-1dfe08cd973b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "94144a46-30b5-4e53-97a5-1dfe08cd973b" (UID: "94144a46-30b5-4e53-97a5-1dfe08cd973b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:03:55.391045 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:55.391021    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4f0d68c-86ae-44e5-8581-d87ac05130a6-kube-api-access-dnwqq" (OuterVolumeSpecName: "kube-api-access-dnwqq") pod "a4f0d68c-86ae-44e5-8581-d87ac05130a6" (UID: "a4f0d68c-86ae-44e5-8581-d87ac05130a6"). InnerVolumeSpecName "kube-api-access-dnwqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:03:55.391313 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:55.391295    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94144a46-30b5-4e53-97a5-1dfe08cd973b-kube-api-access-flrqt" (OuterVolumeSpecName: "kube-api-access-flrqt") pod "94144a46-30b5-4e53-97a5-1dfe08cd973b" (UID: "94144a46-30b5-4e53-97a5-1dfe08cd973b"). InnerVolumeSpecName "kube-api-access-flrqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:03:55.489261 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:55.489207    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnwqq\" (UniqueName: \"kubernetes.io/projected/a4f0d68c-86ae-44e5-8581-d87ac05130a6-kube-api-access-dnwqq\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:03:55.489261 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:55.489253    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4f0d68c-86ae-44e5-8581-d87ac05130a6-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:03:55.489261 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:55.489264    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flrqt\" (UniqueName: \"kubernetes.io/projected/94144a46-30b5-4e53-97a5-1dfe08cd973b-kube-api-access-flrqt\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:03:55.489261 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:55.489275    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94144a46-30b5-4e53-97a5-1dfe08cd973b-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:03:55.772563 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:55.772524    2559 generic.go:356] "Generic (PLEG): container finished" podID="94144a46-30b5-4e53-97a5-1dfe08cd973b" containerID="ec7d43ab0e15404cb1533045e08a2c9926a248b245c4af1e9eb6d915da9becd9" exitCode=0
Jul 29 19:03:55.772758 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:55.772593    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b5315ce-7d74c7966d-mm8zm"
Jul 29 19:03:55.772758 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:55.772614    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b5315ce-7d74c7966d-mm8zm" event={"ID":"94144a46-30b5-4e53-97a5-1dfe08cd973b","Type":"ContainerDied","Data":"ec7d43ab0e15404cb1533045e08a2c9926a248b245c4af1e9eb6d915da9becd9"}
Jul 29 19:03:55.772758 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:55.772702    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b5315ce-7d74c7966d-mm8zm" event={"ID":"94144a46-30b5-4e53-97a5-1dfe08cd973b","Type":"ContainerDied","Data":"528a903ad2b47c8e1a83cf4877a6835d75cb5bb6c8882c8c42645a6db05e6b2d"}
Jul 29 19:03:55.772758 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:55.772737    2559 scope.go:117] "RemoveContainer" containerID="ec7d43ab0e15404cb1533045e08a2c9926a248b245c4af1e9eb6d915da9becd9"
Jul 29 19:03:55.773930 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:55.773908    2559 generic.go:356] "Generic (PLEG): container finished" podID="a4f0d68c-86ae-44e5-8581-d87ac05130a6" containerID="bfea59549659f99795953af8937e0264997c16ee652488c0f4efe44e0ff6455a" exitCode=0
Jul 29 19:03:55.774023 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:55.773952    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1b5315ce-766469487d-gndw2" event={"ID":"a4f0d68c-86ae-44e5-8581-d87ac05130a6","Type":"ContainerDied","Data":"bfea59549659f99795953af8937e0264997c16ee652488c0f4efe44e0ff6455a"}
Jul 29 19:03:55.774023 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:55.773962    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1b5315ce-766469487d-gndw2"
Jul 29 19:03:55.774023 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:55.773988    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1b5315ce-766469487d-gndw2" event={"ID":"a4f0d68c-86ae-44e5-8581-d87ac05130a6","Type":"ContainerDied","Data":"34ecfe102cc4f867c0e35432f209baf993ffd1c66e419b4a7c6d418ae766344b"}
Jul 29 19:03:55.782944 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:55.782820    2559 scope.go:117] "RemoveContainer" containerID="ec7d43ab0e15404cb1533045e08a2c9926a248b245c4af1e9eb6d915da9becd9"
Jul 29 19:03:55.783276 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:03:55.783251    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec7d43ab0e15404cb1533045e08a2c9926a248b245c4af1e9eb6d915da9becd9\": container with ID starting with ec7d43ab0e15404cb1533045e08a2c9926a248b245c4af1e9eb6d915da9becd9 not found: ID does not exist" containerID="ec7d43ab0e15404cb1533045e08a2c9926a248b245c4af1e9eb6d915da9becd9"
Jul 29 19:03:55.783367 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:55.783287    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec7d43ab0e15404cb1533045e08a2c9926a248b245c4af1e9eb6d915da9becd9"} err="failed to get container status \"ec7d43ab0e15404cb1533045e08a2c9926a248b245c4af1e9eb6d915da9becd9\": rpc error: code = NotFound desc = could not find container \"ec7d43ab0e15404cb1533045e08a2c9926a248b245c4af1e9eb6d915da9becd9\": container with ID starting with ec7d43ab0e15404cb1533045e08a2c9926a248b245c4af1e9eb6d915da9becd9 not found: ID does not exist"
Jul 29 19:03:55.783367 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:55.783313    2559 scope.go:117] "RemoveContainer" containerID="bfea59549659f99795953af8937e0264997c16ee652488c0f4efe44e0ff6455a"
Jul 29 19:03:55.793189 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:55.793172    2559 scope.go:117] "RemoveContainer" containerID="bfea59549659f99795953af8937e0264997c16ee652488c0f4efe44e0ff6455a"
Jul 29 19:03:55.793553 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:03:55.793525    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfea59549659f99795953af8937e0264997c16ee652488c0f4efe44e0ff6455a\": container with ID starting with bfea59549659f99795953af8937e0264997c16ee652488c0f4efe44e0ff6455a not found: ID does not exist" containerID="bfea59549659f99795953af8937e0264997c16ee652488c0f4efe44e0ff6455a"
Jul 29 19:03:55.793655 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:55.793556    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfea59549659f99795953af8937e0264997c16ee652488c0f4efe44e0ff6455a"} err="failed to get container status \"bfea59549659f99795953af8937e0264997c16ee652488c0f4efe44e0ff6455a\": rpc error: code = NotFound desc = could not find container \"bfea59549659f99795953af8937e0264997c16ee652488c0f4efe44e0ff6455a\": container with ID starting with bfea59549659f99795953af8937e0264997c16ee652488c0f4efe44e0ff6455a not found: ID does not exist"
Jul 29 19:03:55.799394 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:55.799352    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1b5315ce-7d74c7966d-mm8zm"]
Jul 29 19:03:55.804824 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:55.804795    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1b5315ce-7d74c7966d-mm8zm"]
Jul 29 19:03:55.814892 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:55.814861    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1b5315ce-766469487d-gndw2"]
Jul 29 19:03:55.822624 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:55.822589    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1b5315ce-766469487d-gndw2"]
Jul 29 19:03:56.150487 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:56.150459    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94144a46-30b5-4e53-97a5-1dfe08cd973b" path="/var/lib/kubelet/pods/94144a46-30b5-4e53-97a5-1dfe08cd973b/volumes"
Jul 29 19:03:56.150846 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:56.150801    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4f0d68c-86ae-44e5-8581-d87ac05130a6" path="/var/lib/kubelet/pods/a4f0d68c-86ae-44e5-8581-d87ac05130a6/volumes"
Jul 29 19:03:56.151068 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:56.151056    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf5f0152-31e8-4de1-a8fa-248dbf7efcb5" path="/var/lib/kubelet/pods/bf5f0152-31e8-4de1-a8fa-248dbf7efcb5/volumes"
Jul 29 19:03:56.186889 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:56.186851    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-v9nsr_7f8b4737-d4c2-4598-bd40-208015542831/console-operator/1.log"
Jul 29 19:03:56.188681 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:56.188653    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-v9nsr_7f8b4737-d4c2-4598-bd40-208015542831/console-operator/1.log"
Jul 29 19:03:56.194982 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:56.194958    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjdrx_60f444e1-7e31-443c-8ec7-a4d04c64bf4b/ovn-acl-logging/0.log"
Jul 29 19:03:56.196933 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:56.196913    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjdrx_60f444e1-7e31-443c-8ec7-a4d04c64bf4b/ovn-acl-logging/0.log"
Jul 29 19:03:59.137900 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:03:59.137859    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:04:00.197526 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:00.197487    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-77117817-97df84c48-lg56k"]
Jul 29 19:04:00.197925 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:00.197823    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4f0d68c-86ae-44e5-8581-d87ac05130a6" containerName="payload-processing"
Jul 29 19:04:00.197925 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:00.197834    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4f0d68c-86ae-44e5-8581-d87ac05130a6" containerName="payload-processing"
Jul 29 19:04:00.197925 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:00.197850    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf5f0152-31e8-4de1-a8fa-248dbf7efcb5" containerName="revoke-keys"
Jul 29 19:04:00.197925 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:00.197855    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf5f0152-31e8-4de1-a8fa-248dbf7efcb5" containerName="revoke-keys"
Jul 29 19:04:00.197925 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:00.197870    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94144a46-30b5-4e53-97a5-1dfe08cd973b" containerName="payload-pre-processing"
Jul 29 19:04:00.197925 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:00.197876    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="94144a46-30b5-4e53-97a5-1dfe08cd973b" containerName="payload-pre-processing"
Jul 29 19:04:00.198159 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:00.197931    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf5f0152-31e8-4de1-a8fa-248dbf7efcb5" containerName="revoke-keys"
Jul 29 19:04:00.198159 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:00.197941    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="94144a46-30b5-4e53-97a5-1dfe08cd973b" containerName="payload-pre-processing"
Jul 29 19:04:00.198159 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:00.197949    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4f0d68c-86ae-44e5-8581-d87ac05130a6" containerName="payload-processing"
Jul 29 19:04:00.202454 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:00.202431    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-77117817-97df84c48-lg56k"
Jul 29 19:04:00.205050 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:00.205028    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-77117817\""
Jul 29 19:04:00.210214 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:00.210189    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-77117817-97df84c48-lg56k"]
Jul 29 19:04:00.231514 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:00.231474    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-77117817-6848f9f758-9rmp2"]
Jul 29 19:04:00.234904 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:00.234881    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-77117817-6848f9f758-9rmp2"
Jul 29 19:04:00.237212 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:00.237189    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-77117817-dockercfg-cxjwd\""
Jul 29 19:04:00.245540 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:00.245506    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-77117817-6848f9f758-9rmp2"]
Jul 29 19:04:00.334134 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:00.334100    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-789fj\" (UniqueName: \"kubernetes.io/projected/4748ab16-ab1f-4879-86cc-480bfa4b5196-kube-api-access-789fj\") pod \"payload-processing-e2e-derive-77117817-6848f9f758-9rmp2\" (UID: \"4748ab16-ab1f-4879-86cc-480bfa4b5196\") " pod="openshift-ingress/payload-processing-e2e-derive-77117817-6848f9f758-9rmp2"
Jul 29 19:04:00.334134 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:00.334137    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4748ab16-ab1f-4879-86cc-480bfa4b5196-plugins-config-volume\") pod \"payload-processing-e2e-derive-77117817-6848f9f758-9rmp2\" (UID: \"4748ab16-ab1f-4879-86cc-480bfa4b5196\") " pod="openshift-ingress/payload-processing-e2e-derive-77117817-6848f9f758-9rmp2"
Jul 29 19:04:00.334326 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:00.334244    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2ntz\" (UniqueName: \"kubernetes.io/projected/592094ee-9275-4c37-b09f-a61260cfb6f5-kube-api-access-h2ntz\") pod \"payload-pre-processing-e2e-derive-77117817-97df84c48-lg56k\" (UID: \"592094ee-9275-4c37-b09f-a61260cfb6f5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-77117817-97df84c48-lg56k"
Jul 29 19:04:00.334326 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:00.334272    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/592094ee-9275-4c37-b09f-a61260cfb6f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-77117817-97df84c48-lg56k\" (UID: \"592094ee-9275-4c37-b09f-a61260cfb6f5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-77117817-97df84c48-lg56k"
Jul 29 19:04:00.435073 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:00.435027    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-789fj\" (UniqueName: \"kubernetes.io/projected/4748ab16-ab1f-4879-86cc-480bfa4b5196-kube-api-access-789fj\") pod \"payload-processing-e2e-derive-77117817-6848f9f758-9rmp2\" (UID: \"4748ab16-ab1f-4879-86cc-480bfa4b5196\") " pod="openshift-ingress/payload-processing-e2e-derive-77117817-6848f9f758-9rmp2"
Jul 29 19:04:00.435270 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:00.435083    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4748ab16-ab1f-4879-86cc-480bfa4b5196-plugins-config-volume\") pod \"payload-processing-e2e-derive-77117817-6848f9f758-9rmp2\" (UID: \"4748ab16-ab1f-4879-86cc-480bfa4b5196\") " pod="openshift-ingress/payload-processing-e2e-derive-77117817-6848f9f758-9rmp2"
Jul 29 19:04:00.435270 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:00.435165    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2ntz\" (UniqueName: \"kubernetes.io/projected/592094ee-9275-4c37-b09f-a61260cfb6f5-kube-api-access-h2ntz\") pod \"payload-pre-processing-e2e-derive-77117817-97df84c48-lg56k\" (UID: \"592094ee-9275-4c37-b09f-a61260cfb6f5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-77117817-97df84c48-lg56k"
Jul 29 19:04:00.435270 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:00.435191    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/592094ee-9275-4c37-b09f-a61260cfb6f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-77117817-97df84c48-lg56k\" (UID: \"592094ee-9275-4c37-b09f-a61260cfb6f5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-77117817-97df84c48-lg56k"
Jul 29 19:04:00.436080 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:00.436055    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/592094ee-9275-4c37-b09f-a61260cfb6f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-77117817-97df84c48-lg56k\" (UID: \"592094ee-9275-4c37-b09f-a61260cfb6f5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-77117817-97df84c48-lg56k"
Jul 29 19:04:00.436135 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:00.436055    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4748ab16-ab1f-4879-86cc-480bfa4b5196-plugins-config-volume\") pod \"payload-processing-e2e-derive-77117817-6848f9f758-9rmp2\" (UID: \"4748ab16-ab1f-4879-86cc-480bfa4b5196\") " pod="openshift-ingress/payload-processing-e2e-derive-77117817-6848f9f758-9rmp2"
Jul 29 19:04:00.444870 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:00.444843    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2ntz\" (UniqueName: \"kubernetes.io/projected/592094ee-9275-4c37-b09f-a61260cfb6f5-kube-api-access-h2ntz\") pod \"payload-pre-processing-e2e-derive-77117817-97df84c48-lg56k\" (UID: \"592094ee-9275-4c37-b09f-a61260cfb6f5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-77117817-97df84c48-lg56k"
Jul 29 19:04:00.445202 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:00.445183    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-789fj\" (UniqueName: \"kubernetes.io/projected/4748ab16-ab1f-4879-86cc-480bfa4b5196-kube-api-access-789fj\") pod \"payload-processing-e2e-derive-77117817-6848f9f758-9rmp2\" (UID: \"4748ab16-ab1f-4879-86cc-480bfa4b5196\") " pod="openshift-ingress/payload-processing-e2e-derive-77117817-6848f9f758-9rmp2"
Jul 29 19:04:00.514440 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:00.514317    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-77117817-97df84c48-lg56k"
Jul 29 19:04:00.547156 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:00.547124    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-77117817-6848f9f758-9rmp2"
Jul 29 19:04:00.686591 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:00.686546    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-77117817-97df84c48-lg56k"]
Jul 29 19:04:00.691324 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:04:00.691273    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod592094ee_9275_4c37_b09f_a61260cfb6f5.slice/crio-2d5cd74e8c4e0b80030114d3679c3ab3b7ccfdadbd61ae8f26219495427a7c67 WatchSource:0}: Error finding container 2d5cd74e8c4e0b80030114d3679c3ab3b7ccfdadbd61ae8f26219495427a7c67: Status 404 returned error can't find the container with id 2d5cd74e8c4e0b80030114d3679c3ab3b7ccfdadbd61ae8f26219495427a7c67
Jul 29 19:04:00.723403 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:00.723354    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-77117817-6848f9f758-9rmp2"]
Jul 29 19:04:00.726458 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:04:00.726433    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4748ab16_ab1f_4879_86cc_480bfa4b5196.slice/crio-430fb5cdb60bcdfb61a4f861220959745f73539b3ad891bbd8d992889af47ed8 WatchSource:0}: Error finding container 430fb5cdb60bcdfb61a4f861220959745f73539b3ad891bbd8d992889af47ed8: Status 404 returned error can't find the container with id 430fb5cdb60bcdfb61a4f861220959745f73539b3ad891bbd8d992889af47ed8
Jul 29 19:04:00.802536 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:00.802496    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-77117817-97df84c48-lg56k" event={"ID":"592094ee-9275-4c37-b09f-a61260cfb6f5","Type":"ContainerStarted","Data":"273194a3b8a173ad4cccd2ce964f301df5f6df1fdaa825e1fee99f29ed47b002"}
Jul 29 19:04:00.802757 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:00.802565    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-77117817-97df84c48-lg56k" event={"ID":"592094ee-9275-4c37-b09f-a61260cfb6f5","Type":"ContainerStarted","Data":"2d5cd74e8c4e0b80030114d3679c3ab3b7ccfdadbd61ae8f26219495427a7c67"}
Jul 29 19:04:00.802757 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:00.802598    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-77117817-97df84c48-lg56k"
Jul 29 19:04:00.804052 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:00.804022    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-77117817-6848f9f758-9rmp2" event={"ID":"4748ab16-ab1f-4879-86cc-480bfa4b5196","Type":"ContainerStarted","Data":"779c3bdf3df3847a4a5fc574fb16fcba37e607c6154ee84d4c3998390e6fd08c"}
Jul 29 19:04:00.804199 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:00.804064    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-77117817-6848f9f758-9rmp2" event={"ID":"4748ab16-ab1f-4879-86cc-480bfa4b5196","Type":"ContainerStarted","Data":"430fb5cdb60bcdfb61a4f861220959745f73539b3ad891bbd8d992889af47ed8"}
Jul 29 19:04:00.804199 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:00.804128    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-77117817-6848f9f758-9rmp2"
Jul 29 19:04:00.820141 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:00.820085    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-77117817-97df84c48-lg56k" podStartSLOduration=0.820069798 podStartE2EDuration="820.069798ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:04:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:04:00.817983653 +0000 UTC m=+1805.087752794" watchObservedRunningTime="2026-07-29 19:04:00.820069798 +0000 UTC m=+1805.089838885"
Jul 29 19:04:00.840677 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:00.840619    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-77117817-6848f9f758-9rmp2" podStartSLOduration=0.840599367 podStartE2EDuration="840.599367ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:04:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:04:00.838842334 +0000 UTC m=+1805.108611417" watchObservedRunningTime="2026-07-29 19:04:00.840599367 +0000 UTC m=+1805.110368453"
Jul 29 19:04:10.831071 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:10.831030    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:04:11.811916 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:11.811886    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-77117817-6848f9f758-9rmp2"
Jul 29 19:04:11.812085 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:11.811954    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-77117817-97df84c48-lg56k"
Jul 29 19:04:24.278874 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:24.278839    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-77117817-5k2lb"]
Jul 29 19:04:24.283637 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:24.283616    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-77117817-5k2lb"
Jul 29 19:04:24.286208 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:24.286184    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 19:04:24.286208 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:24.286202    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vgb9z\""
Jul 29 19:04:24.291763 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:24.291048    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-77117817-5k2lb"]
Jul 29 19:04:24.341567 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:24.341528    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acea07db-3f43-463f-a7f5-58b2d0dcac77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-77117817-5k2lb\" (UID: \"acea07db-3f43-463f-a7f5-58b2d0dcac77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-77117817-5k2lb"
Jul 29 19:04:24.442627 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:24.442585    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acea07db-3f43-463f-a7f5-58b2d0dcac77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-77117817-5k2lb\" (UID: \"acea07db-3f43-463f-a7f5-58b2d0dcac77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-77117817-5k2lb"
Jul 29 19:04:24.443261 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:24.443238    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acea07db-3f43-463f-a7f5-58b2d0dcac77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-77117817-5k2lb\" (UID: \"acea07db-3f43-463f-a7f5-58b2d0dcac77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-77117817-5k2lb"
Jul 29 19:04:24.595237 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:24.595206    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-77117817-5k2lb"
Jul 29 19:04:24.733985 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:24.733912    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-77117817-5k2lb"]
Jul 29 19:04:24.736299 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:04:24.736273    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacea07db_3f43_463f_a7f5_58b2d0dcac77.slice/crio-5cb283aff9993217d42a1eb073926a349b2947f21d0ea875dd4fbb5394dfd538 WatchSource:0}: Error finding container 5cb283aff9993217d42a1eb073926a349b2947f21d0ea875dd4fbb5394dfd538: Status 404 returned error can't find the container with id 5cb283aff9993217d42a1eb073926a349b2947f21d0ea875dd4fbb5394dfd538
Jul 29 19:04:24.926851 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:24.926769    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-77117817-5k2lb" event={"ID":"acea07db-3f43-463f-a7f5-58b2d0dcac77","Type":"ContainerStarted","Data":"5cb283aff9993217d42a1eb073926a349b2947f21d0ea875dd4fbb5394dfd538"}
Jul 29 19:04:25.934522 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:25.934482    2559 generic.go:356] "Generic (PLEG): container finished" podID="acea07db-3f43-463f-a7f5-58b2d0dcac77" containerID="f6ac213d42d01707ffaaeafa11db4dc6fdc1c90d0440f5cb5c330ebe9dd4265c" exitCode=0
Jul 29 19:04:25.934923 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:25.934570    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-77117817-5k2lb" event={"ID":"acea07db-3f43-463f-a7f5-58b2d0dcac77","Type":"ContainerDied","Data":"f6ac213d42d01707ffaaeafa11db4dc6fdc1c90d0440f5cb5c330ebe9dd4265c"}
Jul 29 19:04:27.080355 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:27.080324    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-77117817-5k2lb"
Jul 29 19:04:27.168775 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:27.168743    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acea07db-3f43-463f-a7f5-58b2d0dcac77-maas-api-service-ca\") pod \"acea07db-3f43-463f-a7f5-58b2d0dcac77\" (UID: \"acea07db-3f43-463f-a7f5-58b2d0dcac77\") "
Jul 29 19:04:27.169164 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:27.169139    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acea07db-3f43-463f-a7f5-58b2d0dcac77-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "acea07db-3f43-463f-a7f5-58b2d0dcac77" (UID: "acea07db-3f43-463f-a7f5-58b2d0dcac77"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:04:27.270323 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:27.270202    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acea07db-3f43-463f-a7f5-58b2d0dcac77-maas-api-service-ca\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:04:27.945528 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:27.945499    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-77117817-5k2lb"
Jul 29 19:04:27.945742 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:27.945499    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-77117817-5k2lb" event={"ID":"acea07db-3f43-463f-a7f5-58b2d0dcac77","Type":"ContainerDied","Data":"5cb283aff9993217d42a1eb073926a349b2947f21d0ea875dd4fbb5394dfd538"}
Jul 29 19:04:27.945742 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:27.945633    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cb283aff9993217d42a1eb073926a349b2947f21d0ea875dd4fbb5394dfd538"
Jul 29 19:04:30.025985 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:30.025942    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-77117817-6848f9f758-9rmp2"]
Jul 29 19:04:30.026503 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:30.026148    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-77117817-6848f9f758-9rmp2" podUID="4748ab16-ab1f-4879-86cc-480bfa4b5196" containerName="payload-processing" containerID="cri-o://779c3bdf3df3847a4a5fc574fb16fcba37e607c6154ee84d4c3998390e6fd08c" gracePeriod=30
Jul 29 19:04:30.041835 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:30.041800    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-77117817-97df84c48-lg56k"]
Jul 29 19:04:30.042086 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:30.042061    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-77117817-97df84c48-lg56k" podUID="592094ee-9275-4c37-b09f-a61260cfb6f5" containerName="payload-pre-processing" containerID="cri-o://273194a3b8a173ad4cccd2ce964f301df5f6df1fdaa825e1fee99f29ed47b002" gracePeriod=30
Jul 29 19:04:30.190320 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:30.190282    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-77117817-5k2lb"]
Jul 29 19:04:30.194511 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:30.194475    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-77117817-5k2lb"]
Jul 29 19:04:30.321516 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:30.321131    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-77117817-6848f9f758-9rmp2"
Jul 29 19:04:30.326361 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:30.326249    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-77117817-97df84c48-lg56k"
Jul 29 19:04:30.399328 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:30.399296    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4748ab16-ab1f-4879-86cc-480bfa4b5196-plugins-config-volume\") pod \"4748ab16-ab1f-4879-86cc-480bfa4b5196\" (UID: \"4748ab16-ab1f-4879-86cc-480bfa4b5196\") "
Jul 29 19:04:30.399583 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:30.399410    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-789fj\" (UniqueName: \"kubernetes.io/projected/4748ab16-ab1f-4879-86cc-480bfa4b5196-kube-api-access-789fj\") pod \"4748ab16-ab1f-4879-86cc-480bfa4b5196\" (UID: \"4748ab16-ab1f-4879-86cc-480bfa4b5196\") "
Jul 29 19:04:30.399583 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:30.399458    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/592094ee-9275-4c37-b09f-a61260cfb6f5-plugins-config-volume\") pod \"592094ee-9275-4c37-b09f-a61260cfb6f5\" (UID: \"592094ee-9275-4c37-b09f-a61260cfb6f5\") "
Jul 29 19:04:30.399583 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:30.399483    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2ntz\" (UniqueName: \"kubernetes.io/projected/592094ee-9275-4c37-b09f-a61260cfb6f5-kube-api-access-h2ntz\") pod \"592094ee-9275-4c37-b09f-a61260cfb6f5\" (UID: \"592094ee-9275-4c37-b09f-a61260cfb6f5\") "
Jul 29 19:04:30.399855 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:30.399819    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4748ab16-ab1f-4879-86cc-480bfa4b5196-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4748ab16-ab1f-4879-86cc-480bfa4b5196" (UID: "4748ab16-ab1f-4879-86cc-480bfa4b5196"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:04:30.399855 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:30.399846    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/592094ee-9275-4c37-b09f-a61260cfb6f5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "592094ee-9275-4c37-b09f-a61260cfb6f5" (UID: "592094ee-9275-4c37-b09f-a61260cfb6f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:04:30.401826 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:30.401792    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4748ab16-ab1f-4879-86cc-480bfa4b5196-kube-api-access-789fj" (OuterVolumeSpecName: "kube-api-access-789fj") pod "4748ab16-ab1f-4879-86cc-480bfa4b5196" (UID: "4748ab16-ab1f-4879-86cc-480bfa4b5196"). InnerVolumeSpecName "kube-api-access-789fj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:04:30.402075 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:30.402051    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/592094ee-9275-4c37-b09f-a61260cfb6f5-kube-api-access-h2ntz" (OuterVolumeSpecName: "kube-api-access-h2ntz") pod "592094ee-9275-4c37-b09f-a61260cfb6f5" (UID: "592094ee-9275-4c37-b09f-a61260cfb6f5"). InnerVolumeSpecName "kube-api-access-h2ntz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:04:30.501067 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:30.501023    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/592094ee-9275-4c37-b09f-a61260cfb6f5-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:04:30.501067 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:30.501061    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2ntz\" (UniqueName: \"kubernetes.io/projected/592094ee-9275-4c37-b09f-a61260cfb6f5-kube-api-access-h2ntz\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:04:30.501067 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:30.501070    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4748ab16-ab1f-4879-86cc-480bfa4b5196-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:04:30.501067 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:30.501078    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-789fj\" (UniqueName: \"kubernetes.io/projected/4748ab16-ab1f-4879-86cc-480bfa4b5196-kube-api-access-789fj\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:04:30.960932 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:30.960887    2559 generic.go:356] "Generic (PLEG): container finished" podID="4748ab16-ab1f-4879-86cc-480bfa4b5196" containerID="779c3bdf3df3847a4a5fc574fb16fcba37e607c6154ee84d4c3998390e6fd08c" exitCode=0
Jul 29 19:04:30.961107 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:30.960940    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-77117817-6848f9f758-9rmp2"
Jul 29 19:04:30.961107 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:30.960970    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-77117817-6848f9f758-9rmp2" event={"ID":"4748ab16-ab1f-4879-86cc-480bfa4b5196","Type":"ContainerDied","Data":"779c3bdf3df3847a4a5fc574fb16fcba37e607c6154ee84d4c3998390e6fd08c"}
Jul 29 19:04:30.961107 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:30.961027    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-77117817-6848f9f758-9rmp2" event={"ID":"4748ab16-ab1f-4879-86cc-480bfa4b5196","Type":"ContainerDied","Data":"430fb5cdb60bcdfb61a4f861220959745f73539b3ad891bbd8d992889af47ed8"}
Jul 29 19:04:30.961107 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:30.961056    2559 scope.go:117] "RemoveContainer" containerID="779c3bdf3df3847a4a5fc574fb16fcba37e607c6154ee84d4c3998390e6fd08c"
Jul 29 19:04:30.962365 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:30.962342    2559 generic.go:356] "Generic (PLEG): container finished" podID="592094ee-9275-4c37-b09f-a61260cfb6f5" containerID="273194a3b8a173ad4cccd2ce964f301df5f6df1fdaa825e1fee99f29ed47b002" exitCode=0
Jul 29 19:04:30.962483 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:30.962397    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-77117817-97df84c48-lg56k" event={"ID":"592094ee-9275-4c37-b09f-a61260cfb6f5","Type":"ContainerDied","Data":"273194a3b8a173ad4cccd2ce964f301df5f6df1fdaa825e1fee99f29ed47b002"}
Jul 29 19:04:30.962483 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:30.962404    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-77117817-97df84c48-lg56k"
Jul 29 19:04:30.962483 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:30.962436    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-77117817-97df84c48-lg56k" event={"ID":"592094ee-9275-4c37-b09f-a61260cfb6f5","Type":"ContainerDied","Data":"2d5cd74e8c4e0b80030114d3679c3ab3b7ccfdadbd61ae8f26219495427a7c67"}
Jul 29 19:04:30.970726 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:30.970704    2559 scope.go:117] "RemoveContainer" containerID="779c3bdf3df3847a4a5fc574fb16fcba37e607c6154ee84d4c3998390e6fd08c"
Jul 29 19:04:30.971021 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:04:30.971001    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"779c3bdf3df3847a4a5fc574fb16fcba37e607c6154ee84d4c3998390e6fd08c\": container with ID starting with 779c3bdf3df3847a4a5fc574fb16fcba37e607c6154ee84d4c3998390e6fd08c not found: ID does not exist" containerID="779c3bdf3df3847a4a5fc574fb16fcba37e607c6154ee84d4c3998390e6fd08c"
Jul 29 19:04:30.971089 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:30.971040    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"779c3bdf3df3847a4a5fc574fb16fcba37e607c6154ee84d4c3998390e6fd08c"} err="failed to get container status \"779c3bdf3df3847a4a5fc574fb16fcba37e607c6154ee84d4c3998390e6fd08c\": rpc error: code = NotFound desc = could not find container \"779c3bdf3df3847a4a5fc574fb16fcba37e607c6154ee84d4c3998390e6fd08c\": container with ID starting with 779c3bdf3df3847a4a5fc574fb16fcba37e607c6154ee84d4c3998390e6fd08c not found: ID does not exist"
Jul 29 19:04:30.971089 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:30.971062    2559 scope.go:117] "RemoveContainer" containerID="273194a3b8a173ad4cccd2ce964f301df5f6df1fdaa825e1fee99f29ed47b002"
Jul 29 19:04:30.979640 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:30.979613    2559 scope.go:117] "RemoveContainer" containerID="273194a3b8a173ad4cccd2ce964f301df5f6df1fdaa825e1fee99f29ed47b002"
Jul 29 19:04:30.979931 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:04:30.979912    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"273194a3b8a173ad4cccd2ce964f301df5f6df1fdaa825e1fee99f29ed47b002\": container with ID starting with 273194a3b8a173ad4cccd2ce964f301df5f6df1fdaa825e1fee99f29ed47b002 not found: ID does not exist" containerID="273194a3b8a173ad4cccd2ce964f301df5f6df1fdaa825e1fee99f29ed47b002"
Jul 29 19:04:30.979998 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:30.979943    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"273194a3b8a173ad4cccd2ce964f301df5f6df1fdaa825e1fee99f29ed47b002"} err="failed to get container status \"273194a3b8a173ad4cccd2ce964f301df5f6df1fdaa825e1fee99f29ed47b002\": rpc error: code = NotFound desc = could not find container \"273194a3b8a173ad4cccd2ce964f301df5f6df1fdaa825e1fee99f29ed47b002\": container with ID starting with 273194a3b8a173ad4cccd2ce964f301df5f6df1fdaa825e1fee99f29ed47b002 not found: ID does not exist"
Jul 29 19:04:30.986621 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:30.986593    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-77117817-6848f9f758-9rmp2"]
Jul 29 19:04:30.989461 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:30.989435    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-77117817-6848f9f758-9rmp2"]
Jul 29 19:04:31.003505 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:31.003446    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-77117817-97df84c48-lg56k"]
Jul 29 19:04:31.005608 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:31.005582    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-77117817-97df84c48-lg56k"]
Jul 29 19:04:32.149856 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:32.149823    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4748ab16-ab1f-4879-86cc-480bfa4b5196" path="/var/lib/kubelet/pods/4748ab16-ab1f-4879-86cc-480bfa4b5196/volumes"
Jul 29 19:04:32.150246 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:32.150167    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="592094ee-9275-4c37-b09f-a61260cfb6f5" path="/var/lib/kubelet/pods/592094ee-9275-4c37-b09f-a61260cfb6f5/volumes"
Jul 29 19:04:32.150460 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:32.150447    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acea07db-3f43-463f-a7f5-58b2d0dcac77" path="/var/lib/kubelet/pods/acea07db-3f43-463f-a7f5-58b2d0dcac77/volumes"
Jul 29 19:04:33.840467 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:33.840427    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:04:44.239087 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:04:44.239051    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:05:57.417274 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:05:57.417235    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:06:03.642185 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:03.642143    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:06:12.847054 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:12.847012    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:06:17.944614 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:17.944571    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b5792115-748fd4f476-g6dp5"]
Jul 29 19:06:17.945030 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:17.944935    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acea07db-3f43-463f-a7f5-58b2d0dcac77" containerName="revoke-keys"
Jul 29 19:06:17.945030 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:17.944947    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="acea07db-3f43-463f-a7f5-58b2d0dcac77" containerName="revoke-keys"
Jul 29 19:06:17.945030 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:17.944956    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="592094ee-9275-4c37-b09f-a61260cfb6f5" containerName="payload-pre-processing"
Jul 29 19:06:17.945030 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:17.944962    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="592094ee-9275-4c37-b09f-a61260cfb6f5" containerName="payload-pre-processing"
Jul 29 19:06:17.945030 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:17.944977    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4748ab16-ab1f-4879-86cc-480bfa4b5196" containerName="payload-processing"
Jul 29 19:06:17.945030 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:17.944983    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4748ab16-ab1f-4879-86cc-480bfa4b5196" containerName="payload-processing"
Jul 29 19:06:17.945223 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:17.945045    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="acea07db-3f43-463f-a7f5-58b2d0dcac77" containerName="revoke-keys"
Jul 29 19:06:17.945223 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:17.945056    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="592094ee-9275-4c37-b09f-a61260cfb6f5" containerName="payload-pre-processing"
Jul 29 19:06:17.945223 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:17.945062    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4748ab16-ab1f-4879-86cc-480bfa4b5196" containerName="payload-processing"
Jul 29 19:06:17.948275 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:17.948255    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5792115-748fd4f476-g6dp5"
Jul 29 19:06:17.950949 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:17.950925    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-b5792115\""
Jul 29 19:06:17.951105 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:17.950971    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-b5792115-dockercfg-4rghw\""
Jul 29 19:06:17.959603 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:17.959539    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b5792115-748fd4f476-g6dp5"]
Jul 29 19:06:17.985441 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:17.985401    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b5792115-96c6479fd-4r6f7"]
Jul 29 19:06:17.988836 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:17.988815    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b5792115-96c6479fd-4r6f7"
Jul 29 19:06:17.999084 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:17.999054    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b5792115-96c6479fd-4r6f7"]
Jul 29 19:06:18.015512 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:18.015470    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea30e9d3-83a0-4e58-b985-24df572fa955-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b5792115-748fd4f476-g6dp5\" (UID: \"ea30e9d3-83a0-4e58-b985-24df572fa955\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b5792115-748fd4f476-g6dp5"
Jul 29 19:06:18.015512 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:18.015516    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg26s\" (UniqueName: \"kubernetes.io/projected/ea30e9d3-83a0-4e58-b985-24df572fa955-kube-api-access-kg26s\") pod \"payload-pre-processing-e2e-mt-b5792115-748fd4f476-g6dp5\" (UID: \"ea30e9d3-83a0-4e58-b985-24df572fa955\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b5792115-748fd4f476-g6dp5"
Jul 29 19:06:18.117140 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:18.117092    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd875773-041d-4a60-ac04-0faae9bb4a61-plugins-config-volume\") pod \"payload-processing-e2e-mt-b5792115-96c6479fd-4r6f7\" (UID: \"dd875773-041d-4a60-ac04-0faae9bb4a61\") " pod="openshift-ingress/payload-processing-e2e-mt-b5792115-96c6479fd-4r6f7"
Jul 29 19:06:18.117370 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:18.117153    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea30e9d3-83a0-4e58-b985-24df572fa955-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b5792115-748fd4f476-g6dp5\" (UID: \"ea30e9d3-83a0-4e58-b985-24df572fa955\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b5792115-748fd4f476-g6dp5"
Jul 29 19:06:18.117370 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:18.117219    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg26s\" (UniqueName: \"kubernetes.io/projected/ea30e9d3-83a0-4e58-b985-24df572fa955-kube-api-access-kg26s\") pod \"payload-pre-processing-e2e-mt-b5792115-748fd4f476-g6dp5\" (UID: \"ea30e9d3-83a0-4e58-b985-24df572fa955\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b5792115-748fd4f476-g6dp5"
Jul 29 19:06:18.117370 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:18.117361    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4jn8\" (UniqueName: \"kubernetes.io/projected/dd875773-041d-4a60-ac04-0faae9bb4a61-kube-api-access-n4jn8\") pod \"payload-processing-e2e-mt-b5792115-96c6479fd-4r6f7\" (UID: \"dd875773-041d-4a60-ac04-0faae9bb4a61\") " pod="openshift-ingress/payload-processing-e2e-mt-b5792115-96c6479fd-4r6f7"
Jul 29 19:06:18.118087 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:18.118062    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea30e9d3-83a0-4e58-b985-24df572fa955-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b5792115-748fd4f476-g6dp5\" (UID: \"ea30e9d3-83a0-4e58-b985-24df572fa955\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b5792115-748fd4f476-g6dp5"
Jul 29 19:06:18.126034 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:18.125999    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg26s\" (UniqueName: \"kubernetes.io/projected/ea30e9d3-83a0-4e58-b985-24df572fa955-kube-api-access-kg26s\") pod \"payload-pre-processing-e2e-mt-b5792115-748fd4f476-g6dp5\" (UID: \"ea30e9d3-83a0-4e58-b985-24df572fa955\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b5792115-748fd4f476-g6dp5"
Jul 29 19:06:18.218904 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:18.218821    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4jn8\" (UniqueName: \"kubernetes.io/projected/dd875773-041d-4a60-ac04-0faae9bb4a61-kube-api-access-n4jn8\") pod \"payload-processing-e2e-mt-b5792115-96c6479fd-4r6f7\" (UID: \"dd875773-041d-4a60-ac04-0faae9bb4a61\") " pod="openshift-ingress/payload-processing-e2e-mt-b5792115-96c6479fd-4r6f7"
Jul 29 19:06:18.218904 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:18.218884    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd875773-041d-4a60-ac04-0faae9bb4a61-plugins-config-volume\") pod \"payload-processing-e2e-mt-b5792115-96c6479fd-4r6f7\" (UID: \"dd875773-041d-4a60-ac04-0faae9bb4a61\") " pod="openshift-ingress/payload-processing-e2e-mt-b5792115-96c6479fd-4r6f7"
Jul 29 19:06:18.219681 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:18.219659    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd875773-041d-4a60-ac04-0faae9bb4a61-plugins-config-volume\") pod \"payload-processing-e2e-mt-b5792115-96c6479fd-4r6f7\" (UID: \"dd875773-041d-4a60-ac04-0faae9bb4a61\") " pod="openshift-ingress/payload-processing-e2e-mt-b5792115-96c6479fd-4r6f7"
Jul 29 19:06:18.229741 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:18.229709    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4jn8\" (UniqueName: \"kubernetes.io/projected/dd875773-041d-4a60-ac04-0faae9bb4a61-kube-api-access-n4jn8\") pod \"payload-processing-e2e-mt-b5792115-96c6479fd-4r6f7\" (UID: \"dd875773-041d-4a60-ac04-0faae9bb4a61\") " pod="openshift-ingress/payload-processing-e2e-mt-b5792115-96c6479fd-4r6f7"
Jul 29 19:06:18.260438 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:18.260367    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5792115-748fd4f476-g6dp5"
Jul 29 19:06:18.300922 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:18.300867    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b5792115-96c6479fd-4r6f7"
Jul 29 19:06:18.437842 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:18.437579    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b5792115-748fd4f476-g6dp5"]
Jul 29 19:06:18.440999 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:06:18.440970    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea30e9d3_83a0_4e58_b985_24df572fa955.slice/crio-a968df10095a8d5e18b84aba510d3757d19502308bddfeae820085a50b82a1b2 WatchSource:0}: Error finding container a968df10095a8d5e18b84aba510d3757d19502308bddfeae820085a50b82a1b2: Status 404 returned error can't find the container with id a968df10095a8d5e18b84aba510d3757d19502308bddfeae820085a50b82a1b2
Jul 29 19:06:18.474835 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:18.474807    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b5792115-96c6479fd-4r6f7"]
Jul 29 19:06:18.478635 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:06:18.478596    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd875773_041d_4a60_ac04_0faae9bb4a61.slice/crio-4736e26e607747f5acec98d9f39030f1e7d3b4a82413530e0bb722a9c0b36e63 WatchSource:0}: Error finding container 4736e26e607747f5acec98d9f39030f1e7d3b4a82413530e0bb722a9c0b36e63: Status 404 returned error can't find the container with id 4736e26e607747f5acec98d9f39030f1e7d3b4a82413530e0bb722a9c0b36e63
Jul 29 19:06:18.484687 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:18.484658    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5792115-748fd4f476-g6dp5" event={"ID":"ea30e9d3-83a0-4e58-b985-24df572fa955","Type":"ContainerStarted","Data":"a968df10095a8d5e18b84aba510d3757d19502308bddfeae820085a50b82a1b2"}
Jul 29 19:06:19.491856 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:19.491816    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b5792115-96c6479fd-4r6f7" event={"ID":"dd875773-041d-4a60-ac04-0faae9bb4a61","Type":"ContainerStarted","Data":"fe75a906d9da581b943d8d39d84d1a49bd6c6fd67c4949f1e41abe5cc9f4fc20"}
Jul 29 19:06:19.492350 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:19.491878    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b5792115-96c6479fd-4r6f7" event={"ID":"dd875773-041d-4a60-ac04-0faae9bb4a61","Type":"ContainerStarted","Data":"4736e26e607747f5acec98d9f39030f1e7d3b4a82413530e0bb722a9c0b36e63"}
Jul 29 19:06:19.492350 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:19.491956    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-b5792115-96c6479fd-4r6f7"
Jul 29 19:06:19.493578 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:19.493550    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5792115-748fd4f476-g6dp5" event={"ID":"ea30e9d3-83a0-4e58-b985-24df572fa955","Type":"ContainerStarted","Data":"b49a185b429a9d5228943e57c026642456155ebf4cced0c78e65aad305fe67f5"}
Jul 29 19:06:19.493701 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:19.493600    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5792115-748fd4f476-g6dp5"
Jul 29 19:06:19.515274 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:19.515216    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-b5792115-96c6479fd-4r6f7" podStartSLOduration=2.515199208 podStartE2EDuration="2.515199208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:06:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:06:19.514367203 +0000 UTC m=+1943.784136292" watchObservedRunningTime="2026-07-29 19:06:19.515199208 +0000 UTC m=+1943.784968295"
Jul 29 19:06:19.536962 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:19.536883    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5792115-748fd4f476-g6dp5" podStartSLOduration=2.5368648179999997 podStartE2EDuration="2.536864818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:06:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:06:19.535785133 +0000 UTC m=+1943.805554234" watchObservedRunningTime="2026-07-29 19:06:19.536864818 +0000 UTC m=+1943.806633906"
Jul 29 19:06:21.237215 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:21.237176    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:06:30.501809 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:30.501777    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-b5792115-96c6479fd-4r6f7"
Jul 29 19:06:30.502304 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:30.501850    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5792115-748fd4f476-g6dp5"
Jul 29 19:06:49.425373 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:49.425285    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5792115-hdbgw"]
Jul 29 19:06:49.429035 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:49.429012    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5792115-hdbgw"
Jul 29 19:06:49.432529 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:49.432505    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vgb9z\""
Jul 29 19:06:49.433222 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:49.433198    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 19:06:49.445326 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:49.445294    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5792115-hdbgw"]
Jul 29 19:06:49.596209 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:49.596159    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/436b689b-3825-473d-9491-05b047ad53f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b5792115-hdbgw\" (UID: \"436b689b-3825-473d-9491-05b047ad53f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5792115-hdbgw"
Jul 29 19:06:49.697331 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:49.697238    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/436b689b-3825-473d-9491-05b047ad53f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b5792115-hdbgw\" (UID: \"436b689b-3825-473d-9491-05b047ad53f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5792115-hdbgw"
Jul 29 19:06:49.697920 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:49.697897    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/436b689b-3825-473d-9491-05b047ad53f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b5792115-hdbgw\" (UID: \"436b689b-3825-473d-9491-05b047ad53f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5792115-hdbgw"
Jul 29 19:06:49.739773 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:49.739737    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5792115-hdbgw"
Jul 29 19:06:49.881655 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:49.881543    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5792115-hdbgw"]
Jul 29 19:06:49.884488 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:06:49.884461    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod436b689b_3825_473d_9491_05b047ad53f2.slice/crio-0ec6afbaf81067da9805fe7e4d396fa1800e0d3c5e3010ad494afc6ac8d9aa73 WatchSource:0}: Error finding container 0ec6afbaf81067da9805fe7e4d396fa1800e0d3c5e3010ad494afc6ac8d9aa73: Status 404 returned error can't find the container with id 0ec6afbaf81067da9805fe7e4d396fa1800e0d3c5e3010ad494afc6ac8d9aa73
Jul 29 19:06:50.654335 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:50.654301    2559 generic.go:356] "Generic (PLEG): container finished" podID="436b689b-3825-473d-9491-05b047ad53f2" containerID="c9fc886d1a1b547c357429401e190fdad8a26982763b99370e5b3a426f963043" exitCode=0
Jul 29 19:06:50.654762 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:50.654436    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5792115-hdbgw" event={"ID":"436b689b-3825-473d-9491-05b047ad53f2","Type":"ContainerDied","Data":"c9fc886d1a1b547c357429401e190fdad8a26982763b99370e5b3a426f963043"}
Jul 29 19:06:50.654762 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:50.654503    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5792115-hdbgw" event={"ID":"436b689b-3825-473d-9491-05b047ad53f2","Type":"ContainerStarted","Data":"0ec6afbaf81067da9805fe7e4d396fa1800e0d3c5e3010ad494afc6ac8d9aa73"}
Jul 29 19:06:51.812094 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:51.812067    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5792115-hdbgw"
Jul 29 19:06:51.814069 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:51.814047    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/436b689b-3825-473d-9491-05b047ad53f2-maas-api-service-ca\") pod \"436b689b-3825-473d-9491-05b047ad53f2\" (UID: \"436b689b-3825-473d-9491-05b047ad53f2\") "
Jul 29 19:06:51.814407 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:51.814364    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/436b689b-3825-473d-9491-05b047ad53f2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "436b689b-3825-473d-9491-05b047ad53f2" (UID: "436b689b-3825-473d-9491-05b047ad53f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:06:51.915321 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:51.915220    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/436b689b-3825-473d-9491-05b047ad53f2-maas-api-service-ca\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:06:52.665557 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:52.665519    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5792115-hdbgw"
Jul 29 19:06:52.665756 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:52.665552    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5792115-hdbgw" event={"ID":"436b689b-3825-473d-9491-05b047ad53f2","Type":"ContainerDied","Data":"0ec6afbaf81067da9805fe7e4d396fa1800e0d3c5e3010ad494afc6ac8d9aa73"}
Jul 29 19:06:52.665756 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:52.665642    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ec6afbaf81067da9805fe7e4d396fa1800e0d3c5e3010ad494afc6ac8d9aa73"
Jul 29 19:06:59.920559 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:59.920518    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b5792115-96c6479fd-4r6f7"]
Jul 29 19:06:59.921355 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:59.921321    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-b5792115-96c6479fd-4r6f7" podUID="dd875773-041d-4a60-ac04-0faae9bb4a61" containerName="payload-processing" containerID="cri-o://fe75a906d9da581b943d8d39d84d1a49bd6c6fd67c4949f1e41abe5cc9f4fc20" gracePeriod=30
Jul 29 19:06:59.927774 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:59.927739    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b5792115-748fd4f476-g6dp5"]
Jul 29 19:06:59.928030 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:06:59.927994    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5792115-748fd4f476-g6dp5" podUID="ea30e9d3-83a0-4e58-b985-24df572fa955" containerName="payload-pre-processing" containerID="cri-o://b49a185b429a9d5228943e57c026642456155ebf4cced0c78e65aad305fe67f5" gracePeriod=30
Jul 29 19:07:00.060976 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:00.059823    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5792115-hdbgw"]
Jul 29 19:07:00.062081 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:00.062049    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5792115-hdbgw"]
Jul 29 19:07:00.155800 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:00.155767    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="436b689b-3825-473d-9491-05b047ad53f2" path="/var/lib/kubelet/pods/436b689b-3825-473d-9491-05b047ad53f2/volumes"
Jul 29 19:07:00.219486 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:00.219460    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b5792115-96c6479fd-4r6f7"
Jul 29 19:07:00.222507 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:00.222481    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5792115-748fd4f476-g6dp5"
Jul 29 19:07:00.282147 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:00.282106    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd875773-041d-4a60-ac04-0faae9bb4a61-plugins-config-volume\") pod \"dd875773-041d-4a60-ac04-0faae9bb4a61\" (UID: \"dd875773-041d-4a60-ac04-0faae9bb4a61\") "
Jul 29 19:07:00.282350 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:00.282209    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kg26s\" (UniqueName: \"kubernetes.io/projected/ea30e9d3-83a0-4e58-b985-24df572fa955-kube-api-access-kg26s\") pod \"ea30e9d3-83a0-4e58-b985-24df572fa955\" (UID: \"ea30e9d3-83a0-4e58-b985-24df572fa955\") "
Jul 29 19:07:00.282350 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:00.282241    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4jn8\" (UniqueName: \"kubernetes.io/projected/dd875773-041d-4a60-ac04-0faae9bb4a61-kube-api-access-n4jn8\") pod \"dd875773-041d-4a60-ac04-0faae9bb4a61\" (UID: \"dd875773-041d-4a60-ac04-0faae9bb4a61\") "
Jul 29 19:07:00.282350 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:00.282283    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea30e9d3-83a0-4e58-b985-24df572fa955-plugins-config-volume\") pod \"ea30e9d3-83a0-4e58-b985-24df572fa955\" (UID: \"ea30e9d3-83a0-4e58-b985-24df572fa955\") "
Jul 29 19:07:00.282584 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:00.282496    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd875773-041d-4a60-ac04-0faae9bb4a61-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dd875773-041d-4a60-ac04-0faae9bb4a61" (UID: "dd875773-041d-4a60-ac04-0faae9bb4a61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:07:00.282637 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:00.282599    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd875773-041d-4a60-ac04-0faae9bb4a61-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:07:00.282726 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:00.282703    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea30e9d3-83a0-4e58-b985-24df572fa955-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ea30e9d3-83a0-4e58-b985-24df572fa955" (UID: "ea30e9d3-83a0-4e58-b985-24df572fa955"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:07:00.284616 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:00.284591    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea30e9d3-83a0-4e58-b985-24df572fa955-kube-api-access-kg26s" (OuterVolumeSpecName: "kube-api-access-kg26s") pod "ea30e9d3-83a0-4e58-b985-24df572fa955" (UID: "ea30e9d3-83a0-4e58-b985-24df572fa955"). InnerVolumeSpecName "kube-api-access-kg26s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:07:00.284716 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:00.284651    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd875773-041d-4a60-ac04-0faae9bb4a61-kube-api-access-n4jn8" (OuterVolumeSpecName: "kube-api-access-n4jn8") pod "dd875773-041d-4a60-ac04-0faae9bb4a61" (UID: "dd875773-041d-4a60-ac04-0faae9bb4a61"). InnerVolumeSpecName "kube-api-access-n4jn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:07:00.383960 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:00.383912    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg26s\" (UniqueName: \"kubernetes.io/projected/ea30e9d3-83a0-4e58-b985-24df572fa955-kube-api-access-kg26s\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:07:00.383960 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:00.383954    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4jn8\" (UniqueName: \"kubernetes.io/projected/dd875773-041d-4a60-ac04-0faae9bb4a61-kube-api-access-n4jn8\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:07:00.383960 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:00.383964    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea30e9d3-83a0-4e58-b985-24df572fa955-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:07:00.716539 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:00.716494    2559 generic.go:356] "Generic (PLEG): container finished" podID="dd875773-041d-4a60-ac04-0faae9bb4a61" containerID="fe75a906d9da581b943d8d39d84d1a49bd6c6fd67c4949f1e41abe5cc9f4fc20" exitCode=0
Jul 29 19:07:00.716539 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:00.716546    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b5792115-96c6479fd-4r6f7"
Jul 29 19:07:00.716850 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:00.716586    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b5792115-96c6479fd-4r6f7" event={"ID":"dd875773-041d-4a60-ac04-0faae9bb4a61","Type":"ContainerDied","Data":"fe75a906d9da581b943d8d39d84d1a49bd6c6fd67c4949f1e41abe5cc9f4fc20"}
Jul 29 19:07:00.716850 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:00.716650    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b5792115-96c6479fd-4r6f7" event={"ID":"dd875773-041d-4a60-ac04-0faae9bb4a61","Type":"ContainerDied","Data":"4736e26e607747f5acec98d9f39030f1e7d3b4a82413530e0bb722a9c0b36e63"}
Jul 29 19:07:00.716850 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:00.716680    2559 scope.go:117] "RemoveContainer" containerID="fe75a906d9da581b943d8d39d84d1a49bd6c6fd67c4949f1e41abe5cc9f4fc20"
Jul 29 19:07:00.718087 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:00.718059    2559 generic.go:356] "Generic (PLEG): container finished" podID="ea30e9d3-83a0-4e58-b985-24df572fa955" containerID="b49a185b429a9d5228943e57c026642456155ebf4cced0c78e65aad305fe67f5" exitCode=0
Jul 29 19:07:00.718204 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:00.718094    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5792115-748fd4f476-g6dp5" event={"ID":"ea30e9d3-83a0-4e58-b985-24df572fa955","Type":"ContainerDied","Data":"b49a185b429a9d5228943e57c026642456155ebf4cced0c78e65aad305fe67f5"}
Jul 29 19:07:00.718204 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:00.718151    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5792115-748fd4f476-g6dp5" event={"ID":"ea30e9d3-83a0-4e58-b985-24df572fa955","Type":"ContainerDied","Data":"a968df10095a8d5e18b84aba510d3757d19502308bddfeae820085a50b82a1b2"}
Jul 29 19:07:00.718204 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:00.718101    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5792115-748fd4f476-g6dp5"
Jul 29 19:07:00.726965 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:00.726941    2559 scope.go:117] "RemoveContainer" containerID="fe75a906d9da581b943d8d39d84d1a49bd6c6fd67c4949f1e41abe5cc9f4fc20"
Jul 29 19:07:00.727327 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:07:00.727298    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe75a906d9da581b943d8d39d84d1a49bd6c6fd67c4949f1e41abe5cc9f4fc20\": container with ID starting with fe75a906d9da581b943d8d39d84d1a49bd6c6fd67c4949f1e41abe5cc9f4fc20 not found: ID does not exist" containerID="fe75a906d9da581b943d8d39d84d1a49bd6c6fd67c4949f1e41abe5cc9f4fc20"
Jul 29 19:07:00.727460 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:00.727346    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe75a906d9da581b943d8d39d84d1a49bd6c6fd67c4949f1e41abe5cc9f4fc20"} err="failed to get container status \"fe75a906d9da581b943d8d39d84d1a49bd6c6fd67c4949f1e41abe5cc9f4fc20\": rpc error: code = NotFound desc = could not find container \"fe75a906d9da581b943d8d39d84d1a49bd6c6fd67c4949f1e41abe5cc9f4fc20\": container with ID starting with fe75a906d9da581b943d8d39d84d1a49bd6c6fd67c4949f1e41abe5cc9f4fc20 not found: ID does not exist"
Jul 29 19:07:00.727460 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:00.727373    2559 scope.go:117] "RemoveContainer" containerID="b49a185b429a9d5228943e57c026642456155ebf4cced0c78e65aad305fe67f5"
Jul 29 19:07:00.736826 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:00.736805    2559 scope.go:117] "RemoveContainer" containerID="b49a185b429a9d5228943e57c026642456155ebf4cced0c78e65aad305fe67f5"
Jul 29 19:07:00.737136 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:07:00.737113    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b49a185b429a9d5228943e57c026642456155ebf4cced0c78e65aad305fe67f5\": container with ID starting with b49a185b429a9d5228943e57c026642456155ebf4cced0c78e65aad305fe67f5 not found: ID does not exist" containerID="b49a185b429a9d5228943e57c026642456155ebf4cced0c78e65aad305fe67f5"
Jul 29 19:07:00.737203 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:00.737149    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b49a185b429a9d5228943e57c026642456155ebf4cced0c78e65aad305fe67f5"} err="failed to get container status \"b49a185b429a9d5228943e57c026642456155ebf4cced0c78e65aad305fe67f5\": rpc error: code = NotFound desc = could not find container \"b49a185b429a9d5228943e57c026642456155ebf4cced0c78e65aad305fe67f5\": container with ID starting with b49a185b429a9d5228943e57c026642456155ebf4cced0c78e65aad305fe67f5 not found: ID does not exist"
Jul 29 19:07:00.742460 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:00.742428    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b5792115-96c6479fd-4r6f7"]
Jul 29 19:07:00.747258 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:00.747227    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b5792115-96c6479fd-4r6f7"]
Jul 29 19:07:00.758029 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:00.757997    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b5792115-748fd4f476-g6dp5"]
Jul 29 19:07:00.761975 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:00.761945    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b5792115-748fd4f476-g6dp5"]
Jul 29 19:07:02.150012 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:02.149975    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd875773-041d-4a60-ac04-0faae9bb4a61" path="/var/lib/kubelet/pods/dd875773-041d-4a60-ac04-0faae9bb4a61/volumes"
Jul 29 19:07:02.150509 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:02.150344    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea30e9d3-83a0-4e58-b985-24df572fa955" path="/var/lib/kubelet/pods/ea30e9d3-83a0-4e58-b985-24df572fa955/volumes"
Jul 29 19:07:02.748788 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:02.748749    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:07:30.884585 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:30.884547    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8eafda81-6dc79db4bb-g6bc7"]
Jul 29 19:07:30.885021 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:30.884903    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd875773-041d-4a60-ac04-0faae9bb4a61" containerName="payload-processing"
Jul 29 19:07:30.885021 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:30.884916    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd875773-041d-4a60-ac04-0faae9bb4a61" containerName="payload-processing"
Jul 29 19:07:30.885021 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:30.884926    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea30e9d3-83a0-4e58-b985-24df572fa955" containerName="payload-pre-processing"
Jul 29 19:07:30.885021 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:30.884933    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea30e9d3-83a0-4e58-b985-24df572fa955" containerName="payload-pre-processing"
Jul 29 19:07:30.885021 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:30.884949    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="436b689b-3825-473d-9491-05b047ad53f2" containerName="revoke-keys"
Jul 29 19:07:30.885021 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:30.884954    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="436b689b-3825-473d-9491-05b047ad53f2" containerName="revoke-keys"
Jul 29 19:07:30.885021 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:30.885010    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="436b689b-3825-473d-9491-05b047ad53f2" containerName="revoke-keys"
Jul 29 19:07:30.885021 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:30.885022    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea30e9d3-83a0-4e58-b985-24df572fa955" containerName="payload-pre-processing"
Jul 29 19:07:30.885276 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:30.885030    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd875773-041d-4a60-ac04-0faae9bb4a61" containerName="payload-processing"
Jul 29 19:07:30.888117 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:30.888098    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8eafda81-6dc79db4bb-g6bc7"
Jul 29 19:07:30.891367 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:30.891338    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-8eafda81\""
Jul 29 19:07:30.912282 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:30.912237    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8eafda81-6dc79db4bb-g6bc7"]
Jul 29 19:07:30.950030 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:30.949989    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8eafda81-5cc8f4c5fb-qnwhp"]
Jul 29 19:07:30.953509 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:30.953487    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8eafda81-5cc8f4c5fb-qnwhp"
Jul 29 19:07:30.958786 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:30.958760    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-8eafda81-dockercfg-mzmwq\""
Jul 29 19:07:30.970924 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:30.970886    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8eafda81-5cc8f4c5fb-qnwhp"]
Jul 29 19:07:31.044680 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:31.044634    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cffee44-40f7-4f83-b91b-8bc676fdd428-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8eafda81-5cc8f4c5fb-qnwhp\" (UID: \"5cffee44-40f7-4f83-b91b-8bc676fdd428\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8eafda81-5cc8f4c5fb-qnwhp"
Jul 29 19:07:31.044680 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:31.044677    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kwnl\" (UniqueName: \"kubernetes.io/projected/5cffee44-40f7-4f83-b91b-8bc676fdd428-kube-api-access-9kwnl\") pod \"payload-pre-processing-e2e-mt-8eafda81-5cc8f4c5fb-qnwhp\" (UID: \"5cffee44-40f7-4f83-b91b-8bc676fdd428\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8eafda81-5cc8f4c5fb-qnwhp"
Jul 29 19:07:31.044927 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:31.044702    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxzjm\" (UniqueName: \"kubernetes.io/projected/068168b4-c06b-4041-a979-345750b43c6c-kube-api-access-sxzjm\") pod \"payload-processing-e2e-mt-8eafda81-6dc79db4bb-g6bc7\" (UID: \"068168b4-c06b-4041-a979-345750b43c6c\") " pod="openshift-ingress/payload-processing-e2e-mt-8eafda81-6dc79db4bb-g6bc7"
Jul 29 19:07:31.044927 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:31.044795    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/068168b4-c06b-4041-a979-345750b43c6c-plugins-config-volume\") pod \"payload-processing-e2e-mt-8eafda81-6dc79db4bb-g6bc7\" (UID: \"068168b4-c06b-4041-a979-345750b43c6c\") " pod="openshift-ingress/payload-processing-e2e-mt-8eafda81-6dc79db4bb-g6bc7"
Jul 29 19:07:31.145338 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:31.145244    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/068168b4-c06b-4041-a979-345750b43c6c-plugins-config-volume\") pod \"payload-processing-e2e-mt-8eafda81-6dc79db4bb-g6bc7\" (UID: \"068168b4-c06b-4041-a979-345750b43c6c\") " pod="openshift-ingress/payload-processing-e2e-mt-8eafda81-6dc79db4bb-g6bc7"
Jul 29 19:07:31.145540 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:31.145341    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cffee44-40f7-4f83-b91b-8bc676fdd428-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8eafda81-5cc8f4c5fb-qnwhp\" (UID: \"5cffee44-40f7-4f83-b91b-8bc676fdd428\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8eafda81-5cc8f4c5fb-qnwhp"
Jul 29 19:07:31.145540 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:31.145374    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kwnl\" (UniqueName: \"kubernetes.io/projected/5cffee44-40f7-4f83-b91b-8bc676fdd428-kube-api-access-9kwnl\") pod \"payload-pre-processing-e2e-mt-8eafda81-5cc8f4c5fb-qnwhp\" (UID: \"5cffee44-40f7-4f83-b91b-8bc676fdd428\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8eafda81-5cc8f4c5fb-qnwhp"
Jul 29 19:07:31.145540 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:31.145446    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxzjm\" (UniqueName: \"kubernetes.io/projected/068168b4-c06b-4041-a979-345750b43c6c-kube-api-access-sxzjm\") pod \"payload-processing-e2e-mt-8eafda81-6dc79db4bb-g6bc7\" (UID: \"068168b4-c06b-4041-a979-345750b43c6c\") " pod="openshift-ingress/payload-processing-e2e-mt-8eafda81-6dc79db4bb-g6bc7"
Jul 29 19:07:31.146029 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:31.146009    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/068168b4-c06b-4041-a979-345750b43c6c-plugins-config-volume\") pod \"payload-processing-e2e-mt-8eafda81-6dc79db4bb-g6bc7\" (UID: \"068168b4-c06b-4041-a979-345750b43c6c\") " pod="openshift-ingress/payload-processing-e2e-mt-8eafda81-6dc79db4bb-g6bc7"
Jul 29 19:07:31.146076 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:31.146062    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cffee44-40f7-4f83-b91b-8bc676fdd428-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8eafda81-5cc8f4c5fb-qnwhp\" (UID: \"5cffee44-40f7-4f83-b91b-8bc676fdd428\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8eafda81-5cc8f4c5fb-qnwhp"
Jul 29 19:07:31.170925 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:31.170884    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxzjm\" (UniqueName: \"kubernetes.io/projected/068168b4-c06b-4041-a979-345750b43c6c-kube-api-access-sxzjm\") pod \"payload-processing-e2e-mt-8eafda81-6dc79db4bb-g6bc7\" (UID: \"068168b4-c06b-4041-a979-345750b43c6c\") " pod="openshift-ingress/payload-processing-e2e-mt-8eafda81-6dc79db4bb-g6bc7"
Jul 29 19:07:31.175257 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:31.175228    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kwnl\" (UniqueName: \"kubernetes.io/projected/5cffee44-40f7-4f83-b91b-8bc676fdd428-kube-api-access-9kwnl\") pod \"payload-pre-processing-e2e-mt-8eafda81-5cc8f4c5fb-qnwhp\" (UID: \"5cffee44-40f7-4f83-b91b-8bc676fdd428\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8eafda81-5cc8f4c5fb-qnwhp"
Jul 29 19:07:31.199263 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:31.199220    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8eafda81-6dc79db4bb-g6bc7"
Jul 29 19:07:31.268889 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:31.268831    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8eafda81-5cc8f4c5fb-qnwhp"
Jul 29 19:07:31.377879 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:31.377846    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8eafda81-6dc79db4bb-g6bc7"]
Jul 29 19:07:31.380280 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:07:31.380249    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod068168b4_c06b_4041_a979_345750b43c6c.slice/crio-5aa9a3cc5731792d0a71d7cf01fe3804173284071a2499137f44637ac0e2dc16 WatchSource:0}: Error finding container 5aa9a3cc5731792d0a71d7cf01fe3804173284071a2499137f44637ac0e2dc16: Status 404 returned error can't find the container with id 5aa9a3cc5731792d0a71d7cf01fe3804173284071a2499137f44637ac0e2dc16
Jul 29 19:07:31.449658 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:31.449628    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8eafda81-5cc8f4c5fb-qnwhp"]
Jul 29 19:07:31.452187 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:07:31.452147    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cffee44_40f7_4f83_b91b_8bc676fdd428.slice/crio-5bf82fb0fb945aa5653255747d468262d2be171bb5a72a8e9b87c00880710abb WatchSource:0}: Error finding container 5bf82fb0fb945aa5653255747d468262d2be171bb5a72a8e9b87c00880710abb: Status 404 returned error can't find the container with id 5bf82fb0fb945aa5653255747d468262d2be171bb5a72a8e9b87c00880710abb
Jul 29 19:07:31.874117 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:31.874075    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8eafda81-5cc8f4c5fb-qnwhp" event={"ID":"5cffee44-40f7-4f83-b91b-8bc676fdd428","Type":"ContainerStarted","Data":"1b6a662376ef7fcc8d9fa0909d2f43e5a269dee519100777e57efe32f804f6c9"}
Jul 29 19:07:31.874334 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:31.874255    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8eafda81-5cc8f4c5fb-qnwhp"
Jul 29 19:07:31.874334 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:31.874294    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8eafda81-5cc8f4c5fb-qnwhp" event={"ID":"5cffee44-40f7-4f83-b91b-8bc676fdd428","Type":"ContainerStarted","Data":"5bf82fb0fb945aa5653255747d468262d2be171bb5a72a8e9b87c00880710abb"}
Jul 29 19:07:31.875729 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:31.875701    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8eafda81-6dc79db4bb-g6bc7" event={"ID":"068168b4-c06b-4041-a979-345750b43c6c","Type":"ContainerStarted","Data":"9401bc962061f0204630d882351d3c4b564b6802b813ea2e1bb8c74b4e96f245"}
Jul 29 19:07:31.875848 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:31.875748    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8eafda81-6dc79db4bb-g6bc7" event={"ID":"068168b4-c06b-4041-a979-345750b43c6c","Type":"ContainerStarted","Data":"5aa9a3cc5731792d0a71d7cf01fe3804173284071a2499137f44637ac0e2dc16"}
Jul 29 19:07:31.875848 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:31.875777    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-8eafda81-6dc79db4bb-g6bc7"
Jul 29 19:07:31.940356 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:31.940301    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-8eafda81-6dc79db4bb-g6bc7" podStartSLOduration=1.940281471 podStartE2EDuration="1.940281471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:07:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:07:31.937570542 +0000 UTC m=+2016.207339641" watchObservedRunningTime="2026-07-29 19:07:31.940281471 +0000 UTC m=+2016.210050558"
Jul 29 19:07:31.940786 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:31.940404    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-8eafda81-5cc8f4c5fb-qnwhp" podStartSLOduration=1.9403977129999999 podStartE2EDuration="1.940397713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:07:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:07:31.908008253 +0000 UTC m=+2016.177777342" watchObservedRunningTime="2026-07-29 19:07:31.940397713 +0000 UTC m=+2016.210166794"
Jul 29 19:07:36.766524 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:36.766483    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:07:42.883224 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:42.883195    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8eafda81-5cc8f4c5fb-qnwhp"
Jul 29 19:07:42.883721 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:07:42.883265    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-8eafda81-6dc79db4bb-g6bc7"
Jul 29 19:08:07.185712 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:07.185670    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-8eafda81/e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx"]
Jul 29 19:08:07.189267 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:07.189248    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8eafda81/e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx"
Jul 29 19:08:07.192163 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:07.192136    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-8eafda81\"/\"kube-root-ca.crt\""
Jul 29 19:08:07.192926 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:07.192907    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-8eafda81\"/\"openshift-service-ca.crt\""
Jul 29 19:08:07.193047 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:07.192921    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-8eafda81\"/\"default-dockercfg-2hs84\""
Jul 29 19:08:07.193047 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:07.192949    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-8eafda81\"/\"e2e-lifecycle-model-8eafda81-kserve-self-signed-certs\""
Jul 29 19:08:07.204463 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:07.204436    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-8eafda81/e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx"]
Jul 29 19:08:07.268851 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:07.268813    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d484b41d-4a92-4593-8992-156493609c27-tls-certs\") pod \"e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx\" (UID: \"d484b41d-4a92-4593-8992-156493609c27\") " pod="ai-tenant-e2e-mt-8eafda81/e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx"
Jul 29 19:08:07.269043 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:07.268860    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxnck\" (UniqueName: \"kubernetes.io/projected/d484b41d-4a92-4593-8992-156493609c27-kube-api-access-mxnck\") pod \"e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx\" (UID: \"d484b41d-4a92-4593-8992-156493609c27\") " pod="ai-tenant-e2e-mt-8eafda81/e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx"
Jul 29 19:08:07.269043 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:07.268886    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d484b41d-4a92-4593-8992-156493609c27-tmp-dir\") pod \"e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx\" (UID: \"d484b41d-4a92-4593-8992-156493609c27\") " pod="ai-tenant-e2e-mt-8eafda81/e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx"
Jul 29 19:08:07.269043 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:07.268953    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d484b41d-4a92-4593-8992-156493609c27-dshm\") pod \"e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx\" (UID: \"d484b41d-4a92-4593-8992-156493609c27\") " pod="ai-tenant-e2e-mt-8eafda81/e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx"
Jul 29 19:08:07.269043 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:07.268999    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d484b41d-4a92-4593-8992-156493609c27-home\") pod \"e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx\" (UID: \"d484b41d-4a92-4593-8992-156493609c27\") " pod="ai-tenant-e2e-mt-8eafda81/e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx"
Jul 29 19:08:07.269043 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:07.269014    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d484b41d-4a92-4593-8992-156493609c27-model-cache\") pod \"e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx\" (UID: \"d484b41d-4a92-4593-8992-156493609c27\") " pod="ai-tenant-e2e-mt-8eafda81/e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx"
Jul 29 19:08:07.369958 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:07.369914    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d484b41d-4a92-4593-8992-156493609c27-tmp-dir\") pod \"e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx\" (UID: \"d484b41d-4a92-4593-8992-156493609c27\") " pod="ai-tenant-e2e-mt-8eafda81/e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx"
Jul 29 19:08:07.369958 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:07.369965    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d484b41d-4a92-4593-8992-156493609c27-dshm\") pod \"e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx\" (UID: \"d484b41d-4a92-4593-8992-156493609c27\") " pod="ai-tenant-e2e-mt-8eafda81/e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx"
Jul 29 19:08:07.370206 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:07.369998    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d484b41d-4a92-4593-8992-156493609c27-home\") pod \"e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx\" (UID: \"d484b41d-4a92-4593-8992-156493609c27\") " pod="ai-tenant-e2e-mt-8eafda81/e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx"
Jul 29 19:08:07.370206 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:07.370013    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d484b41d-4a92-4593-8992-156493609c27-model-cache\") pod \"e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx\" (UID: \"d484b41d-4a92-4593-8992-156493609c27\") " pod="ai-tenant-e2e-mt-8eafda81/e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx"
Jul 29 19:08:07.370206 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:07.370073    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d484b41d-4a92-4593-8992-156493609c27-tls-certs\") pod \"e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx\" (UID: \"d484b41d-4a92-4593-8992-156493609c27\") " pod="ai-tenant-e2e-mt-8eafda81/e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx"
Jul 29 19:08:07.370206 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:07.370122    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxnck\" (UniqueName: \"kubernetes.io/projected/d484b41d-4a92-4593-8992-156493609c27-kube-api-access-mxnck\") pod \"e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx\" (UID: \"d484b41d-4a92-4593-8992-156493609c27\") " pod="ai-tenant-e2e-mt-8eafda81/e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx"
Jul 29 19:08:07.370451 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:07.370360    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d484b41d-4a92-4593-8992-156493609c27-tmp-dir\") pod \"e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx\" (UID: \"d484b41d-4a92-4593-8992-156493609c27\") " pod="ai-tenant-e2e-mt-8eafda81/e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx"
Jul 29 19:08:07.370518 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:07.370457    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d484b41d-4a92-4593-8992-156493609c27-home\") pod \"e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx\" (UID: \"d484b41d-4a92-4593-8992-156493609c27\") " pod="ai-tenant-e2e-mt-8eafda81/e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx"
Jul 29 19:08:07.370596 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:07.370573    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d484b41d-4a92-4593-8992-156493609c27-model-cache\") pod \"e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx\" (UID: \"d484b41d-4a92-4593-8992-156493609c27\") " pod="ai-tenant-e2e-mt-8eafda81/e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx"
Jul 29 19:08:07.372568 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:07.372545    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d484b41d-4a92-4593-8992-156493609c27-dshm\") pod \"e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx\" (UID: \"d484b41d-4a92-4593-8992-156493609c27\") " pod="ai-tenant-e2e-mt-8eafda81/e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx"
Jul 29 19:08:07.372773 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:07.372760    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d484b41d-4a92-4593-8992-156493609c27-tls-certs\") pod \"e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx\" (UID: \"d484b41d-4a92-4593-8992-156493609c27\") " pod="ai-tenant-e2e-mt-8eafda81/e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx"
Jul 29 19:08:07.384945 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:07.384906    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxnck\" (UniqueName: \"kubernetes.io/projected/d484b41d-4a92-4593-8992-156493609c27-kube-api-access-mxnck\") pod \"e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx\" (UID: \"d484b41d-4a92-4593-8992-156493609c27\") " pod="ai-tenant-e2e-mt-8eafda81/e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx"
Jul 29 19:08:07.498545 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:07.498451    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8eafda81/e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx"
Jul 29 19:08:07.659179 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:07.659110    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-8eafda81/e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx"]
Jul 29 19:08:07.661563 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:08:07.661528    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd484b41d_4a92_4593_8992_156493609c27.slice/crio-83ad9bd5ebe51b22d2e7485aef7cf8866f129ad0b1bc059c240da7cc93edf5a7 WatchSource:0}: Error finding container 83ad9bd5ebe51b22d2e7485aef7cf8866f129ad0b1bc059c240da7cc93edf5a7: Status 404 returned error can't find the container with id 83ad9bd5ebe51b22d2e7485aef7cf8866f129ad0b1bc059c240da7cc93edf5a7
Jul 29 19:08:08.061838 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:08.061798    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8eafda81/e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx" event={"ID":"d484b41d-4a92-4593-8992-156493609c27","Type":"ContainerStarted","Data":"333d9183c51204973c1d825a100c7849df249989af152d7df94498d752be5f2b"}
Jul 29 19:08:08.062047 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:08.061888    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8eafda81/e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx" event={"ID":"d484b41d-4a92-4593-8992-156493609c27","Type":"ContainerStarted","Data":"83ad9bd5ebe51b22d2e7485aef7cf8866f129ad0b1bc059c240da7cc93edf5a7"}
Jul 29 19:08:08.081604 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:08.081550    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-8eafda81/e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx" podStartSLOduration=1.081535317 podStartE2EDuration="1.081535317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:08:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:08:08.080306418 +0000 UTC m=+2052.350075506" watchObservedRunningTime="2026-07-29 19:08:08.081535317 +0000 UTC m=+2052.351304404"
Jul 29 19:08:10.140258 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:10.140220    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:08:17.499495 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:17.499369    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-8eafda81/e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx"
Jul 29 19:08:17.499495 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:17.499454    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-8eafda81/e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx"
Jul 29 19:08:17.512237 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:17.512202    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-8eafda81/e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx"
Jul 29 19:08:18.124310 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:18.124281    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-8eafda81/e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx"
Jul 29 19:08:25.239851 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:25.239813    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:08:29.647288 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:29.647250    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eafda81-j742z"]
Jul 29 19:08:29.650861 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:29.650839    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eafda81-j742z"
Jul 29 19:08:29.657872 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:29.657708    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vgb9z\""
Jul 29 19:08:29.658183 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:29.658161    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 19:08:29.661286 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:29.661254    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eafda81-j742z"]
Jul 29 19:08:29.778324 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:29.778286    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8c41cda-13ec-418d-ba4b-7cb8951b022a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8eafda81-j742z\" (UID: \"e8c41cda-13ec-418d-ba4b-7cb8951b022a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eafda81-j742z"
Jul 29 19:08:29.879664 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:29.879625    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8c41cda-13ec-418d-ba4b-7cb8951b022a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8eafda81-j742z\" (UID: \"e8c41cda-13ec-418d-ba4b-7cb8951b022a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eafda81-j742z"
Jul 29 19:08:29.880289 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:29.880266    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8c41cda-13ec-418d-ba4b-7cb8951b022a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8eafda81-j742z\" (UID: \"e8c41cda-13ec-418d-ba4b-7cb8951b022a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eafda81-j742z"
Jul 29 19:08:29.970905 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:29.970817    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eafda81-j742z"
Jul 29 19:08:30.118433 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:30.118404    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eafda81-j742z"]
Jul 29 19:08:30.121408 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:08:30.121362    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8c41cda_13ec_418d_ba4b_7cb8951b022a.slice/crio-0d83257ff451970fda28ebebfc6be107354055c336c3f00c619e9e5f5dcec0cb WatchSource:0}: Error finding container 0d83257ff451970fda28ebebfc6be107354055c336c3f00c619e9e5f5dcec0cb: Status 404 returned error can't find the container with id 0d83257ff451970fda28ebebfc6be107354055c336c3f00c619e9e5f5dcec0cb
Jul 29 19:08:30.172678 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:30.172642    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eafda81-j742z" event={"ID":"e8c41cda-13ec-418d-ba4b-7cb8951b022a","Type":"ContainerStarted","Data":"0d83257ff451970fda28ebebfc6be107354055c336c3f00c619e9e5f5dcec0cb"}
Jul 29 19:08:31.179980 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:31.179885    2559 generic.go:356] "Generic (PLEG): container finished" podID="e8c41cda-13ec-418d-ba4b-7cb8951b022a" containerID="7a58b555641831a3424a06055bc980723f38e4bc64f923e93a6fc5dcb47e17e1" exitCode=0
Jul 29 19:08:31.179980 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:31.179967    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eafda81-j742z" event={"ID":"e8c41cda-13ec-418d-ba4b-7cb8951b022a","Type":"ContainerDied","Data":"7a58b555641831a3424a06055bc980723f38e4bc64f923e93a6fc5dcb47e17e1"}
Jul 29 19:08:32.328317 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:32.328290    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eafda81-j742z"
Jul 29 19:08:32.501698 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:32.501604    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8c41cda-13ec-418d-ba4b-7cb8951b022a-maas-api-service-ca\") pod \"e8c41cda-13ec-418d-ba4b-7cb8951b022a\" (UID: \"e8c41cda-13ec-418d-ba4b-7cb8951b022a\") "
Jul 29 19:08:32.501950 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:32.501928    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8c41cda-13ec-418d-ba4b-7cb8951b022a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e8c41cda-13ec-418d-ba4b-7cb8951b022a" (UID: "e8c41cda-13ec-418d-ba4b-7cb8951b022a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:08:32.602644 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:32.602607    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8c41cda-13ec-418d-ba4b-7cb8951b022a-maas-api-service-ca\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:08:33.191617 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:33.191585    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eafda81-j742z"
Jul 29 19:08:33.191617 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:33.191594    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eafda81-j742z" event={"ID":"e8c41cda-13ec-418d-ba4b-7cb8951b022a","Type":"ContainerDied","Data":"0d83257ff451970fda28ebebfc6be107354055c336c3f00c619e9e5f5dcec0cb"}
Jul 29 19:08:33.191836 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:33.191644    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d83257ff451970fda28ebebfc6be107354055c336c3f00c619e9e5f5dcec0cb"
Jul 29 19:08:43.642004 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:43.641960    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:08:45.163690 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:45.163651    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8eafda81-6dc79db4bb-g6bc7"]
Jul 29 19:08:45.164174 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:45.164012    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-8eafda81-6dc79db4bb-g6bc7" podUID="068168b4-c06b-4041-a979-345750b43c6c" containerName="payload-processing" containerID="cri-o://9401bc962061f0204630d882351d3c4b564b6802b813ea2e1bb8c74b4e96f245" gracePeriod=30
Jul 29 19:08:45.170692 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:45.170663    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8eafda81-5cc8f4c5fb-qnwhp"]
Jul 29 19:08:45.172318 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:45.170896    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-8eafda81-5cc8f4c5fb-qnwhp" podUID="5cffee44-40f7-4f83-b91b-8bc676fdd428" containerName="payload-pre-processing" containerID="cri-o://1b6a662376ef7fcc8d9fa0909d2f43e5a269dee519100777e57efe32f804f6c9" gracePeriod=30
Jul 29 19:08:45.366820 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:45.366782    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eafda81-j742z"]
Jul 29 19:08:45.369756 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:45.369728    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eafda81-j742z"]
Jul 29 19:08:45.462033 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:45.461986    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8eafda81-6dc79db4bb-g6bc7"
Jul 29 19:08:45.464947 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:45.464921    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8eafda81-5cc8f4c5fb-qnwhp"
Jul 29 19:08:45.521664 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:45.521626    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxzjm\" (UniqueName: \"kubernetes.io/projected/068168b4-c06b-4041-a979-345750b43c6c-kube-api-access-sxzjm\") pod \"068168b4-c06b-4041-a979-345750b43c6c\" (UID: \"068168b4-c06b-4041-a979-345750b43c6c\") "
Jul 29 19:08:45.521664 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:45.521666    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kwnl\" (UniqueName: \"kubernetes.io/projected/5cffee44-40f7-4f83-b91b-8bc676fdd428-kube-api-access-9kwnl\") pod \"5cffee44-40f7-4f83-b91b-8bc676fdd428\" (UID: \"5cffee44-40f7-4f83-b91b-8bc676fdd428\") "
Jul 29 19:08:45.521881 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:45.521727    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/068168b4-c06b-4041-a979-345750b43c6c-plugins-config-volume\") pod \"068168b4-c06b-4041-a979-345750b43c6c\" (UID: \"068168b4-c06b-4041-a979-345750b43c6c\") "
Jul 29 19:08:45.521881 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:45.521797    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cffee44-40f7-4f83-b91b-8bc676fdd428-plugins-config-volume\") pod \"5cffee44-40f7-4f83-b91b-8bc676fdd428\" (UID: \"5cffee44-40f7-4f83-b91b-8bc676fdd428\") "
Jul 29 19:08:45.522223 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:45.522190    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/068168b4-c06b-4041-a979-345750b43c6c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "068168b4-c06b-4041-a979-345750b43c6c" (UID: "068168b4-c06b-4041-a979-345750b43c6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:08:45.522317 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:45.522200    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cffee44-40f7-4f83-b91b-8bc676fdd428-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5cffee44-40f7-4f83-b91b-8bc676fdd428" (UID: "5cffee44-40f7-4f83-b91b-8bc676fdd428"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:08:45.523928 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:45.523905    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/068168b4-c06b-4041-a979-345750b43c6c-kube-api-access-sxzjm" (OuterVolumeSpecName: "kube-api-access-sxzjm") pod "068168b4-c06b-4041-a979-345750b43c6c" (UID: "068168b4-c06b-4041-a979-345750b43c6c"). InnerVolumeSpecName "kube-api-access-sxzjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:08:45.523992 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:45.523930    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cffee44-40f7-4f83-b91b-8bc676fdd428-kube-api-access-9kwnl" (OuterVolumeSpecName: "kube-api-access-9kwnl") pod "5cffee44-40f7-4f83-b91b-8bc676fdd428" (UID: "5cffee44-40f7-4f83-b91b-8bc676fdd428"). InnerVolumeSpecName "kube-api-access-9kwnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:08:45.623255 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:45.623217    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/068168b4-c06b-4041-a979-345750b43c6c-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:08:45.623255 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:45.623246    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cffee44-40f7-4f83-b91b-8bc676fdd428-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:08:45.623255 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:45.623256    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxzjm\" (UniqueName: \"kubernetes.io/projected/068168b4-c06b-4041-a979-345750b43c6c-kube-api-access-sxzjm\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:08:45.623255 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:45.623267    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kwnl\" (UniqueName: \"kubernetes.io/projected/5cffee44-40f7-4f83-b91b-8bc676fdd428-kube-api-access-9kwnl\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:08:46.149831 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:46.149795    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8c41cda-13ec-418d-ba4b-7cb8951b022a" path="/var/lib/kubelet/pods/e8c41cda-13ec-418d-ba4b-7cb8951b022a/volumes"
Jul 29 19:08:46.260799 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:46.260762    2559 generic.go:356] "Generic (PLEG): container finished" podID="5cffee44-40f7-4f83-b91b-8bc676fdd428" containerID="1b6a662376ef7fcc8d9fa0909d2f43e5a269dee519100777e57efe32f804f6c9" exitCode=0
Jul 29 19:08:46.261277 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:46.260816    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8eafda81-5cc8f4c5fb-qnwhp"
Jul 29 19:08:46.261277 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:46.260848    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8eafda81-5cc8f4c5fb-qnwhp" event={"ID":"5cffee44-40f7-4f83-b91b-8bc676fdd428","Type":"ContainerDied","Data":"1b6a662376ef7fcc8d9fa0909d2f43e5a269dee519100777e57efe32f804f6c9"}
Jul 29 19:08:46.261277 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:46.260922    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8eafda81-5cc8f4c5fb-qnwhp" event={"ID":"5cffee44-40f7-4f83-b91b-8bc676fdd428","Type":"ContainerDied","Data":"5bf82fb0fb945aa5653255747d468262d2be171bb5a72a8e9b87c00880710abb"}
Jul 29 19:08:46.261277 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:46.260959    2559 scope.go:117] "RemoveContainer" containerID="1b6a662376ef7fcc8d9fa0909d2f43e5a269dee519100777e57efe32f804f6c9"
Jul 29 19:08:46.262572 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:46.262543    2559 generic.go:356] "Generic (PLEG): container finished" podID="068168b4-c06b-4041-a979-345750b43c6c" containerID="9401bc962061f0204630d882351d3c4b564b6802b813ea2e1bb8c74b4e96f245" exitCode=0
Jul 29 19:08:46.262823 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:46.262595    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8eafda81-6dc79db4bb-g6bc7"
Jul 29 19:08:46.262823 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:46.262635    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8eafda81-6dc79db4bb-g6bc7" event={"ID":"068168b4-c06b-4041-a979-345750b43c6c","Type":"ContainerDied","Data":"9401bc962061f0204630d882351d3c4b564b6802b813ea2e1bb8c74b4e96f245"}
Jul 29 19:08:46.262823 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:46.262692    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8eafda81-6dc79db4bb-g6bc7" event={"ID":"068168b4-c06b-4041-a979-345750b43c6c","Type":"ContainerDied","Data":"5aa9a3cc5731792d0a71d7cf01fe3804173284071a2499137f44637ac0e2dc16"}
Jul 29 19:08:46.270460 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:46.270436    2559 scope.go:117] "RemoveContainer" containerID="1b6a662376ef7fcc8d9fa0909d2f43e5a269dee519100777e57efe32f804f6c9"
Jul 29 19:08:46.270783 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:08:46.270760    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b6a662376ef7fcc8d9fa0909d2f43e5a269dee519100777e57efe32f804f6c9\": container with ID starting with 1b6a662376ef7fcc8d9fa0909d2f43e5a269dee519100777e57efe32f804f6c9 not found: ID does not exist" containerID="1b6a662376ef7fcc8d9fa0909d2f43e5a269dee519100777e57efe32f804f6c9"
Jul 29 19:08:46.270835 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:46.270798    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b6a662376ef7fcc8d9fa0909d2f43e5a269dee519100777e57efe32f804f6c9"} err="failed to get container status \"1b6a662376ef7fcc8d9fa0909d2f43e5a269dee519100777e57efe32f804f6c9\": rpc error: code = NotFound desc = could not find container \"1b6a662376ef7fcc8d9fa0909d2f43e5a269dee519100777e57efe32f804f6c9\": container with ID starting with 1b6a662376ef7fcc8d9fa0909d2f43e5a269dee519100777e57efe32f804f6c9 not found: ID does not exist"
Jul 29 19:08:46.270835 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:46.270820    2559 scope.go:117] "RemoveContainer" containerID="9401bc962061f0204630d882351d3c4b564b6802b813ea2e1bb8c74b4e96f245"
Jul 29 19:08:46.279947 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:46.279924    2559 scope.go:117] "RemoveContainer" containerID="9401bc962061f0204630d882351d3c4b564b6802b813ea2e1bb8c74b4e96f245"
Jul 29 19:08:46.280322 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:08:46.280294    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9401bc962061f0204630d882351d3c4b564b6802b813ea2e1bb8c74b4e96f245\": container with ID starting with 9401bc962061f0204630d882351d3c4b564b6802b813ea2e1bb8c74b4e96f245 not found: ID does not exist" containerID="9401bc962061f0204630d882351d3c4b564b6802b813ea2e1bb8c74b4e96f245"
Jul 29 19:08:46.280431 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:46.280334    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9401bc962061f0204630d882351d3c4b564b6802b813ea2e1bb8c74b4e96f245"} err="failed to get container status \"9401bc962061f0204630d882351d3c4b564b6802b813ea2e1bb8c74b4e96f245\": rpc error: code = NotFound desc = could not find container \"9401bc962061f0204630d882351d3c4b564b6802b813ea2e1bb8c74b4e96f245\": container with ID starting with 9401bc962061f0204630d882351d3c4b564b6802b813ea2e1bb8c74b4e96f245 not found: ID does not exist"
Jul 29 19:08:46.280899 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:46.280881    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8eafda81-5cc8f4c5fb-qnwhp"]
Jul 29 19:08:46.283726 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:46.283704    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8eafda81-5cc8f4c5fb-qnwhp"]
Jul 29 19:08:46.293634 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:46.293607    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8eafda81-6dc79db4bb-g6bc7"]
Jul 29 19:08:46.297776 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:46.297747    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8eafda81-6dc79db4bb-g6bc7"]
Jul 29 19:08:48.151905 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:48.151862    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="068168b4-c06b-4041-a979-345750b43c6c" path="/var/lib/kubelet/pods/068168b4-c06b-4041-a979-345750b43c6c/volumes"
Jul 29 19:08:48.153093 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:48.153061    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cffee44-40f7-4f83-b91b-8bc676fdd428" path="/var/lib/kubelet/pods/5cffee44-40f7-4f83-b91b-8bc676fdd428/volumes"
Jul 29 19:08:53.732558 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:53.732523    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-8eafda81/e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx"]
Jul 29 19:08:54.559798 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:54.559755    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:08:56.217952 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:56.217924    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-v9nsr_7f8b4737-d4c2-4598-bd40-208015542831/console-operator/1.log"
Jul 29 19:08:56.219788 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:56.219764    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-v9nsr_7f8b4737-d4c2-4598-bd40-208015542831/console-operator/1.log"
Jul 29 19:08:56.225575 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:56.225553    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjdrx_60f444e1-7e31-443c-8ec7-a4d04c64bf4b/ovn-acl-logging/0.log"
Jul 29 19:08:56.227353 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:56.227332    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjdrx_60f444e1-7e31-443c-8ec7-a4d04c64bf4b/ovn-acl-logging/0.log"
Jul 29 19:08:56.326012 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:08:56.325978    2559 scope.go:117] "RemoveContainer" containerID="6d0b44aa2c19b473b19ae00afc340cdd5eb1b8a1b0a5fcf92364834c84b08fd6"
Jul 29 19:09:08.752694 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:09:08.752648    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-8eafda81/e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx" podUID="d484b41d-4a92-4593-8992-156493609c27" containerName="main" containerID="cri-o://333d9183c51204973c1d825a100c7849df249989af152d7df94498d752be5f2b" gracePeriod=45
Jul 29 19:09:09.017038 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:09:09.016966    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8eafda81/e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx"
Jul 29 19:09:09.133103 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:09:09.133063    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d484b41d-4a92-4593-8992-156493609c27-dshm\") pod \"d484b41d-4a92-4593-8992-156493609c27\" (UID: \"d484b41d-4a92-4593-8992-156493609c27\") "
Jul 29 19:09:09.133283 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:09:09.133123    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxnck\" (UniqueName: \"kubernetes.io/projected/d484b41d-4a92-4593-8992-156493609c27-kube-api-access-mxnck\") pod \"d484b41d-4a92-4593-8992-156493609c27\" (UID: \"d484b41d-4a92-4593-8992-156493609c27\") "
Jul 29 19:09:09.133283 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:09:09.133181    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d484b41d-4a92-4593-8992-156493609c27-tmp-dir\") pod \"d484b41d-4a92-4593-8992-156493609c27\" (UID: \"d484b41d-4a92-4593-8992-156493609c27\") "
Jul 29 19:09:09.133283 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:09:09.133211    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d484b41d-4a92-4593-8992-156493609c27-home\") pod \"d484b41d-4a92-4593-8992-156493609c27\" (UID: \"d484b41d-4a92-4593-8992-156493609c27\") "
Jul 29 19:09:09.133283 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:09:09.133269    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d484b41d-4a92-4593-8992-156493609c27-tls-certs\") pod \"d484b41d-4a92-4593-8992-156493609c27\" (UID: \"d484b41d-4a92-4593-8992-156493609c27\") "
Jul 29 19:09:09.133505 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:09:09.133292    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d484b41d-4a92-4593-8992-156493609c27-model-cache\") pod \"d484b41d-4a92-4593-8992-156493609c27\" (UID: \"d484b41d-4a92-4593-8992-156493609c27\") "
Jul 29 19:09:09.133563 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:09:09.133507    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d484b41d-4a92-4593-8992-156493609c27-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d484b41d-4a92-4593-8992-156493609c27" (UID: "d484b41d-4a92-4593-8992-156493609c27"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:09:09.133664 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:09:09.133636    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d484b41d-4a92-4593-8992-156493609c27-home" (OuterVolumeSpecName: "home") pod "d484b41d-4a92-4593-8992-156493609c27" (UID: "d484b41d-4a92-4593-8992-156493609c27"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:09:09.133740 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:09:09.133691    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d484b41d-4a92-4593-8992-156493609c27-model-cache" (OuterVolumeSpecName: "model-cache") pod "d484b41d-4a92-4593-8992-156493609c27" (UID: "d484b41d-4a92-4593-8992-156493609c27"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:09:09.133740 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:09:09.133720    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d484b41d-4a92-4593-8992-156493609c27-tmp-dir\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:09:09.133740 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:09:09.133736    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d484b41d-4a92-4593-8992-156493609c27-home\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:09:09.135703 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:09:09.135671    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d484b41d-4a92-4593-8992-156493609c27-dshm" (OuterVolumeSpecName: "dshm") pod "d484b41d-4a92-4593-8992-156493609c27" (UID: "d484b41d-4a92-4593-8992-156493609c27"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:09:09.135703 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:09:09.135685    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d484b41d-4a92-4593-8992-156493609c27-kube-api-access-mxnck" (OuterVolumeSpecName: "kube-api-access-mxnck") pod "d484b41d-4a92-4593-8992-156493609c27" (UID: "d484b41d-4a92-4593-8992-156493609c27"). InnerVolumeSpecName "kube-api-access-mxnck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:09:09.135856 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:09:09.135743    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d484b41d-4a92-4593-8992-156493609c27-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d484b41d-4a92-4593-8992-156493609c27" (UID: "d484b41d-4a92-4593-8992-156493609c27"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:09:09.234832 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:09:09.234789    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d484b41d-4a92-4593-8992-156493609c27-tls-certs\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:09:09.234832 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:09:09.234825    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d484b41d-4a92-4593-8992-156493609c27-model-cache\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:09:09.234832 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:09:09.234834    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d484b41d-4a92-4593-8992-156493609c27-dshm\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:09:09.234832 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:09:09.234843    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxnck\" (UniqueName: \"kubernetes.io/projected/d484b41d-4a92-4593-8992-156493609c27-kube-api-access-mxnck\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:09:09.389072 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:09:09.389037    2559 generic.go:356] "Generic (PLEG): container finished" podID="d484b41d-4a92-4593-8992-156493609c27" containerID="333d9183c51204973c1d825a100c7849df249989af152d7df94498d752be5f2b" exitCode=0
Jul 29 19:09:09.389269 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:09:09.389083    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8eafda81/e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx"
Jul 29 19:09:09.389269 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:09:09.389094    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8eafda81/e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx" event={"ID":"d484b41d-4a92-4593-8992-156493609c27","Type":"ContainerDied","Data":"333d9183c51204973c1d825a100c7849df249989af152d7df94498d752be5f2b"}
Jul 29 19:09:09.389269 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:09:09.389128    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8eafda81/e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx" event={"ID":"d484b41d-4a92-4593-8992-156493609c27","Type":"ContainerDied","Data":"83ad9bd5ebe51b22d2e7485aef7cf8866f129ad0b1bc059c240da7cc93edf5a7"}
Jul 29 19:09:09.389269 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:09:09.389144    2559 scope.go:117] "RemoveContainer" containerID="333d9183c51204973c1d825a100c7849df249989af152d7df94498d752be5f2b"
Jul 29 19:09:09.398871 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:09:09.398851    2559 scope.go:117] "RemoveContainer" containerID="333d9183c51204973c1d825a100c7849df249989af152d7df94498d752be5f2b"
Jul 29 19:09:09.399176 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:09:09.399156    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"333d9183c51204973c1d825a100c7849df249989af152d7df94498d752be5f2b\": container with ID starting with 333d9183c51204973c1d825a100c7849df249989af152d7df94498d752be5f2b not found: ID does not exist" containerID="333d9183c51204973c1d825a100c7849df249989af152d7df94498d752be5f2b"
Jul 29 19:09:09.399224 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:09:09.399186    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"333d9183c51204973c1d825a100c7849df249989af152d7df94498d752be5f2b"} err="failed to get container status \"333d9183c51204973c1d825a100c7849df249989af152d7df94498d752be5f2b\": rpc error: code = NotFound desc = could not find container \"333d9183c51204973c1d825a100c7849df249989af152d7df94498d752be5f2b\": container with ID starting with 333d9183c51204973c1d825a100c7849df249989af152d7df94498d752be5f2b not found: ID does not exist"
Jul 29 19:09:09.411272 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:09:09.411238    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-8eafda81/e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx"]
Jul 29 19:09:09.416270 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:09:09.416244    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-8eafda81/e2e-lifecycle-model-8eafda81-kserve-547d6ff867-l7dvx"]
Jul 29 19:09:10.150122 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:09:10.150090    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d484b41d-4a92-4593-8992-156493609c27" path="/var/lib/kubelet/pods/d484b41d-4a92-4593-8992-156493609c27/volumes"
Jul 29 19:09:40.844510 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:09:40.844474    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:09:52.576391 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:09:52.576348    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:09:56.357799 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:09:56.357760    2559 scope.go:117] "RemoveContainer" containerID="e28908aa6bffcb2ae48a4ac2425b5bc8705e89aa8ecf0d74bea976248aae5587"
Jul 29 19:09:56.366795 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:09:56.366772    2559 scope.go:117] "RemoveContainer" containerID="791abd3d75dce515eceaf7a672cc680c627e139494a7776ede0aa2f787538a05"
Jul 29 19:10:37.988908 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:37.988869    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e3eaaf-64f4f84984-b8wdj"]
Jul 29 19:10:37.989450 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:37.989253    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cffee44-40f7-4f83-b91b-8bc676fdd428" containerName="payload-pre-processing"
Jul 29 19:10:37.989450 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:37.989267    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cffee44-40f7-4f83-b91b-8bc676fdd428" containerName="payload-pre-processing"
Jul 29 19:10:37.989450 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:37.989278    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8c41cda-13ec-418d-ba4b-7cb8951b022a" containerName="revoke-keys"
Jul 29 19:10:37.989450 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:37.989284    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8c41cda-13ec-418d-ba4b-7cb8951b022a" containerName="revoke-keys"
Jul 29 19:10:37.989450 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:37.989298    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="068168b4-c06b-4041-a979-345750b43c6c" containerName="payload-processing"
Jul 29 19:10:37.989450 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:37.989303    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="068168b4-c06b-4041-a979-345750b43c6c" containerName="payload-processing"
Jul 29 19:10:37.989450 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:37.989309    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d484b41d-4a92-4593-8992-156493609c27" containerName="main"
Jul 29 19:10:37.989450 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:37.989314    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d484b41d-4a92-4593-8992-156493609c27" containerName="main"
Jul 29 19:10:37.989450 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:37.989369    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cffee44-40f7-4f83-b91b-8bc676fdd428" containerName="payload-pre-processing"
Jul 29 19:10:37.989450 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:37.989374    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="068168b4-c06b-4041-a979-345750b43c6c" containerName="payload-processing"
Jul 29 19:10:37.989450 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:37.989396    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d484b41d-4a92-4593-8992-156493609c27" containerName="main"
Jul 29 19:10:37.989450 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:37.989403    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8c41cda-13ec-418d-ba4b-7cb8951b022a" containerName="revoke-keys"
Jul 29 19:10:37.992442 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:37.992426    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3eaaf-64f4f84984-b8wdj"
Jul 29 19:10:37.996099 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:37.996075    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-e3eaaf-dockercfg-gspmc\""
Jul 29 19:10:37.996340 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:37.996324    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-e3eaaf\""
Jul 29 19:10:38.006364 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:38.006333    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b968b2f-bde7-4f5c-8be1-056ad58fb47c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e3eaaf-64f4f84984-b8wdj\" (UID: \"8b968b2f-bde7-4f5c-8be1-056ad58fb47c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3eaaf-64f4f84984-b8wdj"
Jul 29 19:10:38.006525 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:38.006417    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t6l8\" (UniqueName: \"kubernetes.io/projected/8b968b2f-bde7-4f5c-8be1-056ad58fb47c-kube-api-access-2t6l8\") pod \"payload-pre-processing-e2e-inf-a-e3eaaf-64f4f84984-b8wdj\" (UID: \"8b968b2f-bde7-4f5c-8be1-056ad58fb47c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3eaaf-64f4f84984-b8wdj"
Jul 29 19:10:38.013876 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:38.013839    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e3eaaf-64f4f84984-b8wdj"]
Jul 29 19:10:38.107290 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:38.107250    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t6l8\" (UniqueName: \"kubernetes.io/projected/8b968b2f-bde7-4f5c-8be1-056ad58fb47c-kube-api-access-2t6l8\") pod \"payload-pre-processing-e2e-inf-a-e3eaaf-64f4f84984-b8wdj\" (UID: \"8b968b2f-bde7-4f5c-8be1-056ad58fb47c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3eaaf-64f4f84984-b8wdj"
Jul 29 19:10:38.107579 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:38.107554    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b968b2f-bde7-4f5c-8be1-056ad58fb47c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e3eaaf-64f4f84984-b8wdj\" (UID: \"8b968b2f-bde7-4f5c-8be1-056ad58fb47c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3eaaf-64f4f84984-b8wdj"
Jul 29 19:10:38.108271 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:38.108245    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b968b2f-bde7-4f5c-8be1-056ad58fb47c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e3eaaf-64f4f84984-b8wdj\" (UID: \"8b968b2f-bde7-4f5c-8be1-056ad58fb47c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3eaaf-64f4f84984-b8wdj"
Jul 29 19:10:38.119012 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:38.118976    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e3eaaf-6b9549cf7b-2bzx9"]
Jul 29 19:10:38.121079 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:38.121053    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t6l8\" (UniqueName: \"kubernetes.io/projected/8b968b2f-bde7-4f5c-8be1-056ad58fb47c-kube-api-access-2t6l8\") pod \"payload-pre-processing-e2e-inf-a-e3eaaf-64f4f84984-b8wdj\" (UID: \"8b968b2f-bde7-4f5c-8be1-056ad58fb47c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3eaaf-64f4f84984-b8wdj"
Jul 29 19:10:38.122477 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:38.122459    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e3eaaf-6b9549cf7b-2bzx9"
Jul 29 19:10:38.133707 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:38.133679    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e3eaaf-6b9549cf7b-2bzx9"]
Jul 29 19:10:38.208428 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:38.208353    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e4e42e4-84c8-4fa7-ae50-dbede7ec6e5c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e3eaaf-6b9549cf7b-2bzx9\" (UID: \"9e4e42e4-84c8-4fa7-ae50-dbede7ec6e5c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e3eaaf-6b9549cf7b-2bzx9"
Jul 29 19:10:38.208428 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:38.208426    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch8qc\" (UniqueName: \"kubernetes.io/projected/9e4e42e4-84c8-4fa7-ae50-dbede7ec6e5c-kube-api-access-ch8qc\") pod \"payload-processing-e2e-inf-a-e3eaaf-6b9549cf7b-2bzx9\" (UID: \"9e4e42e4-84c8-4fa7-ae50-dbede7ec6e5c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e3eaaf-6b9549cf7b-2bzx9"
Jul 29 19:10:38.303198 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:38.303097    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3eaaf-64f4f84984-b8wdj"
Jul 29 19:10:38.309409 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:38.309363    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e4e42e4-84c8-4fa7-ae50-dbede7ec6e5c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e3eaaf-6b9549cf7b-2bzx9\" (UID: \"9e4e42e4-84c8-4fa7-ae50-dbede7ec6e5c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e3eaaf-6b9549cf7b-2bzx9"
Jul 29 19:10:38.309409 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:38.309413    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch8qc\" (UniqueName: \"kubernetes.io/projected/9e4e42e4-84c8-4fa7-ae50-dbede7ec6e5c-kube-api-access-ch8qc\") pod \"payload-processing-e2e-inf-a-e3eaaf-6b9549cf7b-2bzx9\" (UID: \"9e4e42e4-84c8-4fa7-ae50-dbede7ec6e5c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e3eaaf-6b9549cf7b-2bzx9"
Jul 29 19:10:38.310116 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:38.310091    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e4e42e4-84c8-4fa7-ae50-dbede7ec6e5c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e3eaaf-6b9549cf7b-2bzx9\" (UID: \"9e4e42e4-84c8-4fa7-ae50-dbede7ec6e5c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e3eaaf-6b9549cf7b-2bzx9"
Jul 29 19:10:38.318958 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:38.318899    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch8qc\" (UniqueName: \"kubernetes.io/projected/9e4e42e4-84c8-4fa7-ae50-dbede7ec6e5c-kube-api-access-ch8qc\") pod \"payload-processing-e2e-inf-a-e3eaaf-6b9549cf7b-2bzx9\" (UID: \"9e4e42e4-84c8-4fa7-ae50-dbede7ec6e5c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e3eaaf-6b9549cf7b-2bzx9"
Jul 29 19:10:38.440878 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:38.440839    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e3eaaf-6b9549cf7b-2bzx9"
Jul 29 19:10:38.452413 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:38.452357    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e3eaaf-64f4f84984-b8wdj"]
Jul 29 19:10:38.456227 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:10:38.456198    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b968b2f_bde7_4f5c_8be1_056ad58fb47c.slice/crio-ce7a16b9c1bcc8bf916e74256ebb45396536336a6517a00071e85ff43f71412d WatchSource:0}: Error finding container ce7a16b9c1bcc8bf916e74256ebb45396536336a6517a00071e85ff43f71412d: Status 404 returned error can't find the container with id ce7a16b9c1bcc8bf916e74256ebb45396536336a6517a00071e85ff43f71412d
Jul 29 19:10:38.596002 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:38.595973    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e3eaaf-6b9549cf7b-2bzx9"]
Jul 29 19:10:38.598562 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:10:38.598533    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e4e42e4_84c8_4fa7_ae50_dbede7ec6e5c.slice/crio-517818a2514369f1a1e14510b2535c8af2b66e59788e7c00b03fd9ebd12f9361 WatchSource:0}: Error finding container 517818a2514369f1a1e14510b2535c8af2b66e59788e7c00b03fd9ebd12f9361: Status 404 returned error can't find the container with id 517818a2514369f1a1e14510b2535c8af2b66e59788e7c00b03fd9ebd12f9361
Jul 29 19:10:38.821758 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:38.821714    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3eaaf-64f4f84984-b8wdj" event={"ID":"8b968b2f-bde7-4f5c-8be1-056ad58fb47c","Type":"ContainerStarted","Data":"161dba36d900c1ede7ded59890e066e8f26c47c9a09ec0406ca7757697737c6d"}
Jul 29 19:10:38.821949 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:38.821787    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3eaaf-64f4f84984-b8wdj"
Jul 29 19:10:38.821949 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:38.821810    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3eaaf-64f4f84984-b8wdj" event={"ID":"8b968b2f-bde7-4f5c-8be1-056ad58fb47c","Type":"ContainerStarted","Data":"ce7a16b9c1bcc8bf916e74256ebb45396536336a6517a00071e85ff43f71412d"}
Jul 29 19:10:38.823474 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:38.823434    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e3eaaf-6b9549cf7b-2bzx9" event={"ID":"9e4e42e4-84c8-4fa7-ae50-dbede7ec6e5c","Type":"ContainerStarted","Data":"63ee3db55d3d8e01a155bd1b14aef0eef5a50d77acf97eca8e3f3f91e443d29b"}
Jul 29 19:10:38.823609 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:38.823490    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e3eaaf-6b9549cf7b-2bzx9" event={"ID":"9e4e42e4-84c8-4fa7-ae50-dbede7ec6e5c","Type":"ContainerStarted","Data":"517818a2514369f1a1e14510b2535c8af2b66e59788e7c00b03fd9ebd12f9361"}
Jul 29 19:10:38.823609 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:38.823601    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e3eaaf-6b9549cf7b-2bzx9"
Jul 29 19:10:38.840613 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:38.840560    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3eaaf-64f4f84984-b8wdj" podStartSLOduration=1.840543657 podStartE2EDuration="1.840543657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:10:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:10:38.838409871 +0000 UTC m=+2203.108178953" watchObservedRunningTime="2026-07-29 19:10:38.840543657 +0000 UTC m=+2203.110312746"
Jul 29 19:10:38.860355 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:38.860244    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-e3eaaf-6b9549cf7b-2bzx9" podStartSLOduration=0.860228486 podStartE2EDuration="860.228486ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:10:38.858622081 +0000 UTC m=+2203.128391167" watchObservedRunningTime="2026-07-29 19:10:38.860228486 +0000 UTC m=+2203.129997573"
Jul 29 19:10:41.568561 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:41.568518    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:10:49.833032 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:49.832996    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e3eaaf-6b9549cf7b-2bzx9"
Jul 29 19:10:49.833476 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:49.833351    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3eaaf-64f4f84984-b8wdj"
Jul 29 19:10:56.398784 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:10:56.398750    2559 scope.go:117] "RemoveContainer" containerID="f6ac213d42d01707ffaaeafa11db4dc6fdc1c90d0440f5cb5c330ebe9dd4265c"
Jul 29 19:11:06.142228 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:06.142192    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g"]
Jul 29 19:11:06.146727 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:06.146698    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g"
Jul 29 19:11:06.150877 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:06.150852    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-5b4a8e-serving-cert\""
Jul 29 19:11:06.151872 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:06.151843    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Jul 29 19:11:06.152012 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:06.151885    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mglnh\""
Jul 29 19:11:06.152012 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:06.151894    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 29 19:11:06.154828 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:06.154809    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 19:11:06.158811 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:06.158782    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g"]
Jul 29 19:11:06.251731 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:06.251689    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk69d\" (UniqueName: \"kubernetes.io/projected/51dd7f67-7b58-4010-8716-305a4ec58ccf-kube-api-access-rk69d\") pod \"maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g\" (UID: \"51dd7f67-7b58-4010-8716-305a4ec58ccf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g"
Jul 29 19:11:06.251952 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:06.251742    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51dd7f67-7b58-4010-8716-305a4ec58ccf-metrics-certs\") pod \"maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g\" (UID: \"51dd7f67-7b58-4010-8716-305a4ec58ccf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g"
Jul 29 19:11:06.251952 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:06.251881    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51dd7f67-7b58-4010-8716-305a4ec58ccf-maas-api-tls\") pod \"maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g\" (UID: \"51dd7f67-7b58-4010-8716-305a4ec58ccf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g"
Jul 29 19:11:06.251952 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:06.251906    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51dd7f67-7b58-4010-8716-305a4ec58ccf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g\" (UID: \"51dd7f67-7b58-4010-8716-305a4ec58ccf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g"
Jul 29 19:11:06.352749 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:06.352706    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51dd7f67-7b58-4010-8716-305a4ec58ccf-maas-api-tls\") pod \"maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g\" (UID: \"51dd7f67-7b58-4010-8716-305a4ec58ccf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g"
Jul 29 19:11:06.352749 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:06.352747    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51dd7f67-7b58-4010-8716-305a4ec58ccf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g\" (UID: \"51dd7f67-7b58-4010-8716-305a4ec58ccf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g"
Jul 29 19:11:06.353059 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:06.352771    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk69d\" (UniqueName: \"kubernetes.io/projected/51dd7f67-7b58-4010-8716-305a4ec58ccf-kube-api-access-rk69d\") pod \"maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g\" (UID: \"51dd7f67-7b58-4010-8716-305a4ec58ccf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g"
Jul 29 19:11:06.353059 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:06.352810    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51dd7f67-7b58-4010-8716-305a4ec58ccf-metrics-certs\") pod \"maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g\" (UID: \"51dd7f67-7b58-4010-8716-305a4ec58ccf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g"
Jul 29 19:11:06.353059 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:11:06.352866    2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-5b4a8e-serving-cert: secret "maas-api-e2e-inf-b-5b4a8e-serving-cert" not found
Jul 29 19:11:06.353059 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:11:06.352951    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51dd7f67-7b58-4010-8716-305a4ec58ccf-maas-api-tls podName:51dd7f67-7b58-4010-8716-305a4ec58ccf nodeName:}" failed. No retries permitted until 2026-07-29 19:11:06.852931528 +0000 UTC m=+2231.122700593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/51dd7f67-7b58-4010-8716-305a4ec58ccf-maas-api-tls") pod "maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g" (UID: "51dd7f67-7b58-4010-8716-305a4ec58ccf") : secret "maas-api-e2e-inf-b-5b4a8e-serving-cert" not found
Jul 29 19:11:06.353589 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:06.353562    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51dd7f67-7b58-4010-8716-305a4ec58ccf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g\" (UID: \"51dd7f67-7b58-4010-8716-305a4ec58ccf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g"
Jul 29 19:11:06.355659 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:06.355636    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51dd7f67-7b58-4010-8716-305a4ec58ccf-metrics-certs\") pod \"maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g\" (UID: \"51dd7f67-7b58-4010-8716-305a4ec58ccf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g"
Jul 29 19:11:06.364942 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:06.364904    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk69d\" (UniqueName: \"kubernetes.io/projected/51dd7f67-7b58-4010-8716-305a4ec58ccf-kube-api-access-rk69d\") pod \"maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g\" (UID: \"51dd7f67-7b58-4010-8716-305a4ec58ccf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g"
Jul 29 19:11:06.857754 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:06.857712    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51dd7f67-7b58-4010-8716-305a4ec58ccf-maas-api-tls\") pod \"maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g\" (UID: \"51dd7f67-7b58-4010-8716-305a4ec58ccf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g"
Jul 29 19:11:06.860600 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:06.860571    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51dd7f67-7b58-4010-8716-305a4ec58ccf-maas-api-tls\") pod \"maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g\" (UID: \"51dd7f67-7b58-4010-8716-305a4ec58ccf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g"
Jul 29 19:11:06.870168 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:06.870137    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5b4a8e-68f7d49cd4-4v5kg"]
Jul 29 19:11:06.873722 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:06.873706    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5b4a8e-68f7d49cd4-4v5kg"
Jul 29 19:11:06.877165 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:06.877139    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-5b4a8e\""
Jul 29 19:11:06.894757 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:06.894716    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5b4a8e-68f7d49cd4-4v5kg"]
Jul 29 19:11:06.906140 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:06.906105    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5b4a8e-6449784d58-8mqvc"]
Jul 29 19:11:06.909635 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:06.909611    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5b4a8e-6449784d58-8mqvc"
Jul 29 19:11:06.914438 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:06.914412    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-5b4a8e-dockercfg-wmtk6\""
Jul 29 19:11:06.933111 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:06.933077    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5b4a8e-6449784d58-8mqvc"]
Jul 29 19:11:06.958441 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:06.958355    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsvqb\" (UniqueName: \"kubernetes.io/projected/e94f994e-3d76-4b7e-ad2c-a3d2a5ed87db-kube-api-access-dsvqb\") pod \"payload-pre-processing-e2e-inf-b-5b4a8e-68f7d49cd4-4v5kg\" (UID: \"e94f994e-3d76-4b7e-ad2c-a3d2a5ed87db\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5b4a8e-68f7d49cd4-4v5kg"
Jul 29 19:11:06.958669 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:06.958462    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e94f994e-3d76-4b7e-ad2c-a3d2a5ed87db-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5b4a8e-68f7d49cd4-4v5kg\" (UID: \"e94f994e-3d76-4b7e-ad2c-a3d2a5ed87db\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5b4a8e-68f7d49cd4-4v5kg"
Jul 29 19:11:07.059504 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:07.059463    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsvqb\" (UniqueName: \"kubernetes.io/projected/e94f994e-3d76-4b7e-ad2c-a3d2a5ed87db-kube-api-access-dsvqb\") pod \"payload-pre-processing-e2e-inf-b-5b4a8e-68f7d49cd4-4v5kg\" (UID: \"e94f994e-3d76-4b7e-ad2c-a3d2a5ed87db\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5b4a8e-68f7d49cd4-4v5kg"
Jul 29 19:11:07.059504 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:07.059505    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e94f994e-3d76-4b7e-ad2c-a3d2a5ed87db-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5b4a8e-68f7d49cd4-4v5kg\" (UID: \"e94f994e-3d76-4b7e-ad2c-a3d2a5ed87db\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5b4a8e-68f7d49cd4-4v5kg"
Jul 29 19:11:07.059766 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:07.059551    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m624\" (UniqueName: \"kubernetes.io/projected/feadcbd5-235f-4380-9ea4-33e7dc69571b-kube-api-access-9m624\") pod \"payload-processing-e2e-inf-b-5b4a8e-6449784d58-8mqvc\" (UID: \"feadcbd5-235f-4380-9ea4-33e7dc69571b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5b4a8e-6449784d58-8mqvc"
Jul 29 19:11:07.059766 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:07.059583    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feadcbd5-235f-4380-9ea4-33e7dc69571b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5b4a8e-6449784d58-8mqvc\" (UID: \"feadcbd5-235f-4380-9ea4-33e7dc69571b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5b4a8e-6449784d58-8mqvc"
Jul 29 19:11:07.059766 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:07.059740    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g"
Jul 29 19:11:07.060351 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:07.060328    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e94f994e-3d76-4b7e-ad2c-a3d2a5ed87db-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5b4a8e-68f7d49cd4-4v5kg\" (UID: \"e94f994e-3d76-4b7e-ad2c-a3d2a5ed87db\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5b4a8e-68f7d49cd4-4v5kg"
Jul 29 19:11:07.071693 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:07.071663    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsvqb\" (UniqueName: \"kubernetes.io/projected/e94f994e-3d76-4b7e-ad2c-a3d2a5ed87db-kube-api-access-dsvqb\") pod \"payload-pre-processing-e2e-inf-b-5b4a8e-68f7d49cd4-4v5kg\" (UID: \"e94f994e-3d76-4b7e-ad2c-a3d2a5ed87db\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5b4a8e-68f7d49cd4-4v5kg"
Jul 29 19:11:07.160233 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:07.160200    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feadcbd5-235f-4380-9ea4-33e7dc69571b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5b4a8e-6449784d58-8mqvc\" (UID: \"feadcbd5-235f-4380-9ea4-33e7dc69571b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5b4a8e-6449784d58-8mqvc"
Jul 29 19:11:07.160699 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:07.160321    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m624\" (UniqueName: \"kubernetes.io/projected/feadcbd5-235f-4380-9ea4-33e7dc69571b-kube-api-access-9m624\") pod \"payload-processing-e2e-inf-b-5b4a8e-6449784d58-8mqvc\" (UID: \"feadcbd5-235f-4380-9ea4-33e7dc69571b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5b4a8e-6449784d58-8mqvc"
Jul 29 19:11:07.160997 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:07.160976    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feadcbd5-235f-4380-9ea4-33e7dc69571b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5b4a8e-6449784d58-8mqvc\" (UID: \"feadcbd5-235f-4380-9ea4-33e7dc69571b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5b4a8e-6449784d58-8mqvc"
Jul 29 19:11:07.184634 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:07.184404    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5b4a8e-68f7d49cd4-4v5kg"
Jul 29 19:11:07.184952 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:07.184927    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m624\" (UniqueName: \"kubernetes.io/projected/feadcbd5-235f-4380-9ea4-33e7dc69571b-kube-api-access-9m624\") pod \"payload-processing-e2e-inf-b-5b4a8e-6449784d58-8mqvc\" (UID: \"feadcbd5-235f-4380-9ea4-33e7dc69571b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5b4a8e-6449784d58-8mqvc"
Jul 29 19:11:07.204897 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:07.204803    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g"]
Jul 29 19:11:07.207718 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:11:07.207688    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51dd7f67_7b58_4010_8716_305a4ec58ccf.slice/crio-42f599570b780eb43e56ea12fc4df93d36d57f7654fa3120ffe953b1c23f419e WatchSource:0}: Error finding container 42f599570b780eb43e56ea12fc4df93d36d57f7654fa3120ffe953b1c23f419e: Status 404 returned error can't find the container with id 42f599570b780eb43e56ea12fc4df93d36d57f7654fa3120ffe953b1c23f419e
Jul 29 19:11:07.209159 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:07.209139    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 19:11:07.221524 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:07.221499    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5b4a8e-6449784d58-8mqvc"
Jul 29 19:11:07.347316 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:11:07.347275    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94f994e_3d76_4b7e_ad2c_a3d2a5ed87db.slice/crio-895840636d5df0553076abf22a248b28bacac881eda2d3ecc12c4f43dc89b1dc WatchSource:0}: Error finding container 895840636d5df0553076abf22a248b28bacac881eda2d3ecc12c4f43dc89b1dc: Status 404 returned error can't find the container with id 895840636d5df0553076abf22a248b28bacac881eda2d3ecc12c4f43dc89b1dc
Jul 29 19:11:07.347606 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:07.347584    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5b4a8e-68f7d49cd4-4v5kg"]
Jul 29 19:11:07.595551 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:07.595520    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5b4a8e-6449784d58-8mqvc"]
Jul 29 19:11:07.596429 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:11:07.596401    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeadcbd5_235f_4380_9ea4_33e7dc69571b.slice/crio-7501c0c1b49cd15517e878e47875f63349d167878554db5c3b5b8fda9b79a83a WatchSource:0}: Error finding container 7501c0c1b49cd15517e878e47875f63349d167878554db5c3b5b8fda9b79a83a: Status 404 returned error can't find the container with id 7501c0c1b49cd15517e878e47875f63349d167878554db5c3b5b8fda9b79a83a
Jul 29 19:11:07.969527 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:07.969433    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5b4a8e-68f7d49cd4-4v5kg" event={"ID":"e94f994e-3d76-4b7e-ad2c-a3d2a5ed87db","Type":"ContainerStarted","Data":"2e2e0db79f5163cba1e8c0f3a607e64851238a271eedd36e49f3faaed9a0f866"}
Jul 29 19:11:07.969854 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:07.969606    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5b4a8e-68f7d49cd4-4v5kg"
Jul 29 19:11:07.969854 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:07.969653    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5b4a8e-68f7d49cd4-4v5kg" event={"ID":"e94f994e-3d76-4b7e-ad2c-a3d2a5ed87db","Type":"ContainerStarted","Data":"895840636d5df0553076abf22a248b28bacac881eda2d3ecc12c4f43dc89b1dc"}
Jul 29 19:11:07.971362 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:07.971310    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5b4a8e-6449784d58-8mqvc" event={"ID":"feadcbd5-235f-4380-9ea4-33e7dc69571b","Type":"ContainerStarted","Data":"8170013af2e62ae534cd9e54f4ae447349fd3cf56b4dabbcabf39d663567b27e"}
Jul 29 19:11:07.971362 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:07.971354    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5b4a8e-6449784d58-8mqvc" event={"ID":"feadcbd5-235f-4380-9ea4-33e7dc69571b","Type":"ContainerStarted","Data":"7501c0c1b49cd15517e878e47875f63349d167878554db5c3b5b8fda9b79a83a"}
Jul 29 19:11:07.971362 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:07.971410    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5b4a8e-6449784d58-8mqvc"
Jul 29 19:11:07.972811 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:07.972783    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g" event={"ID":"51dd7f67-7b58-4010-8716-305a4ec58ccf","Type":"ContainerStarted","Data":"42f599570b780eb43e56ea12fc4df93d36d57f7654fa3120ffe953b1c23f419e"}
Jul 29 19:11:07.988973 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:07.988882    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5b4a8e-68f7d49cd4-4v5kg" podStartSLOduration=1.988863592 podStartE2EDuration="1.988863592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:11:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:11:07.986024387 +0000 UTC m=+2232.255793475" watchObservedRunningTime="2026-07-29 19:11:07.988863592 +0000 UTC m=+2232.258632700"
Jul 29 19:11:08.008083 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:08.008016    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-5b4a8e-6449784d58-8mqvc" podStartSLOduration=2.007994142 podStartE2EDuration="2.007994142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:11:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:11:08.007590081 +0000 UTC m=+2232.277359169" watchObservedRunningTime="2026-07-29 19:11:08.007994142 +0000 UTC m=+2232.277763229"
Jul 29 19:11:09.993917 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:09.993871    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g" event={"ID":"51dd7f67-7b58-4010-8716-305a4ec58ccf","Type":"ContainerStarted","Data":"30041b87c5d74341cd9f77ca9177c0071b700e9030da58d16de66735c121175f"}
Jul 29 19:11:09.994355 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:09.993975    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g"
Jul 29 19:11:10.012596 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:10.012547    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g" podStartSLOduration=1.7121279280000001 podStartE2EDuration="4.012531602s" totalImagesPullingTime="2.300403674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:11:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:11:10.011891216 +0000 UTC m=+2234.281660332" watchObservedRunningTime="2026-07-29 19:11:10.012531602 +0000 UTC m=+2234.282300689"
Jul 29 19:11:13.139913 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:13.139871    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:11:16.004084 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:16.004054    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g"
Jul 29 19:11:18.988883 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:18.988803    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5b4a8e-6449784d58-8mqvc"
Jul 29 19:11:18.989437 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:18.989141    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5b4a8e-68f7d49cd4-4v5kg"
Jul 29 19:11:37.693773 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:37.693739    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-e3eaaf/test-model-e3eaaf-kserve-69fbf46cbc-85b4w"]
Jul 29 19:11:37.697419 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:37.697399    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e3eaaf/test-model-e3eaaf-kserve-69fbf46cbc-85b4w"
Jul 29 19:11:37.702155 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:37.702123    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e3eaaf\"/\"openshift-service-ca.crt\""
Jul 29 19:11:37.702317 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:37.702163    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e3eaaf\"/\"kube-root-ca.crt\""
Jul 29 19:11:37.702317 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:37.702124    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e3eaaf\"/\"test-model-e3eaaf-kserve-self-signed-certs\""
Jul 29 19:11:37.702317 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:37.702124    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e3eaaf\"/\"default-dockercfg-4s2r9\""
Jul 29 19:11:37.709623 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:37.709593    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e3eaaf/test-model-e3eaaf-kserve-69fbf46cbc-85b4w"]
Jul 29 19:11:37.731979 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:37.731941    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/459ec5b7-7dce-4591-92b3-5a2a4ec6257d-tmp-dir\") pod \"test-model-e3eaaf-kserve-69fbf46cbc-85b4w\" (UID: \"459ec5b7-7dce-4591-92b3-5a2a4ec6257d\") " pod="ai-tenant-e2e-inf-a-e3eaaf/test-model-e3eaaf-kserve-69fbf46cbc-85b4w"
Jul 29 19:11:37.731979 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:37.731984    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/459ec5b7-7dce-4591-92b3-5a2a4ec6257d-home\") pod \"test-model-e3eaaf-kserve-69fbf46cbc-85b4w\" (UID: \"459ec5b7-7dce-4591-92b3-5a2a4ec6257d\") " pod="ai-tenant-e2e-inf-a-e3eaaf/test-model-e3eaaf-kserve-69fbf46cbc-85b4w"
Jul 29 19:11:37.732237 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:37.732010    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hx45\" (UniqueName: \"kubernetes.io/projected/459ec5b7-7dce-4591-92b3-5a2a4ec6257d-kube-api-access-2hx45\") pod \"test-model-e3eaaf-kserve-69fbf46cbc-85b4w\" (UID: \"459ec5b7-7dce-4591-92b3-5a2a4ec6257d\") " pod="ai-tenant-e2e-inf-a-e3eaaf/test-model-e3eaaf-kserve-69fbf46cbc-85b4w"
Jul 29 19:11:37.732237 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:37.732078    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/459ec5b7-7dce-4591-92b3-5a2a4ec6257d-tls-certs\") pod \"test-model-e3eaaf-kserve-69fbf46cbc-85b4w\" (UID: \"459ec5b7-7dce-4591-92b3-5a2a4ec6257d\") " pod="ai-tenant-e2e-inf-a-e3eaaf/test-model-e3eaaf-kserve-69fbf46cbc-85b4w"
Jul 29 19:11:37.732237 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:37.732151    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/459ec5b7-7dce-4591-92b3-5a2a4ec6257d-dshm\") pod \"test-model-e3eaaf-kserve-69fbf46cbc-85b4w\" (UID: \"459ec5b7-7dce-4591-92b3-5a2a4ec6257d\") " pod="ai-tenant-e2e-inf-a-e3eaaf/test-model-e3eaaf-kserve-69fbf46cbc-85b4w"
Jul 29 19:11:37.732237 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:37.732200    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/459ec5b7-7dce-4591-92b3-5a2a4ec6257d-model-cache\") pod \"test-model-e3eaaf-kserve-69fbf46cbc-85b4w\" (UID: \"459ec5b7-7dce-4591-92b3-5a2a4ec6257d\") " pod="ai-tenant-e2e-inf-a-e3eaaf/test-model-e3eaaf-kserve-69fbf46cbc-85b4w"
Jul 29 19:11:37.833675 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:37.833624    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/459ec5b7-7dce-4591-92b3-5a2a4ec6257d-dshm\") pod \"test-model-e3eaaf-kserve-69fbf46cbc-85b4w\" (UID: \"459ec5b7-7dce-4591-92b3-5a2a4ec6257d\") " pod="ai-tenant-e2e-inf-a-e3eaaf/test-model-e3eaaf-kserve-69fbf46cbc-85b4w"
Jul 29 19:11:37.833887 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:37.833690    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/459ec5b7-7dce-4591-92b3-5a2a4ec6257d-model-cache\") pod \"test-model-e3eaaf-kserve-69fbf46cbc-85b4w\" (UID: \"459ec5b7-7dce-4591-92b3-5a2a4ec6257d\") " pod="ai-tenant-e2e-inf-a-e3eaaf/test-model-e3eaaf-kserve-69fbf46cbc-85b4w"
Jul 29 19:11:37.833887 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:37.833751    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/459ec5b7-7dce-4591-92b3-5a2a4ec6257d-tmp-dir\") pod \"test-model-e3eaaf-kserve-69fbf46cbc-85b4w\" (UID: \"459ec5b7-7dce-4591-92b3-5a2a4ec6257d\") " pod="ai-tenant-e2e-inf-a-e3eaaf/test-model-e3eaaf-kserve-69fbf46cbc-85b4w"
Jul 29 19:11:37.833887 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:37.833781    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/459ec5b7-7dce-4591-92b3-5a2a4ec6257d-home\") pod \"test-model-e3eaaf-kserve-69fbf46cbc-85b4w\" (UID: \"459ec5b7-7dce-4591-92b3-5a2a4ec6257d\") " pod="ai-tenant-e2e-inf-a-e3eaaf/test-model-e3eaaf-kserve-69fbf46cbc-85b4w"
Jul 29 19:11:37.833887 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:37.833820    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hx45\" (UniqueName: \"kubernetes.io/projected/459ec5b7-7dce-4591-92b3-5a2a4ec6257d-kube-api-access-2hx45\") pod \"test-model-e3eaaf-kserve-69fbf46cbc-85b4w\" (UID: \"459ec5b7-7dce-4591-92b3-5a2a4ec6257d\") " pod="ai-tenant-e2e-inf-a-e3eaaf/test-model-e3eaaf-kserve-69fbf46cbc-85b4w"
Jul 29 19:11:37.833887 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:37.833848    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/459ec5b7-7dce-4591-92b3-5a2a4ec6257d-tls-certs\") pod \"test-model-e3eaaf-kserve-69fbf46cbc-85b4w\" (UID: \"459ec5b7-7dce-4591-92b3-5a2a4ec6257d\") " pod="ai-tenant-e2e-inf-a-e3eaaf/test-model-e3eaaf-kserve-69fbf46cbc-85b4w"
Jul 29 19:11:37.834167 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:37.834124    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/459ec5b7-7dce-4591-92b3-5a2a4ec6257d-model-cache\") pod \"test-model-e3eaaf-kserve-69fbf46cbc-85b4w\" (UID: \"459ec5b7-7dce-4591-92b3-5a2a4ec6257d\") " pod="ai-tenant-e2e-inf-a-e3eaaf/test-model-e3eaaf-kserve-69fbf46cbc-85b4w"
Jul 29 19:11:37.834225 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:37.834197    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/459ec5b7-7dce-4591-92b3-5a2a4ec6257d-home\") pod \"test-model-e3eaaf-kserve-69fbf46cbc-85b4w\" (UID: \"459ec5b7-7dce-4591-92b3-5a2a4ec6257d\") " pod="ai-tenant-e2e-inf-a-e3eaaf/test-model-e3eaaf-kserve-69fbf46cbc-85b4w"
Jul 29 19:11:37.834282 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:37.834231    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/459ec5b7-7dce-4591-92b3-5a2a4ec6257d-tmp-dir\") pod \"test-model-e3eaaf-kserve-69fbf46cbc-85b4w\" (UID: \"459ec5b7-7dce-4591-92b3-5a2a4ec6257d\") " pod="ai-tenant-e2e-inf-a-e3eaaf/test-model-e3eaaf-kserve-69fbf46cbc-85b4w"
Jul 29 19:11:37.836129 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:37.836105    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/459ec5b7-7dce-4591-92b3-5a2a4ec6257d-dshm\") pod \"test-model-e3eaaf-kserve-69fbf46cbc-85b4w\" (UID: \"459ec5b7-7dce-4591-92b3-5a2a4ec6257d\") " pod="ai-tenant-e2e-inf-a-e3eaaf/test-model-e3eaaf-kserve-69fbf46cbc-85b4w"
Jul 29 19:11:37.836495 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:37.836473    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/459ec5b7-7dce-4591-92b3-5a2a4ec6257d-tls-certs\") pod \"test-model-e3eaaf-kserve-69fbf46cbc-85b4w\" (UID: \"459ec5b7-7dce-4591-92b3-5a2a4ec6257d\") " pod="ai-tenant-e2e-inf-a-e3eaaf/test-model-e3eaaf-kserve-69fbf46cbc-85b4w"
Jul 29 19:11:37.842168 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:37.842145    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hx45\" (UniqueName: \"kubernetes.io/projected/459ec5b7-7dce-4591-92b3-5a2a4ec6257d-kube-api-access-2hx45\") pod \"test-model-e3eaaf-kserve-69fbf46cbc-85b4w\" (UID: \"459ec5b7-7dce-4591-92b3-5a2a4ec6257d\") " pod="ai-tenant-e2e-inf-a-e3eaaf/test-model-e3eaaf-kserve-69fbf46cbc-85b4w"
Jul 29 19:11:38.006910 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:38.006813    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e3eaaf/test-model-e3eaaf-kserve-69fbf46cbc-85b4w"
Jul 29 19:11:38.157239 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:38.157214    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e3eaaf/test-model-e3eaaf-kserve-69fbf46cbc-85b4w"]
Jul 29 19:11:38.159948 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:11:38.159917    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod459ec5b7_7dce_4591_92b3_5a2a4ec6257d.slice/crio-5db79cdc822e08ac2f69eba4cb6d1b48843c5a06f27bab65e3f137aec2fb2bcd WatchSource:0}: Error finding container 5db79cdc822e08ac2f69eba4cb6d1b48843c5a06f27bab65e3f137aec2fb2bcd: Status 404 returned error can't find the container with id 5db79cdc822e08ac2f69eba4cb6d1b48843c5a06f27bab65e3f137aec2fb2bcd
Jul 29 19:11:39.141391 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:39.141353    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e3eaaf/test-model-e3eaaf-kserve-69fbf46cbc-85b4w" event={"ID":"459ec5b7-7dce-4591-92b3-5a2a4ec6257d","Type":"ContainerStarted","Data":"647cfe6659564642a0012f8d9efe2ad9f2e7dc237853c5d92bb2589be676a0dd"}
Jul 29 19:11:39.141793 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:39.141422    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e3eaaf/test-model-e3eaaf-kserve-69fbf46cbc-85b4w" event={"ID":"459ec5b7-7dce-4591-92b3-5a2a4ec6257d","Type":"ContainerStarted","Data":"5db79cdc822e08ac2f69eba4cb6d1b48843c5a06f27bab65e3f137aec2fb2bcd"}
Jul 29 19:11:39.159366 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:39.159315    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-e3eaaf/test-model-e3eaaf-kserve-69fbf46cbc-85b4w" podStartSLOduration=2.159298639 podStartE2EDuration="2.159298639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:11:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:11:39.159078355 +0000 UTC m=+2263.428847443" watchObservedRunningTime="2026-07-29 19:11:39.159298639 +0000 UTC m=+2263.429067776"
Jul 29 19:11:40.937225 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:40.937183    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:11:48.007717 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:48.007675    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-e3eaaf/test-model-e3eaaf-kserve-69fbf46cbc-85b4w"
Jul 29 19:11:48.007717 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:48.007731    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-e3eaaf/test-model-e3eaaf-kserve-69fbf46cbc-85b4w"
Jul 29 19:11:48.020123 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:48.020091    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-e3eaaf/test-model-e3eaaf-kserve-69fbf46cbc-85b4w"
Jul 29 19:11:48.198737 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:48.198702    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-e3eaaf/test-model-e3eaaf-kserve-69fbf46cbc-85b4w"
Jul 29 19:11:53.950692 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:11:53.950652    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:12:08.638838 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:12:08.638798    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:12:25.150303 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:12:25.150265    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:12:46.165621 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:12:46.165582    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e3eaaf/test-model-e3eaaf-kserve-69fbf46cbc-85b4w"]
Jul 29 19:12:50.534308 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:12:50.534214    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:12:56.427221 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:12:56.427186    2559 scope.go:117] "RemoveContainer" containerID="c9fc886d1a1b547c357429401e190fdad8a26982763b99370e5b3a426f963043"
Jul 29 19:12:57.745833 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:12:57.745668    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e3eaaf-6b9549cf7b-2bzx9"]
Jul 29 19:12:57.746311 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:12:57.745930    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-e3eaaf-6b9549cf7b-2bzx9" podUID="9e4e42e4-84c8-4fa7-ae50-dbede7ec6e5c" containerName="payload-processing" containerID="cri-o://63ee3db55d3d8e01a155bd1b14aef0eef5a50d77acf97eca8e3f3f91e443d29b" gracePeriod=30
Jul 29 19:12:57.757599 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:12:57.757560    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e3eaaf-64f4f84984-b8wdj"]
Jul 29 19:12:57.757853 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:12:57.757822    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3eaaf-64f4f84984-b8wdj" podUID="8b968b2f-bde7-4f5c-8be1-056ad58fb47c" containerName="payload-pre-processing" containerID="cri-o://161dba36d900c1ede7ded59890e066e8f26c47c9a09ec0406ca7757697737c6d" gracePeriod=30
Jul 29 19:12:58.045635 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:12:58.045587    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e3eaaf-6b9549cf7b-2bzx9"
Jul 29 19:12:58.048848 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:12:58.048822    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3eaaf-64f4f84984-b8wdj"
Jul 29 19:12:58.174226 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:12:58.174179    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ch8qc\" (UniqueName: \"kubernetes.io/projected/9e4e42e4-84c8-4fa7-ae50-dbede7ec6e5c-kube-api-access-ch8qc\") pod \"9e4e42e4-84c8-4fa7-ae50-dbede7ec6e5c\" (UID: \"9e4e42e4-84c8-4fa7-ae50-dbede7ec6e5c\") "
Jul 29 19:12:58.174226 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:12:58.174229    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e4e42e4-84c8-4fa7-ae50-dbede7ec6e5c-plugins-config-volume\") pod \"9e4e42e4-84c8-4fa7-ae50-dbede7ec6e5c\" (UID: \"9e4e42e4-84c8-4fa7-ae50-dbede7ec6e5c\") "
Jul 29 19:12:58.174536 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:12:58.174252    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2t6l8\" (UniqueName: \"kubernetes.io/projected/8b968b2f-bde7-4f5c-8be1-056ad58fb47c-kube-api-access-2t6l8\") pod \"8b968b2f-bde7-4f5c-8be1-056ad58fb47c\" (UID: \"8b968b2f-bde7-4f5c-8be1-056ad58fb47c\") "
Jul 29 19:12:58.174536 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:12:58.174322    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b968b2f-bde7-4f5c-8be1-056ad58fb47c-plugins-config-volume\") pod \"8b968b2f-bde7-4f5c-8be1-056ad58fb47c\" (UID: \"8b968b2f-bde7-4f5c-8be1-056ad58fb47c\") "
Jul 29 19:12:58.174716 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:12:58.174687    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e4e42e4-84c8-4fa7-ae50-dbede7ec6e5c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9e4e42e4-84c8-4fa7-ae50-dbede7ec6e5c" (UID: "9e4e42e4-84c8-4fa7-ae50-dbede7ec6e5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:12:58.174716 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:12:58.174701    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b968b2f-bde7-4f5c-8be1-056ad58fb47c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8b968b2f-bde7-4f5c-8be1-056ad58fb47c" (UID: "8b968b2f-bde7-4f5c-8be1-056ad58fb47c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:12:58.176715 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:12:58.176684    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b968b2f-bde7-4f5c-8be1-056ad58fb47c-kube-api-access-2t6l8" (OuterVolumeSpecName: "kube-api-access-2t6l8") pod "8b968b2f-bde7-4f5c-8be1-056ad58fb47c" (UID: "8b968b2f-bde7-4f5c-8be1-056ad58fb47c"). InnerVolumeSpecName "kube-api-access-2t6l8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:12:58.176813 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:12:58.176738    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e4e42e4-84c8-4fa7-ae50-dbede7ec6e5c-kube-api-access-ch8qc" (OuterVolumeSpecName: "kube-api-access-ch8qc") pod "9e4e42e4-84c8-4fa7-ae50-dbede7ec6e5c" (UID: "9e4e42e4-84c8-4fa7-ae50-dbede7ec6e5c"). InnerVolumeSpecName "kube-api-access-ch8qc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:12:58.276096 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:12:58.275819    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch8qc\" (UniqueName: \"kubernetes.io/projected/9e4e42e4-84c8-4fa7-ae50-dbede7ec6e5c-kube-api-access-ch8qc\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:12:58.276096 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:12:58.275858    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e4e42e4-84c8-4fa7-ae50-dbede7ec6e5c-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:12:58.276096 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:12:58.275873    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t6l8\" (UniqueName: \"kubernetes.io/projected/8b968b2f-bde7-4f5c-8be1-056ad58fb47c-kube-api-access-2t6l8\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:12:58.276096 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:12:58.275887    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b968b2f-bde7-4f5c-8be1-056ad58fb47c-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:12:58.549284 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:12:58.549189    2559 generic.go:356] "Generic (PLEG): container finished" podID="8b968b2f-bde7-4f5c-8be1-056ad58fb47c" containerID="161dba36d900c1ede7ded59890e066e8f26c47c9a09ec0406ca7757697737c6d" exitCode=0
Jul 29 19:12:58.549284 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:12:58.549233    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3eaaf-64f4f84984-b8wdj"
Jul 29 19:12:58.549536 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:12:58.549278    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3eaaf-64f4f84984-b8wdj" event={"ID":"8b968b2f-bde7-4f5c-8be1-056ad58fb47c","Type":"ContainerDied","Data":"161dba36d900c1ede7ded59890e066e8f26c47c9a09ec0406ca7757697737c6d"}
Jul 29 19:12:58.549536 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:12:58.549329    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3eaaf-64f4f84984-b8wdj" event={"ID":"8b968b2f-bde7-4f5c-8be1-056ad58fb47c","Type":"ContainerDied","Data":"ce7a16b9c1bcc8bf916e74256ebb45396536336a6517a00071e85ff43f71412d"}
Jul 29 19:12:58.549536 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:12:58.549361    2559 scope.go:117] "RemoveContainer" containerID="161dba36d900c1ede7ded59890e066e8f26c47c9a09ec0406ca7757697737c6d"
Jul 29 19:12:58.550730 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:12:58.550704    2559 generic.go:356] "Generic (PLEG): container finished" podID="9e4e42e4-84c8-4fa7-ae50-dbede7ec6e5c" containerID="63ee3db55d3d8e01a155bd1b14aef0eef5a50d77acf97eca8e3f3f91e443d29b" exitCode=0
Jul 29 19:12:58.550859 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:12:58.550754    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e3eaaf-6b9549cf7b-2bzx9"
Jul 29 19:12:58.550859 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:12:58.550782    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e3eaaf-6b9549cf7b-2bzx9" event={"ID":"9e4e42e4-84c8-4fa7-ae50-dbede7ec6e5c","Type":"ContainerDied","Data":"63ee3db55d3d8e01a155bd1b14aef0eef5a50d77acf97eca8e3f3f91e443d29b"}
Jul 29 19:12:58.550859 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:12:58.550822    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e3eaaf-6b9549cf7b-2bzx9" event={"ID":"9e4e42e4-84c8-4fa7-ae50-dbede7ec6e5c","Type":"ContainerDied","Data":"517818a2514369f1a1e14510b2535c8af2b66e59788e7c00b03fd9ebd12f9361"}
Jul 29 19:12:58.559371 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:12:58.559349    2559 scope.go:117] "RemoveContainer" containerID="161dba36d900c1ede7ded59890e066e8f26c47c9a09ec0406ca7757697737c6d"
Jul 29 19:12:58.559727 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:12:58.559700    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"161dba36d900c1ede7ded59890e066e8f26c47c9a09ec0406ca7757697737c6d\": container with ID starting with 161dba36d900c1ede7ded59890e066e8f26c47c9a09ec0406ca7757697737c6d not found: ID does not exist" containerID="161dba36d900c1ede7ded59890e066e8f26c47c9a09ec0406ca7757697737c6d"
Jul 29 19:12:58.559813 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:12:58.559729    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"161dba36d900c1ede7ded59890e066e8f26c47c9a09ec0406ca7757697737c6d"} err="failed to get container status \"161dba36d900c1ede7ded59890e066e8f26c47c9a09ec0406ca7757697737c6d\": rpc error: code = NotFound desc = could not find container \"161dba36d900c1ede7ded59890e066e8f26c47c9a09ec0406ca7757697737c6d\": container with ID starting with 161dba36d900c1ede7ded59890e066e8f26c47c9a09ec0406ca7757697737c6d not found: ID does not exist"
Jul 29 19:12:58.559813 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:12:58.559746    2559 scope.go:117] "RemoveContainer" containerID="63ee3db55d3d8e01a155bd1b14aef0eef5a50d77acf97eca8e3f3f91e443d29b"
Jul 29 19:12:58.568407 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:12:58.568369    2559 scope.go:117] "RemoveContainer" containerID="63ee3db55d3d8e01a155bd1b14aef0eef5a50d77acf97eca8e3f3f91e443d29b"
Jul 29 19:12:58.568703 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:12:58.568681    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63ee3db55d3d8e01a155bd1b14aef0eef5a50d77acf97eca8e3f3f91e443d29b\": container with ID starting with 63ee3db55d3d8e01a155bd1b14aef0eef5a50d77acf97eca8e3f3f91e443d29b not found: ID does not exist" containerID="63ee3db55d3d8e01a155bd1b14aef0eef5a50d77acf97eca8e3f3f91e443d29b"
Jul 29 19:12:58.568748 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:12:58.568716    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63ee3db55d3d8e01a155bd1b14aef0eef5a50d77acf97eca8e3f3f91e443d29b"} err="failed to get container status \"63ee3db55d3d8e01a155bd1b14aef0eef5a50d77acf97eca8e3f3f91e443d29b\": rpc error: code = NotFound desc = could not find container \"63ee3db55d3d8e01a155bd1b14aef0eef5a50d77acf97eca8e3f3f91e443d29b\": container with ID starting with 63ee3db55d3d8e01a155bd1b14aef0eef5a50d77acf97eca8e3f3f91e443d29b not found: ID does not exist"
Jul 29 19:12:58.574705 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:12:58.574676    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e3eaaf-64f4f84984-b8wdj"]
Jul 29 19:12:58.578478 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:12:58.578447    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e3eaaf-64f4f84984-b8wdj"]
Jul 29 19:12:58.589640 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:12:58.589604    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e3eaaf-6b9549cf7b-2bzx9"]
Jul 29 19:12:58.593025 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:12:58.592997    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e3eaaf-6b9549cf7b-2bzx9"]
Jul 29 19:13:00.151835 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:00.151798    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b968b2f-bde7-4f5c-8be1-056ad58fb47c" path="/var/lib/kubelet/pods/8b968b2f-bde7-4f5c-8be1-056ad58fb47c/volumes"
Jul 29 19:13:00.152248 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:00.152152    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e4e42e4-84c8-4fa7-ae50-dbede7ec6e5c" path="/var/lib/kubelet/pods/9e4e42e4-84c8-4fa7-ae50-dbede7ec6e5c/volumes"
Jul 29 19:13:01.185976 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:01.185911    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-e3eaaf/test-model-e3eaaf-kserve-69fbf46cbc-85b4w" podUID="459ec5b7-7dce-4591-92b3-5a2a4ec6257d" containerName="main" containerID="cri-o://647cfe6659564642a0012f8d9efe2ad9f2e7dc237853c5d92bb2589be676a0dd" gracePeriod=45
Jul 29 19:13:01.453166 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:01.453134    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e3eaaf/test-model-e3eaaf-kserve-69fbf46cbc-85b4w"
Jul 29 19:13:01.504025 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:01.503989    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/459ec5b7-7dce-4591-92b3-5a2a4ec6257d-dshm\") pod \"459ec5b7-7dce-4591-92b3-5a2a4ec6257d\" (UID: \"459ec5b7-7dce-4591-92b3-5a2a4ec6257d\") "
Jul 29 19:13:01.504025 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:01.504033    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/459ec5b7-7dce-4591-92b3-5a2a4ec6257d-tls-certs\") pod \"459ec5b7-7dce-4591-92b3-5a2a4ec6257d\" (UID: \"459ec5b7-7dce-4591-92b3-5a2a4ec6257d\") "
Jul 29 19:13:01.504299 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:01.504078    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/459ec5b7-7dce-4591-92b3-5a2a4ec6257d-home\") pod \"459ec5b7-7dce-4591-92b3-5a2a4ec6257d\" (UID: \"459ec5b7-7dce-4591-92b3-5a2a4ec6257d\") "
Jul 29 19:13:01.504299 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:01.504096    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/459ec5b7-7dce-4591-92b3-5a2a4ec6257d-model-cache\") pod \"459ec5b7-7dce-4591-92b3-5a2a4ec6257d\" (UID: \"459ec5b7-7dce-4591-92b3-5a2a4ec6257d\") "
Jul 29 19:13:01.504299 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:01.504122    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/459ec5b7-7dce-4591-92b3-5a2a4ec6257d-tmp-dir\") pod \"459ec5b7-7dce-4591-92b3-5a2a4ec6257d\" (UID: \"459ec5b7-7dce-4591-92b3-5a2a4ec6257d\") "
Jul 29 19:13:01.504299 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:01.504154    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hx45\" (UniqueName: \"kubernetes.io/projected/459ec5b7-7dce-4591-92b3-5a2a4ec6257d-kube-api-access-2hx45\") pod \"459ec5b7-7dce-4591-92b3-5a2a4ec6257d\" (UID: \"459ec5b7-7dce-4591-92b3-5a2a4ec6257d\") "
Jul 29 19:13:01.504530 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:01.504428    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/459ec5b7-7dce-4591-92b3-5a2a4ec6257d-model-cache" (OuterVolumeSpecName: "model-cache") pod "459ec5b7-7dce-4591-92b3-5a2a4ec6257d" (UID: "459ec5b7-7dce-4591-92b3-5a2a4ec6257d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:13:01.504530 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:01.504437    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/459ec5b7-7dce-4591-92b3-5a2a4ec6257d-home" (OuterVolumeSpecName: "home") pod "459ec5b7-7dce-4591-92b3-5a2a4ec6257d" (UID: "459ec5b7-7dce-4591-92b3-5a2a4ec6257d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:13:01.504530 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:01.504454    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/459ec5b7-7dce-4591-92b3-5a2a4ec6257d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "459ec5b7-7dce-4591-92b3-5a2a4ec6257d" (UID: "459ec5b7-7dce-4591-92b3-5a2a4ec6257d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:13:01.506654 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:01.506627    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/459ec5b7-7dce-4591-92b3-5a2a4ec6257d-kube-api-access-2hx45" (OuterVolumeSpecName: "kube-api-access-2hx45") pod "459ec5b7-7dce-4591-92b3-5a2a4ec6257d" (UID: "459ec5b7-7dce-4591-92b3-5a2a4ec6257d"). InnerVolumeSpecName "kube-api-access-2hx45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:13:01.506909 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:01.506892    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/459ec5b7-7dce-4591-92b3-5a2a4ec6257d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "459ec5b7-7dce-4591-92b3-5a2a4ec6257d" (UID: "459ec5b7-7dce-4591-92b3-5a2a4ec6257d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:13:01.506983 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:01.506896    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/459ec5b7-7dce-4591-92b3-5a2a4ec6257d-dshm" (OuterVolumeSpecName: "dshm") pod "459ec5b7-7dce-4591-92b3-5a2a4ec6257d" (UID: "459ec5b7-7dce-4591-92b3-5a2a4ec6257d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:13:01.568455 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:01.568415    2559 generic.go:356] "Generic (PLEG): container finished" podID="459ec5b7-7dce-4591-92b3-5a2a4ec6257d" containerID="647cfe6659564642a0012f8d9efe2ad9f2e7dc237853c5d92bb2589be676a0dd" exitCode=0
Jul 29 19:13:01.568455 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:01.568461    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e3eaaf/test-model-e3eaaf-kserve-69fbf46cbc-85b4w"
Jul 29 19:13:01.568713 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:01.568501    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e3eaaf/test-model-e3eaaf-kserve-69fbf46cbc-85b4w" event={"ID":"459ec5b7-7dce-4591-92b3-5a2a4ec6257d","Type":"ContainerDied","Data":"647cfe6659564642a0012f8d9efe2ad9f2e7dc237853c5d92bb2589be676a0dd"}
Jul 29 19:13:01.568713 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:01.568544    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e3eaaf/test-model-e3eaaf-kserve-69fbf46cbc-85b4w" event={"ID":"459ec5b7-7dce-4591-92b3-5a2a4ec6257d","Type":"ContainerDied","Data":"5db79cdc822e08ac2f69eba4cb6d1b48843c5a06f27bab65e3f137aec2fb2bcd"}
Jul 29 19:13:01.568713 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:01.568564    2559 scope.go:117] "RemoveContainer" containerID="647cfe6659564642a0012f8d9efe2ad9f2e7dc237853c5d92bb2589be676a0dd"
Jul 29 19:13:01.577785 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:01.577766    2559 scope.go:117] "RemoveContainer" containerID="647cfe6659564642a0012f8d9efe2ad9f2e7dc237853c5d92bb2589be676a0dd"
Jul 29 19:13:01.578096 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:13:01.578060    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"647cfe6659564642a0012f8d9efe2ad9f2e7dc237853c5d92bb2589be676a0dd\": container with ID starting with 647cfe6659564642a0012f8d9efe2ad9f2e7dc237853c5d92bb2589be676a0dd not found: ID does not exist" containerID="647cfe6659564642a0012f8d9efe2ad9f2e7dc237853c5d92bb2589be676a0dd"
Jul 29 19:13:01.578155 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:01.578089    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"647cfe6659564642a0012f8d9efe2ad9f2e7dc237853c5d92bb2589be676a0dd"} err="failed to get container status \"647cfe6659564642a0012f8d9efe2ad9f2e7dc237853c5d92bb2589be676a0dd\": rpc error: code = NotFound desc = could not find container \"647cfe6659564642a0012f8d9efe2ad9f2e7dc237853c5d92bb2589be676a0dd\": container with ID starting with 647cfe6659564642a0012f8d9efe2ad9f2e7dc237853c5d92bb2589be676a0dd not found: ID does not exist"
Jul 29 19:13:01.591814 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:01.591780    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e3eaaf/test-model-e3eaaf-kserve-69fbf46cbc-85b4w"]
Jul 29 19:13:01.596688 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:01.596657    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-e3eaaf/test-model-e3eaaf-kserve-69fbf46cbc-85b4w"]
Jul 29 19:13:01.605766 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:01.605744    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/459ec5b7-7dce-4591-92b3-5a2a4ec6257d-dshm\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:13:01.605766 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:01.605765    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/459ec5b7-7dce-4591-92b3-5a2a4ec6257d-tls-certs\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:13:01.605921 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:01.605774    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/459ec5b7-7dce-4591-92b3-5a2a4ec6257d-home\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:13:01.605921 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:01.605782    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/459ec5b7-7dce-4591-92b3-5a2a4ec6257d-model-cache\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:13:01.605921 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:01.605790    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/459ec5b7-7dce-4591-92b3-5a2a4ec6257d-tmp-dir\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:13:01.605921 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:01.605799    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hx45\" (UniqueName: \"kubernetes.io/projected/459ec5b7-7dce-4591-92b3-5a2a4ec6257d-kube-api-access-2hx45\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:13:02.151239 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:02.151202    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="459ec5b7-7dce-4591-92b3-5a2a4ec6257d" path="/var/lib/kubelet/pods/459ec5b7-7dce-4591-92b3-5a2a4ec6257d/volumes"
Jul 29 19:13:03.336775 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:03.336733    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:13:15.570778 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:15.570730    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g"]
Jul 29 19:13:15.571237 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:15.570967    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g" podUID="51dd7f67-7b58-4010-8716-305a4ec58ccf" containerName="maas-api" containerID="cri-o://30041b87c5d74341cd9f77ca9177c0071b700e9030da58d16de66735c121175f" gracePeriod=30
Jul 29 19:13:15.696498 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:15.696465    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5b4a8e-6449784d58-8mqvc"]
Jul 29 19:13:15.696713 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:15.696686    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-5b4a8e-6449784d58-8mqvc" podUID="feadcbd5-235f-4380-9ea4-33e7dc69571b" containerName="payload-processing" containerID="cri-o://8170013af2e62ae534cd9e54f4ae447349fd3cf56b4dabbcabf39d663567b27e" gracePeriod=30
Jul 29 19:13:15.713975 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:15.713940    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5b4a8e-68f7d49cd4-4v5kg"]
Jul 29 19:13:15.714249 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:15.714159    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5b4a8e-68f7d49cd4-4v5kg" podUID="e94f994e-3d76-4b7e-ad2c-a3d2a5ed87db" containerName="payload-pre-processing" containerID="cri-o://2e2e0db79f5163cba1e8c0f3a607e64851238a271eedd36e49f3faaed9a0f866" gracePeriod=30
Jul 29 19:13:15.850817 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:15.850786    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g"
Jul 29 19:13:15.933975 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:15.933928    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51dd7f67-7b58-4010-8716-305a4ec58ccf-maas-api-tls\") pod \"51dd7f67-7b58-4010-8716-305a4ec58ccf\" (UID: \"51dd7f67-7b58-4010-8716-305a4ec58ccf\") "
Jul 29 19:13:15.934149 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:15.934001    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51dd7f67-7b58-4010-8716-305a4ec58ccf-metrics-certs\") pod \"51dd7f67-7b58-4010-8716-305a4ec58ccf\" (UID: \"51dd7f67-7b58-4010-8716-305a4ec58ccf\") "
Jul 29 19:13:15.934149 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:15.934049    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51dd7f67-7b58-4010-8716-305a4ec58ccf-openshift-service-ca\") pod \"51dd7f67-7b58-4010-8716-305a4ec58ccf\" (UID: \"51dd7f67-7b58-4010-8716-305a4ec58ccf\") "
Jul 29 19:13:15.934149 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:15.934105    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rk69d\" (UniqueName: \"kubernetes.io/projected/51dd7f67-7b58-4010-8716-305a4ec58ccf-kube-api-access-rk69d\") pod \"51dd7f67-7b58-4010-8716-305a4ec58ccf\" (UID: \"51dd7f67-7b58-4010-8716-305a4ec58ccf\") "
Jul 29 19:13:15.934762 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:15.934727    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51dd7f67-7b58-4010-8716-305a4ec58ccf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "51dd7f67-7b58-4010-8716-305a4ec58ccf" (UID: "51dd7f67-7b58-4010-8716-305a4ec58ccf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:13:15.936597 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:15.936564    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51dd7f67-7b58-4010-8716-305a4ec58ccf-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "51dd7f67-7b58-4010-8716-305a4ec58ccf" (UID: "51dd7f67-7b58-4010-8716-305a4ec58ccf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:13:15.936847 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:15.936822    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51dd7f67-7b58-4010-8716-305a4ec58ccf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "51dd7f67-7b58-4010-8716-305a4ec58ccf" (UID: "51dd7f67-7b58-4010-8716-305a4ec58ccf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:13:15.937073 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:15.937046    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51dd7f67-7b58-4010-8716-305a4ec58ccf-kube-api-access-rk69d" (OuterVolumeSpecName: "kube-api-access-rk69d") pod "51dd7f67-7b58-4010-8716-305a4ec58ccf" (UID: "51dd7f67-7b58-4010-8716-305a4ec58ccf"). InnerVolumeSpecName "kube-api-access-rk69d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:13:15.987839 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:15.987815    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5b4a8e-68f7d49cd4-4v5kg"
Jul 29 19:13:15.992666 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:15.992640    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5b4a8e-6449784d58-8mqvc"
Jul 29 19:13:16.034717 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:16.034685    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9m624\" (UniqueName: \"kubernetes.io/projected/feadcbd5-235f-4380-9ea4-33e7dc69571b-kube-api-access-9m624\") pod \"feadcbd5-235f-4380-9ea4-33e7dc69571b\" (UID: \"feadcbd5-235f-4380-9ea4-33e7dc69571b\") "
Jul 29 19:13:16.034918 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:16.034778    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e94f994e-3d76-4b7e-ad2c-a3d2a5ed87db-plugins-config-volume\") pod \"e94f994e-3d76-4b7e-ad2c-a3d2a5ed87db\" (UID: \"e94f994e-3d76-4b7e-ad2c-a3d2a5ed87db\") "
Jul 29 19:13:16.034918 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:16.034796    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feadcbd5-235f-4380-9ea4-33e7dc69571b-plugins-config-volume\") pod \"feadcbd5-235f-4380-9ea4-33e7dc69571b\" (UID: \"feadcbd5-235f-4380-9ea4-33e7dc69571b\") "
Jul 29 19:13:16.034918 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:16.034837    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dsvqb\" (UniqueName: \"kubernetes.io/projected/e94f994e-3d76-4b7e-ad2c-a3d2a5ed87db-kube-api-access-dsvqb\") pod \"e94f994e-3d76-4b7e-ad2c-a3d2a5ed87db\" (UID: \"e94f994e-3d76-4b7e-ad2c-a3d2a5ed87db\") "
Jul 29 19:13:16.035096 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:16.035013    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51dd7f67-7b58-4010-8716-305a4ec58ccf-metrics-certs\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:13:16.035096 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:16.035032    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51dd7f67-7b58-4010-8716-305a4ec58ccf-openshift-service-ca\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:13:16.035096 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:16.035049    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk69d\" (UniqueName: \"kubernetes.io/projected/51dd7f67-7b58-4010-8716-305a4ec58ccf-kube-api-access-rk69d\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:13:16.035096 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:16.035062    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51dd7f67-7b58-4010-8716-305a4ec58ccf-maas-api-tls\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:13:16.035256 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:16.035139    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e94f994e-3d76-4b7e-ad2c-a3d2a5ed87db-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e94f994e-3d76-4b7e-ad2c-a3d2a5ed87db" (UID: "e94f994e-3d76-4b7e-ad2c-a3d2a5ed87db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:13:16.035256 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:16.035214    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feadcbd5-235f-4380-9ea4-33e7dc69571b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "feadcbd5-235f-4380-9ea4-33e7dc69571b" (UID: "feadcbd5-235f-4380-9ea4-33e7dc69571b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:13:16.037138 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:16.037112    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feadcbd5-235f-4380-9ea4-33e7dc69571b-kube-api-access-9m624" (OuterVolumeSpecName: "kube-api-access-9m624") pod "feadcbd5-235f-4380-9ea4-33e7dc69571b" (UID: "feadcbd5-235f-4380-9ea4-33e7dc69571b"). InnerVolumeSpecName "kube-api-access-9m624". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:13:16.037138 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:16.037123    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e94f994e-3d76-4b7e-ad2c-a3d2a5ed87db-kube-api-access-dsvqb" (OuterVolumeSpecName: "kube-api-access-dsvqb") pod "e94f994e-3d76-4b7e-ad2c-a3d2a5ed87db" (UID: "e94f994e-3d76-4b7e-ad2c-a3d2a5ed87db"). InnerVolumeSpecName "kube-api-access-dsvqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:13:16.136172 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:16.136129    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e94f994e-3d76-4b7e-ad2c-a3d2a5ed87db-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:13:16.136172 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:16.136163    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feadcbd5-235f-4380-9ea4-33e7dc69571b-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:13:16.136172 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:16.136176    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsvqb\" (UniqueName: \"kubernetes.io/projected/e94f994e-3d76-4b7e-ad2c-a3d2a5ed87db-kube-api-access-dsvqb\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:13:16.136172 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:16.136184    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m624\" (UniqueName: \"kubernetes.io/projected/feadcbd5-235f-4380-9ea4-33e7dc69571b-kube-api-access-9m624\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:13:16.656693 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:16.656656    2559 generic.go:356] "Generic (PLEG): container finished" podID="e94f994e-3d76-4b7e-ad2c-a3d2a5ed87db" containerID="2e2e0db79f5163cba1e8c0f3a607e64851238a271eedd36e49f3faaed9a0f866" exitCode=0
Jul 29 19:13:16.657209 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:16.656718    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5b4a8e-68f7d49cd4-4v5kg"
Jul 29 19:13:16.657209 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:16.656741    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5b4a8e-68f7d49cd4-4v5kg" event={"ID":"e94f994e-3d76-4b7e-ad2c-a3d2a5ed87db","Type":"ContainerDied","Data":"2e2e0db79f5163cba1e8c0f3a607e64851238a271eedd36e49f3faaed9a0f866"}
Jul 29 19:13:16.657209 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:16.656798    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5b4a8e-68f7d49cd4-4v5kg" event={"ID":"e94f994e-3d76-4b7e-ad2c-a3d2a5ed87db","Type":"ContainerDied","Data":"895840636d5df0553076abf22a248b28bacac881eda2d3ecc12c4f43dc89b1dc"}
Jul 29 19:13:16.657209 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:16.656819    2559 scope.go:117] "RemoveContainer" containerID="2e2e0db79f5163cba1e8c0f3a607e64851238a271eedd36e49f3faaed9a0f866"
Jul 29 19:13:16.658184 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:16.658115    2559 generic.go:356] "Generic (PLEG): container finished" podID="feadcbd5-235f-4380-9ea4-33e7dc69571b" containerID="8170013af2e62ae534cd9e54f4ae447349fd3cf56b4dabbcabf39d663567b27e" exitCode=0
Jul 29 19:13:16.658184 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:16.658167    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5b4a8e-6449784d58-8mqvc"
Jul 29 19:13:16.658184 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:16.658163    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5b4a8e-6449784d58-8mqvc" event={"ID":"feadcbd5-235f-4380-9ea4-33e7dc69571b","Type":"ContainerDied","Data":"8170013af2e62ae534cd9e54f4ae447349fd3cf56b4dabbcabf39d663567b27e"}
Jul 29 19:13:16.658358 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:16.658217    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5b4a8e-6449784d58-8mqvc" event={"ID":"feadcbd5-235f-4380-9ea4-33e7dc69571b","Type":"ContainerDied","Data":"7501c0c1b49cd15517e878e47875f63349d167878554db5c3b5b8fda9b79a83a"}
Jul 29 19:13:16.659824 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:16.659770    2559 generic.go:356] "Generic (PLEG): container finished" podID="51dd7f67-7b58-4010-8716-305a4ec58ccf" containerID="30041b87c5d74341cd9f77ca9177c0071b700e9030da58d16de66735c121175f" exitCode=0
Jul 29 19:13:16.659940 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:16.659825    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g" event={"ID":"51dd7f67-7b58-4010-8716-305a4ec58ccf","Type":"ContainerDied","Data":"30041b87c5d74341cd9f77ca9177c0071b700e9030da58d16de66735c121175f"}
Jul 29 19:13:16.659940 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:16.659865    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g" event={"ID":"51dd7f67-7b58-4010-8716-305a4ec58ccf","Type":"ContainerDied","Data":"42f599570b780eb43e56ea12fc4df93d36d57f7654fa3120ffe953b1c23f419e"}
Jul 29 19:13:16.659940 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:16.659831    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g"
Jul 29 19:13:16.670618 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:16.670592    2559 scope.go:117] "RemoveContainer" containerID="2e2e0db79f5163cba1e8c0f3a607e64851238a271eedd36e49f3faaed9a0f866"
Jul 29 19:13:16.670979 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:13:16.670953    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e2e0db79f5163cba1e8c0f3a607e64851238a271eedd36e49f3faaed9a0f866\": container with ID starting with 2e2e0db79f5163cba1e8c0f3a607e64851238a271eedd36e49f3faaed9a0f866 not found: ID does not exist" containerID="2e2e0db79f5163cba1e8c0f3a607e64851238a271eedd36e49f3faaed9a0f866"
Jul 29 19:13:16.671081 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:16.670998    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e2e0db79f5163cba1e8c0f3a607e64851238a271eedd36e49f3faaed9a0f866"} err="failed to get container status \"2e2e0db79f5163cba1e8c0f3a607e64851238a271eedd36e49f3faaed9a0f866\": rpc error: code = NotFound desc = could not find container \"2e2e0db79f5163cba1e8c0f3a607e64851238a271eedd36e49f3faaed9a0f866\": container with ID starting with 2e2e0db79f5163cba1e8c0f3a607e64851238a271eedd36e49f3faaed9a0f866 not found: ID does not exist"
Jul 29 19:13:16.671081 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:16.671015    2559 scope.go:117] "RemoveContainer" containerID="8170013af2e62ae534cd9e54f4ae447349fd3cf56b4dabbcabf39d663567b27e"
Jul 29 19:13:16.679671 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:16.679649    2559 scope.go:117] "RemoveContainer" containerID="8170013af2e62ae534cd9e54f4ae447349fd3cf56b4dabbcabf39d663567b27e"
Jul 29 19:13:16.679994 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:13:16.679972    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8170013af2e62ae534cd9e54f4ae447349fd3cf56b4dabbcabf39d663567b27e\": container with ID starting with 8170013af2e62ae534cd9e54f4ae447349fd3cf56b4dabbcabf39d663567b27e not found: ID does not exist" containerID="8170013af2e62ae534cd9e54f4ae447349fd3cf56b4dabbcabf39d663567b27e"
Jul 29 19:13:16.680074 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:16.680005    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8170013af2e62ae534cd9e54f4ae447349fd3cf56b4dabbcabf39d663567b27e"} err="failed to get container status \"8170013af2e62ae534cd9e54f4ae447349fd3cf56b4dabbcabf39d663567b27e\": rpc error: code = NotFound desc = could not find container \"8170013af2e62ae534cd9e54f4ae447349fd3cf56b4dabbcabf39d663567b27e\": container with ID starting with 8170013af2e62ae534cd9e54f4ae447349fd3cf56b4dabbcabf39d663567b27e not found: ID does not exist"
Jul 29 19:13:16.680074 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:16.680030    2559 scope.go:117] "RemoveContainer" containerID="30041b87c5d74341cd9f77ca9177c0071b700e9030da58d16de66735c121175f"
Jul 29 19:13:16.683429 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:16.683406    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5b4a8e-68f7d49cd4-4v5kg"]
Jul 29 19:13:16.687047 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:16.687024    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5b4a8e-68f7d49cd4-4v5kg"]
Jul 29 19:13:16.689277 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:16.689241    2559 scope.go:117] "RemoveContainer" containerID="30041b87c5d74341cd9f77ca9177c0071b700e9030da58d16de66735c121175f"
Jul 29 19:13:16.689610 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:13:16.689593    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30041b87c5d74341cd9f77ca9177c0071b700e9030da58d16de66735c121175f\": container with ID starting with 30041b87c5d74341cd9f77ca9177c0071b700e9030da58d16de66735c121175f not found: ID does not exist" containerID="30041b87c5d74341cd9f77ca9177c0071b700e9030da58d16de66735c121175f"
Jul 29 19:13:16.689674 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:16.689617    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30041b87c5d74341cd9f77ca9177c0071b700e9030da58d16de66735c121175f"} err="failed to get container status \"30041b87c5d74341cd9f77ca9177c0071b700e9030da58d16de66735c121175f\": rpc error: code = NotFound desc = could not find container \"30041b87c5d74341cd9f77ca9177c0071b700e9030da58d16de66735c121175f\": container with ID starting with 30041b87c5d74341cd9f77ca9177c0071b700e9030da58d16de66735c121175f not found: ID does not exist"
Jul 29 19:13:16.699506 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:16.699475    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g"]
Jul 29 19:13:16.705740 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:16.705714    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5b4a8e-5c75db895-wdb8g"]
Jul 29 19:13:16.717488 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:16.717457    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5b4a8e-6449784d58-8mqvc"]
Jul 29 19:13:16.728714 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:16.728681    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5b4a8e-6449784d58-8mqvc"]
Jul 29 19:13:18.150880 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:18.150849    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51dd7f67-7b58-4010-8716-305a4ec58ccf" path="/var/lib/kubelet/pods/51dd7f67-7b58-4010-8716-305a4ec58ccf/volumes"
Jul 29 19:13:18.151347 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:18.151192    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e94f994e-3d76-4b7e-ad2c-a3d2a5ed87db" path="/var/lib/kubelet/pods/e94f994e-3d76-4b7e-ad2c-a3d2a5ed87db/volumes"
Jul 29 19:13:18.151511 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:18.151499    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="feadcbd5-235f-4380-9ea4-33e7dc69571b" path="/var/lib/kubelet/pods/feadcbd5-235f-4380-9ea4-33e7dc69571b/volumes"
Jul 29 19:13:18.440778 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:18.440698    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:13:26.961845 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:26.961810    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk"]
Jul 29 19:13:26.962231 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:26.962159    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feadcbd5-235f-4380-9ea4-33e7dc69571b" containerName="payload-processing"
Jul 29 19:13:26.962231 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:26.962170    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="feadcbd5-235f-4380-9ea4-33e7dc69571b" containerName="payload-processing"
Jul 29 19:13:26.962231 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:26.962179    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e4e42e4-84c8-4fa7-ae50-dbede7ec6e5c" containerName="payload-processing"
Jul 29 19:13:26.962231 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:26.962185    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e4e42e4-84c8-4fa7-ae50-dbede7ec6e5c" containerName="payload-processing"
Jul 29 19:13:26.962231 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:26.962194    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e94f994e-3d76-4b7e-ad2c-a3d2a5ed87db" containerName="payload-pre-processing"
Jul 29 19:13:26.962231 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:26.962200    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e94f994e-3d76-4b7e-ad2c-a3d2a5ed87db" containerName="payload-pre-processing"
Jul 29 19:13:26.962231 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:26.962209    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b968b2f-bde7-4f5c-8be1-056ad58fb47c" containerName="payload-pre-processing"
Jul 29 19:13:26.962231 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:26.962214    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b968b2f-bde7-4f5c-8be1-056ad58fb47c" containerName="payload-pre-processing"
Jul 29 19:13:26.962231 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:26.962221    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="459ec5b7-7dce-4591-92b3-5a2a4ec6257d" containerName="main"
Jul 29 19:13:26.962231 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:26.962226    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="459ec5b7-7dce-4591-92b3-5a2a4ec6257d" containerName="main"
Jul 29 19:13:26.962231 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:26.962236    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51dd7f67-7b58-4010-8716-305a4ec58ccf" containerName="maas-api"
Jul 29 19:13:26.962584 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:26.962241    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="51dd7f67-7b58-4010-8716-305a4ec58ccf" containerName="maas-api"
Jul 29 19:13:26.962584 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:26.962301    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="51dd7f67-7b58-4010-8716-305a4ec58ccf" containerName="maas-api"
Jul 29 19:13:26.962584 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:26.962308    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b968b2f-bde7-4f5c-8be1-056ad58fb47c" containerName="payload-pre-processing"
Jul 29 19:13:26.962584 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:26.962315    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="459ec5b7-7dce-4591-92b3-5a2a4ec6257d" containerName="main"
Jul 29 19:13:26.962584 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:26.962320    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="feadcbd5-235f-4380-9ea4-33e7dc69571b" containerName="payload-processing"
Jul 29 19:13:26.962584 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:26.962326    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e94f994e-3d76-4b7e-ad2c-a3d2a5ed87db" containerName="payload-pre-processing"
Jul 29 19:13:26.962584 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:26.962333    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e4e42e4-84c8-4fa7-ae50-dbede7ec6e5c" containerName="payload-processing"
Jul 29 19:13:26.967243 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:26.967222    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk"
Jul 29 19:13:26.971268 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:26.971237    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Jul 29 19:13:26.971268 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:26.971259    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mglnh\""
Jul 29 19:13:26.971541 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:26.971248    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-1dcc27-serving-cert\""
Jul 29 19:13:26.971541 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:26.971284    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 19:13:26.971715 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:26.971692    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 29 19:13:26.983446 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:26.983416    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk"]
Jul 29 19:13:27.036475 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:27.036434    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd22b3e4-db32-4bcd-9a7d-451907714a5f-maas-api-tls\") pod \"maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk\" (UID: \"dd22b3e4-db32-4bcd-9a7d-451907714a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk"
Jul 29 19:13:27.036475 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:27.036473    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd22b3e4-db32-4bcd-9a7d-451907714a5f-metrics-certs\") pod \"maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk\" (UID: \"dd22b3e4-db32-4bcd-9a7d-451907714a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk"
Jul 29 19:13:27.036717 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:27.036553    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpqx4\" (UniqueName: \"kubernetes.io/projected/dd22b3e4-db32-4bcd-9a7d-451907714a5f-kube-api-access-tpqx4\") pod \"maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk\" (UID: \"dd22b3e4-db32-4bcd-9a7d-451907714a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk"
Jul 29 19:13:27.036717 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:27.036622    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd22b3e4-db32-4bcd-9a7d-451907714a5f-openshift-service-ca\") pod \"maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk\" (UID: \"dd22b3e4-db32-4bcd-9a7d-451907714a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk"
Jul 29 19:13:27.137579 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:27.137540    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd22b3e4-db32-4bcd-9a7d-451907714a5f-maas-api-tls\") pod \"maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk\" (UID: \"dd22b3e4-db32-4bcd-9a7d-451907714a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk"
Jul 29 19:13:27.137579 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:27.137578    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd22b3e4-db32-4bcd-9a7d-451907714a5f-metrics-certs\") pod \"maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk\" (UID: \"dd22b3e4-db32-4bcd-9a7d-451907714a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk"
Jul 29 19:13:27.137809 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:27.137636    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpqx4\" (UniqueName: \"kubernetes.io/projected/dd22b3e4-db32-4bcd-9a7d-451907714a5f-kube-api-access-tpqx4\") pod \"maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk\" (UID: \"dd22b3e4-db32-4bcd-9a7d-451907714a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk"
Jul 29 19:13:27.137809 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:27.137683    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd22b3e4-db32-4bcd-9a7d-451907714a5f-openshift-service-ca\") pod \"maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk\" (UID: \"dd22b3e4-db32-4bcd-9a7d-451907714a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk"
Jul 29 19:13:27.137809 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:13:27.137732    2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-1dcc27-serving-cert: secret "maas-api-e2e-api-a-1dcc27-serving-cert" not found
Jul 29 19:13:27.137935 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:13:27.137822    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd22b3e4-db32-4bcd-9a7d-451907714a5f-maas-api-tls podName:dd22b3e4-db32-4bcd-9a7d-451907714a5f nodeName:}" failed. No retries permitted until 2026-07-29 19:13:27.637798797 +0000 UTC m=+2371.907567869 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dd22b3e4-db32-4bcd-9a7d-451907714a5f-maas-api-tls") pod "maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk" (UID: "dd22b3e4-db32-4bcd-9a7d-451907714a5f") : secret "maas-api-e2e-api-a-1dcc27-serving-cert" not found
Jul 29 19:13:27.138430 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:27.138409    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd22b3e4-db32-4bcd-9a7d-451907714a5f-openshift-service-ca\") pod \"maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk\" (UID: \"dd22b3e4-db32-4bcd-9a7d-451907714a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk"
Jul 29 19:13:27.140434 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:27.140407    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd22b3e4-db32-4bcd-9a7d-451907714a5f-metrics-certs\") pod \"maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk\" (UID: \"dd22b3e4-db32-4bcd-9a7d-451907714a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk"
Jul 29 19:13:27.153061 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:27.153028    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpqx4\" (UniqueName: \"kubernetes.io/projected/dd22b3e4-db32-4bcd-9a7d-451907714a5f-kube-api-access-tpqx4\") pod \"maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk\" (UID: \"dd22b3e4-db32-4bcd-9a7d-451907714a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk"
Jul 29 19:13:27.643615 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:27.643576    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd22b3e4-db32-4bcd-9a7d-451907714a5f-maas-api-tls\") pod \"maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk\" (UID: \"dd22b3e4-db32-4bcd-9a7d-451907714a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk"
Jul 29 19:13:27.646438 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:27.646408    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd22b3e4-db32-4bcd-9a7d-451907714a5f-maas-api-tls\") pod \"maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk\" (UID: \"dd22b3e4-db32-4bcd-9a7d-451907714a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk"
Jul 29 19:13:27.651430 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:27.651397    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1dcc27-86999c6664-gccmf"]
Jul 29 19:13:27.655093 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:27.655071    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1dcc27-86999c6664-gccmf"
Jul 29 19:13:27.657690 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:27.657669    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-1dcc27\""
Jul 29 19:13:27.666895 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:27.666862    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1dcc27-86999c6664-gccmf"]
Jul 29 19:13:27.700340 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:27.700305    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1dcc27-6b855b6744-6wg4c"]
Jul 29 19:13:27.703698 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:27.703680    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1dcc27-6b855b6744-6wg4c"
Jul 29 19:13:27.706221 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:27.706197    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-1dcc27-dockercfg-ptfzl\""
Jul 29 19:13:27.715832 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:27.715767    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1dcc27-6b855b6744-6wg4c"]
Jul 29 19:13:27.745045 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:27.745001    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmmvf\" (UniqueName: \"kubernetes.io/projected/835ecc18-94c5-42c8-ae60-7887e93a9cab-kube-api-access-xmmvf\") pod \"payload-pre-processing-e2e-api-a-1dcc27-86999c6664-gccmf\" (UID: \"835ecc18-94c5-42c8-ae60-7887e93a9cab\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1dcc27-86999c6664-gccmf"
Jul 29 19:13:27.745230 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:27.745067    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/835ecc18-94c5-42c8-ae60-7887e93a9cab-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1dcc27-86999c6664-gccmf\" (UID: \"835ecc18-94c5-42c8-ae60-7887e93a9cab\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1dcc27-86999c6664-gccmf"
Jul 29 19:13:27.845886 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:27.845844    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmmvf\" (UniqueName: \"kubernetes.io/projected/835ecc18-94c5-42c8-ae60-7887e93a9cab-kube-api-access-xmmvf\") pod \"payload-pre-processing-e2e-api-a-1dcc27-86999c6664-gccmf\" (UID: \"835ecc18-94c5-42c8-ae60-7887e93a9cab\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1dcc27-86999c6664-gccmf"
Jul 29 19:13:27.846074 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:27.845898    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cdd8b6c-a030-43d8-927a-631d5474c974-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1dcc27-6b855b6744-6wg4c\" (UID: \"1cdd8b6c-a030-43d8-927a-631d5474c974\") " pod="openshift-ingress/payload-processing-e2e-api-a-1dcc27-6b855b6744-6wg4c"
Jul 29 19:13:27.846074 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:27.845935    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/835ecc18-94c5-42c8-ae60-7887e93a9cab-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1dcc27-86999c6664-gccmf\" (UID: \"835ecc18-94c5-42c8-ae60-7887e93a9cab\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1dcc27-86999c6664-gccmf"
Jul 29 19:13:27.846074 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:27.845951    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b6gz\" (UniqueName: \"kubernetes.io/projected/1cdd8b6c-a030-43d8-927a-631d5474c974-kube-api-access-5b6gz\") pod \"payload-processing-e2e-api-a-1dcc27-6b855b6744-6wg4c\" (UID: \"1cdd8b6c-a030-43d8-927a-631d5474c974\") " pod="openshift-ingress/payload-processing-e2e-api-a-1dcc27-6b855b6744-6wg4c"
Jul 29 19:13:27.846690 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:27.846670    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/835ecc18-94c5-42c8-ae60-7887e93a9cab-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1dcc27-86999c6664-gccmf\" (UID: \"835ecc18-94c5-42c8-ae60-7887e93a9cab\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1dcc27-86999c6664-gccmf"
Jul 29 19:13:27.854287 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:27.854251    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmmvf\" (UniqueName: \"kubernetes.io/projected/835ecc18-94c5-42c8-ae60-7887e93a9cab-kube-api-access-xmmvf\") pod \"payload-pre-processing-e2e-api-a-1dcc27-86999c6664-gccmf\" (UID: \"835ecc18-94c5-42c8-ae60-7887e93a9cab\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1dcc27-86999c6664-gccmf"
Jul 29 19:13:27.880177 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:27.880145    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk"
Jul 29 19:13:27.946745 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:27.946684    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cdd8b6c-a030-43d8-927a-631d5474c974-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1dcc27-6b855b6744-6wg4c\" (UID: \"1cdd8b6c-a030-43d8-927a-631d5474c974\") " pod="openshift-ingress/payload-processing-e2e-api-a-1dcc27-6b855b6744-6wg4c"
Jul 29 19:13:27.946933 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:27.946766    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b6gz\" (UniqueName: \"kubernetes.io/projected/1cdd8b6c-a030-43d8-927a-631d5474c974-kube-api-access-5b6gz\") pod \"payload-processing-e2e-api-a-1dcc27-6b855b6744-6wg4c\" (UID: \"1cdd8b6c-a030-43d8-927a-631d5474c974\") " pod="openshift-ingress/payload-processing-e2e-api-a-1dcc27-6b855b6744-6wg4c"
Jul 29 19:13:27.947748 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:27.947716    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cdd8b6c-a030-43d8-927a-631d5474c974-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1dcc27-6b855b6744-6wg4c\" (UID: \"1cdd8b6c-a030-43d8-927a-631d5474c974\") " pod="openshift-ingress/payload-processing-e2e-api-a-1dcc27-6b855b6744-6wg4c"
Jul 29 19:13:27.957601 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:27.957575    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b6gz\" (UniqueName: \"kubernetes.io/projected/1cdd8b6c-a030-43d8-927a-631d5474c974-kube-api-access-5b6gz\") pod \"payload-processing-e2e-api-a-1dcc27-6b855b6744-6wg4c\" (UID: \"1cdd8b6c-a030-43d8-927a-631d5474c974\") " pod="openshift-ingress/payload-processing-e2e-api-a-1dcc27-6b855b6744-6wg4c"
Jul 29 19:13:27.966186 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:27.966155    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1dcc27-86999c6664-gccmf"
Jul 29 19:13:28.014684 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:28.014614    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1dcc27-6b855b6744-6wg4c"
Jul 29 19:13:28.032068 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:28.030886    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk"]
Jul 29 19:13:28.034422 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:13:28.034371    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd22b3e4_db32_4bcd_9a7d_451907714a5f.slice/crio-078e07d135ffe940b3db463e0630304521707c4fee9e271ea7f379ef848aa6b9 WatchSource:0}: Error finding container 078e07d135ffe940b3db463e0630304521707c4fee9e271ea7f379ef848aa6b9: Status 404 returned error can't find the container with id 078e07d135ffe940b3db463e0630304521707c4fee9e271ea7f379ef848aa6b9
Jul 29 19:13:28.134254 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:28.134216    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1dcc27-86999c6664-gccmf"]
Jul 29 19:13:28.138676 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:13:28.138589    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod835ecc18_94c5_42c8_ae60_7887e93a9cab.slice/crio-648745b3e887bfe44211c56dcd9829191c5a59304fec5f3e17fcaae18aafbc16 WatchSource:0}: Error finding container 648745b3e887bfe44211c56dcd9829191c5a59304fec5f3e17fcaae18aafbc16: Status 404 returned error can't find the container with id 648745b3e887bfe44211c56dcd9829191c5a59304fec5f3e17fcaae18aafbc16
Jul 29 19:13:28.189593 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:28.189569    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1dcc27-6b855b6744-6wg4c"]
Jul 29 19:13:28.733036 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:28.732978    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1dcc27-6b855b6744-6wg4c" event={"ID":"1cdd8b6c-a030-43d8-927a-631d5474c974","Type":"ContainerStarted","Data":"13232b132d85040adb1645cfe4736a81b48ca57ce6ea2572bcd4d6022fa1ef86"}
Jul 29 19:13:28.733251 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:28.733099    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1dcc27-6b855b6744-6wg4c" event={"ID":"1cdd8b6c-a030-43d8-927a-631d5474c974","Type":"ContainerStarted","Data":"e4dc7b5be7cdfa269b281a8427cd91b2f8079efd3538ef35153d1c3c674258b8"}
Jul 29 19:13:28.733251 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:28.733131    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-1dcc27-6b855b6744-6wg4c"
Jul 29 19:13:28.734832 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:28.734803    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk" event={"ID":"dd22b3e4-db32-4bcd-9a7d-451907714a5f","Type":"ContainerStarted","Data":"2dc70d3d1183ca30de25baa7eeb59c36f22c636e2ae991de709a53097237023a"}
Jul 29 19:13:28.734969 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:28.734856    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk" event={"ID":"dd22b3e4-db32-4bcd-9a7d-451907714a5f","Type":"ContainerStarted","Data":"078e07d135ffe940b3db463e0630304521707c4fee9e271ea7f379ef848aa6b9"}
Jul 29 19:13:28.734969 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:28.734909    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk"
Jul 29 19:13:28.736606 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:28.736582    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1dcc27-86999c6664-gccmf" event={"ID":"835ecc18-94c5-42c8-ae60-7887e93a9cab","Type":"ContainerStarted","Data":"be5fdded1176add1fa429a82a3ee9c35f620c9f76b302101ec1893da53c6507f"}
Jul 29 19:13:28.736735 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:28.736627    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1dcc27-86999c6664-gccmf" event={"ID":"835ecc18-94c5-42c8-ae60-7887e93a9cab","Type":"ContainerStarted","Data":"648745b3e887bfe44211c56dcd9829191c5a59304fec5f3e17fcaae18aafbc16"}
Jul 29 19:13:28.736735 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:28.736676    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1dcc27-86999c6664-gccmf"
Jul 29 19:13:28.753145 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:28.753089    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-1dcc27-6b855b6744-6wg4c" podStartSLOduration=1.7530736999999998 podStartE2EDuration="1.7530737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:13:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:13:28.751410712 +0000 UTC m=+2373.021179801" watchObservedRunningTime="2026-07-29 19:13:28.7530737 +0000 UTC m=+2373.022842789"
Jul 29 19:13:28.767643 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:28.767584    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1dcc27-86999c6664-gccmf" podStartSLOduration=1.767571201 podStartE2EDuration="1.767571201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:13:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:13:28.766546821 +0000 UTC m=+2373.036315910" watchObservedRunningTime="2026-07-29 19:13:28.767571201 +0000 UTC m=+2373.037340288"
Jul 29 19:13:28.784572 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:28.784507    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk" podStartSLOduration=2.241925051 podStartE2EDuration="2.784486638s" totalImagesPullingTime="542.561587ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:13:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:13:28.783226973 +0000 UTC m=+2373.052996061" watchObservedRunningTime="2026-07-29 19:13:28.784486638 +0000 UTC m=+2373.054255726"
Jul 29 19:13:31.751192 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:31.751150    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:13:34.752511 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:34.752478    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk"
Jul 29 19:13:39.748812 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:39.748778    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-1dcc27-6b855b6744-6wg4c"
Jul 29 19:13:39.749309 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:39.748867    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1dcc27-86999c6664-gccmf"
Jul 29 19:13:56.168559 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:56.168515    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-32da20-54fdb4dc65-btm88"]
Jul 29 19:13:56.173125 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:56.173096    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32da20-54fdb4dc65-btm88"
Jul 29 19:13:56.175817 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:56.175790    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-32da20-serving-cert\""
Jul 29 19:13:56.187283 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:56.187250    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-32da20-54fdb4dc65-btm88"]
Jul 29 19:13:56.202741 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:56.202696    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c57ae251-9588-4e85-b8e7-ce611f3f9949-metrics-certs\") pod \"maas-api-e2e-api-b-32da20-54fdb4dc65-btm88\" (UID: \"c57ae251-9588-4e85-b8e7-ce611f3f9949\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32da20-54fdb4dc65-btm88"
Jul 29 19:13:56.202741 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:56.202734    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c57ae251-9588-4e85-b8e7-ce611f3f9949-openshift-service-ca\") pod \"maas-api-e2e-api-b-32da20-54fdb4dc65-btm88\" (UID: \"c57ae251-9588-4e85-b8e7-ce611f3f9949\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32da20-54fdb4dc65-btm88"
Jul 29 19:13:56.203028 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:56.202841    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q24mp\" (UniqueName: \"kubernetes.io/projected/c57ae251-9588-4e85-b8e7-ce611f3f9949-kube-api-access-q24mp\") pod \"maas-api-e2e-api-b-32da20-54fdb4dc65-btm88\" (UID: \"c57ae251-9588-4e85-b8e7-ce611f3f9949\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32da20-54fdb4dc65-btm88"
Jul 29 19:13:56.203028 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:56.202884    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c57ae251-9588-4e85-b8e7-ce611f3f9949-maas-api-tls\") pod \"maas-api-e2e-api-b-32da20-54fdb4dc65-btm88\" (UID: \"c57ae251-9588-4e85-b8e7-ce611f3f9949\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32da20-54fdb4dc65-btm88"
Jul 29 19:13:56.251064 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:56.251032    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-v9nsr_7f8b4737-d4c2-4598-bd40-208015542831/console-operator/1.log"
Jul 29 19:13:56.253796 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:56.253769    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-v9nsr_7f8b4737-d4c2-4598-bd40-208015542831/console-operator/1.log"
Jul 29 19:13:56.259824 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:56.259801    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjdrx_60f444e1-7e31-443c-8ec7-a4d04c64bf4b/ovn-acl-logging/0.log"
Jul 29 19:13:56.262714 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:56.262694    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjdrx_60f444e1-7e31-443c-8ec7-a4d04c64bf4b/ovn-acl-logging/0.log"
Jul 29 19:13:56.303857 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:56.303818    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q24mp\" (UniqueName: \"kubernetes.io/projected/c57ae251-9588-4e85-b8e7-ce611f3f9949-kube-api-access-q24mp\") pod \"maas-api-e2e-api-b-32da20-54fdb4dc65-btm88\" (UID: \"c57ae251-9588-4e85-b8e7-ce611f3f9949\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32da20-54fdb4dc65-btm88"
Jul 29 19:13:56.304055 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:56.303869    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c57ae251-9588-4e85-b8e7-ce611f3f9949-maas-api-tls\") pod \"maas-api-e2e-api-b-32da20-54fdb4dc65-btm88\" (UID: \"c57ae251-9588-4e85-b8e7-ce611f3f9949\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32da20-54fdb4dc65-btm88"
Jul 29 19:13:56.304055 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:56.303899    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c57ae251-9588-4e85-b8e7-ce611f3f9949-metrics-certs\") pod \"maas-api-e2e-api-b-32da20-54fdb4dc65-btm88\" (UID: \"c57ae251-9588-4e85-b8e7-ce611f3f9949\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32da20-54fdb4dc65-btm88"
Jul 29 19:13:56.304055 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:56.303923    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c57ae251-9588-4e85-b8e7-ce611f3f9949-openshift-service-ca\") pod \"maas-api-e2e-api-b-32da20-54fdb4dc65-btm88\" (UID: \"c57ae251-9588-4e85-b8e7-ce611f3f9949\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32da20-54fdb4dc65-btm88"
Jul 29 19:13:56.304745 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:56.304717    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c57ae251-9588-4e85-b8e7-ce611f3f9949-openshift-service-ca\") pod \"maas-api-e2e-api-b-32da20-54fdb4dc65-btm88\" (UID: \"c57ae251-9588-4e85-b8e7-ce611f3f9949\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32da20-54fdb4dc65-btm88"
Jul 29 19:13:56.306679 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:56.306651    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c57ae251-9588-4e85-b8e7-ce611f3f9949-metrics-certs\") pod \"maas-api-e2e-api-b-32da20-54fdb4dc65-btm88\" (UID: \"c57ae251-9588-4e85-b8e7-ce611f3f9949\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32da20-54fdb4dc65-btm88"
Jul 29 19:13:56.306679 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:56.306679    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c57ae251-9588-4e85-b8e7-ce611f3f9949-maas-api-tls\") pod \"maas-api-e2e-api-b-32da20-54fdb4dc65-btm88\" (UID: \"c57ae251-9588-4e85-b8e7-ce611f3f9949\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32da20-54fdb4dc65-btm88"
Jul 29 19:13:56.322230 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:56.322199    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q24mp\" (UniqueName: \"kubernetes.io/projected/c57ae251-9588-4e85-b8e7-ce611f3f9949-kube-api-access-q24mp\") pod \"maas-api-e2e-api-b-32da20-54fdb4dc65-btm88\" (UID: \"c57ae251-9588-4e85-b8e7-ce611f3f9949\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32da20-54fdb4dc65-btm88"
Jul 29 19:13:56.487006 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:56.486976    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32da20-54fdb4dc65-btm88"
Jul 29 19:13:56.644529 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:56.644503    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-32da20-54fdb4dc65-btm88"]
Jul 29 19:13:56.647038 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:13:56.647003    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc57ae251_9588_4e85_b8e7_ce611f3f9949.slice/crio-5baac70cbbf84a671ac837216a4d56663703a0f36f8905db6e3789937407a8fa WatchSource:0}: Error finding container 5baac70cbbf84a671ac837216a4d56663703a0f36f8905db6e3789937407a8fa: Status 404 returned error can't find the container with id 5baac70cbbf84a671ac837216a4d56663703a0f36f8905db6e3789937407a8fa
Jul 29 19:13:56.888249 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:56.888214    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32da20-54fdb4dc65-btm88" event={"ID":"c57ae251-9588-4e85-b8e7-ce611f3f9949","Type":"ContainerStarted","Data":"5baac70cbbf84a671ac837216a4d56663703a0f36f8905db6e3789937407a8fa"}
Jul 29 19:13:57.195324 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:57.195289    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-32da20-fd754c45b-gvtmt"]
Jul 29 19:13:57.198800 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:57.198780    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-32da20-fd754c45b-gvtmt"
Jul 29 19:13:57.202069 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:57.202041    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-32da20\""
Jul 29 19:13:57.202216 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:57.202043    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-32da20-dockercfg-xbhjw\""
Jul 29 19:13:57.220080 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:57.220028    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-32da20-fd754c45b-gvtmt"]
Jul 29 19:13:57.274264 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:57.274229    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-32da20-fbd958b46-l97q4"]
Jul 29 19:13:57.277835 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:57.277814    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-32da20-fbd958b46-l97q4"
Jul 29 19:13:57.295292 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:57.295248    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-32da20-fbd958b46-l97q4"]
Jul 29 19:13:57.313989 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:57.313952    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b7bbbe2-f8e9-48d1-85c1-7548e55f4dbe-plugins-config-volume\") pod \"payload-processing-e2e-api-b-32da20-fbd958b46-l97q4\" (UID: \"0b7bbbe2-f8e9-48d1-85c1-7548e55f4dbe\") " pod="openshift-ingress/payload-processing-e2e-api-b-32da20-fbd958b46-l97q4"
Jul 29 19:13:57.314165 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:57.313998    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e1d8cde-aebd-4554-b7f7-dd9916cbd4de-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-32da20-fd754c45b-gvtmt\" (UID: \"9e1d8cde-aebd-4554-b7f7-dd9916cbd4de\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-32da20-fd754c45b-gvtmt"
Jul 29 19:13:57.314165 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:57.314035    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwwnf\" (UniqueName: \"kubernetes.io/projected/9e1d8cde-aebd-4554-b7f7-dd9916cbd4de-kube-api-access-fwwnf\") pod \"payload-pre-processing-e2e-api-b-32da20-fd754c45b-gvtmt\" (UID: \"9e1d8cde-aebd-4554-b7f7-dd9916cbd4de\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-32da20-fd754c45b-gvtmt"
Jul 29 19:13:57.314165 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:57.314108    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b9th\" (UniqueName: \"kubernetes.io/projected/0b7bbbe2-f8e9-48d1-85c1-7548e55f4dbe-kube-api-access-5b9th\") pod \"payload-processing-e2e-api-b-32da20-fbd958b46-l97q4\" (UID: \"0b7bbbe2-f8e9-48d1-85c1-7548e55f4dbe\") " pod="openshift-ingress/payload-processing-e2e-api-b-32da20-fbd958b46-l97q4"
Jul 29 19:13:57.415615 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:57.415529    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e1d8cde-aebd-4554-b7f7-dd9916cbd4de-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-32da20-fd754c45b-gvtmt\" (UID: \"9e1d8cde-aebd-4554-b7f7-dd9916cbd4de\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-32da20-fd754c45b-gvtmt"
Jul 29 19:13:57.415615 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:57.415578    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwwnf\" (UniqueName: \"kubernetes.io/projected/9e1d8cde-aebd-4554-b7f7-dd9916cbd4de-kube-api-access-fwwnf\") pod \"payload-pre-processing-e2e-api-b-32da20-fd754c45b-gvtmt\" (UID: \"9e1d8cde-aebd-4554-b7f7-dd9916cbd4de\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-32da20-fd754c45b-gvtmt"
Jul 29 19:13:57.415824 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:57.415643    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b9th\" (UniqueName: \"kubernetes.io/projected/0b7bbbe2-f8e9-48d1-85c1-7548e55f4dbe-kube-api-access-5b9th\") pod \"payload-processing-e2e-api-b-32da20-fbd958b46-l97q4\" (UID: \"0b7bbbe2-f8e9-48d1-85c1-7548e55f4dbe\") " pod="openshift-ingress/payload-processing-e2e-api-b-32da20-fbd958b46-l97q4"
Jul 29 19:13:57.415824 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:57.415679    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b7bbbe2-f8e9-48d1-85c1-7548e55f4dbe-plugins-config-volume\") pod \"payload-processing-e2e-api-b-32da20-fbd958b46-l97q4\" (UID: \"0b7bbbe2-f8e9-48d1-85c1-7548e55f4dbe\") " pod="openshift-ingress/payload-processing-e2e-api-b-32da20-fbd958b46-l97q4"
Jul 29 19:13:57.416403 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:57.416362    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e1d8cde-aebd-4554-b7f7-dd9916cbd4de-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-32da20-fd754c45b-gvtmt\" (UID: \"9e1d8cde-aebd-4554-b7f7-dd9916cbd4de\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-32da20-fd754c45b-gvtmt"
Jul 29 19:13:57.416466 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:57.416370    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b7bbbe2-f8e9-48d1-85c1-7548e55f4dbe-plugins-config-volume\") pod \"payload-processing-e2e-api-b-32da20-fbd958b46-l97q4\" (UID: \"0b7bbbe2-f8e9-48d1-85c1-7548e55f4dbe\") " pod="openshift-ingress/payload-processing-e2e-api-b-32da20-fbd958b46-l97q4"
Jul 29 19:13:57.425194 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:57.425163    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b9th\" (UniqueName: \"kubernetes.io/projected/0b7bbbe2-f8e9-48d1-85c1-7548e55f4dbe-kube-api-access-5b9th\") pod \"payload-processing-e2e-api-b-32da20-fbd958b46-l97q4\" (UID: \"0b7bbbe2-f8e9-48d1-85c1-7548e55f4dbe\") " pod="openshift-ingress/payload-processing-e2e-api-b-32da20-fbd958b46-l97q4"
Jul 29 19:13:57.425359 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:57.425199    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwwnf\" (UniqueName: \"kubernetes.io/projected/9e1d8cde-aebd-4554-b7f7-dd9916cbd4de-kube-api-access-fwwnf\") pod \"payload-pre-processing-e2e-api-b-32da20-fd754c45b-gvtmt\" (UID: \"9e1d8cde-aebd-4554-b7f7-dd9916cbd4de\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-32da20-fd754c45b-gvtmt"
Jul 29 19:13:57.509316 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:57.509274    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-32da20-fd754c45b-gvtmt"
Jul 29 19:13:57.588946 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:57.588893    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-32da20-fbd958b46-l97q4"
Jul 29 19:13:57.679738 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:57.679637    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-32da20-fd754c45b-gvtmt"]
Jul 29 19:13:57.682065 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:13:57.682037    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e1d8cde_aebd_4554_b7f7_dd9916cbd4de.slice/crio-1aa3b1048ab66f4432d44874af9c04f6631ac8651fe409b6ef418d75cdf5983a WatchSource:0}: Error finding container 1aa3b1048ab66f4432d44874af9c04f6631ac8651fe409b6ef418d75cdf5983a: Status 404 returned error can't find the container with id 1aa3b1048ab66f4432d44874af9c04f6631ac8651fe409b6ef418d75cdf5983a
Jul 29 19:13:57.753406 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:57.753364    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-32da20-fbd958b46-l97q4"]
Jul 29 19:13:57.756022 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:13:57.755990    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b7bbbe2_f8e9_48d1_85c1_7548e55f4dbe.slice/crio-b3e2d2b8ce44cb7a5fc64ee70d62123302c4b6609a1a9a732d1a2f8c6629777a WatchSource:0}: Error finding container b3e2d2b8ce44cb7a5fc64ee70d62123302c4b6609a1a9a732d1a2f8c6629777a: Status 404 returned error can't find the container with id b3e2d2b8ce44cb7a5fc64ee70d62123302c4b6609a1a9a732d1a2f8c6629777a
Jul 29 19:13:57.896132 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:57.896086    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32da20-54fdb4dc65-btm88" event={"ID":"c57ae251-9588-4e85-b8e7-ce611f3f9949","Type":"ContainerStarted","Data":"95e306d469520da30a43705272a464633c8b2828cab449ccd366f1b6eef47e38"}
Jul 29 19:13:57.896351 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:57.896263    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32da20-54fdb4dc65-btm88"
Jul 29 19:13:57.897835 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:57.897803    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-32da20-fbd958b46-l97q4" event={"ID":"0b7bbbe2-f8e9-48d1-85c1-7548e55f4dbe","Type":"ContainerStarted","Data":"ca43dd22b349464c0e50ba76b4c9a7a52efe84ca3f770050c5b829aa2f3003dc"}
Jul 29 19:13:57.897975 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:57.897845    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-32da20-fbd958b46-l97q4" event={"ID":"0b7bbbe2-f8e9-48d1-85c1-7548e55f4dbe","Type":"ContainerStarted","Data":"b3e2d2b8ce44cb7a5fc64ee70d62123302c4b6609a1a9a732d1a2f8c6629777a"}
Jul 29 19:13:57.897975 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:57.897907    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-32da20-fbd958b46-l97q4"
Jul 29 19:13:57.899477 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:57.899449    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-32da20-fd754c45b-gvtmt" event={"ID":"9e1d8cde-aebd-4554-b7f7-dd9916cbd4de","Type":"ContainerStarted","Data":"bfc08ef6213fbdda87446e2ae2ced68d9da15518fa63d132f2304feecebeca5b"}
Jul 29 19:13:57.899580 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:57.899497    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-32da20-fd754c45b-gvtmt" event={"ID":"9e1d8cde-aebd-4554-b7f7-dd9916cbd4de","Type":"ContainerStarted","Data":"1aa3b1048ab66f4432d44874af9c04f6631ac8651fe409b6ef418d75cdf5983a"}
Jul 29 19:13:57.899580 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:57.899548    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-32da20-fd754c45b-gvtmt"
Jul 29 19:13:57.918981 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:57.918915    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32da20-54fdb4dc65-btm88" podStartSLOduration=1.441822529 podStartE2EDuration="1.918896095s" totalImagesPullingTime="477.073566ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:13:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:13:57.916510578 +0000 UTC m=+2402.186279672" watchObservedRunningTime="2026-07-29 19:13:57.918896095 +0000 UTC m=+2402.188665183"
Jul 29 19:13:57.936244 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:57.936131    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-32da20-fbd958b46-l97q4" podStartSLOduration=0.936113101 podStartE2EDuration="936.113101ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:13:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:13:57.934367584 +0000 UTC m=+2402.204136673" watchObservedRunningTime="2026-07-29 19:13:57.936113101 +0000 UTC m=+2402.205882188"
Jul 29 19:13:57.952472 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:13:57.952420    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-32da20-fd754c45b-gvtmt" podStartSLOduration=0.952402231 podStartE2EDuration="952.402231ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:13:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:13:57.952161077 +0000 UTC m=+2402.221930164" watchObservedRunningTime="2026-07-29 19:13:57.952402231 +0000 UTC m=+2402.222171310"
Jul 29 19:14:03.570249 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:03.570204    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:14:03.913006 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:03.912974    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32da20-54fdb4dc65-btm88"
Jul 29 19:14:08.910515 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:08.910483    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-32da20-fbd958b46-l97q4"
Jul 29 19:14:08.910961 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:08.910571    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-32da20-fd754c45b-gvtmt"
Jul 29 19:14:32.983625 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:32.983581    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk"]
Jul 29 19:14:32.984210 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:32.983855    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk" podUID="dd22b3e4-db32-4bcd-9a7d-451907714a5f" containerName="maas-api" containerID="cri-o://2dc70d3d1183ca30de25baa7eeb59c36f22c636e2ae991de709a53097237023a" gracePeriod=30
Jul 29 19:14:33.057744 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:33.057707    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1dcc27-6b855b6744-6wg4c"]
Jul 29 19:14:33.058044 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:33.057973    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-1dcc27-6b855b6744-6wg4c" podUID="1cdd8b6c-a030-43d8-927a-631d5474c974" containerName="payload-processing" containerID="cri-o://13232b132d85040adb1645cfe4736a81b48ca57ce6ea2572bcd4d6022fa1ef86" gracePeriod=30
Jul 29 19:14:33.084823 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:33.084787    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1dcc27-86999c6664-gccmf"]
Jul 29 19:14:33.085181 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:33.085091    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1dcc27-86999c6664-gccmf" podUID="835ecc18-94c5-42c8-ae60-7887e93a9cab" containerName="payload-pre-processing" containerID="cri-o://be5fdded1176add1fa429a82a3ee9c35f620c9f76b302101ec1893da53c6507f" gracePeriod=30
Jul 29 19:14:33.317872 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:33.317824    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk"
Jul 29 19:14:33.339302 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:33.339248    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd22b3e4-db32-4bcd-9a7d-451907714a5f-openshift-service-ca\") pod \"dd22b3e4-db32-4bcd-9a7d-451907714a5f\" (UID: \"dd22b3e4-db32-4bcd-9a7d-451907714a5f\") "
Jul 29 19:14:33.339524 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:33.339317    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd22b3e4-db32-4bcd-9a7d-451907714a5f-metrics-certs\") pod \"dd22b3e4-db32-4bcd-9a7d-451907714a5f\" (UID: \"dd22b3e4-db32-4bcd-9a7d-451907714a5f\") "
Jul 29 19:14:33.339524 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:33.339350    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd22b3e4-db32-4bcd-9a7d-451907714a5f-maas-api-tls\") pod \"dd22b3e4-db32-4bcd-9a7d-451907714a5f\" (UID: \"dd22b3e4-db32-4bcd-9a7d-451907714a5f\") "
Jul 29 19:14:33.339524 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:33.339429    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpqx4\" (UniqueName: \"kubernetes.io/projected/dd22b3e4-db32-4bcd-9a7d-451907714a5f-kube-api-access-tpqx4\") pod \"dd22b3e4-db32-4bcd-9a7d-451907714a5f\" (UID: \"dd22b3e4-db32-4bcd-9a7d-451907714a5f\") "
Jul 29 19:14:33.340154 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:33.340083    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd22b3e4-db32-4bcd-9a7d-451907714a5f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dd22b3e4-db32-4bcd-9a7d-451907714a5f" (UID: "dd22b3e4-db32-4bcd-9a7d-451907714a5f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:14:33.343638 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:33.343592    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd22b3e4-db32-4bcd-9a7d-451907714a5f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dd22b3e4-db32-4bcd-9a7d-451907714a5f" (UID: "dd22b3e4-db32-4bcd-9a7d-451907714a5f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:14:33.344983 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:33.344944    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd22b3e4-db32-4bcd-9a7d-451907714a5f-kube-api-access-tpqx4" (OuterVolumeSpecName: "kube-api-access-tpqx4") pod "dd22b3e4-db32-4bcd-9a7d-451907714a5f" (UID: "dd22b3e4-db32-4bcd-9a7d-451907714a5f"). InnerVolumeSpecName "kube-api-access-tpqx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:14:33.345106 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:33.345005    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd22b3e4-db32-4bcd-9a7d-451907714a5f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "dd22b3e4-db32-4bcd-9a7d-451907714a5f" (UID: "dd22b3e4-db32-4bcd-9a7d-451907714a5f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:14:33.377578 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:33.377548    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1dcc27-86999c6664-gccmf"
Jul 29 19:14:33.391099 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:33.391074    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1dcc27-6b855b6744-6wg4c"
Jul 29 19:14:33.440609 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:33.440570    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cdd8b6c-a030-43d8-927a-631d5474c974-plugins-config-volume\") pod \"1cdd8b6c-a030-43d8-927a-631d5474c974\" (UID: \"1cdd8b6c-a030-43d8-927a-631d5474c974\") "
Jul 29 19:14:33.440818 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:33.440638    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5b6gz\" (UniqueName: \"kubernetes.io/projected/1cdd8b6c-a030-43d8-927a-631d5474c974-kube-api-access-5b6gz\") pod \"1cdd8b6c-a030-43d8-927a-631d5474c974\" (UID: \"1cdd8b6c-a030-43d8-927a-631d5474c974\") "
Jul 29 19:14:33.440818 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:33.440685    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmmvf\" (UniqueName: \"kubernetes.io/projected/835ecc18-94c5-42c8-ae60-7887e93a9cab-kube-api-access-xmmvf\") pod \"835ecc18-94c5-42c8-ae60-7887e93a9cab\" (UID: \"835ecc18-94c5-42c8-ae60-7887e93a9cab\") "
Jul 29 19:14:33.440818 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:33.440733    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/835ecc18-94c5-42c8-ae60-7887e93a9cab-plugins-config-volume\") pod \"835ecc18-94c5-42c8-ae60-7887e93a9cab\" (UID: \"835ecc18-94c5-42c8-ae60-7887e93a9cab\") "
Jul 29 19:14:33.441015 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:33.440995    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpqx4\" (UniqueName: \"kubernetes.io/projected/dd22b3e4-db32-4bcd-9a7d-451907714a5f-kube-api-access-tpqx4\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:14:33.441075 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:33.441018    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd22b3e4-db32-4bcd-9a7d-451907714a5f-openshift-service-ca\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:14:33.441075 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:33.441032    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd22b3e4-db32-4bcd-9a7d-451907714a5f-metrics-certs\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:14:33.441075 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:33.441048    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd22b3e4-db32-4bcd-9a7d-451907714a5f-maas-api-tls\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:14:33.441235 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:33.441096    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cdd8b6c-a030-43d8-927a-631d5474c974-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1cdd8b6c-a030-43d8-927a-631d5474c974" (UID: "1cdd8b6c-a030-43d8-927a-631d5474c974"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:14:33.441235 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:33.441123    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/835ecc18-94c5-42c8-ae60-7887e93a9cab-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "835ecc18-94c5-42c8-ae60-7887e93a9cab" (UID: "835ecc18-94c5-42c8-ae60-7887e93a9cab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:14:33.443063 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:33.443030    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/835ecc18-94c5-42c8-ae60-7887e93a9cab-kube-api-access-xmmvf" (OuterVolumeSpecName: "kube-api-access-xmmvf") pod "835ecc18-94c5-42c8-ae60-7887e93a9cab" (UID: "835ecc18-94c5-42c8-ae60-7887e93a9cab"). InnerVolumeSpecName "kube-api-access-xmmvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:14:33.443178 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:33.443063    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cdd8b6c-a030-43d8-927a-631d5474c974-kube-api-access-5b6gz" (OuterVolumeSpecName: "kube-api-access-5b6gz") pod "1cdd8b6c-a030-43d8-927a-631d5474c974" (UID: "1cdd8b6c-a030-43d8-927a-631d5474c974"). InnerVolumeSpecName "kube-api-access-5b6gz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:14:33.541557 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:33.541470    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/835ecc18-94c5-42c8-ae60-7887e93a9cab-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:14:33.541557 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:33.541500    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cdd8b6c-a030-43d8-927a-631d5474c974-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:14:33.541557 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:33.541510    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b6gz\" (UniqueName: \"kubernetes.io/projected/1cdd8b6c-a030-43d8-927a-631d5474c974-kube-api-access-5b6gz\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:14:33.541557 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:33.541520    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmmvf\" (UniqueName: \"kubernetes.io/projected/835ecc18-94c5-42c8-ae60-7887e93a9cab-kube-api-access-xmmvf\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:14:34.092669 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:34.092636    2559 generic.go:356] "Generic (PLEG): container finished" podID="835ecc18-94c5-42c8-ae60-7887e93a9cab" containerID="be5fdded1176add1fa429a82a3ee9c35f620c9f76b302101ec1893da53c6507f" exitCode=0
Jul 29 19:14:34.093186 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:34.092684    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1dcc27-86999c6664-gccmf"
Jul 29 19:14:34.093186 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:34.092732    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1dcc27-86999c6664-gccmf" event={"ID":"835ecc18-94c5-42c8-ae60-7887e93a9cab","Type":"ContainerDied","Data":"be5fdded1176add1fa429a82a3ee9c35f620c9f76b302101ec1893da53c6507f"}
Jul 29 19:14:34.093186 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:34.092793    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1dcc27-86999c6664-gccmf" event={"ID":"835ecc18-94c5-42c8-ae60-7887e93a9cab","Type":"ContainerDied","Data":"648745b3e887bfe44211c56dcd9829191c5a59304fec5f3e17fcaae18aafbc16"}
Jul 29 19:14:34.093186 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:34.092822    2559 scope.go:117] "RemoveContainer" containerID="be5fdded1176add1fa429a82a3ee9c35f620c9f76b302101ec1893da53c6507f"
Jul 29 19:14:34.094343 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:34.094323    2559 generic.go:356] "Generic (PLEG): container finished" podID="1cdd8b6c-a030-43d8-927a-631d5474c974" containerID="13232b132d85040adb1645cfe4736a81b48ca57ce6ea2572bcd4d6022fa1ef86" exitCode=0
Jul 29 19:14:34.094471 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:34.094364    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1dcc27-6b855b6744-6wg4c"
Jul 29 19:14:34.094538 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:34.094466    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1dcc27-6b855b6744-6wg4c" event={"ID":"1cdd8b6c-a030-43d8-927a-631d5474c974","Type":"ContainerDied","Data":"13232b132d85040adb1645cfe4736a81b48ca57ce6ea2572bcd4d6022fa1ef86"}
Jul 29 19:14:34.094538 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:34.094508    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1dcc27-6b855b6744-6wg4c" event={"ID":"1cdd8b6c-a030-43d8-927a-631d5474c974","Type":"ContainerDied","Data":"e4dc7b5be7cdfa269b281a8427cd91b2f8079efd3538ef35153d1c3c674258b8"}
Jul 29 19:14:34.095959 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:34.095934    2559 generic.go:356] "Generic (PLEG): container finished" podID="dd22b3e4-db32-4bcd-9a7d-451907714a5f" containerID="2dc70d3d1183ca30de25baa7eeb59c36f22c636e2ae991de709a53097237023a" exitCode=0
Jul 29 19:14:34.096088 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:34.096024    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk" event={"ID":"dd22b3e4-db32-4bcd-9a7d-451907714a5f","Type":"ContainerDied","Data":"2dc70d3d1183ca30de25baa7eeb59c36f22c636e2ae991de709a53097237023a"}
Jul 29 19:14:34.096088 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:34.096066    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk" event={"ID":"dd22b3e4-db32-4bcd-9a7d-451907714a5f","Type":"ContainerDied","Data":"078e07d135ffe940b3db463e0630304521707c4fee9e271ea7f379ef848aa6b9"}
Jul 29 19:14:34.096217 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:34.096088    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk"
Jul 29 19:14:34.106635 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:34.106601    2559 scope.go:117] "RemoveContainer" containerID="be5fdded1176add1fa429a82a3ee9c35f620c9f76b302101ec1893da53c6507f"
Jul 29 19:14:34.107074 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:14:34.106971    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be5fdded1176add1fa429a82a3ee9c35f620c9f76b302101ec1893da53c6507f\": container with ID starting with be5fdded1176add1fa429a82a3ee9c35f620c9f76b302101ec1893da53c6507f not found: ID does not exist" containerID="be5fdded1176add1fa429a82a3ee9c35f620c9f76b302101ec1893da53c6507f"
Jul 29 19:14:34.107074 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:34.107006    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be5fdded1176add1fa429a82a3ee9c35f620c9f76b302101ec1893da53c6507f"} err="failed to get container status \"be5fdded1176add1fa429a82a3ee9c35f620c9f76b302101ec1893da53c6507f\": rpc error: code = NotFound desc = could not find container \"be5fdded1176add1fa429a82a3ee9c35f620c9f76b302101ec1893da53c6507f\": container with ID starting with be5fdded1176add1fa429a82a3ee9c35f620c9f76b302101ec1893da53c6507f not found: ID does not exist"
Jul 29 19:14:34.107074 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:34.107025    2559 scope.go:117] "RemoveContainer" containerID="13232b132d85040adb1645cfe4736a81b48ca57ce6ea2572bcd4d6022fa1ef86"
Jul 29 19:14:34.116319 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:34.116295    2559 scope.go:117] "RemoveContainer" containerID="13232b132d85040adb1645cfe4736a81b48ca57ce6ea2572bcd4d6022fa1ef86"
Jul 29 19:14:34.116630 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:14:34.116611    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13232b132d85040adb1645cfe4736a81b48ca57ce6ea2572bcd4d6022fa1ef86\": container with ID starting with 13232b132d85040adb1645cfe4736a81b48ca57ce6ea2572bcd4d6022fa1ef86 not found: ID does not exist" containerID="13232b132d85040adb1645cfe4736a81b48ca57ce6ea2572bcd4d6022fa1ef86"
Jul 29 19:14:34.116731 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:34.116637    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13232b132d85040adb1645cfe4736a81b48ca57ce6ea2572bcd4d6022fa1ef86"} err="failed to get container status \"13232b132d85040adb1645cfe4736a81b48ca57ce6ea2572bcd4d6022fa1ef86\": rpc error: code = NotFound desc = could not find container \"13232b132d85040adb1645cfe4736a81b48ca57ce6ea2572bcd4d6022fa1ef86\": container with ID starting with 13232b132d85040adb1645cfe4736a81b48ca57ce6ea2572bcd4d6022fa1ef86 not found: ID does not exist"
Jul 29 19:14:34.116731 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:34.116653    2559 scope.go:117] "RemoveContainer" containerID="2dc70d3d1183ca30de25baa7eeb59c36f22c636e2ae991de709a53097237023a"
Jul 29 19:14:34.122922 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:34.122899    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1dcc27-86999c6664-gccmf"]
Jul 29 19:14:34.126243 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:34.126222    2559 scope.go:117] "RemoveContainer" containerID="2dc70d3d1183ca30de25baa7eeb59c36f22c636e2ae991de709a53097237023a"
Jul 29 19:14:34.126571 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:14:34.126552    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dc70d3d1183ca30de25baa7eeb59c36f22c636e2ae991de709a53097237023a\": container with ID starting with 2dc70d3d1183ca30de25baa7eeb59c36f22c636e2ae991de709a53097237023a not found: ID does not exist" containerID="2dc70d3d1183ca30de25baa7eeb59c36f22c636e2ae991de709a53097237023a"
Jul 29 19:14:34.126636 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:34.126579    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dc70d3d1183ca30de25baa7eeb59c36f22c636e2ae991de709a53097237023a"} err="failed to get container status \"2dc70d3d1183ca30de25baa7eeb59c36f22c636e2ae991de709a53097237023a\": rpc error: code = NotFound desc = could not find container \"2dc70d3d1183ca30de25baa7eeb59c36f22c636e2ae991de709a53097237023a\": container with ID starting with 2dc70d3d1183ca30de25baa7eeb59c36f22c636e2ae991de709a53097237023a not found: ID does not exist"
Jul 29 19:14:34.130957 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:34.130933    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1dcc27-86999c6664-gccmf"]
Jul 29 19:14:34.151093 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:34.151057    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="835ecc18-94c5-42c8-ae60-7887e93a9cab" path="/var/lib/kubelet/pods/835ecc18-94c5-42c8-ae60-7887e93a9cab/volumes"
Jul 29 19:14:34.151341 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:34.151326    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk"]
Jul 29 19:14:34.151429 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:34.151354    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1dcc27-6c56c89b78-6khdk"]
Jul 29 19:14:34.162805 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:34.162771    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1dcc27-6b855b6744-6wg4c"]
Jul 29 19:14:34.165896 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:34.165868    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1dcc27-6b855b6744-6wg4c"]
Jul 29 19:14:36.143269 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:36.143229    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:14:36.150083 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:36.150052    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cdd8b6c-a030-43d8-927a-631d5474c974" path="/var/lib/kubelet/pods/1cdd8b6c-a030-43d8-927a-631d5474c974/volumes"
Jul 29 19:14:36.150544 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:36.150530    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd22b3e4-db32-4bcd-9a7d-451907714a5f" path="/var/lib/kubelet/pods/dd22b3e4-db32-4bcd-9a7d-451907714a5f/volumes"
Jul 29 19:14:55.593216 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:55.593170    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-32da20-54fdb4dc65-btm88"]
Jul 29 19:14:55.594046 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:55.594017    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32da20-54fdb4dc65-btm88" podUID="c57ae251-9588-4e85-b8e7-ce611f3f9949" containerName="maas-api" containerID="cri-o://95e306d469520da30a43705272a464633c8b2828cab449ccd366f1b6eef47e38" gracePeriod=30
Jul 29 19:14:55.691986 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:55.691938    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-32da20-fbd958b46-l97q4"]
Jul 29 19:14:55.692274 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:55.692220    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-32da20-fbd958b46-l97q4" podUID="0b7bbbe2-f8e9-48d1-85c1-7548e55f4dbe" containerName="payload-processing" containerID="cri-o://ca43dd22b349464c0e50ba76b4c9a7a52efe84ca3f770050c5b829aa2f3003dc" gracePeriod=30
Jul 29 19:14:55.701261 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:55.701216    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-32da20-fd754c45b-gvtmt"]
Jul 29 19:14:55.702873 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:55.701500    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-32da20-fd754c45b-gvtmt" podUID="9e1d8cde-aebd-4554-b7f7-dd9916cbd4de" containerName="payload-pre-processing" containerID="cri-o://bfc08ef6213fbdda87446e2ae2ced68d9da15518fa63d132f2304feecebeca5b" gracePeriod=30
Jul 29 19:14:55.875857 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:55.875822    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32da20-54fdb4dc65-btm88"
Jul 29 19:14:55.943515 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:55.943479    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q24mp\" (UniqueName: \"kubernetes.io/projected/c57ae251-9588-4e85-b8e7-ce611f3f9949-kube-api-access-q24mp\") pod \"c57ae251-9588-4e85-b8e7-ce611f3f9949\" (UID: \"c57ae251-9588-4e85-b8e7-ce611f3f9949\") "
Jul 29 19:14:55.943710 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:55.943534    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c57ae251-9588-4e85-b8e7-ce611f3f9949-maas-api-tls\") pod \"c57ae251-9588-4e85-b8e7-ce611f3f9949\" (UID: \"c57ae251-9588-4e85-b8e7-ce611f3f9949\") "
Jul 29 19:14:55.943710 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:55.943637    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c57ae251-9588-4e85-b8e7-ce611f3f9949-metrics-certs\") pod \"c57ae251-9588-4e85-b8e7-ce611f3f9949\" (UID: \"c57ae251-9588-4e85-b8e7-ce611f3f9949\") "
Jul 29 19:14:55.943883 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:55.943709    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c57ae251-9588-4e85-b8e7-ce611f3f9949-openshift-service-ca\") pod \"c57ae251-9588-4e85-b8e7-ce611f3f9949\" (UID: \"c57ae251-9588-4e85-b8e7-ce611f3f9949\") "
Jul 29 19:14:55.944122 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:55.944091    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c57ae251-9588-4e85-b8e7-ce611f3f9949-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c57ae251-9588-4e85-b8e7-ce611f3f9949" (UID: "c57ae251-9588-4e85-b8e7-ce611f3f9949"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:14:55.945952 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:55.945926    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c57ae251-9588-4e85-b8e7-ce611f3f9949-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c57ae251-9588-4e85-b8e7-ce611f3f9949" (UID: "c57ae251-9588-4e85-b8e7-ce611f3f9949"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:14:55.946031 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:55.945954    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c57ae251-9588-4e85-b8e7-ce611f3f9949-kube-api-access-q24mp" (OuterVolumeSpecName: "kube-api-access-q24mp") pod "c57ae251-9588-4e85-b8e7-ce611f3f9949" (UID: "c57ae251-9588-4e85-b8e7-ce611f3f9949"). InnerVolumeSpecName "kube-api-access-q24mp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:14:55.946604 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:55.946588    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c57ae251-9588-4e85-b8e7-ce611f3f9949-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c57ae251-9588-4e85-b8e7-ce611f3f9949" (UID: "c57ae251-9588-4e85-b8e7-ce611f3f9949"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:14:55.961290 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:55.961264    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-32da20-fbd958b46-l97q4"
Jul 29 19:14:56.001855 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.001827    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-32da20-fd754c45b-gvtmt"
Jul 29 19:14:56.045183 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.045138    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b7bbbe2-f8e9-48d1-85c1-7548e55f4dbe-plugins-config-volume\") pod \"0b7bbbe2-f8e9-48d1-85c1-7548e55f4dbe\" (UID: \"0b7bbbe2-f8e9-48d1-85c1-7548e55f4dbe\") "
Jul 29 19:14:56.045411 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.045234    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5b9th\" (UniqueName: \"kubernetes.io/projected/0b7bbbe2-f8e9-48d1-85c1-7548e55f4dbe-kube-api-access-5b9th\") pod \"0b7bbbe2-f8e9-48d1-85c1-7548e55f4dbe\" (UID: \"0b7bbbe2-f8e9-48d1-85c1-7548e55f4dbe\") "
Jul 29 19:14:56.045605 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.045585    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c57ae251-9588-4e85-b8e7-ce611f3f9949-openshift-service-ca\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:14:56.045668 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.045594    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b7bbbe2-f8e9-48d1-85c1-7548e55f4dbe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0b7bbbe2-f8e9-48d1-85c1-7548e55f4dbe" (UID: "0b7bbbe2-f8e9-48d1-85c1-7548e55f4dbe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:14:56.045668 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.045610    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q24mp\" (UniqueName: \"kubernetes.io/projected/c57ae251-9588-4e85-b8e7-ce611f3f9949-kube-api-access-q24mp\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:14:56.045668 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.045630    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c57ae251-9588-4e85-b8e7-ce611f3f9949-maas-api-tls\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:14:56.045668 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.045644    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c57ae251-9588-4e85-b8e7-ce611f3f9949-metrics-certs\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:14:56.047515 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.047488    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b7bbbe2-f8e9-48d1-85c1-7548e55f4dbe-kube-api-access-5b9th" (OuterVolumeSpecName: "kube-api-access-5b9th") pod "0b7bbbe2-f8e9-48d1-85c1-7548e55f4dbe" (UID: "0b7bbbe2-f8e9-48d1-85c1-7548e55f4dbe"). InnerVolumeSpecName "kube-api-access-5b9th". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:14:56.146109 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.146007    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e1d8cde-aebd-4554-b7f7-dd9916cbd4de-plugins-config-volume\") pod \"9e1d8cde-aebd-4554-b7f7-dd9916cbd4de\" (UID: \"9e1d8cde-aebd-4554-b7f7-dd9916cbd4de\") "
Jul 29 19:14:56.146109 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.146108    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwwnf\" (UniqueName: \"kubernetes.io/projected/9e1d8cde-aebd-4554-b7f7-dd9916cbd4de-kube-api-access-fwwnf\") pod \"9e1d8cde-aebd-4554-b7f7-dd9916cbd4de\" (UID: \"9e1d8cde-aebd-4554-b7f7-dd9916cbd4de\") "
Jul 29 19:14:56.146343 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.146315    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b7bbbe2-f8e9-48d1-85c1-7548e55f4dbe-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:14:56.146343 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.146336    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b9th\" (UniqueName: \"kubernetes.io/projected/0b7bbbe2-f8e9-48d1-85c1-7548e55f4dbe-kube-api-access-5b9th\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:14:56.146451 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.146418    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e1d8cde-aebd-4554-b7f7-dd9916cbd4de-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9e1d8cde-aebd-4554-b7f7-dd9916cbd4de" (UID: "9e1d8cde-aebd-4554-b7f7-dd9916cbd4de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:14:56.148529 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.148499    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e1d8cde-aebd-4554-b7f7-dd9916cbd4de-kube-api-access-fwwnf" (OuterVolumeSpecName: "kube-api-access-fwwnf") pod "9e1d8cde-aebd-4554-b7f7-dd9916cbd4de" (UID: "9e1d8cde-aebd-4554-b7f7-dd9916cbd4de"). InnerVolumeSpecName "kube-api-access-fwwnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:14:56.212585 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.212550    2559 generic.go:356] "Generic (PLEG): container finished" podID="c57ae251-9588-4e85-b8e7-ce611f3f9949" containerID="95e306d469520da30a43705272a464633c8b2828cab449ccd366f1b6eef47e38" exitCode=0
Jul 29 19:14:56.212776 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.212596    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32da20-54fdb4dc65-btm88"
Jul 29 19:14:56.212776 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.212639    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32da20-54fdb4dc65-btm88" event={"ID":"c57ae251-9588-4e85-b8e7-ce611f3f9949","Type":"ContainerDied","Data":"95e306d469520da30a43705272a464633c8b2828cab449ccd366f1b6eef47e38"}
Jul 29 19:14:56.212776 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.212697    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32da20-54fdb4dc65-btm88" event={"ID":"c57ae251-9588-4e85-b8e7-ce611f3f9949","Type":"ContainerDied","Data":"5baac70cbbf84a671ac837216a4d56663703a0f36f8905db6e3789937407a8fa"}
Jul 29 19:14:56.212776 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.212724    2559 scope.go:117] "RemoveContainer" containerID="95e306d469520da30a43705272a464633c8b2828cab449ccd366f1b6eef47e38"
Jul 29 19:14:56.214130 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.214108    2559 generic.go:356] "Generic (PLEG): container finished" podID="0b7bbbe2-f8e9-48d1-85c1-7548e55f4dbe" containerID="ca43dd22b349464c0e50ba76b4c9a7a52efe84ca3f770050c5b829aa2f3003dc" exitCode=0
Jul 29 19:14:56.214238 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.214158    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-32da20-fbd958b46-l97q4"
Jul 29 19:14:56.214238 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.214194    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-32da20-fbd958b46-l97q4" event={"ID":"0b7bbbe2-f8e9-48d1-85c1-7548e55f4dbe","Type":"ContainerDied","Data":"ca43dd22b349464c0e50ba76b4c9a7a52efe84ca3f770050c5b829aa2f3003dc"}
Jul 29 19:14:56.214404 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.214243    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-32da20-fbd958b46-l97q4" event={"ID":"0b7bbbe2-f8e9-48d1-85c1-7548e55f4dbe","Type":"ContainerDied","Data":"b3e2d2b8ce44cb7a5fc64ee70d62123302c4b6609a1a9a732d1a2f8c6629777a"}
Jul 29 19:14:56.219909 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.219843    2559 generic.go:356] "Generic (PLEG): container finished" podID="9e1d8cde-aebd-4554-b7f7-dd9916cbd4de" containerID="bfc08ef6213fbdda87446e2ae2ced68d9da15518fa63d132f2304feecebeca5b" exitCode=0
Jul 29 19:14:56.220079 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.219897    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-32da20-fd754c45b-gvtmt" event={"ID":"9e1d8cde-aebd-4554-b7f7-dd9916cbd4de","Type":"ContainerDied","Data":"bfc08ef6213fbdda87446e2ae2ced68d9da15518fa63d132f2304feecebeca5b"}
Jul 29 19:14:56.220079 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.219958    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-32da20-fd754c45b-gvtmt" event={"ID":"9e1d8cde-aebd-4554-b7f7-dd9916cbd4de","Type":"ContainerDied","Data":"1aa3b1048ab66f4432d44874af9c04f6631ac8651fe409b6ef418d75cdf5983a"}
Jul 29 19:14:56.220079 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.219894    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-32da20-fd754c45b-gvtmt"
Jul 29 19:14:56.225106 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.225089    2559 scope.go:117] "RemoveContainer" containerID="95e306d469520da30a43705272a464633c8b2828cab449ccd366f1b6eef47e38"
Jul 29 19:14:56.225417 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:14:56.225399    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95e306d469520da30a43705272a464633c8b2828cab449ccd366f1b6eef47e38\": container with ID starting with 95e306d469520da30a43705272a464633c8b2828cab449ccd366f1b6eef47e38 not found: ID does not exist" containerID="95e306d469520da30a43705272a464633c8b2828cab449ccd366f1b6eef47e38"
Jul 29 19:14:56.225471 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.225426    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95e306d469520da30a43705272a464633c8b2828cab449ccd366f1b6eef47e38"} err="failed to get container status \"95e306d469520da30a43705272a464633c8b2828cab449ccd366f1b6eef47e38\": rpc error: code = NotFound desc = could not find container \"95e306d469520da30a43705272a464633c8b2828cab449ccd366f1b6eef47e38\": container with ID starting with 95e306d469520da30a43705272a464633c8b2828cab449ccd366f1b6eef47e38 not found: ID does not exist"
Jul 29 19:14:56.225471 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.225442    2559 scope.go:117] "RemoveContainer" containerID="ca43dd22b349464c0e50ba76b4c9a7a52efe84ca3f770050c5b829aa2f3003dc"
Jul 29 19:14:56.232928 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.232900    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-32da20-54fdb4dc65-btm88"]
Jul 29 19:14:56.234365 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.234344    2559 scope.go:117] "RemoveContainer" containerID="ca43dd22b349464c0e50ba76b4c9a7a52efe84ca3f770050c5b829aa2f3003dc"
Jul 29 19:14:56.234999 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:14:56.234974    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca43dd22b349464c0e50ba76b4c9a7a52efe84ca3f770050c5b829aa2f3003dc\": container with ID starting with ca43dd22b349464c0e50ba76b4c9a7a52efe84ca3f770050c5b829aa2f3003dc not found: ID does not exist" containerID="ca43dd22b349464c0e50ba76b4c9a7a52efe84ca3f770050c5b829aa2f3003dc"
Jul 29 19:14:56.235108 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.235004    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca43dd22b349464c0e50ba76b4c9a7a52efe84ca3f770050c5b829aa2f3003dc"} err="failed to get container status \"ca43dd22b349464c0e50ba76b4c9a7a52efe84ca3f770050c5b829aa2f3003dc\": rpc error: code = NotFound desc = could not find container \"ca43dd22b349464c0e50ba76b4c9a7a52efe84ca3f770050c5b829aa2f3003dc\": container with ID starting with ca43dd22b349464c0e50ba76b4c9a7a52efe84ca3f770050c5b829aa2f3003dc not found: ID does not exist"
Jul 29 19:14:56.235108 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.235024    2559 scope.go:117] "RemoveContainer" containerID="bfc08ef6213fbdda87446e2ae2ced68d9da15518fa63d132f2304feecebeca5b"
Jul 29 19:14:56.236617 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.236598    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-32da20-54fdb4dc65-btm88"]
Jul 29 19:14:56.243536 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.243514    2559 scope.go:117] "RemoveContainer" containerID="bfc08ef6213fbdda87446e2ae2ced68d9da15518fa63d132f2304feecebeca5b"
Jul 29 19:14:56.243791 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:14:56.243774    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfc08ef6213fbdda87446e2ae2ced68d9da15518fa63d132f2304feecebeca5b\": container with ID starting with bfc08ef6213fbdda87446e2ae2ced68d9da15518fa63d132f2304feecebeca5b not found: ID does not exist" containerID="bfc08ef6213fbdda87446e2ae2ced68d9da15518fa63d132f2304feecebeca5b"
Jul 29 19:14:56.243842 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.243800    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfc08ef6213fbdda87446e2ae2ced68d9da15518fa63d132f2304feecebeca5b"} err="failed to get container status \"bfc08ef6213fbdda87446e2ae2ced68d9da15518fa63d132f2304feecebeca5b\": rpc error: code = NotFound desc = could not find container \"bfc08ef6213fbdda87446e2ae2ced68d9da15518fa63d132f2304feecebeca5b\": container with ID starting with bfc08ef6213fbdda87446e2ae2ced68d9da15518fa63d132f2304feecebeca5b not found: ID does not exist"
Jul 29 19:14:56.247330 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.247083    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e1d8cde-aebd-4554-b7f7-dd9916cbd4de-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:14:56.247330 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.247121    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwwnf\" (UniqueName: \"kubernetes.io/projected/9e1d8cde-aebd-4554-b7f7-dd9916cbd4de-kube-api-access-fwwnf\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:14:56.247945 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.247921    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-32da20-fbd958b46-l97q4"]
Jul 29 19:14:56.249553 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.249534    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-32da20-fbd958b46-l97q4"]
Jul 29 19:14:56.260310 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.260277    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-32da20-fd754c45b-gvtmt"]
Jul 29 19:14:56.264594 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.264569    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-32da20-fd754c45b-gvtmt"]
Jul 29 19:14:56.480977 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:56.480879    2559 scope.go:117] "RemoveContainer" containerID="7a58b555641831a3424a06055bc980723f38e4bc64f923e93a6fc5dcb47e17e1"
Jul 29 19:14:58.151151 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:58.151111    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b7bbbe2-f8e9-48d1-85c1-7548e55f4dbe" path="/var/lib/kubelet/pods/0b7bbbe2-f8e9-48d1-85c1-7548e55f4dbe/volumes"
Jul 29 19:14:58.151568 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:58.151496    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e1d8cde-aebd-4554-b7f7-dd9916cbd4de" path="/var/lib/kubelet/pods/9e1d8cde-aebd-4554-b7f7-dd9916cbd4de/volumes"
Jul 29 19:14:58.151831 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:14:58.151818    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c57ae251-9588-4e85-b8e7-ce611f3f9949" path="/var/lib/kubelet/pods/c57ae251-9588-4e85-b8e7-ce611f3f9949/volumes"
Jul 29 19:15:00.136800 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:00.136763    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755875-9r79b"]
Jul 29 19:15:00.137199 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:00.137111    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e1d8cde-aebd-4554-b7f7-dd9916cbd4de" containerName="payload-pre-processing"
Jul 29 19:15:00.137199 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:00.137122    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e1d8cde-aebd-4554-b7f7-dd9916cbd4de" containerName="payload-pre-processing"
Jul 29 19:15:00.137199 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:00.137135    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd22b3e4-db32-4bcd-9a7d-451907714a5f" containerName="maas-api"
Jul 29 19:15:00.137199 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:00.137140    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd22b3e4-db32-4bcd-9a7d-451907714a5f" containerName="maas-api"
Jul 29 19:15:00.137199 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:00.137147    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b7bbbe2-f8e9-48d1-85c1-7548e55f4dbe" containerName="payload-processing"
Jul 29 19:15:00.137199 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:00.137153    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b7bbbe2-f8e9-48d1-85c1-7548e55f4dbe" containerName="payload-processing"
Jul 29 19:15:00.137199 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:00.137161    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c57ae251-9588-4e85-b8e7-ce611f3f9949" containerName="maas-api"
Jul 29 19:15:00.137199 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:00.137167    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c57ae251-9588-4e85-b8e7-ce611f3f9949" containerName="maas-api"
Jul 29 19:15:00.137199 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:00.137179    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cdd8b6c-a030-43d8-927a-631d5474c974" containerName="payload-processing"
Jul 29 19:15:00.137199 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:00.137184    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cdd8b6c-a030-43d8-927a-631d5474c974" containerName="payload-processing"
Jul 29 19:15:00.137199 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:00.137193    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="835ecc18-94c5-42c8-ae60-7887e93a9cab" containerName="payload-pre-processing"
Jul 29 19:15:00.137199 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:00.137198    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="835ecc18-94c5-42c8-ae60-7887e93a9cab" containerName="payload-pre-processing"
Jul 29 19:15:00.137658 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:00.137253    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cdd8b6c-a030-43d8-927a-631d5474c974" containerName="payload-processing"
Jul 29 19:15:00.137658 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:00.137261    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="835ecc18-94c5-42c8-ae60-7887e93a9cab" containerName="payload-pre-processing"
Jul 29 19:15:00.137658 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:00.137267    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd22b3e4-db32-4bcd-9a7d-451907714a5f" containerName="maas-api"
Jul 29 19:15:00.137658 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:00.137273    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c57ae251-9588-4e85-b8e7-ce611f3f9949" containerName="maas-api"
Jul 29 19:15:00.137658 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:00.137280    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e1d8cde-aebd-4554-b7f7-dd9916cbd4de" containerName="payload-pre-processing"
Jul 29 19:15:00.137658 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:00.137285    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b7bbbe2-f8e9-48d1-85c1-7548e55f4dbe" containerName="payload-processing"
Jul 29 19:15:00.142109 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:00.142088    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755875-9r79b"
Jul 29 19:15:00.144840 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:00.144817    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vgb9z\""
Jul 29 19:15:00.152140 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:00.152111    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755875-9r79b"]
Jul 29 19:15:00.155434 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:00.155414    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755875-9r79b"
Jul 29 19:15:00.297205 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:00.297177    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755875-9r79b"]
Jul 29 19:15:00.299642 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:15:00.299612    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd04bdf68_c320_46c7_83ae_d14e6fadbba8.slice/crio-60f5a0561d1f576205eb0a89157e53bde3631cd2031c3ea42d10aeda57043d0c WatchSource:0}: Error finding container 60f5a0561d1f576205eb0a89157e53bde3631cd2031c3ea42d10aeda57043d0c: Status 404 returned error can't find the container with id 60f5a0561d1f576205eb0a89157e53bde3631cd2031c3ea42d10aeda57043d0c
Jul 29 19:15:00.639022 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:00.638991    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:15:01.247449 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:01.247417    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755875-9r79b" event={"ID":"d04bdf68-c320-46c7-83ae-d14e6fadbba8","Type":"ContainerStarted","Data":"c422632de762beafc4f324de797e8f66515c1c29e48559056d52d51a93c6e854"}
Jul 29 19:15:01.247784 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:01.247468    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755875-9r79b" event={"ID":"d04bdf68-c320-46c7-83ae-d14e6fadbba8","Type":"ContainerStarted","Data":"60f5a0561d1f576205eb0a89157e53bde3631cd2031c3ea42d10aeda57043d0c"}
Jul 29 19:15:01.269252 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:01.269197    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755875-9r79b" podStartSLOduration=1.269179961 podStartE2EDuration="1.269179961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:15:01.268011687 +0000 UTC m=+2465.537780775" watchObservedRunningTime="2026-07-29 19:15:01.269179961 +0000 UTC m=+2465.538949199"
Jul 29 19:15:02.253829 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:02.253791    2559 generic.go:356] "Generic (PLEG): container finished" podID="d04bdf68-c320-46c7-83ae-d14e6fadbba8" containerID="c422632de762beafc4f324de797e8f66515c1c29e48559056d52d51a93c6e854" exitCode=0
Jul 29 19:15:02.254209 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:02.253885    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755875-9r79b" event={"ID":"d04bdf68-c320-46c7-83ae-d14e6fadbba8","Type":"ContainerDied","Data":"c422632de762beafc4f324de797e8f66515c1c29e48559056d52d51a93c6e854"}
Jul 29 19:15:03.403734 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:03.403704    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755875-9r79b"
Jul 29 19:15:04.264870 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:04.264840    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755875-9r79b"
Jul 29 19:15:04.265126 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:04.264882    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755875-9r79b" event={"ID":"d04bdf68-c320-46c7-83ae-d14e6fadbba8","Type":"ContainerDied","Data":"60f5a0561d1f576205eb0a89157e53bde3631cd2031c3ea42d10aeda57043d0c"}
Jul 29 19:15:04.265126 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:04.264934    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60f5a0561d1f576205eb0a89157e53bde3631cd2031c3ea42d10aeda57043d0c"
Jul 29 19:15:06.357898 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:06.357861    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d04442-76994dcbf5-pzbdh"]
Jul 29 19:15:06.358464 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:06.358445    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d04bdf68-c320-46c7-83ae-d14e6fadbba8" containerName="cleanup"
Jul 29 19:15:06.358520 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:06.358468    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d04bdf68-c320-46c7-83ae-d14e6fadbba8" containerName="cleanup"
Jul 29 19:15:06.358586 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:06.358574    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d04bdf68-c320-46c7-83ae-d14e6fadbba8" containerName="cleanup"
Jul 29 19:15:06.362691 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:06.362664    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d04442-76994dcbf5-pzbdh"
Jul 29 19:15:06.366084 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:06.366038    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-d04442-serving-cert\""
Jul 29 19:15:06.367045 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:06.367023    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 19:15:06.367882 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:06.367861    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mglnh\""
Jul 29 19:15:06.368030 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:06.368000    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Jul 29 19:15:06.368110 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:06.368088    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 29 19:15:06.380246 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:06.380216    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d04442-76994dcbf5-pzbdh"]
Jul 29 19:15:06.442542 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:06.442499    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08ef2ec9-453f-42c0-be3f-a801a7f2812d-metrics-certs\") pod \"maas-api-e2e-shared-a-d04442-76994dcbf5-pzbdh\" (UID: \"08ef2ec9-453f-42c0-be3f-a801a7f2812d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d04442-76994dcbf5-pzbdh"
Jul 29 19:15:06.442735 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:06.442557    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08ef2ec9-453f-42c0-be3f-a801a7f2812d-maas-api-tls\") pod \"maas-api-e2e-shared-a-d04442-76994dcbf5-pzbdh\" (UID: \"08ef2ec9-453f-42c0-be3f-a801a7f2812d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d04442-76994dcbf5-pzbdh"
Jul 29 19:15:06.442735 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:06.442617    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxddr\" (UniqueName: \"kubernetes.io/projected/08ef2ec9-453f-42c0-be3f-a801a7f2812d-kube-api-access-hxddr\") pod \"maas-api-e2e-shared-a-d04442-76994dcbf5-pzbdh\" (UID: \"08ef2ec9-453f-42c0-be3f-a801a7f2812d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d04442-76994dcbf5-pzbdh"
Jul 29 19:15:06.442735 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:06.442698    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08ef2ec9-453f-42c0-be3f-a801a7f2812d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-d04442-76994dcbf5-pzbdh\" (UID: \"08ef2ec9-453f-42c0-be3f-a801a7f2812d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d04442-76994dcbf5-pzbdh"
Jul 29 19:15:06.543577 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:06.543542    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxddr\" (UniqueName: \"kubernetes.io/projected/08ef2ec9-453f-42c0-be3f-a801a7f2812d-kube-api-access-hxddr\") pod \"maas-api-e2e-shared-a-d04442-76994dcbf5-pzbdh\" (UID: \"08ef2ec9-453f-42c0-be3f-a801a7f2812d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d04442-76994dcbf5-pzbdh"
Jul 29 19:15:06.543810 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:06.543598    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08ef2ec9-453f-42c0-be3f-a801a7f2812d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-d04442-76994dcbf5-pzbdh\" (UID: \"08ef2ec9-453f-42c0-be3f-a801a7f2812d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d04442-76994dcbf5-pzbdh"
Jul 29 19:15:06.543810 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:06.543641    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08ef2ec9-453f-42c0-be3f-a801a7f2812d-metrics-certs\") pod \"maas-api-e2e-shared-a-d04442-76994dcbf5-pzbdh\" (UID: \"08ef2ec9-453f-42c0-be3f-a801a7f2812d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d04442-76994dcbf5-pzbdh"
Jul 29 19:15:06.543810 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:06.543664    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08ef2ec9-453f-42c0-be3f-a801a7f2812d-maas-api-tls\") pod \"maas-api-e2e-shared-a-d04442-76994dcbf5-pzbdh\" (UID: \"08ef2ec9-453f-42c0-be3f-a801a7f2812d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d04442-76994dcbf5-pzbdh"
Jul 29 19:15:06.544343 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:06.544318    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08ef2ec9-453f-42c0-be3f-a801a7f2812d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-d04442-76994dcbf5-pzbdh\" (UID: \"08ef2ec9-453f-42c0-be3f-a801a7f2812d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d04442-76994dcbf5-pzbdh"
Jul 29 19:15:06.546456 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:06.546428    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08ef2ec9-453f-42c0-be3f-a801a7f2812d-metrics-certs\") pod \"maas-api-e2e-shared-a-d04442-76994dcbf5-pzbdh\" (UID: \"08ef2ec9-453f-42c0-be3f-a801a7f2812d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d04442-76994dcbf5-pzbdh"
Jul 29 19:15:06.546552 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:06.546477    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08ef2ec9-453f-42c0-be3f-a801a7f2812d-maas-api-tls\") pod \"maas-api-e2e-shared-a-d04442-76994dcbf5-pzbdh\" (UID: \"08ef2ec9-453f-42c0-be3f-a801a7f2812d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d04442-76994dcbf5-pzbdh"
Jul 29 19:15:06.555298 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:06.555268    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxddr\" (UniqueName: \"kubernetes.io/projected/08ef2ec9-453f-42c0-be3f-a801a7f2812d-kube-api-access-hxddr\") pod \"maas-api-e2e-shared-a-d04442-76994dcbf5-pzbdh\" (UID: \"08ef2ec9-453f-42c0-be3f-a801a7f2812d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d04442-76994dcbf5-pzbdh"
Jul 29 19:15:06.675993 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:06.675885    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d04442-76994dcbf5-pzbdh"
Jul 29 19:15:06.836698 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:06.836672    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d04442-76994dcbf5-pzbdh"]
Jul 29 19:15:06.839112 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:15:06.839075    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08ef2ec9_453f_42c0_be3f_a801a7f2812d.slice/crio-1f7bbc4ea92ffa3f8e4db55871a3272490f15e2dd223933cf831c9557e56103c WatchSource:0}: Error finding container 1f7bbc4ea92ffa3f8e4db55871a3272490f15e2dd223933cf831c9557e56103c: Status 404 returned error can't find the container with id 1f7bbc4ea92ffa3f8e4db55871a3272490f15e2dd223933cf831c9557e56103c
Jul 29 19:15:07.117765 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:07.117727    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-d04442-7c85966bb-l9q2w"]
Jul 29 19:15:07.122563 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:07.122533    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-d04442-7c85966bb-l9q2w"
Jul 29 19:15:07.126770 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:07.126739    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-d04442-dockercfg-rsj7j\""
Jul 29 19:15:07.126917 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:07.126783    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-d04442\""
Jul 29 19:15:07.133367 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:07.133325    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-d04442-7c85966bb-l9q2w"]
Jul 29 19:15:07.145228 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:07.145195    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-d04442-5598777f68-svhg8"]
Jul 29 19:15:07.148776 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:07.148753    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-d04442-5598777f68-svhg8"
Jul 29 19:15:07.173227 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:07.173192    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-d04442-5598777f68-svhg8"]
Jul 29 19:15:07.252663 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:07.252635    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5nsx\" (UniqueName: \"kubernetes.io/projected/f137423f-ecf1-424b-adfc-da0e242b17a2-kube-api-access-p5nsx\") pod \"payload-processing-e2e-shared-a-d04442-7c85966bb-l9q2w\" (UID: \"f137423f-ecf1-424b-adfc-da0e242b17a2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-d04442-7c85966bb-l9q2w"
Jul 29 19:15:07.252784 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:07.252684    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4jsm\" (UniqueName: \"kubernetes.io/projected/2c5bec84-2a80-466e-9ba7-6efcdea6a38e-kube-api-access-t4jsm\") pod \"payload-pre-processing-e2e-shared-a-d04442-5598777f68-svhg8\" (UID: \"2c5bec84-2a80-466e-9ba7-6efcdea6a38e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-d04442-5598777f68-svhg8"
Jul 29 19:15:07.252784 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:07.252719    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c5bec84-2a80-466e-9ba7-6efcdea6a38e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-d04442-5598777f68-svhg8\" (UID: \"2c5bec84-2a80-466e-9ba7-6efcdea6a38e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-d04442-5598777f68-svhg8"
Jul 29 19:15:07.252944 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:07.252782    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f137423f-ecf1-424b-adfc-da0e242b17a2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-d04442-7c85966bb-l9q2w\" (UID: \"f137423f-ecf1-424b-adfc-da0e242b17a2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-d04442-7c85966bb-l9q2w"
Jul 29 19:15:07.281467 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:07.281432    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d04442-76994dcbf5-pzbdh" event={"ID":"08ef2ec9-453f-42c0-be3f-a801a7f2812d","Type":"ContainerStarted","Data":"1f7bbc4ea92ffa3f8e4db55871a3272490f15e2dd223933cf831c9557e56103c"}
Jul 29 19:15:07.353515 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:07.353471    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5nsx\" (UniqueName: \"kubernetes.io/projected/f137423f-ecf1-424b-adfc-da0e242b17a2-kube-api-access-p5nsx\") pod \"payload-processing-e2e-shared-a-d04442-7c85966bb-l9q2w\" (UID: \"f137423f-ecf1-424b-adfc-da0e242b17a2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-d04442-7c85966bb-l9q2w"
Jul 29 19:15:07.353515 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:07.353524    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4jsm\" (UniqueName: \"kubernetes.io/projected/2c5bec84-2a80-466e-9ba7-6efcdea6a38e-kube-api-access-t4jsm\") pod \"payload-pre-processing-e2e-shared-a-d04442-5598777f68-svhg8\" (UID: \"2c5bec84-2a80-466e-9ba7-6efcdea6a38e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-d04442-5598777f68-svhg8"
Jul 29 19:15:07.353796 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:07.353553    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c5bec84-2a80-466e-9ba7-6efcdea6a38e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-d04442-5598777f68-svhg8\" (UID: \"2c5bec84-2a80-466e-9ba7-6efcdea6a38e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-d04442-5598777f68-svhg8"
Jul 29 19:15:07.353796 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:07.353629    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f137423f-ecf1-424b-adfc-da0e242b17a2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-d04442-7c85966bb-l9q2w\" (UID: \"f137423f-ecf1-424b-adfc-da0e242b17a2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-d04442-7c85966bb-l9q2w"
Jul 29 19:15:07.354525 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:07.354500    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f137423f-ecf1-424b-adfc-da0e242b17a2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-d04442-7c85966bb-l9q2w\" (UID: \"f137423f-ecf1-424b-adfc-da0e242b17a2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-d04442-7c85966bb-l9q2w"
Jul 29 19:15:07.354590 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:07.354500    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c5bec84-2a80-466e-9ba7-6efcdea6a38e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-d04442-5598777f68-svhg8\" (UID: \"2c5bec84-2a80-466e-9ba7-6efcdea6a38e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-d04442-5598777f68-svhg8"
Jul 29 19:15:07.366575 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:07.366548    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5nsx\" (UniqueName: \"kubernetes.io/projected/f137423f-ecf1-424b-adfc-da0e242b17a2-kube-api-access-p5nsx\") pod \"payload-processing-e2e-shared-a-d04442-7c85966bb-l9q2w\" (UID: \"f137423f-ecf1-424b-adfc-da0e242b17a2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-d04442-7c85966bb-l9q2w"
Jul 29 19:15:07.369798 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:07.369747    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4jsm\" (UniqueName: \"kubernetes.io/projected/2c5bec84-2a80-466e-9ba7-6efcdea6a38e-kube-api-access-t4jsm\") pod \"payload-pre-processing-e2e-shared-a-d04442-5598777f68-svhg8\" (UID: \"2c5bec84-2a80-466e-9ba7-6efcdea6a38e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-d04442-5598777f68-svhg8"
Jul 29 19:15:07.434749 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:07.434711    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-d04442-7c85966bb-l9q2w"
Jul 29 19:15:07.460300 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:07.460269    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-d04442-5598777f68-svhg8"
Jul 29 19:15:07.597895 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:07.597812    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-d04442-7c85966bb-l9q2w"]
Jul 29 19:15:07.601336 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:15:07.601301    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf137423f_ecf1_424b_adfc_da0e242b17a2.slice/crio-95895affd348422fe0f4b98719f0fca14cfb03388c94332773dc50f474eaec77 WatchSource:0}: Error finding container 95895affd348422fe0f4b98719f0fca14cfb03388c94332773dc50f474eaec77: Status 404 returned error can't find the container with id 95895affd348422fe0f4b98719f0fca14cfb03388c94332773dc50f474eaec77
Jul 29 19:15:07.633832 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:07.633806    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-d04442-5598777f68-svhg8"]
Jul 29 19:15:07.639723 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:15:07.639696    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c5bec84_2a80_466e_9ba7_6efcdea6a38e.slice/crio-185533def893b040c4d2688bcadbe9a3e3b9de6850a601bbebb331d0a961394a WatchSource:0}: Error finding container 185533def893b040c4d2688bcadbe9a3e3b9de6850a601bbebb331d0a961394a: Status 404 returned error can't find the container with id 185533def893b040c4d2688bcadbe9a3e3b9de6850a601bbebb331d0a961394a
Jul 29 19:15:08.288560 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:08.288520    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d04442-76994dcbf5-pzbdh" event={"ID":"08ef2ec9-453f-42c0-be3f-a801a7f2812d","Type":"ContainerStarted","Data":"fdf69b6b6abb0d1d8dc02d6975a15064b26c1acc71b5fad48426fabccb0ef8d4"}
Jul 29 19:15:08.288752 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:08.288653    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d04442-76994dcbf5-pzbdh"
Jul 29 19:15:08.290188 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:08.290160    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-d04442-5598777f68-svhg8" event={"ID":"2c5bec84-2a80-466e-9ba7-6efcdea6a38e","Type":"ContainerStarted","Data":"8b9016cd495bd85963f655011f7a3e9925b42fa814d5ed040dfd48b7fbfb1399"}
Jul 29 19:15:08.290327 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:08.290204    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-d04442-5598777f68-svhg8" event={"ID":"2c5bec84-2a80-466e-9ba7-6efcdea6a38e","Type":"ContainerStarted","Data":"185533def893b040c4d2688bcadbe9a3e3b9de6850a601bbebb331d0a961394a"}
Jul 29 19:15:08.290327 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:08.290251    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-d04442-5598777f68-svhg8"
Jul 29 19:15:08.291694 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:08.291666    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-d04442-7c85966bb-l9q2w" event={"ID":"f137423f-ecf1-424b-adfc-da0e242b17a2","Type":"ContainerStarted","Data":"bf9d25031b54ce3c73300ce60956ff1dea285a53cf2055dd6a55625fe01f44f2"}
Jul 29 19:15:08.291783 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:08.291711    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-d04442-7c85966bb-l9q2w" event={"ID":"f137423f-ecf1-424b-adfc-da0e242b17a2","Type":"ContainerStarted","Data":"95895affd348422fe0f4b98719f0fca14cfb03388c94332773dc50f474eaec77"}
Jul 29 19:15:08.291838 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:08.291787    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-d04442-7c85966bb-l9q2w"
Jul 29 19:15:08.306636 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:08.306587    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d04442-76994dcbf5-pzbdh" podStartSLOduration=1.896697409 podStartE2EDuration="2.306570161s" totalImagesPullingTime="409.872752ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:15:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:15:08.306071246 +0000 UTC m=+2472.575840336" watchObservedRunningTime="2026-07-29 19:15:08.306570161 +0000 UTC m=+2472.576339249"
Jul 29 19:15:08.325128 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:08.325071    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-d04442-7c85966bb-l9q2w" podStartSLOduration=1.325053995 podStartE2EDuration="1.325053995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:15:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:15:08.323118834 +0000 UTC m=+2472.592887922" watchObservedRunningTime="2026-07-29 19:15:08.325053995 +0000 UTC m=+2472.594823082"
Jul 29 19:15:08.339183 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:08.339122    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-d04442-5598777f68-svhg8" podStartSLOduration=1.339104518 podStartE2EDuration="1.339104518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:15:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:15:08.338064979 +0000 UTC m=+2472.607834067" watchObservedRunningTime="2026-07-29 19:15:08.339104518 +0000 UTC m=+2472.608873605"
Jul 29 19:15:12.041395 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:12.041353    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:15:14.302821 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:14.302792    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d04442-76994dcbf5-pzbdh"
Jul 29 19:15:19.300142 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:19.300102    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-d04442-5598777f68-svhg8"
Jul 29 19:15:19.300634 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:19.300612    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-d04442-7c85966bb-l9q2w"
Jul 29 19:15:35.687746 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:35.687710    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-422d30-5886654bcc-8tg5t"]
Jul 29 19:15:35.691265 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:35.691242    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-422d30-5886654bcc-8tg5t"
Jul 29 19:15:35.694362 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:35.694342    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-422d30-serving-cert\""
Jul 29 19:15:35.713840 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:35.713805    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-422d30-5886654bcc-8tg5t"]
Jul 29 19:15:35.785730 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:35.785682    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e5ca931-aa77-48a6-a941-2d8e6c6483fe-metrics-certs\") pod \"maas-api-e2e-shared-b-422d30-5886654bcc-8tg5t\" (UID: \"9e5ca931-aa77-48a6-a941-2d8e6c6483fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-422d30-5886654bcc-8tg5t"
Jul 29 19:15:35.785934 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:35.785744    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e5ca931-aa77-48a6-a941-2d8e6c6483fe-maas-api-tls\") pod \"maas-api-e2e-shared-b-422d30-5886654bcc-8tg5t\" (UID: \"9e5ca931-aa77-48a6-a941-2d8e6c6483fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-422d30-5886654bcc-8tg5t"
Jul 29 19:15:35.785934 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:35.785780    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e5ca931-aa77-48a6-a941-2d8e6c6483fe-openshift-service-ca\") pod \"maas-api-e2e-shared-b-422d30-5886654bcc-8tg5t\" (UID: \"9e5ca931-aa77-48a6-a941-2d8e6c6483fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-422d30-5886654bcc-8tg5t"
Jul 29 19:15:35.785934 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:35.785824    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdtmd\" (UniqueName: \"kubernetes.io/projected/9e5ca931-aa77-48a6-a941-2d8e6c6483fe-kube-api-access-xdtmd\") pod \"maas-api-e2e-shared-b-422d30-5886654bcc-8tg5t\" (UID: \"9e5ca931-aa77-48a6-a941-2d8e6c6483fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-422d30-5886654bcc-8tg5t"
Jul 29 19:15:35.886618 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:35.886579    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e5ca931-aa77-48a6-a941-2d8e6c6483fe-metrics-certs\") pod \"maas-api-e2e-shared-b-422d30-5886654bcc-8tg5t\" (UID: \"9e5ca931-aa77-48a6-a941-2d8e6c6483fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-422d30-5886654bcc-8tg5t"
Jul 29 19:15:35.886813 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:35.886658    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e5ca931-aa77-48a6-a941-2d8e6c6483fe-maas-api-tls\") pod \"maas-api-e2e-shared-b-422d30-5886654bcc-8tg5t\" (UID: \"9e5ca931-aa77-48a6-a941-2d8e6c6483fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-422d30-5886654bcc-8tg5t"
Jul 29 19:15:35.886813 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:35.886695    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e5ca931-aa77-48a6-a941-2d8e6c6483fe-openshift-service-ca\") pod \"maas-api-e2e-shared-b-422d30-5886654bcc-8tg5t\" (UID: \"9e5ca931-aa77-48a6-a941-2d8e6c6483fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-422d30-5886654bcc-8tg5t"
Jul 29 19:15:35.886938 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:35.886843    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdtmd\" (UniqueName: \"kubernetes.io/projected/9e5ca931-aa77-48a6-a941-2d8e6c6483fe-kube-api-access-xdtmd\") pod \"maas-api-e2e-shared-b-422d30-5886654bcc-8tg5t\" (UID: \"9e5ca931-aa77-48a6-a941-2d8e6c6483fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-422d30-5886654bcc-8tg5t"
Jul 29 19:15:35.887400 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:35.887355    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e5ca931-aa77-48a6-a941-2d8e6c6483fe-openshift-service-ca\") pod \"maas-api-e2e-shared-b-422d30-5886654bcc-8tg5t\" (UID: \"9e5ca931-aa77-48a6-a941-2d8e6c6483fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-422d30-5886654bcc-8tg5t"
Jul 29 19:15:35.889293 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:35.889272    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e5ca931-aa77-48a6-a941-2d8e6c6483fe-maas-api-tls\") pod \"maas-api-e2e-shared-b-422d30-5886654bcc-8tg5t\" (UID: \"9e5ca931-aa77-48a6-a941-2d8e6c6483fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-422d30-5886654bcc-8tg5t"
Jul 29 19:15:35.889432 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:35.889329    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e5ca931-aa77-48a6-a941-2d8e6c6483fe-metrics-certs\") pod \"maas-api-e2e-shared-b-422d30-5886654bcc-8tg5t\" (UID: \"9e5ca931-aa77-48a6-a941-2d8e6c6483fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-422d30-5886654bcc-8tg5t"
Jul 29 19:15:35.895715 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:35.895687    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdtmd\" (UniqueName: \"kubernetes.io/projected/9e5ca931-aa77-48a6-a941-2d8e6c6483fe-kube-api-access-xdtmd\") pod \"maas-api-e2e-shared-b-422d30-5886654bcc-8tg5t\" (UID: \"9e5ca931-aa77-48a6-a941-2d8e6c6483fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-422d30-5886654bcc-8tg5t"
Jul 29 19:15:36.002307 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:36.002215    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-422d30-5886654bcc-8tg5t"
Jul 29 19:15:36.153357 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:36.153332    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-422d30-5886654bcc-8tg5t"]
Jul 29 19:15:36.156035 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:15:36.156008    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e5ca931_aa77_48a6_a941_2d8e6c6483fe.slice/crio-369e0487739f2d096b6f1d1db8241523ce7ef2f918efafeefe4971d7d227cc93 WatchSource:0}: Error finding container 369e0487739f2d096b6f1d1db8241523ce7ef2f918efafeefe4971d7d227cc93: Status 404 returned error can't find the container with id 369e0487739f2d096b6f1d1db8241523ce7ef2f918efafeefe4971d7d227cc93
Jul 29 19:15:36.379203 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:36.379168    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-422d30-68f8cc9484-mb5d9"]
Jul 29 19:15:36.383739 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:36.383719    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-422d30-68f8cc9484-mb5d9"
Jul 29 19:15:36.386872 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:36.386850    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-422d30-dockercfg-q7wt9\""
Jul 29 19:15:36.387092 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:36.387079    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-422d30\""
Jul 29 19:15:36.395653 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:36.395621    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-422d30-68f8cc9484-mb5d9"]
Jul 29 19:15:36.432637 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:36.432602    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-422d30-5886654bcc-8tg5t" event={"ID":"9e5ca931-aa77-48a6-a941-2d8e6c6483fe","Type":"ContainerStarted","Data":"369e0487739f2d096b6f1d1db8241523ce7ef2f918efafeefe4971d7d227cc93"}
Jul 29 19:15:36.493349 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:36.493307    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/888ca0f7-0649-4536-a2d5-8ff65029f5f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-422d30-68f8cc9484-mb5d9\" (UID: \"888ca0f7-0649-4536-a2d5-8ff65029f5f1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-422d30-68f8cc9484-mb5d9"
Jul 29 19:15:36.493555 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:36.493432    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5nxv\" (UniqueName: \"kubernetes.io/projected/888ca0f7-0649-4536-a2d5-8ff65029f5f1-kube-api-access-t5nxv\") pod \"payload-pre-processing-e2e-shared-b-422d30-68f8cc9484-mb5d9\" (UID: \"888ca0f7-0649-4536-a2d5-8ff65029f5f1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-422d30-68f8cc9484-mb5d9"
Jul 29 19:15:36.594719 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:36.594693    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/888ca0f7-0649-4536-a2d5-8ff65029f5f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-422d30-68f8cc9484-mb5d9\" (UID: \"888ca0f7-0649-4536-a2d5-8ff65029f5f1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-422d30-68f8cc9484-mb5d9"
Jul 29 19:15:36.594873 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:36.594765    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5nxv\" (UniqueName: \"kubernetes.io/projected/888ca0f7-0649-4536-a2d5-8ff65029f5f1-kube-api-access-t5nxv\") pod \"payload-pre-processing-e2e-shared-b-422d30-68f8cc9484-mb5d9\" (UID: \"888ca0f7-0649-4536-a2d5-8ff65029f5f1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-422d30-68f8cc9484-mb5d9"
Jul 29 19:15:36.595414 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:36.595375    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/888ca0f7-0649-4536-a2d5-8ff65029f5f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-422d30-68f8cc9484-mb5d9\" (UID: \"888ca0f7-0649-4536-a2d5-8ff65029f5f1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-422d30-68f8cc9484-mb5d9"
Jul 29 19:15:36.604393 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:36.604359    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5nxv\" (UniqueName: \"kubernetes.io/projected/888ca0f7-0649-4536-a2d5-8ff65029f5f1-kube-api-access-t5nxv\") pod \"payload-pre-processing-e2e-shared-b-422d30-68f8cc9484-mb5d9\" (UID: \"888ca0f7-0649-4536-a2d5-8ff65029f5f1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-422d30-68f8cc9484-mb5d9"
Jul 29 19:15:36.695283 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:36.695176    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-422d30-68f8cc9484-mb5d9"
Jul 29 19:15:37.056178 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:37.056082    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-422d30-68f8cc9484-mb5d9"]
Jul 29 19:15:37.064845 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:15:37.064812    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod888ca0f7_0649_4536_a2d5_8ff65029f5f1.slice/crio-5fcbae0a22dfb065d202c79cb828ce23b12404c0aebb93e57e69d92b2a5aba53 WatchSource:0}: Error finding container 5fcbae0a22dfb065d202c79cb828ce23b12404c0aebb93e57e69d92b2a5aba53: Status 404 returned error can't find the container with id 5fcbae0a22dfb065d202c79cb828ce23b12404c0aebb93e57e69d92b2a5aba53
Jul 29 19:15:37.438763 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:37.438721    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-422d30-68f8cc9484-mb5d9" event={"ID":"888ca0f7-0649-4536-a2d5-8ff65029f5f1","Type":"ContainerStarted","Data":"09a6d04d8f14a2fbd049b48946ec6f0fcbdfa94f76ff2a6306e37b53061ee108"}
Jul 29 19:15:37.438971 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:37.438790    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-422d30-68f8cc9484-mb5d9" event={"ID":"888ca0f7-0649-4536-a2d5-8ff65029f5f1","Type":"ContainerStarted","Data":"5fcbae0a22dfb065d202c79cb828ce23b12404c0aebb93e57e69d92b2a5aba53"}
Jul 29 19:15:37.438971 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:37.438818    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-422d30-68f8cc9484-mb5d9"
Jul 29 19:15:37.440168 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:37.440134    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-422d30-5886654bcc-8tg5t" event={"ID":"9e5ca931-aa77-48a6-a941-2d8e6c6483fe","Type":"ContainerStarted","Data":"eed0e4d05079e3e4bc20930d50af62600d3e7ec0f5d4e7764d94bbceaae8eef7"}
Jul 29 19:15:37.440283 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:37.440199    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-422d30-5886654bcc-8tg5t"
Jul 29 19:15:37.457757 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:37.457705    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-422d30-68f8cc9484-mb5d9" podStartSLOduration=1.457688191 podStartE2EDuration="1.457688191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:15:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:15:37.457175207 +0000 UTC m=+2501.726944296" watchObservedRunningTime="2026-07-29 19:15:37.457688191 +0000 UTC m=+2501.727457279"
Jul 29 19:15:37.475231 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:37.475178    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-422d30-5886654bcc-8tg5t" podStartSLOduration=2.083706077 podStartE2EDuration="2.475160503s" totalImagesPullingTime="391.454426ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:15:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:15:37.474147001 +0000 UTC m=+2501.743916086" watchObservedRunningTime="2026-07-29 19:15:37.475160503 +0000 UTC m=+2501.744929590"
Jul 29 19:15:43.447361 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:43.447313    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:15:43.455071 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:43.455038    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-422d30-5886654bcc-8tg5t"
Jul 29 19:15:48.447148 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:15:48.447069    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-422d30-68f8cc9484-mb5d9"
Jul 29 19:16:22.444170 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:16:22.444128    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:16:54.777661 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:16:54.777627    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-422d30/auth-test-model-422d30-kserve-6676f64755-s7xsl"]
Jul 29 19:16:54.781307 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:16:54.781284    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-422d30/auth-test-model-422d30-kserve-6676f64755-s7xsl"
Jul 29 19:16:54.784779 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:16:54.784755    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-422d30\"/\"kube-root-ca.crt\""
Jul 29 19:16:54.784965 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:16:54.784755    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-422d30\"/\"default-dockercfg-8b8fl\""
Jul 29 19:16:54.784965 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:16:54.784756    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-422d30\"/\"openshift-service-ca.crt\""
Jul 29 19:16:54.784965 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:16:54.784816    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-422d30\"/\"auth-test-model-422d30-kserve-self-signed-certs\""
Jul 29 19:16:54.790057 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:16:54.790014    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-422d30/auth-test-model-422d30-kserve-6676f64755-s7xsl"]
Jul 29 19:16:54.902789 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:16:54.902752    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0-model-cache\") pod \"auth-test-model-422d30-kserve-6676f64755-s7xsl\" (UID: \"e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0\") " pod="ai-tenant-e2e-shared-b-422d30/auth-test-model-422d30-kserve-6676f64755-s7xsl"
Jul 29 19:16:54.903003 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:16:54.902829    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0-tmp-dir\") pod \"auth-test-model-422d30-kserve-6676f64755-s7xsl\" (UID: \"e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0\") " pod="ai-tenant-e2e-shared-b-422d30/auth-test-model-422d30-kserve-6676f64755-s7xsl"
Jul 29 19:16:54.903003 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:16:54.902890    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0-dshm\") pod \"auth-test-model-422d30-kserve-6676f64755-s7xsl\" (UID: \"e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0\") " pod="ai-tenant-e2e-shared-b-422d30/auth-test-model-422d30-kserve-6676f64755-s7xsl"
Jul 29 19:16:54.903003 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:16:54.902974    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0-home\") pod \"auth-test-model-422d30-kserve-6676f64755-s7xsl\" (UID: \"e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0\") " pod="ai-tenant-e2e-shared-b-422d30/auth-test-model-422d30-kserve-6676f64755-s7xsl"
Jul 29 19:16:54.903119 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:16:54.903004    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0-tls-certs\") pod \"auth-test-model-422d30-kserve-6676f64755-s7xsl\" (UID: \"e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0\") " pod="ai-tenant-e2e-shared-b-422d30/auth-test-model-422d30-kserve-6676f64755-s7xsl"
Jul 29 19:16:54.903119 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:16:54.903048    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cswjx\" (UniqueName: \"kubernetes.io/projected/e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0-kube-api-access-cswjx\") pod \"auth-test-model-422d30-kserve-6676f64755-s7xsl\" (UID: \"e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0\") " pod="ai-tenant-e2e-shared-b-422d30/auth-test-model-422d30-kserve-6676f64755-s7xsl"
Jul 29 19:16:55.003565 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:16:55.003526    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0-tmp-dir\") pod \"auth-test-model-422d30-kserve-6676f64755-s7xsl\" (UID: \"e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0\") " pod="ai-tenant-e2e-shared-b-422d30/auth-test-model-422d30-kserve-6676f64755-s7xsl"
Jul 29 19:16:55.003773 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:16:55.003587    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0-dshm\") pod \"auth-test-model-422d30-kserve-6676f64755-s7xsl\" (UID: \"e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0\") " pod="ai-tenant-e2e-shared-b-422d30/auth-test-model-422d30-kserve-6676f64755-s7xsl"
Jul 29 19:16:55.003773 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:16:55.003610    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0-home\") pod \"auth-test-model-422d30-kserve-6676f64755-s7xsl\" (UID: \"e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0\") " pod="ai-tenant-e2e-shared-b-422d30/auth-test-model-422d30-kserve-6676f64755-s7xsl"
Jul 29 19:16:55.003773 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:16:55.003624    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0-tls-certs\") pod \"auth-test-model-422d30-kserve-6676f64755-s7xsl\" (UID: \"e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0\") " pod="ai-tenant-e2e-shared-b-422d30/auth-test-model-422d30-kserve-6676f64755-s7xsl"
Jul 29 19:16:55.003773 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:16:55.003653    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cswjx\" (UniqueName: \"kubernetes.io/projected/e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0-kube-api-access-cswjx\") pod \"auth-test-model-422d30-kserve-6676f64755-s7xsl\" (UID: \"e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0\") " pod="ai-tenant-e2e-shared-b-422d30/auth-test-model-422d30-kserve-6676f64755-s7xsl"
Jul 29 19:16:55.003773 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:16:55.003693    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0-model-cache\") pod \"auth-test-model-422d30-kserve-6676f64755-s7xsl\" (UID: \"e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0\") " pod="ai-tenant-e2e-shared-b-422d30/auth-test-model-422d30-kserve-6676f64755-s7xsl"
Jul 29 19:16:55.004038 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:16:55.003978    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0-tmp-dir\") pod \"auth-test-model-422d30-kserve-6676f64755-s7xsl\" (UID: \"e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0\") " pod="ai-tenant-e2e-shared-b-422d30/auth-test-model-422d30-kserve-6676f64755-s7xsl"
Jul 29 19:16:55.004038 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:16:55.004035    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0-home\") pod \"auth-test-model-422d30-kserve-6676f64755-s7xsl\" (UID: \"e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0\") " pod="ai-tenant-e2e-shared-b-422d30/auth-test-model-422d30-kserve-6676f64755-s7xsl"
Jul 29 19:16:55.004117 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:16:55.004082    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0-model-cache\") pod \"auth-test-model-422d30-kserve-6676f64755-s7xsl\" (UID: \"e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0\") " pod="ai-tenant-e2e-shared-b-422d30/auth-test-model-422d30-kserve-6676f64755-s7xsl"
Jul 29 19:16:55.006222 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:16:55.006196    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0-dshm\") pod \"auth-test-model-422d30-kserve-6676f64755-s7xsl\" (UID: \"e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0\") " pod="ai-tenant-e2e-shared-b-422d30/auth-test-model-422d30-kserve-6676f64755-s7xsl"
Jul 29 19:16:55.006332 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:16:55.006317    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0-tls-certs\") pod \"auth-test-model-422d30-kserve-6676f64755-s7xsl\" (UID: \"e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0\") " pod="ai-tenant-e2e-shared-b-422d30/auth-test-model-422d30-kserve-6676f64755-s7xsl"
Jul 29 19:16:55.013636 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:16:55.013615    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cswjx\" (UniqueName: \"kubernetes.io/projected/e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0-kube-api-access-cswjx\") pod \"auth-test-model-422d30-kserve-6676f64755-s7xsl\" (UID: \"e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0\") " pod="ai-tenant-e2e-shared-b-422d30/auth-test-model-422d30-kserve-6676f64755-s7xsl"
Jul 29 19:16:55.091528 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:16:55.091490    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-422d30/auth-test-model-422d30-kserve-6676f64755-s7xsl"
Jul 29 19:16:55.231885 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:16:55.231858    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-422d30/auth-test-model-422d30-kserve-6676f64755-s7xsl"]
Jul 29 19:16:55.234283 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:16:55.234252    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode29dfcbc_4cc2_4a12_98c1_ede92f00c7d0.slice/crio-5bf21a22d1d2055541d0c59502b35faf958d68cf6371c5d7fe82771563f025b9 WatchSource:0}: Error finding container 5bf21a22d1d2055541d0c59502b35faf958d68cf6371c5d7fe82771563f025b9: Status 404 returned error can't find the container with id 5bf21a22d1d2055541d0c59502b35faf958d68cf6371c5d7fe82771563f025b9
Jul 29 19:16:55.850927 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:16:55.850882    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-422d30/auth-test-model-422d30-kserve-6676f64755-s7xsl" event={"ID":"e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0","Type":"ContainerStarted","Data":"2f305780b68944b4a76389342e6bef0ccf00c8f5cc5c361627042a91a9f9260b"}
Jul 29 19:16:55.852561 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:16:55.852526    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-422d30/auth-test-model-422d30-kserve-6676f64755-s7xsl" event={"ID":"e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0","Type":"ContainerStarted","Data":"5bf21a22d1d2055541d0c59502b35faf958d68cf6371c5d7fe82771563f025b9"}
Jul 29 19:16:55.871710 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:16:55.871650    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-422d30/auth-test-model-422d30-kserve-6676f64755-s7xsl" podStartSLOduration=1.8716339180000001 podStartE2EDuration="1.871633918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:16:55.870193182 +0000 UTC m=+2580.139962279" watchObservedRunningTime="2026-07-29 19:16:55.871633918 +0000 UTC m=+2580.141403004"
Jul 29 19:16:58.151213 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:16:58.151179    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:17:05.092524 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:17:05.092480    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-422d30/auth-test-model-422d30-kserve-6676f64755-s7xsl"
Jul 29 19:17:05.092524 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:17:05.092532    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-422d30/auth-test-model-422d30-kserve-6676f64755-s7xsl"
Jul 29 19:17:05.105258 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:17:05.105230    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-422d30/auth-test-model-422d30-kserve-6676f64755-s7xsl"
Jul 29 19:17:05.909768 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:17:05.909736    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-422d30/auth-test-model-422d30-kserve-6676f64755-s7xsl"
Jul 29 19:17:22.341517 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:17:22.341427    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:17:43.449234 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:17:43.449191    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:17:59.056041 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:17:59.056003    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:18:18.952937 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:18:18.952882    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:18:36.944574 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:18:36.944536    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:18:56.282550 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:18:56.282520    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-v9nsr_7f8b4737-d4c2-4598-bd40-208015542831/console-operator/1.log"
Jul 29 19:18:56.287586 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:18:56.287560    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-v9nsr_7f8b4737-d4c2-4598-bd40-208015542831/console-operator/1.log"
Jul 29 19:18:56.290278 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:18:56.290256    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjdrx_60f444e1-7e31-443c-8ec7-a4d04c64bf4b/ovn-acl-logging/0.log"
Jul 29 19:18:56.295169 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:18:56.295151    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjdrx_60f444e1-7e31-443c-8ec7-a4d04c64bf4b/ovn-acl-logging/0.log"
Jul 29 19:19:00.336520 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:19:00.336477    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:19:18.337252 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:19:18.337213    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:19:40.139785 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:19:40.139747    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:19:53.339052 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:19:53.339007    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:20:06.543288 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:20:06.543246    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:20:28.245759 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:20:28.245663    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:20:46.239559 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:20:46.239514    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:21:14.051450 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:21:14.051408    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:21:20.187612 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:21:20.187571    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-422d30/auth-test-model-422d30-kserve-6676f64755-s7xsl"]
Jul 29 19:21:30.546747 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:21:30.546698    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:21:35.207351 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:21:35.207309    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-422d30/auth-test-model-422d30-kserve-6676f64755-s7xsl" podUID="e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0" containerName="main" containerID="cri-o://2f305780b68944b4a76389342e6bef0ccf00c8f5cc5c361627042a91a9f9260b" gracePeriod=45
Jul 29 19:21:35.473751 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:21:35.473673    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-422d30/auth-test-model-422d30-kserve-6676f64755-s7xsl"
Jul 29 19:21:35.535268 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:21:35.535234    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0-model-cache\") pod \"e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0\" (UID: \"e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0\") "
Jul 29 19:21:35.535268 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:21:35.535271    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0-dshm\") pod \"e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0\" (UID: \"e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0\") "
Jul 29 19:21:35.535571 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:21:35.535356    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0-tls-certs\") pod \"e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0\" (UID: \"e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0\") "
Jul 29 19:21:35.535571 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:21:35.535431    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0-home\") pod \"e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0\" (UID: \"e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0\") "
Jul 29 19:21:35.535685 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:21:35.535585    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0-model-cache" (OuterVolumeSpecName: "model-cache") pod "e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0" (UID: "e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:21:35.535685 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:21:35.535594    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cswjx\" (UniqueName: \"kubernetes.io/projected/e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0-kube-api-access-cswjx\") pod \"e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0\" (UID: \"e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0\") "
Jul 29 19:21:35.535685 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:21:35.535645    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0-home" (OuterVolumeSpecName: "home") pod "e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0" (UID: "e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:21:35.535819 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:21:35.535693    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0-tmp-dir\") pod \"e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0\" (UID: \"e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0\") "
Jul 29 19:21:35.535906 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:21:35.535890    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0" (UID: "e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:21:35.536043 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:21:35.536030    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0-tmp-dir\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:21:35.536102 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:21:35.536048    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0-model-cache\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:21:35.536102 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:21:35.536056    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0-home\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:21:35.537879 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:21:35.537853    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0-kube-api-access-cswjx" (OuterVolumeSpecName: "kube-api-access-cswjx") pod "e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0" (UID: "e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0"). InnerVolumeSpecName "kube-api-access-cswjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:21:35.538109 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:21:35.538093    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0-dshm" (OuterVolumeSpecName: "dshm") pod "e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0" (UID: "e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:21:35.538171 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:21:35.538110    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0" (UID: "e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:21:35.637586 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:21:35.637550    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0-dshm\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:21:35.637586 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:21:35.637579    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0-tls-certs\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:21:35.637586 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:21:35.637591    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cswjx\" (UniqueName: \"kubernetes.io/projected/e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0-kube-api-access-cswjx\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:21:36.270563 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:21:36.270529    2559 generic.go:356] "Generic (PLEG): container finished" podID="e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0" containerID="2f305780b68944b4a76389342e6bef0ccf00c8f5cc5c361627042a91a9f9260b" exitCode=0
Jul 29 19:21:36.271067 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:21:36.270582    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-422d30/auth-test-model-422d30-kserve-6676f64755-s7xsl"
Jul 29 19:21:36.271067 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:21:36.270592    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-422d30/auth-test-model-422d30-kserve-6676f64755-s7xsl" event={"ID":"e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0","Type":"ContainerDied","Data":"2f305780b68944b4a76389342e6bef0ccf00c8f5cc5c361627042a91a9f9260b"}
Jul 29 19:21:36.271067 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:21:36.270626    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-422d30/auth-test-model-422d30-kserve-6676f64755-s7xsl" event={"ID":"e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0","Type":"ContainerDied","Data":"5bf21a22d1d2055541d0c59502b35faf958d68cf6371c5d7fe82771563f025b9"}
Jul 29 19:21:36.271067 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:21:36.270642    2559 scope.go:117] "RemoveContainer" containerID="2f305780b68944b4a76389342e6bef0ccf00c8f5cc5c361627042a91a9f9260b"
Jul 29 19:21:36.279759 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:21:36.279717    2559 scope.go:117] "RemoveContainer" containerID="2f305780b68944b4a76389342e6bef0ccf00c8f5cc5c361627042a91a9f9260b"
Jul 29 19:21:36.280050 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:21:36.280023    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f305780b68944b4a76389342e6bef0ccf00c8f5cc5c361627042a91a9f9260b\": container with ID starting with 2f305780b68944b4a76389342e6bef0ccf00c8f5cc5c361627042a91a9f9260b not found: ID does not exist" containerID="2f305780b68944b4a76389342e6bef0ccf00c8f5cc5c361627042a91a9f9260b"
Jul 29 19:21:36.280104 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:21:36.280051    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f305780b68944b4a76389342e6bef0ccf00c8f5cc5c361627042a91a9f9260b"} err="failed to get container status \"2f305780b68944b4a76389342e6bef0ccf00c8f5cc5c361627042a91a9f9260b\": rpc error: code = NotFound desc = could not find container \"2f305780b68944b4a76389342e6bef0ccf00c8f5cc5c361627042a91a9f9260b\": container with ID starting with 2f305780b68944b4a76389342e6bef0ccf00c8f5cc5c361627042a91a9f9260b not found: ID does not exist"
Jul 29 19:21:36.289690 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:21:36.289651    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-422d30/auth-test-model-422d30-kserve-6676f64755-s7xsl"]
Jul 29 19:21:36.295901 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:21:36.295866    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-422d30/auth-test-model-422d30-kserve-6676f64755-s7xsl"]
Jul 29 19:21:38.150488 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:21:38.150449    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0" path="/var/lib/kubelet/pods/e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0/volumes"
Jul 29 19:21:39.444878 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:21:39.444838    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:22:04.141631 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:22:04.141592    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:22:33.743360 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:22:33.743314    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:22:53.451674 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:22:53.451640    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:23:10.184525 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:23:10.184488    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-d04442/rate-test-model-d04442-kserve-54c8cd79ff-z85m9"]
Jul 29 19:23:10.185082 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:23:10.184885    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0" containerName="main"
Jul 29 19:23:10.185082 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:23:10.184902    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0" containerName="main"
Jul 29 19:23:10.185082 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:23:10.185021    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e29dfcbc-4cc2-4a12-98c1-ede92f00c7d0" containerName="main"
Jul 29 19:23:10.188151 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:23:10.188136    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d04442/rate-test-model-d04442-kserve-54c8cd79ff-z85m9"
Jul 29 19:23:10.193492 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:23:10.193468    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d04442\"/\"default-dockercfg-bbg88\""
Jul 29 19:23:10.193650 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:23:10.193475    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-d04442\"/\"kube-root-ca.crt\""
Jul 29 19:23:10.193823 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:23:10.193799    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d04442\"/\"rate-test-model-d04442-kserve-self-signed-certs\""
Jul 29 19:23:10.193993 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:23:10.193974    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-d04442\"/\"openshift-service-ca.crt\""
Jul 29 19:23:10.199417 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:23:10.199373    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d04442/rate-test-model-d04442-kserve-54c8cd79ff-z85m9"]
Jul 29 19:23:10.275907 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:23:10.275869    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6723fc15-8fb4-478c-82c5-73d669b59e48-tmp-dir\") pod \"rate-test-model-d04442-kserve-54c8cd79ff-z85m9\" (UID: \"6723fc15-8fb4-478c-82c5-73d669b59e48\") " pod="ai-tenant-e2e-shared-a-d04442/rate-test-model-d04442-kserve-54c8cd79ff-z85m9"
Jul 29 19:23:10.276094 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:23:10.275918    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6723fc15-8fb4-478c-82c5-73d669b59e48-home\") pod \"rate-test-model-d04442-kserve-54c8cd79ff-z85m9\" (UID: \"6723fc15-8fb4-478c-82c5-73d669b59e48\") " pod="ai-tenant-e2e-shared-a-d04442/rate-test-model-d04442-kserve-54c8cd79ff-z85m9"
Jul 29 19:23:10.276094 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:23:10.275941    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6723fc15-8fb4-478c-82c5-73d669b59e48-model-cache\") pod \"rate-test-model-d04442-kserve-54c8cd79ff-z85m9\" (UID: \"6723fc15-8fb4-478c-82c5-73d669b59e48\") " pod="ai-tenant-e2e-shared-a-d04442/rate-test-model-d04442-kserve-54c8cd79ff-z85m9"
Jul 29 19:23:10.276094 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:23:10.276030    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6723fc15-8fb4-478c-82c5-73d669b59e48-dshm\") pod \"rate-test-model-d04442-kserve-54c8cd79ff-z85m9\" (UID: \"6723fc15-8fb4-478c-82c5-73d669b59e48\") " pod="ai-tenant-e2e-shared-a-d04442/rate-test-model-d04442-kserve-54c8cd79ff-z85m9"
Jul 29 19:23:10.276094 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:23:10.276058    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6723fc15-8fb4-478c-82c5-73d669b59e48-tls-certs\") pod \"rate-test-model-d04442-kserve-54c8cd79ff-z85m9\" (UID: \"6723fc15-8fb4-478c-82c5-73d669b59e48\") " pod="ai-tenant-e2e-shared-a-d04442/rate-test-model-d04442-kserve-54c8cd79ff-z85m9"
Jul 29 19:23:10.276228 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:23:10.276100    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gfdq\" (UniqueName: \"kubernetes.io/projected/6723fc15-8fb4-478c-82c5-73d669b59e48-kube-api-access-7gfdq\") pod \"rate-test-model-d04442-kserve-54c8cd79ff-z85m9\" (UID: \"6723fc15-8fb4-478c-82c5-73d669b59e48\") " pod="ai-tenant-e2e-shared-a-d04442/rate-test-model-d04442-kserve-54c8cd79ff-z85m9"
Jul 29 19:23:10.377274 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:23:10.377231    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6723fc15-8fb4-478c-82c5-73d669b59e48-home\") pod \"rate-test-model-d04442-kserve-54c8cd79ff-z85m9\" (UID: \"6723fc15-8fb4-478c-82c5-73d669b59e48\") " pod="ai-tenant-e2e-shared-a-d04442/rate-test-model-d04442-kserve-54c8cd79ff-z85m9"
Jul 29 19:23:10.377274 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:23:10.377273    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6723fc15-8fb4-478c-82c5-73d669b59e48-model-cache\") pod \"rate-test-model-d04442-kserve-54c8cd79ff-z85m9\" (UID: \"6723fc15-8fb4-478c-82c5-73d669b59e48\") " pod="ai-tenant-e2e-shared-a-d04442/rate-test-model-d04442-kserve-54c8cd79ff-z85m9"
Jul 29 19:23:10.377582 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:23:10.377315    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6723fc15-8fb4-478c-82c5-73d669b59e48-dshm\") pod \"rate-test-model-d04442-kserve-54c8cd79ff-z85m9\" (UID: \"6723fc15-8fb4-478c-82c5-73d669b59e48\") " pod="ai-tenant-e2e-shared-a-d04442/rate-test-model-d04442-kserve-54c8cd79ff-z85m9"
Jul 29 19:23:10.377582 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:23:10.377335    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6723fc15-8fb4-478c-82c5-73d669b59e48-tls-certs\") pod \"rate-test-model-d04442-kserve-54c8cd79ff-z85m9\" (UID: \"6723fc15-8fb4-478c-82c5-73d669b59e48\") " pod="ai-tenant-e2e-shared-a-d04442/rate-test-model-d04442-kserve-54c8cd79ff-z85m9"
Jul 29 19:23:10.377582 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:23:10.377365    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gfdq\" (UniqueName: \"kubernetes.io/projected/6723fc15-8fb4-478c-82c5-73d669b59e48-kube-api-access-7gfdq\") pod \"rate-test-model-d04442-kserve-54c8cd79ff-z85m9\" (UID: \"6723fc15-8fb4-478c-82c5-73d669b59e48\") " pod="ai-tenant-e2e-shared-a-d04442/rate-test-model-d04442-kserve-54c8cd79ff-z85m9"
Jul 29 19:23:10.377582 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:23:10.377461    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6723fc15-8fb4-478c-82c5-73d669b59e48-tmp-dir\") pod \"rate-test-model-d04442-kserve-54c8cd79ff-z85m9\" (UID: \"6723fc15-8fb4-478c-82c5-73d669b59e48\") " pod="ai-tenant-e2e-shared-a-d04442/rate-test-model-d04442-kserve-54c8cd79ff-z85m9"
Jul 29 19:23:10.377803 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:23:10.377732    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6723fc15-8fb4-478c-82c5-73d669b59e48-home\") pod \"rate-test-model-d04442-kserve-54c8cd79ff-z85m9\" (UID: \"6723fc15-8fb4-478c-82c5-73d669b59e48\") " pod="ai-tenant-e2e-shared-a-d04442/rate-test-model-d04442-kserve-54c8cd79ff-z85m9"
Jul 29 19:23:10.377803 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:23:10.377767    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6723fc15-8fb4-478c-82c5-73d669b59e48-model-cache\") pod \"rate-test-model-d04442-kserve-54c8cd79ff-z85m9\" (UID: \"6723fc15-8fb4-478c-82c5-73d669b59e48\") " pod="ai-tenant-e2e-shared-a-d04442/rate-test-model-d04442-kserve-54c8cd79ff-z85m9"
Jul 29 19:23:10.377902 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:23:10.377884    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6723fc15-8fb4-478c-82c5-73d669b59e48-tmp-dir\") pod \"rate-test-model-d04442-kserve-54c8cd79ff-z85m9\" (UID: \"6723fc15-8fb4-478c-82c5-73d669b59e48\") " pod="ai-tenant-e2e-shared-a-d04442/rate-test-model-d04442-kserve-54c8cd79ff-z85m9"
Jul 29 19:23:10.379948 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:23:10.379918    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6723fc15-8fb4-478c-82c5-73d669b59e48-dshm\") pod \"rate-test-model-d04442-kserve-54c8cd79ff-z85m9\" (UID: \"6723fc15-8fb4-478c-82c5-73d669b59e48\") " pod="ai-tenant-e2e-shared-a-d04442/rate-test-model-d04442-kserve-54c8cd79ff-z85m9"
Jul 29 19:23:10.380178 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:23:10.380163    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6723fc15-8fb4-478c-82c5-73d669b59e48-tls-certs\") pod \"rate-test-model-d04442-kserve-54c8cd79ff-z85m9\" (UID: \"6723fc15-8fb4-478c-82c5-73d669b59e48\") " pod="ai-tenant-e2e-shared-a-d04442/rate-test-model-d04442-kserve-54c8cd79ff-z85m9"
Jul 29 19:23:10.386150 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:23:10.386122    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gfdq\" (UniqueName: \"kubernetes.io/projected/6723fc15-8fb4-478c-82c5-73d669b59e48-kube-api-access-7gfdq\") pod \"rate-test-model-d04442-kserve-54c8cd79ff-z85m9\" (UID: \"6723fc15-8fb4-478c-82c5-73d669b59e48\") " pod="ai-tenant-e2e-shared-a-d04442/rate-test-model-d04442-kserve-54c8cd79ff-z85m9"
Jul 29 19:23:10.498443 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:23:10.498349    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d04442/rate-test-model-d04442-kserve-54c8cd79ff-z85m9"
Jul 29 19:23:10.549033 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:23:10.548994    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:23:10.655039 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:23:10.655013    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d04442/rate-test-model-d04442-kserve-54c8cd79ff-z85m9"]
Jul 29 19:23:10.657285 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:23:10.657255    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6723fc15_8fb4_478c_82c5_73d669b59e48.slice/crio-1b26c40310395909ae4959dd6a48c421df59d6a4daab11220b9de9f13190fafd WatchSource:0}: Error finding container 1b26c40310395909ae4959dd6a48c421df59d6a4daab11220b9de9f13190fafd: Status 404 returned error can't find the container with id 1b26c40310395909ae4959dd6a48c421df59d6a4daab11220b9de9f13190fafd
Jul 29 19:23:10.746649 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:23:10.746612    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d04442/rate-test-model-d04442-kserve-54c8cd79ff-z85m9" event={"ID":"6723fc15-8fb4-478c-82c5-73d669b59e48","Type":"ContainerStarted","Data":"fc7e20e9ecb39e89d1cc4b4905ed4302b63ae5bdfc04bea1c911f31623816e3d"}
Jul 29 19:23:10.746831 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:23:10.746662    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d04442/rate-test-model-d04442-kserve-54c8cd79ff-z85m9" event={"ID":"6723fc15-8fb4-478c-82c5-73d669b59e48","Type":"ContainerStarted","Data":"1b26c40310395909ae4959dd6a48c421df59d6a4daab11220b9de9f13190fafd"}
Jul 29 19:23:10.766775 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:23:10.766658    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-d04442/rate-test-model-d04442-kserve-54c8cd79ff-z85m9" podStartSLOduration=0.766640703 podStartE2EDuration="766.640703ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:23:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:23:10.764654518 +0000 UTC m=+2955.034423607" watchObservedRunningTime="2026-07-29 19:23:10.766640703 +0000 UTC m=+2955.036409790"
Jul 29 19:23:19.645835 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:23:19.645745    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:23:20.499156 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:23:20.499111    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-d04442/rate-test-model-d04442-kserve-54c8cd79ff-z85m9"
Jul 29 19:23:20.499156 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:23:20.499168    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-d04442/rate-test-model-d04442-kserve-54c8cd79ff-z85m9"
Jul 29 19:23:20.511924 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:23:20.511894    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-d04442/rate-test-model-d04442-kserve-54c8cd79ff-z85m9"
Jul 29 19:23:20.812455 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:23:20.812362    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-d04442/rate-test-model-d04442-kserve-54c8cd79ff-z85m9"
Jul 29 19:23:37.737512 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:23:37.737464    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:23:56.312600 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:23:56.312571    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-v9nsr_7f8b4737-d4c2-4598-bd40-208015542831/console-operator/1.log"
Jul 29 19:23:56.319257 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:23:56.319226    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-v9nsr_7f8b4737-d4c2-4598-bd40-208015542831/console-operator/1.log"
Jul 29 19:23:56.320579 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:23:56.320558    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjdrx_60f444e1-7e31-443c-8ec7-a4d04c64bf4b/ovn-acl-logging/0.log"
Jul 29 19:23:56.326926 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:23:56.326903    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjdrx_60f444e1-7e31-443c-8ec7-a4d04c64bf4b/ovn-acl-logging/0.log"
Jul 29 19:24:07.745056 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:24:07.745016    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:24:22.249948 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:24:22.249912    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:24:36.843334 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:24:36.843294    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:24:40.785239 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:24:40.785202    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d04442/rate-test-model-d04442-kserve-54c8cd79ff-z85m9"]
Jul 29 19:24:52.875072 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:24:52.875027    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:24:55.806129 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:24:55.806088    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-d04442/rate-test-model-d04442-kserve-54c8cd79ff-z85m9" podUID="6723fc15-8fb4-478c-82c5-73d669b59e48" containerName="main" containerID="cri-o://fc7e20e9ecb39e89d1cc4b4905ed4302b63ae5bdfc04bea1c911f31623816e3d" gracePeriod=45
Jul 29 19:24:56.065946 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:24:56.065873    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d04442/rate-test-model-d04442-kserve-54c8cd79ff-z85m9"
Jul 29 19:24:56.143451 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:24:56.143410    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6723fc15-8fb4-478c-82c5-73d669b59e48-tls-certs\") pod \"6723fc15-8fb4-478c-82c5-73d669b59e48\" (UID: \"6723fc15-8fb4-478c-82c5-73d669b59e48\") "
Jul 29 19:24:56.143451 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:24:56.143457    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6723fc15-8fb4-478c-82c5-73d669b59e48-dshm\") pod \"6723fc15-8fb4-478c-82c5-73d669b59e48\" (UID: \"6723fc15-8fb4-478c-82c5-73d669b59e48\") "
Jul 29 19:24:56.143718 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:24:56.143483    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6723fc15-8fb4-478c-82c5-73d669b59e48-tmp-dir\") pod \"6723fc15-8fb4-478c-82c5-73d669b59e48\" (UID: \"6723fc15-8fb4-478c-82c5-73d669b59e48\") "
Jul 29 19:24:56.143718 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:24:56.143517    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6723fc15-8fb4-478c-82c5-73d669b59e48-home\") pod \"6723fc15-8fb4-478c-82c5-73d669b59e48\" (UID: \"6723fc15-8fb4-478c-82c5-73d669b59e48\") "
Jul 29 19:24:56.143718 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:24:56.143538    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6723fc15-8fb4-478c-82c5-73d669b59e48-model-cache\") pod \"6723fc15-8fb4-478c-82c5-73d669b59e48\" (UID: \"6723fc15-8fb4-478c-82c5-73d669b59e48\") "
Jul 29 19:24:56.143718 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:24:56.143573    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gfdq\" (UniqueName: \"kubernetes.io/projected/6723fc15-8fb4-478c-82c5-73d669b59e48-kube-api-access-7gfdq\") pod \"6723fc15-8fb4-478c-82c5-73d669b59e48\" (UID: \"6723fc15-8fb4-478c-82c5-73d669b59e48\") "
Jul 29 19:24:56.143924 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:24:56.143847    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6723fc15-8fb4-478c-82c5-73d669b59e48-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6723fc15-8fb4-478c-82c5-73d669b59e48" (UID: "6723fc15-8fb4-478c-82c5-73d669b59e48"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:24:56.143924 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:24:56.143866    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6723fc15-8fb4-478c-82c5-73d669b59e48-home" (OuterVolumeSpecName: "home") pod "6723fc15-8fb4-478c-82c5-73d669b59e48" (UID: "6723fc15-8fb4-478c-82c5-73d669b59e48"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:24:56.143924 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:24:56.143902    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6723fc15-8fb4-478c-82c5-73d669b59e48-model-cache" (OuterVolumeSpecName: "model-cache") pod "6723fc15-8fb4-478c-82c5-73d669b59e48" (UID: "6723fc15-8fb4-478c-82c5-73d669b59e48"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:24:56.146053 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:24:56.146021    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6723fc15-8fb4-478c-82c5-73d669b59e48-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6723fc15-8fb4-478c-82c5-73d669b59e48" (UID: "6723fc15-8fb4-478c-82c5-73d669b59e48"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:24:56.146053 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:24:56.146034    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6723fc15-8fb4-478c-82c5-73d669b59e48-dshm" (OuterVolumeSpecName: "dshm") pod "6723fc15-8fb4-478c-82c5-73d669b59e48" (UID: "6723fc15-8fb4-478c-82c5-73d669b59e48"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:24:56.146230 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:24:56.146051    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6723fc15-8fb4-478c-82c5-73d669b59e48-kube-api-access-7gfdq" (OuterVolumeSpecName: "kube-api-access-7gfdq") pod "6723fc15-8fb4-478c-82c5-73d669b59e48" (UID: "6723fc15-8fb4-478c-82c5-73d669b59e48"). InnerVolumeSpecName "kube-api-access-7gfdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:24:56.244772 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:24:56.244734    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6723fc15-8fb4-478c-82c5-73d669b59e48-tmp-dir\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:24:56.244772 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:24:56.244767    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6723fc15-8fb4-478c-82c5-73d669b59e48-home\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:24:56.244772 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:24:56.244776    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6723fc15-8fb4-478c-82c5-73d669b59e48-model-cache\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:24:56.244772 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:24:56.244785    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gfdq\" (UniqueName: \"kubernetes.io/projected/6723fc15-8fb4-478c-82c5-73d669b59e48-kube-api-access-7gfdq\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:24:56.245084 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:24:56.244792    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6723fc15-8fb4-478c-82c5-73d669b59e48-tls-certs\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:24:56.245084 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:24:56.244801    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6723fc15-8fb4-478c-82c5-73d669b59e48-dshm\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:24:56.260315 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:24:56.260278    2559 generic.go:356] "Generic (PLEG): container finished" podID="6723fc15-8fb4-478c-82c5-73d669b59e48" containerID="fc7e20e9ecb39e89d1cc4b4905ed4302b63ae5bdfc04bea1c911f31623816e3d" exitCode=0
Jul 29 19:24:56.260538 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:24:56.260332    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d04442/rate-test-model-d04442-kserve-54c8cd79ff-z85m9"
Jul 29 19:24:56.260538 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:24:56.260331    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d04442/rate-test-model-d04442-kserve-54c8cd79ff-z85m9" event={"ID":"6723fc15-8fb4-478c-82c5-73d669b59e48","Type":"ContainerDied","Data":"fc7e20e9ecb39e89d1cc4b4905ed4302b63ae5bdfc04bea1c911f31623816e3d"}
Jul 29 19:24:56.260538 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:24:56.260461    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d04442/rate-test-model-d04442-kserve-54c8cd79ff-z85m9" event={"ID":"6723fc15-8fb4-478c-82c5-73d669b59e48","Type":"ContainerDied","Data":"1b26c40310395909ae4959dd6a48c421df59d6a4daab11220b9de9f13190fafd"}
Jul 29 19:24:56.260538 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:24:56.260494    2559 scope.go:117] "RemoveContainer" containerID="fc7e20e9ecb39e89d1cc4b4905ed4302b63ae5bdfc04bea1c911f31623816e3d"
Jul 29 19:24:56.270063 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:24:56.270041    2559 scope.go:117] "RemoveContainer" containerID="fc7e20e9ecb39e89d1cc4b4905ed4302b63ae5bdfc04bea1c911f31623816e3d"
Jul 29 19:24:56.270430 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:24:56.270411    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc7e20e9ecb39e89d1cc4b4905ed4302b63ae5bdfc04bea1c911f31623816e3d\": container with ID starting with fc7e20e9ecb39e89d1cc4b4905ed4302b63ae5bdfc04bea1c911f31623816e3d not found: ID does not exist" containerID="fc7e20e9ecb39e89d1cc4b4905ed4302b63ae5bdfc04bea1c911f31623816e3d"
Jul 29 19:24:56.270478 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:24:56.270441    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc7e20e9ecb39e89d1cc4b4905ed4302b63ae5bdfc04bea1c911f31623816e3d"} err="failed to get container status \"fc7e20e9ecb39e89d1cc4b4905ed4302b63ae5bdfc04bea1c911f31623816e3d\": rpc error: code = NotFound desc = could not find container \"fc7e20e9ecb39e89d1cc4b4905ed4302b63ae5bdfc04bea1c911f31623816e3d\": container with ID starting with fc7e20e9ecb39e89d1cc4b4905ed4302b63ae5bdfc04bea1c911f31623816e3d not found: ID does not exist"
Jul 29 19:24:56.279674 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:24:56.279646    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d04442/rate-test-model-d04442-kserve-54c8cd79ff-z85m9"]
Jul 29 19:24:56.285517 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:24:56.285469    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-d04442/rate-test-model-d04442-kserve-54c8cd79ff-z85m9"]
Jul 29 19:24:58.150634 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:24:58.150600    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6723fc15-8fb4-478c-82c5-73d669b59e48" path="/var/lib/kubelet/pods/6723fc15-8fb4-478c-82c5-73d669b59e48/volumes"
Jul 29 19:25:04.477399 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:04.477326    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf"]
Jul 29 19:25:04.478002 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:04.477979    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6723fc15-8fb4-478c-82c5-73d669b59e48" containerName="main"
Jul 29 19:25:04.478117 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:04.478004    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6723fc15-8fb4-478c-82c5-73d669b59e48" containerName="main"
Jul 29 19:25:04.478171 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:04.478129    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6723fc15-8fb4-478c-82c5-73d669b59e48" containerName="main"
Jul 29 19:25:04.483036 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:04.483011    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf"
Jul 29 19:25:04.486205 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:04.486175    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-cc3e80-serving-cert\""
Jul 29 19:25:04.498648 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:04.498612    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf"]
Jul 29 19:25:04.621239 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:04.621140    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2f550c1-979d-40ed-9dcc-627f2281ecb4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf\" (UID: \"a2f550c1-979d-40ed-9dcc-627f2281ecb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf"
Jul 29 19:25:04.621239 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:04.621226    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2f550c1-979d-40ed-9dcc-627f2281ecb4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf\" (UID: \"a2f550c1-979d-40ed-9dcc-627f2281ecb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf"
Jul 29 19:25:04.621513 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:04.621259    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2f550c1-979d-40ed-9dcc-627f2281ecb4-metrics-certs\") pod \"maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf\" (UID: \"a2f550c1-979d-40ed-9dcc-627f2281ecb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf"
Jul 29 19:25:04.621513 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:04.621283    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrbsr\" (UniqueName: \"kubernetes.io/projected/a2f550c1-979d-40ed-9dcc-627f2281ecb4-kube-api-access-nrbsr\") pod \"maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf\" (UID: \"a2f550c1-979d-40ed-9dcc-627f2281ecb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf"
Jul 29 19:25:04.722177 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:04.722125    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2f550c1-979d-40ed-9dcc-627f2281ecb4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf\" (UID: \"a2f550c1-979d-40ed-9dcc-627f2281ecb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf"
Jul 29 19:25:04.722428 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:04.722240    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2f550c1-979d-40ed-9dcc-627f2281ecb4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf\" (UID: \"a2f550c1-979d-40ed-9dcc-627f2281ecb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf"
Jul 29 19:25:04.722428 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:04.722288    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2f550c1-979d-40ed-9dcc-627f2281ecb4-metrics-certs\") pod \"maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf\" (UID: \"a2f550c1-979d-40ed-9dcc-627f2281ecb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf"
Jul 29 19:25:04.722428 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:04.722312    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrbsr\" (UniqueName: \"kubernetes.io/projected/a2f550c1-979d-40ed-9dcc-627f2281ecb4-kube-api-access-nrbsr\") pod \"maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf\" (UID: \"a2f550c1-979d-40ed-9dcc-627f2281ecb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf"
Jul 29 19:25:04.722697 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:25:04.722442    2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc3e80-serving-cert: secret "maas-api-e2e-ipp-a-cc3e80-serving-cert" not found
Jul 29 19:25:04.722697 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:25:04.722544    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2f550c1-979d-40ed-9dcc-627f2281ecb4-maas-api-tls podName:a2f550c1-979d-40ed-9dcc-627f2281ecb4 nodeName:}" failed. No retries permitted until 2026-07-29 19:25:05.222520488 +0000 UTC m=+3069.492289567 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a2f550c1-979d-40ed-9dcc-627f2281ecb4-maas-api-tls") pod "maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf" (UID: "a2f550c1-979d-40ed-9dcc-627f2281ecb4") : secret "maas-api-e2e-ipp-a-cc3e80-serving-cert" not found
Jul 29 19:25:04.723012 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:04.722989    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2f550c1-979d-40ed-9dcc-627f2281ecb4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf\" (UID: \"a2f550c1-979d-40ed-9dcc-627f2281ecb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf"
Jul 29 19:25:04.724979 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:04.724933    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2f550c1-979d-40ed-9dcc-627f2281ecb4-metrics-certs\") pod \"maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf\" (UID: \"a2f550c1-979d-40ed-9dcc-627f2281ecb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf"
Jul 29 19:25:04.734633 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:04.734571    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrbsr\" (UniqueName: \"kubernetes.io/projected/a2f550c1-979d-40ed-9dcc-627f2281ecb4-kube-api-access-nrbsr\") pod \"maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf\" (UID: \"a2f550c1-979d-40ed-9dcc-627f2281ecb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf"
Jul 29 19:25:05.226748 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:05.226708    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2f550c1-979d-40ed-9dcc-627f2281ecb4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf\" (UID: \"a2f550c1-979d-40ed-9dcc-627f2281ecb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf"
Jul 29 19:25:05.229529 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:05.229497    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2f550c1-979d-40ed-9dcc-627f2281ecb4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf\" (UID: \"a2f550c1-979d-40ed-9dcc-627f2281ecb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf"
Jul 29 19:25:05.281515 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:05.281468    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cc3e80-b65b446cd-qprgp"]
Jul 29 19:25:05.285214 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:05.285183    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-cc3e80-b65b446cd-qprgp"
Jul 29 19:25:05.287631 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:05.287606    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-cc3e80\""
Jul 29 19:25:05.287800 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:05.287673    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-cc3e80-dockercfg-ckhql\""
Jul 29 19:25:05.299619 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:05.299581    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cc3e80-b65b446cd-qprgp"]
Jul 29 19:25:05.342783 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:05.342746    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cc3e80-77ff6bccb-h24qx"]
Jul 29 19:25:05.346500 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:05.346479    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cc3e80-77ff6bccb-h24qx"
Jul 29 19:25:05.358111 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:05.358078    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cc3e80-77ff6bccb-h24qx"]
Jul 29 19:25:05.398812 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:05.398760    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf"
Jul 29 19:25:05.428312 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:05.428277    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6988a18-fdfb-484a-ba02-290a8fdc4e1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-cc3e80-77ff6bccb-h24qx\" (UID: \"f6988a18-fdfb-484a-ba02-290a8fdc4e1a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cc3e80-77ff6bccb-h24qx"
Jul 29 19:25:05.428527 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:05.428331    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjf42\" (UniqueName: \"kubernetes.io/projected/f6988a18-fdfb-484a-ba02-290a8fdc4e1a-kube-api-access-bjf42\") pod \"payload-pre-processing-e2e-ipp-a-cc3e80-77ff6bccb-h24qx\" (UID: \"f6988a18-fdfb-484a-ba02-290a8fdc4e1a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cc3e80-77ff6bccb-h24qx"
Jul 29 19:25:05.428527 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:05.428359    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rssck\" (UniqueName: \"kubernetes.io/projected/5a2b03a7-d196-4391-86b3-5ba0ce84031f-kube-api-access-rssck\") pod \"payload-processing-e2e-ipp-a-cc3e80-b65b446cd-qprgp\" (UID: \"5a2b03a7-d196-4391-86b3-5ba0ce84031f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cc3e80-b65b446cd-qprgp"
Jul 29 19:25:05.428527 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:05.428482    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a2b03a7-d196-4391-86b3-5ba0ce84031f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-cc3e80-b65b446cd-qprgp\" (UID: \"5a2b03a7-d196-4391-86b3-5ba0ce84031f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cc3e80-b65b446cd-qprgp"
Jul 29 19:25:05.529242 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:05.529206    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a2b03a7-d196-4391-86b3-5ba0ce84031f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-cc3e80-b65b446cd-qprgp\" (UID: \"5a2b03a7-d196-4391-86b3-5ba0ce84031f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cc3e80-b65b446cd-qprgp"
Jul 29 19:25:05.529740 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:05.529271    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6988a18-fdfb-484a-ba02-290a8fdc4e1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-cc3e80-77ff6bccb-h24qx\" (UID: \"f6988a18-fdfb-484a-ba02-290a8fdc4e1a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cc3e80-77ff6bccb-h24qx"
Jul 29 19:25:05.529740 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:05.529317    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjf42\" (UniqueName: \"kubernetes.io/projected/f6988a18-fdfb-484a-ba02-290a8fdc4e1a-kube-api-access-bjf42\") pod \"payload-pre-processing-e2e-ipp-a-cc3e80-77ff6bccb-h24qx\" (UID: \"f6988a18-fdfb-484a-ba02-290a8fdc4e1a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cc3e80-77ff6bccb-h24qx"
Jul 29 19:25:05.529740 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:05.529345    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rssck\" (UniqueName: \"kubernetes.io/projected/5a2b03a7-d196-4391-86b3-5ba0ce84031f-kube-api-access-rssck\") pod \"payload-processing-e2e-ipp-a-cc3e80-b65b446cd-qprgp\" (UID: \"5a2b03a7-d196-4391-86b3-5ba0ce84031f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cc3e80-b65b446cd-qprgp"
Jul 29 19:25:05.530076 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:05.530041    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6988a18-fdfb-484a-ba02-290a8fdc4e1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-cc3e80-77ff6bccb-h24qx\" (UID: \"f6988a18-fdfb-484a-ba02-290a8fdc4e1a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cc3e80-77ff6bccb-h24qx"
Jul 29 19:25:05.530310 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:05.530285    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a2b03a7-d196-4391-86b3-5ba0ce84031f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-cc3e80-b65b446cd-qprgp\" (UID: \"5a2b03a7-d196-4391-86b3-5ba0ce84031f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cc3e80-b65b446cd-qprgp"
Jul 29 19:25:05.541798 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:05.541757    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rssck\" (UniqueName: \"kubernetes.io/projected/5a2b03a7-d196-4391-86b3-5ba0ce84031f-kube-api-access-rssck\") pod \"payload-processing-e2e-ipp-a-cc3e80-b65b446cd-qprgp\" (UID: \"5a2b03a7-d196-4391-86b3-5ba0ce84031f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cc3e80-b65b446cd-qprgp"
Jul 29 19:25:05.542128 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:05.542104    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjf42\" (UniqueName: \"kubernetes.io/projected/f6988a18-fdfb-484a-ba02-290a8fdc4e1a-kube-api-access-bjf42\") pod \"payload-pre-processing-e2e-ipp-a-cc3e80-77ff6bccb-h24qx\" (UID: \"f6988a18-fdfb-484a-ba02-290a8fdc4e1a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cc3e80-77ff6bccb-h24qx"
Jul 29 19:25:05.562309 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:05.562279    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf"]
Jul 29 19:25:05.564677 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:25:05.564651    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2f550c1_979d_40ed_9dcc_627f2281ecb4.slice/crio-3968de5cb54b3f80c7a35b151fcbd1755b167a473dc9a517d0a9fad6d61e0fac WatchSource:0}: Error finding container 3968de5cb54b3f80c7a35b151fcbd1755b167a473dc9a517d0a9fad6d61e0fac: Status 404 returned error can't find the container with id 3968de5cb54b3f80c7a35b151fcbd1755b167a473dc9a517d0a9fad6d61e0fac
Jul 29 19:25:05.566092 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:05.566062    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 19:25:05.597128 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:05.597085    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-cc3e80-b65b446cd-qprgp"
Jul 29 19:25:05.658169 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:05.658131    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cc3e80-77ff6bccb-h24qx"
Jul 29 19:25:05.766844 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:05.766813    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cc3e80-b65b446cd-qprgp"]
Jul 29 19:25:05.769770 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:25:05.769736    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a2b03a7_d196_4391_86b3_5ba0ce84031f.slice/crio-dc41b3c4fe82c8a74fac2b095009000fecfdb9b01749532d80bb306bd128cccf WatchSource:0}: Error finding container dc41b3c4fe82c8a74fac2b095009000fecfdb9b01749532d80bb306bd128cccf: Status 404 returned error can't find the container with id dc41b3c4fe82c8a74fac2b095009000fecfdb9b01749532d80bb306bd128cccf
Jul 29 19:25:05.832566 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:05.832524    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cc3e80-77ff6bccb-h24qx"]
Jul 29 19:25:05.834820 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:25:05.834785    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6988a18_fdfb_484a_ba02_290a8fdc4e1a.slice/crio-9807ad3af0f7bed9153d7720efae0e78a38f78cc39f6942117f59df97caf8a4b WatchSource:0}: Error finding container 9807ad3af0f7bed9153d7720efae0e78a38f78cc39f6942117f59df97caf8a4b: Status 404 returned error can't find the container with id 9807ad3af0f7bed9153d7720efae0e78a38f78cc39f6942117f59df97caf8a4b
Jul 29 19:25:06.312903 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:06.312866    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-cc3e80-b65b446cd-qprgp" event={"ID":"5a2b03a7-d196-4391-86b3-5ba0ce84031f","Type":"ContainerStarted","Data":"03dcb089975a0471dc7546b7c6bca8bc0daa97c018d04356a16221f317e7ca99"}
Jul 29 19:25:06.313166 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:06.313052    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-cc3e80-b65b446cd-qprgp"
Jul 29 19:25:06.313166 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:06.313092    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-cc3e80-b65b446cd-qprgp" event={"ID":"5a2b03a7-d196-4391-86b3-5ba0ce84031f","Type":"ContainerStarted","Data":"dc41b3c4fe82c8a74fac2b095009000fecfdb9b01749532d80bb306bd128cccf"}
Jul 29 19:25:06.314681 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:06.314611    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cc3e80-77ff6bccb-h24qx" event={"ID":"f6988a18-fdfb-484a-ba02-290a8fdc4e1a","Type":"ContainerStarted","Data":"529b94190c4150fe0b83814b4225f4c9ec9b12a8a89405438ce5d09f85916bd5"}
Jul 29 19:25:06.314681 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:06.314676    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cc3e80-77ff6bccb-h24qx"
Jul 29 19:25:06.314900 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:06.314703    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cc3e80-77ff6bccb-h24qx" event={"ID":"f6988a18-fdfb-484a-ba02-290a8fdc4e1a","Type":"ContainerStarted","Data":"9807ad3af0f7bed9153d7720efae0e78a38f78cc39f6942117f59df97caf8a4b"}
Jul 29 19:25:06.315949 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:06.315922    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf" event={"ID":"a2f550c1-979d-40ed-9dcc-627f2281ecb4","Type":"ContainerStarted","Data":"3968de5cb54b3f80c7a35b151fcbd1755b167a473dc9a517d0a9fad6d61e0fac"}
Jul 29 19:25:06.332771 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:06.332718    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-cc3e80-b65b446cd-qprgp" podStartSLOduration=1.332700382 podStartE2EDuration="1.332700382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:25:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:25:06.331858038 +0000 UTC m=+3070.601627116" watchObservedRunningTime="2026-07-29 19:25:06.332700382 +0000 UTC m=+3070.602469469"
Jul 29 19:25:06.351047 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:06.350992    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cc3e80-77ff6bccb-h24qx" podStartSLOduration=1.350972546 podStartE2EDuration="1.350972546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:25:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:25:06.348701834 +0000 UTC m=+3070.618470922" watchObservedRunningTime="2026-07-29 19:25:06.350972546 +0000 UTC m=+3070.620741807"
Jul 29 19:25:07.323208 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:07.323173    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf" event={"ID":"a2f550c1-979d-40ed-9dcc-627f2281ecb4","Type":"ContainerStarted","Data":"79dd58facb0fe5757b650b766f4c76194dbbb3470b079aedf0401733ead0e187"}
Jul 29 19:25:07.323744 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:07.323272    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf"
Jul 29 19:25:07.342107 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:07.342025    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf" podStartSLOduration=2.396963742 podStartE2EDuration="3.342002387s" totalImagesPullingTime="945.038645ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:25:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:25:07.341718264 +0000 UTC m=+3071.611487351" watchObservedRunningTime="2026-07-29 19:25:07.342002387 +0000 UTC m=+3071.611771478"
Jul 29 19:25:12.686928 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:12.686882    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:25:13.333695 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:13.333665    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf"
Jul 29 19:25:17.325208 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:17.325161    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cc3e80-77ff6bccb-h24qx" podUID="f6988a18-fdfb-484a-ba02-290a8fdc4e1a" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.78:9004: connect: connection refused"
Jul 29 19:25:17.325693 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:17.325610    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-cc3e80-b65b446cd-qprgp"
Jul 29 19:25:27.323931 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:27.323892    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cc3e80-77ff6bccb-h24qx"
Jul 29 19:25:36.843883 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:36.843837    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws"]
Jul 29 19:25:36.847463 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:36.847443    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws"
Jul 29 19:25:36.850728 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:36.850699    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-f0bee1-serving-cert\""
Jul 29 19:25:36.860633 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:36.860602    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws"]
Jul 29 19:25:36.925907 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:36.925861    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa40169f-e600-4800-a452-afefaeabff92-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws\" (UID: \"aa40169f-e600-4800-a452-afefaeabff92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws"
Jul 29 19:25:36.926121 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:36.925956    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa40169f-e600-4800-a452-afefaeabff92-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws\" (UID: \"aa40169f-e600-4800-a452-afefaeabff92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws"
Jul 29 19:25:36.926121 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:36.925981    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n7qt\" (UniqueName: \"kubernetes.io/projected/aa40169f-e600-4800-a452-afefaeabff92-kube-api-access-9n7qt\") pod \"maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws\" (UID: \"aa40169f-e600-4800-a452-afefaeabff92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws"
Jul 29 19:25:36.926121 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:36.925998    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa40169f-e600-4800-a452-afefaeabff92-metrics-certs\") pod \"maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws\" (UID: \"aa40169f-e600-4800-a452-afefaeabff92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws"
Jul 29 19:25:37.026945 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:37.026895    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa40169f-e600-4800-a452-afefaeabff92-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws\" (UID: \"aa40169f-e600-4800-a452-afefaeabff92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws"
Jul 29 19:25:37.026945 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:37.026944    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n7qt\" (UniqueName: \"kubernetes.io/projected/aa40169f-e600-4800-a452-afefaeabff92-kube-api-access-9n7qt\") pod \"maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws\" (UID: \"aa40169f-e600-4800-a452-afefaeabff92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws"
Jul 29 19:25:37.027237 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:37.026965    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa40169f-e600-4800-a452-afefaeabff92-metrics-certs\") pod \"maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws\" (UID: \"aa40169f-e600-4800-a452-afefaeabff92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws"
Jul 29 19:25:37.027237 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:25:37.027087    2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0bee1-serving-cert: secret "maas-api-e2e-ipp-b-f0bee1-serving-cert" not found
Jul 29 19:25:37.027237 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:37.027101    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa40169f-e600-4800-a452-afefaeabff92-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws\" (UID: \"aa40169f-e600-4800-a452-afefaeabff92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws"
Jul 29 19:25:37.027237 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:25:37.027173    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa40169f-e600-4800-a452-afefaeabff92-maas-api-tls podName:aa40169f-e600-4800-a452-afefaeabff92 nodeName:}" failed. No retries permitted until 2026-07-29 19:25:37.52715111 +0000 UTC m=+3101.796920181 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aa40169f-e600-4800-a452-afefaeabff92-maas-api-tls") pod "maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws" (UID: "aa40169f-e600-4800-a452-afefaeabff92") : secret "maas-api-e2e-ipp-b-f0bee1-serving-cert" not found
Jul 29 19:25:37.027823 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:37.027750    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa40169f-e600-4800-a452-afefaeabff92-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws\" (UID: \"aa40169f-e600-4800-a452-afefaeabff92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws"
Jul 29 19:25:37.029812 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:37.029786    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa40169f-e600-4800-a452-afefaeabff92-metrics-certs\") pod \"maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws\" (UID: \"aa40169f-e600-4800-a452-afefaeabff92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws"
Jul 29 19:25:37.038497 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:37.038474    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n7qt\" (UniqueName: \"kubernetes.io/projected/aa40169f-e600-4800-a452-afefaeabff92-kube-api-access-9n7qt\") pod \"maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws\" (UID: \"aa40169f-e600-4800-a452-afefaeabff92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws"
Jul 29 19:25:37.531424 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:37.531357    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa40169f-e600-4800-a452-afefaeabff92-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws\" (UID: \"aa40169f-e600-4800-a452-afefaeabff92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws"
Jul 29 19:25:37.534303 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:37.534267    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa40169f-e600-4800-a452-afefaeabff92-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws\" (UID: \"aa40169f-e600-4800-a452-afefaeabff92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws"
Jul 29 19:25:37.547030 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:37.546995    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f0bee1-6fbbdf9548-bm4zq"]
Jul 29 19:25:37.550898 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:37.550875    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0bee1-6fbbdf9548-bm4zq"
Jul 29 19:25:37.553743 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:37.553713    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f0bee1\""
Jul 29 19:25:37.561221 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:37.561193    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f0bee1-6fbbdf9548-bm4zq"]
Jul 29 19:25:37.632787 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:37.632748    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvl44\" (UniqueName: \"kubernetes.io/projected/ffc38b90-3e01-4eaa-a612-9aeb014ab1b5-kube-api-access-xvl44\") pod \"payload-pre-processing-e2e-ipp-b-f0bee1-6fbbdf9548-bm4zq\" (UID: \"ffc38b90-3e01-4eaa-a612-9aeb014ab1b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0bee1-6fbbdf9548-bm4zq"
Jul 29 19:25:37.633003 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:37.632834    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffc38b90-3e01-4eaa-a612-9aeb014ab1b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f0bee1-6fbbdf9548-bm4zq\" (UID: \"ffc38b90-3e01-4eaa-a612-9aeb014ab1b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0bee1-6fbbdf9548-bm4zq"
Jul 29 19:25:37.733963 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:37.733912    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvl44\" (UniqueName: \"kubernetes.io/projected/ffc38b90-3e01-4eaa-a612-9aeb014ab1b5-kube-api-access-xvl44\") pod \"payload-pre-processing-e2e-ipp-b-f0bee1-6fbbdf9548-bm4zq\" (UID: \"ffc38b90-3e01-4eaa-a612-9aeb014ab1b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0bee1-6fbbdf9548-bm4zq"
Jul 29 19:25:37.734182 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:37.734015    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffc38b90-3e01-4eaa-a612-9aeb014ab1b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f0bee1-6fbbdf9548-bm4zq\" (UID: \"ffc38b90-3e01-4eaa-a612-9aeb014ab1b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0bee1-6fbbdf9548-bm4zq"
Jul 29 19:25:37.734763 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:37.734743    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffc38b90-3e01-4eaa-a612-9aeb014ab1b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f0bee1-6fbbdf9548-bm4zq\" (UID: \"ffc38b90-3e01-4eaa-a612-9aeb014ab1b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0bee1-6fbbdf9548-bm4zq"
Jul 29 19:25:37.743590 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:37.743556    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvl44\" (UniqueName: \"kubernetes.io/projected/ffc38b90-3e01-4eaa-a612-9aeb014ab1b5-kube-api-access-xvl44\") pod \"payload-pre-processing-e2e-ipp-b-f0bee1-6fbbdf9548-bm4zq\" (UID: \"ffc38b90-3e01-4eaa-a612-9aeb014ab1b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0bee1-6fbbdf9548-bm4zq"
Jul 29 19:25:37.760556 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:37.760521    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws"
Jul 29 19:25:37.863735 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:37.863704    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0bee1-6fbbdf9548-bm4zq"
Jul 29 19:25:37.915798 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:37.915769    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws"]
Jul 29 19:25:38.021706 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:38.021672    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f0bee1-6fbbdf9548-bm4zq"]
Jul 29 19:25:38.023506 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:25:38.023483    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffc38b90_3e01_4eaa_a612_9aeb014ab1b5.slice/crio-f86ac99a78f2144622470e6d71e4aab357c3c6aabad15cdeef37b9086c5272cd WatchSource:0}: Error finding container f86ac99a78f2144622470e6d71e4aab357c3c6aabad15cdeef37b9086c5272cd: Status 404 returned error can't find the container with id f86ac99a78f2144622470e6d71e4aab357c3c6aabad15cdeef37b9086c5272cd
Jul 29 19:25:38.495591 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:38.495552    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0bee1-6fbbdf9548-bm4zq" event={"ID":"ffc38b90-3e01-4eaa-a612-9aeb014ab1b5","Type":"ContainerStarted","Data":"c60fde680391cc7bfb839aed823a49a53a07c749152159882c4eb33c567928d0"}
Jul 29 19:25:38.495793 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:38.495695    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0bee1-6fbbdf9548-bm4zq"
Jul 29 19:25:38.495793 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:38.495754    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0bee1-6fbbdf9548-bm4zq" event={"ID":"ffc38b90-3e01-4eaa-a612-9aeb014ab1b5","Type":"ContainerStarted","Data":"f86ac99a78f2144622470e6d71e4aab357c3c6aabad15cdeef37b9086c5272cd"}
Jul 29 19:25:38.497012 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:38.496987    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws" event={"ID":"aa40169f-e600-4800-a452-afefaeabff92","Type":"ContainerStarted","Data":"d55ce1817a3259fb11320d095afa6f85b68b49a6e7d3b92375b703ccabe6778c"}
Jul 29 19:25:38.497140 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:38.497031    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws" event={"ID":"aa40169f-e600-4800-a452-afefaeabff92","Type":"ContainerStarted","Data":"afb67ac5d01879a6b05fa8285e0b86b58d02a01330d9abda7e3e4a26559862b8"}
Jul 29 19:25:38.497140 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:38.497058    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws"
Jul 29 19:25:38.514093 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:38.514032    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0bee1-6fbbdf9548-bm4zq" podStartSLOduration=1.5140156120000001 podStartE2EDuration="1.514015612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:25:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:25:38.512758519 +0000 UTC m=+3102.782527606" watchObservedRunningTime="2026-07-29 19:25:38.514015612 +0000 UTC m=+3102.783784700"
Jul 29 19:25:38.535072 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:38.535013    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws" podStartSLOduration=2.189709734 podStartE2EDuration="2.534995279s" totalImagesPullingTime="345.285545ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:25:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:25:38.532531525 +0000 UTC m=+3102.802300607" watchObservedRunningTime="2026-07-29 19:25:38.534995279 +0000 UTC m=+3102.804764385"
Jul 29 19:25:42.639653 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:42.639614    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:25:44.508354 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:44.508327    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws"
Jul 29 19:25:49.504739 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:25:49.504711    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0bee1-6fbbdf9548-bm4zq"
Jul 29 19:26:17.064845 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:26:17.064751    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:26:48.943945 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:26:48.943900    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:26:59.203219 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:26:59.203181    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-cc3e80-b65b446cd-qprgp_5a2b03a7-d196-4391-86b3-5ba0ce84031f/payload-processing/0.log"
Jul 29 19:27:03.452800 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:03.452762    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6"]
Jul 29 19:27:03.457434 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:03.457408    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6"
Jul 29 19:27:03.460576 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:03.460540    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-21dbec-serving-cert\""
Jul 29 19:27:03.468560 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:03.468522    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6"]
Jul 29 19:27:03.542847 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:03.542806    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29c4311a-1165-4afb-8ea3-4c1e72124021-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6\" (UID: \"29c4311a-1165-4afb-8ea3-4c1e72124021\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6"
Jul 29 19:27:03.542847 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:03.542849    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsp5w\" (UniqueName: \"kubernetes.io/projected/29c4311a-1165-4afb-8ea3-4c1e72124021-kube-api-access-bsp5w\") pod \"maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6\" (UID: \"29c4311a-1165-4afb-8ea3-4c1e72124021\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6"
Jul 29 19:27:03.543115 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:03.542876    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c4311a-1165-4afb-8ea3-4c1e72124021-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6\" (UID: \"29c4311a-1165-4afb-8ea3-4c1e72124021\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6"
Jul 29 19:27:03.543115 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:03.543003    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29c4311a-1165-4afb-8ea3-4c1e72124021-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6\" (UID: \"29c4311a-1165-4afb-8ea3-4c1e72124021\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6"
Jul 29 19:27:03.644310 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:03.644258    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29c4311a-1165-4afb-8ea3-4c1e72124021-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6\" (UID: \"29c4311a-1165-4afb-8ea3-4c1e72124021\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6"
Jul 29 19:27:03.644310 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:03.644313    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsp5w\" (UniqueName: \"kubernetes.io/projected/29c4311a-1165-4afb-8ea3-4c1e72124021-kube-api-access-bsp5w\") pod \"maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6\" (UID: \"29c4311a-1165-4afb-8ea3-4c1e72124021\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6"
Jul 29 19:27:03.644644 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:03.644349    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c4311a-1165-4afb-8ea3-4c1e72124021-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6\" (UID: \"29c4311a-1165-4afb-8ea3-4c1e72124021\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6"
Jul 29 19:27:03.644644 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:03.644461    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29c4311a-1165-4afb-8ea3-4c1e72124021-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6\" (UID: \"29c4311a-1165-4afb-8ea3-4c1e72124021\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6"
Jul 29 19:27:03.644644 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:27:03.644469    2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-21dbec-serving-cert: secret "maas-api-e2e-ipp-cleanup-21dbec-serving-cert" not found
Jul 29 19:27:03.644644 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:27:03.644542    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29c4311a-1165-4afb-8ea3-4c1e72124021-maas-api-tls podName:29c4311a-1165-4afb-8ea3-4c1e72124021 nodeName:}" failed. No retries permitted until 2026-07-29 19:27:04.144518721 +0000 UTC m=+3188.414287786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/29c4311a-1165-4afb-8ea3-4c1e72124021-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6" (UID: "29c4311a-1165-4afb-8ea3-4c1e72124021") : secret "maas-api-e2e-ipp-cleanup-21dbec-serving-cert" not found
Jul 29 19:27:03.645213 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:03.645182    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c4311a-1165-4afb-8ea3-4c1e72124021-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6\" (UID: \"29c4311a-1165-4afb-8ea3-4c1e72124021\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6"
Jul 29 19:27:03.647238 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:03.647210    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29c4311a-1165-4afb-8ea3-4c1e72124021-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6\" (UID: \"29c4311a-1165-4afb-8ea3-4c1e72124021\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6"
Jul 29 19:27:03.653415 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:03.653356    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsp5w\" (UniqueName: \"kubernetes.io/projected/29c4311a-1165-4afb-8ea3-4c1e72124021-kube-api-access-bsp5w\") pod \"maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6\" (UID: \"29c4311a-1165-4afb-8ea3-4c1e72124021\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6"
Jul 29 19:27:04.148993 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:04.148946    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29c4311a-1165-4afb-8ea3-4c1e72124021-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6\" (UID: \"29c4311a-1165-4afb-8ea3-4c1e72124021\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6"
Jul 29 19:27:04.151720 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:04.151694    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29c4311a-1165-4afb-8ea3-4c1e72124021-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6\" (UID: \"29c4311a-1165-4afb-8ea3-4c1e72124021\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6"
Jul 29 19:27:04.370865 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:04.370830    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6"
Jul 29 19:27:04.519823 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:04.519798    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6"]
Jul 29 19:27:04.521859 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:27:04.521832    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c4311a_1165_4afb_8ea3_4c1e72124021.slice/crio-68af1a31ac26ff1f8511585cfed1ca6656405eaa81471abb86c90a5be45ea952 WatchSource:0}: Error finding container 68af1a31ac26ff1f8511585cfed1ca6656405eaa81471abb86c90a5be45ea952: Status 404 returned error can't find the container with id 68af1a31ac26ff1f8511585cfed1ca6656405eaa81471abb86c90a5be45ea952
Jul 29 19:27:04.946531 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:04.946490    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6" event={"ID":"29c4311a-1165-4afb-8ea3-4c1e72124021","Type":"ContainerStarted","Data":"68af1a31ac26ff1f8511585cfed1ca6656405eaa81471abb86c90a5be45ea952"}
Jul 29 19:27:05.952843 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:05.952807    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6" event={"ID":"29c4311a-1165-4afb-8ea3-4c1e72124021","Type":"ContainerStarted","Data":"a7a506a5033afd3e9b347f66102342fe775e1ce288f2cf311b6bfd35bcb3b37a"}
Jul 29 19:27:05.953302 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:05.952875    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6"
Jul 29 19:27:05.976874 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:05.976808    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6" podStartSLOduration=2.495675619 podStartE2EDuration="2.976790113s" totalImagesPullingTime="481.114494ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:27:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:27:05.974444116 +0000 UTC m=+3190.244213212" watchObservedRunningTime="2026-07-29 19:27:05.976790113 +0000 UTC m=+3190.246559261"
Jul 29 19:27:08.844460 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:08.844420    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:27:11.964161 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:11.964128    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6"
Jul 29 19:27:34.217434 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:34.217395    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6"]
Jul 29 19:27:34.218055 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:34.217672    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6" podUID="29c4311a-1165-4afb-8ea3-4c1e72124021" containerName="maas-api" containerID="cri-o://a7a506a5033afd3e9b347f66102342fe775e1ce288f2cf311b6bfd35bcb3b37a" gracePeriod=30
Jul 29 19:27:34.482802 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:34.482727    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6"
Jul 29 19:27:34.542224 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:34.542182    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29c4311a-1165-4afb-8ea3-4c1e72124021-maas-api-tls\") pod \"29c4311a-1165-4afb-8ea3-4c1e72124021\" (UID: \"29c4311a-1165-4afb-8ea3-4c1e72124021\") "
Jul 29 19:27:34.542224 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:34.542225    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29c4311a-1165-4afb-8ea3-4c1e72124021-metrics-certs\") pod \"29c4311a-1165-4afb-8ea3-4c1e72124021\" (UID: \"29c4311a-1165-4afb-8ea3-4c1e72124021\") "
Jul 29 19:27:34.542529 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:34.542262    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bsp5w\" (UniqueName: \"kubernetes.io/projected/29c4311a-1165-4afb-8ea3-4c1e72124021-kube-api-access-bsp5w\") pod \"29c4311a-1165-4afb-8ea3-4c1e72124021\" (UID: \"29c4311a-1165-4afb-8ea3-4c1e72124021\") "
Jul 29 19:27:34.542529 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:34.542315    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c4311a-1165-4afb-8ea3-4c1e72124021-openshift-service-ca\") pod \"29c4311a-1165-4afb-8ea3-4c1e72124021\" (UID: \"29c4311a-1165-4afb-8ea3-4c1e72124021\") "
Jul 29 19:27:34.542828 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:34.542795    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29c4311a-1165-4afb-8ea3-4c1e72124021-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "29c4311a-1165-4afb-8ea3-4c1e72124021" (UID: "29c4311a-1165-4afb-8ea3-4c1e72124021"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:27:34.544813 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:34.544783    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29c4311a-1165-4afb-8ea3-4c1e72124021-kube-api-access-bsp5w" (OuterVolumeSpecName: "kube-api-access-bsp5w") pod "29c4311a-1165-4afb-8ea3-4c1e72124021" (UID: "29c4311a-1165-4afb-8ea3-4c1e72124021"). InnerVolumeSpecName "kube-api-access-bsp5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:27:34.545130 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:34.545107    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29c4311a-1165-4afb-8ea3-4c1e72124021-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "29c4311a-1165-4afb-8ea3-4c1e72124021" (UID: "29c4311a-1165-4afb-8ea3-4c1e72124021"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:27:34.545189 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:34.545147    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29c4311a-1165-4afb-8ea3-4c1e72124021-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "29c4311a-1165-4afb-8ea3-4c1e72124021" (UID: "29c4311a-1165-4afb-8ea3-4c1e72124021"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:27:34.643553 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:34.643510    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29c4311a-1165-4afb-8ea3-4c1e72124021-maas-api-tls\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:27:34.643553 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:34.643549    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29c4311a-1165-4afb-8ea3-4c1e72124021-metrics-certs\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:27:34.643822 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:34.643563    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsp5w\" (UniqueName: \"kubernetes.io/projected/29c4311a-1165-4afb-8ea3-4c1e72124021-kube-api-access-bsp5w\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:27:34.643822 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:34.643578    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c4311a-1165-4afb-8ea3-4c1e72124021-openshift-service-ca\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:27:35.104143 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:35.104105    2559 generic.go:356] "Generic (PLEG): container finished" podID="29c4311a-1165-4afb-8ea3-4c1e72124021" containerID="a7a506a5033afd3e9b347f66102342fe775e1ce288f2cf311b6bfd35bcb3b37a" exitCode=0
Jul 29 19:27:35.104366 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:35.104156    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6"
Jul 29 19:27:35.104366 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:35.104198    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6" event={"ID":"29c4311a-1165-4afb-8ea3-4c1e72124021","Type":"ContainerDied","Data":"a7a506a5033afd3e9b347f66102342fe775e1ce288f2cf311b6bfd35bcb3b37a"}
Jul 29 19:27:35.104366 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:35.104260    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6" event={"ID":"29c4311a-1165-4afb-8ea3-4c1e72124021","Type":"ContainerDied","Data":"68af1a31ac26ff1f8511585cfed1ca6656405eaa81471abb86c90a5be45ea952"}
Jul 29 19:27:35.104366 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:35.104291    2559 scope.go:117] "RemoveContainer" containerID="a7a506a5033afd3e9b347f66102342fe775e1ce288f2cf311b6bfd35bcb3b37a"
Jul 29 19:27:35.114309 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:35.114280    2559 scope.go:117] "RemoveContainer" containerID="a7a506a5033afd3e9b347f66102342fe775e1ce288f2cf311b6bfd35bcb3b37a"
Jul 29 19:27:35.114635 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:27:35.114615    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7a506a5033afd3e9b347f66102342fe775e1ce288f2cf311b6bfd35bcb3b37a\": container with ID starting with a7a506a5033afd3e9b347f66102342fe775e1ce288f2cf311b6bfd35bcb3b37a not found: ID does not exist" containerID="a7a506a5033afd3e9b347f66102342fe775e1ce288f2cf311b6bfd35bcb3b37a"
Jul 29 19:27:35.114723 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:35.114648    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7a506a5033afd3e9b347f66102342fe775e1ce288f2cf311b6bfd35bcb3b37a"} err="failed to get container status \"a7a506a5033afd3e9b347f66102342fe775e1ce288f2cf311b6bfd35bcb3b37a\": rpc error: code = NotFound desc = could not find container \"a7a506a5033afd3e9b347f66102342fe775e1ce288f2cf311b6bfd35bcb3b37a\": container with ID starting with a7a506a5033afd3e9b347f66102342fe775e1ce288f2cf311b6bfd35bcb3b37a not found: ID does not exist"
Jul 29 19:27:35.128149 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:35.128120    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6"]
Jul 29 19:27:35.134434 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:35.134405    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-21dbec-59dbb77f4f-hk2x6"]
Jul 29 19:27:36.152182 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:36.152144    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29c4311a-1165-4afb-8ea3-4c1e72124021" path="/var/lib/kubelet/pods/29c4311a-1165-4afb-8ea3-4c1e72124021/volumes"
Jul 29 19:27:38.944843 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:27:38.944802    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:28:00.605976 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:00.605936    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf"]
Jul 29 19:28:00.606508 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:00.606215    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf" podUID="a2f550c1-979d-40ed-9dcc-627f2281ecb4" containerName="maas-api" containerID="cri-o://79dd58facb0fe5757b650b766f4c76194dbbb3470b079aedf0401733ead0e187" gracePeriod=30
Jul 29 19:28:00.664628 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:00.664580    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cc3e80-b65b446cd-qprgp"]
Jul 29 19:28:00.664866 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:00.664841    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-cc3e80-b65b446cd-qprgp" podUID="5a2b03a7-d196-4391-86b3-5ba0ce84031f" containerName="payload-processing" containerID="cri-o://03dcb089975a0471dc7546b7c6bca8bc0daa97c018d04356a16221f317e7ca99" gracePeriod=30
Jul 29 19:28:00.671489 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:00.671459    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cc3e80-77ff6bccb-h24qx"]
Jul 29 19:28:00.671745 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:00.671697    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cc3e80-77ff6bccb-h24qx" podUID="f6988a18-fdfb-484a-ba02-290a8fdc4e1a" containerName="payload-pre-processing" containerID="cri-o://529b94190c4150fe0b83814b4225f4c9ec9b12a8a89405438ce5d09f85916bd5" gracePeriod=30
Jul 29 19:28:00.983019 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:00.982991    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-cc3e80-b65b446cd-qprgp"
Jul 29 19:28:00.987163 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:00.987134    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cc3e80-77ff6bccb-h24qx"
Jul 29 19:28:00.990862 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:00.990840    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf"
Jul 29 19:28:01.096451 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.096419    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2f550c1-979d-40ed-9dcc-627f2281ecb4-metrics-certs\") pod \"a2f550c1-979d-40ed-9dcc-627f2281ecb4\" (UID: \"a2f550c1-979d-40ed-9dcc-627f2281ecb4\") "
Jul 29 19:28:01.096631 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.096481    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrbsr\" (UniqueName: \"kubernetes.io/projected/a2f550c1-979d-40ed-9dcc-627f2281ecb4-kube-api-access-nrbsr\") pod \"a2f550c1-979d-40ed-9dcc-627f2281ecb4\" (UID: \"a2f550c1-979d-40ed-9dcc-627f2281ecb4\") "
Jul 29 19:28:01.096631 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.096506    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a2b03a7-d196-4391-86b3-5ba0ce84031f-plugins-config-volume\") pod \"5a2b03a7-d196-4391-86b3-5ba0ce84031f\" (UID: \"5a2b03a7-d196-4391-86b3-5ba0ce84031f\") "
Jul 29 19:28:01.096631 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.096568    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6988a18-fdfb-484a-ba02-290a8fdc4e1a-plugins-config-volume\") pod \"f6988a18-fdfb-484a-ba02-290a8fdc4e1a\" (UID: \"f6988a18-fdfb-484a-ba02-290a8fdc4e1a\") "
Jul 29 19:28:01.096631 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.096604    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjf42\" (UniqueName: \"kubernetes.io/projected/f6988a18-fdfb-484a-ba02-290a8fdc4e1a-kube-api-access-bjf42\") pod \"f6988a18-fdfb-484a-ba02-290a8fdc4e1a\" (UID: \"f6988a18-fdfb-484a-ba02-290a8fdc4e1a\") "
Jul 29 19:28:01.096875 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.096668    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2f550c1-979d-40ed-9dcc-627f2281ecb4-openshift-service-ca\") pod \"a2f550c1-979d-40ed-9dcc-627f2281ecb4\" (UID: \"a2f550c1-979d-40ed-9dcc-627f2281ecb4\") "
Jul 29 19:28:01.096875 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.096712    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2f550c1-979d-40ed-9dcc-627f2281ecb4-maas-api-tls\") pod \"a2f550c1-979d-40ed-9dcc-627f2281ecb4\" (UID: \"a2f550c1-979d-40ed-9dcc-627f2281ecb4\") "
Jul 29 19:28:01.096875 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.096753    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rssck\" (UniqueName: \"kubernetes.io/projected/5a2b03a7-d196-4391-86b3-5ba0ce84031f-kube-api-access-rssck\") pod \"5a2b03a7-d196-4391-86b3-5ba0ce84031f\" (UID: \"5a2b03a7-d196-4391-86b3-5ba0ce84031f\") "
Jul 29 19:28:01.097059 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.096921    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a2b03a7-d196-4391-86b3-5ba0ce84031f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5a2b03a7-d196-4391-86b3-5ba0ce84031f" (UID: "5a2b03a7-d196-4391-86b3-5ba0ce84031f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:28:01.097118 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.097061    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6988a18-fdfb-484a-ba02-290a8fdc4e1a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f6988a18-fdfb-484a-ba02-290a8fdc4e1a" (UID: "f6988a18-fdfb-484a-ba02-290a8fdc4e1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:28:01.097118 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.097068    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2f550c1-979d-40ed-9dcc-627f2281ecb4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a2f550c1-979d-40ed-9dcc-627f2281ecb4" (UID: "a2f550c1-979d-40ed-9dcc-627f2281ecb4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:28:01.097219 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.097180    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6988a18-fdfb-484a-ba02-290a8fdc4e1a-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:28:01.097219 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.097201    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2f550c1-979d-40ed-9dcc-627f2281ecb4-openshift-service-ca\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:28:01.097219 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.097216    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a2b03a7-d196-4391-86b3-5ba0ce84031f-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:28:01.098972 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.098941    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2f550c1-979d-40ed-9dcc-627f2281ecb4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a2f550c1-979d-40ed-9dcc-627f2281ecb4" (UID: "a2f550c1-979d-40ed-9dcc-627f2281ecb4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:28:01.098972 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.098940    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6988a18-fdfb-484a-ba02-290a8fdc4e1a-kube-api-access-bjf42" (OuterVolumeSpecName: "kube-api-access-bjf42") pod "f6988a18-fdfb-484a-ba02-290a8fdc4e1a" (UID: "f6988a18-fdfb-484a-ba02-290a8fdc4e1a"). InnerVolumeSpecName "kube-api-access-bjf42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:28:01.099211 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.099197    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a2b03a7-d196-4391-86b3-5ba0ce84031f-kube-api-access-rssck" (OuterVolumeSpecName: "kube-api-access-rssck") pod "5a2b03a7-d196-4391-86b3-5ba0ce84031f" (UID: "5a2b03a7-d196-4391-86b3-5ba0ce84031f"). InnerVolumeSpecName "kube-api-access-rssck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:28:01.099558 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.099537    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2f550c1-979d-40ed-9dcc-627f2281ecb4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a2f550c1-979d-40ed-9dcc-627f2281ecb4" (UID: "a2f550c1-979d-40ed-9dcc-627f2281ecb4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:28:01.099604 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.099584    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2f550c1-979d-40ed-9dcc-627f2281ecb4-kube-api-access-nrbsr" (OuterVolumeSpecName: "kube-api-access-nrbsr") pod "a2f550c1-979d-40ed-9dcc-627f2281ecb4" (UID: "a2f550c1-979d-40ed-9dcc-627f2281ecb4"). InnerVolumeSpecName "kube-api-access-nrbsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:28:01.198299 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.198208    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2f550c1-979d-40ed-9dcc-627f2281ecb4-metrics-certs\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:28:01.198299 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.198240    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrbsr\" (UniqueName: \"kubernetes.io/projected/a2f550c1-979d-40ed-9dcc-627f2281ecb4-kube-api-access-nrbsr\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:28:01.198299 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.198250    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjf42\" (UniqueName: \"kubernetes.io/projected/f6988a18-fdfb-484a-ba02-290a8fdc4e1a-kube-api-access-bjf42\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:28:01.198299 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.198258    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2f550c1-979d-40ed-9dcc-627f2281ecb4-maas-api-tls\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:28:01.198299 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.198267    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rssck\" (UniqueName: \"kubernetes.io/projected/5a2b03a7-d196-4391-86b3-5ba0ce84031f-kube-api-access-rssck\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:28:01.244447 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.244410    2559 generic.go:356] "Generic (PLEG): container finished" podID="5a2b03a7-d196-4391-86b3-5ba0ce84031f" containerID="03dcb089975a0471dc7546b7c6bca8bc0daa97c018d04356a16221f317e7ca99" exitCode=0
Jul 29 19:28:01.244648 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.244463    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-cc3e80-b65b446cd-qprgp"
Jul 29 19:28:01.244648 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.244496    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-cc3e80-b65b446cd-qprgp" event={"ID":"5a2b03a7-d196-4391-86b3-5ba0ce84031f","Type":"ContainerDied","Data":"03dcb089975a0471dc7546b7c6bca8bc0daa97c018d04356a16221f317e7ca99"}
Jul 29 19:28:01.244648 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.244557    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-cc3e80-b65b446cd-qprgp" event={"ID":"5a2b03a7-d196-4391-86b3-5ba0ce84031f","Type":"ContainerDied","Data":"dc41b3c4fe82c8a74fac2b095009000fecfdb9b01749532d80bb306bd128cccf"}
Jul 29 19:28:01.244648 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.244581    2559 scope.go:117] "RemoveContainer" containerID="03dcb089975a0471dc7546b7c6bca8bc0daa97c018d04356a16221f317e7ca99"
Jul 29 19:28:01.245968 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.245935    2559 generic.go:356] "Generic (PLEG): container finished" podID="f6988a18-fdfb-484a-ba02-290a8fdc4e1a" containerID="529b94190c4150fe0b83814b4225f4c9ec9b12a8a89405438ce5d09f85916bd5" exitCode=0
Jul 29 19:28:01.246107 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.245965    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cc3e80-77ff6bccb-h24qx" event={"ID":"f6988a18-fdfb-484a-ba02-290a8fdc4e1a","Type":"ContainerDied","Data":"529b94190c4150fe0b83814b4225f4c9ec9b12a8a89405438ce5d09f85916bd5"}
Jul 29 19:28:01.246107 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.245986    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cc3e80-77ff6bccb-h24qx"
Jul 29 19:28:01.246107 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.246007    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cc3e80-77ff6bccb-h24qx" event={"ID":"f6988a18-fdfb-484a-ba02-290a8fdc4e1a","Type":"ContainerDied","Data":"9807ad3af0f7bed9153d7720efae0e78a38f78cc39f6942117f59df97caf8a4b"}
Jul 29 19:28:01.247444 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.247408    2559 generic.go:356] "Generic (PLEG): container finished" podID="a2f550c1-979d-40ed-9dcc-627f2281ecb4" containerID="79dd58facb0fe5757b650b766f4c76194dbbb3470b079aedf0401733ead0e187" exitCode=0
Jul 29 19:28:01.247564 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.247446    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf"
Jul 29 19:28:01.247564 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.247491    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf" event={"ID":"a2f550c1-979d-40ed-9dcc-627f2281ecb4","Type":"ContainerDied","Data":"79dd58facb0fe5757b650b766f4c76194dbbb3470b079aedf0401733ead0e187"}
Jul 29 19:28:01.247564 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.247527    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf" event={"ID":"a2f550c1-979d-40ed-9dcc-627f2281ecb4","Type":"ContainerDied","Data":"3968de5cb54b3f80c7a35b151fcbd1755b167a473dc9a517d0a9fad6d61e0fac"}
Jul 29 19:28:01.257812 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.257791    2559 scope.go:117] "RemoveContainer" containerID="03dcb089975a0471dc7546b7c6bca8bc0daa97c018d04356a16221f317e7ca99"
Jul 29 19:28:01.258265 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:28:01.258243    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03dcb089975a0471dc7546b7c6bca8bc0daa97c018d04356a16221f317e7ca99\": container with ID starting with 03dcb089975a0471dc7546b7c6bca8bc0daa97c018d04356a16221f317e7ca99 not found: ID does not exist" containerID="03dcb089975a0471dc7546b7c6bca8bc0daa97c018d04356a16221f317e7ca99"
Jul 29 19:28:01.258368 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.258275    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03dcb089975a0471dc7546b7c6bca8bc0daa97c018d04356a16221f317e7ca99"} err="failed to get container status \"03dcb089975a0471dc7546b7c6bca8bc0daa97c018d04356a16221f317e7ca99\": rpc error: code = NotFound desc = could not find container \"03dcb089975a0471dc7546b7c6bca8bc0daa97c018d04356a16221f317e7ca99\": container with ID starting with 03dcb089975a0471dc7546b7c6bca8bc0daa97c018d04356a16221f317e7ca99 not found: ID does not exist"
Jul 29 19:28:01.258368 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.258300    2559 scope.go:117] "RemoveContainer" containerID="529b94190c4150fe0b83814b4225f4c9ec9b12a8a89405438ce5d09f85916bd5"
Jul 29 19:28:01.267702 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.267684    2559 scope.go:117] "RemoveContainer" containerID="529b94190c4150fe0b83814b4225f4c9ec9b12a8a89405438ce5d09f85916bd5"
Jul 29 19:28:01.267986 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:28:01.267967    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"529b94190c4150fe0b83814b4225f4c9ec9b12a8a89405438ce5d09f85916bd5\": container with ID starting with 529b94190c4150fe0b83814b4225f4c9ec9b12a8a89405438ce5d09f85916bd5 not found: ID does not exist" containerID="529b94190c4150fe0b83814b4225f4c9ec9b12a8a89405438ce5d09f85916bd5"
Jul 29 19:28:01.268033 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.267995    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"529b94190c4150fe0b83814b4225f4c9ec9b12a8a89405438ce5d09f85916bd5"} err="failed to get container status \"529b94190c4150fe0b83814b4225f4c9ec9b12a8a89405438ce5d09f85916bd5\": rpc error: code = NotFound desc = could not find container \"529b94190c4150fe0b83814b4225f4c9ec9b12a8a89405438ce5d09f85916bd5\": container with ID starting with 529b94190c4150fe0b83814b4225f4c9ec9b12a8a89405438ce5d09f85916bd5 not found: ID does not exist"
Jul 29 19:28:01.268033 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.268013    2559 scope.go:117] "RemoveContainer" containerID="79dd58facb0fe5757b650b766f4c76194dbbb3470b079aedf0401733ead0e187"
Jul 29 19:28:01.276436 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.276419    2559 scope.go:117] "RemoveContainer" containerID="79dd58facb0fe5757b650b766f4c76194dbbb3470b079aedf0401733ead0e187"
Jul 29 19:28:01.276809 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:28:01.276788    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79dd58facb0fe5757b650b766f4c76194dbbb3470b079aedf0401733ead0e187\": container with ID starting with 79dd58facb0fe5757b650b766f4c76194dbbb3470b079aedf0401733ead0e187 not found: ID does not exist" containerID="79dd58facb0fe5757b650b766f4c76194dbbb3470b079aedf0401733ead0e187"
Jul 29 19:28:01.276886 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.276821    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79dd58facb0fe5757b650b766f4c76194dbbb3470b079aedf0401733ead0e187"} err="failed to get container status \"79dd58facb0fe5757b650b766f4c76194dbbb3470b079aedf0401733ead0e187\": rpc error: code = NotFound desc = could not find container \"79dd58facb0fe5757b650b766f4c76194dbbb3470b079aedf0401733ead0e187\": container with ID starting with 79dd58facb0fe5757b650b766f4c76194dbbb3470b079aedf0401733ead0e187 not found: ID does not exist"
Jul 29 19:28:01.281262 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.281239    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cc3e80-b65b446cd-qprgp"]
Jul 29 19:28:01.286180 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.286156    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cc3e80-b65b446cd-qprgp"]
Jul 29 19:28:01.313887 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.313857    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf"]
Jul 29 19:28:01.323336 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.323304    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cc3e80-7498f9b5cc-r4dwf"]
Jul 29 19:28:01.335999 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.335970    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cc3e80-77ff6bccb-h24qx"]
Jul 29 19:28:01.349336 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.349305    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cc3e80-77ff6bccb-h24qx"]
Jul 29 19:28:01.373592 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:01.373555    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:28:02.151112 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:02.151076    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a2b03a7-d196-4391-86b3-5ba0ce84031f" path="/var/lib/kubelet/pods/5a2b03a7-d196-4391-86b3-5ba0ce84031f/volumes"
Jul 29 19:28:02.151546 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:02.151512    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2f550c1-979d-40ed-9dcc-627f2281ecb4" path="/var/lib/kubelet/pods/a2f550c1-979d-40ed-9dcc-627f2281ecb4/volumes"
Jul 29 19:28:02.151822 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:02.151808    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6988a18-fdfb-484a-ba02-290a8fdc4e1a" path="/var/lib/kubelet/pods/f6988a18-fdfb-484a-ba02-290a8fdc4e1a/volumes"
Jul 29 19:28:11.449693 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:11.449649    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:28:56.343592 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:56.343558    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-v9nsr_7f8b4737-d4c2-4598-bd40-208015542831/console-operator/1.log"
Jul 29 19:28:56.352124 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:56.352096    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjdrx_60f444e1-7e31-443c-8ec7-a4d04c64bf4b/ovn-acl-logging/0.log"
Jul 29 19:28:56.352331 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:56.352226    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-v9nsr_7f8b4737-d4c2-4598-bd40-208015542831/console-operator/1.log"
Jul 29 19:28:56.361277 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:28:56.361253    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjdrx_60f444e1-7e31-443c-8ec7-a4d04c64bf4b/ovn-acl-logging/0.log"
Jul 29 19:29:01.160082 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:01.160046    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws"]
Jul 29 19:29:01.160539 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:01.160278    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws" podUID="aa40169f-e600-4800-a452-afefaeabff92" containerName="maas-api" containerID="cri-o://d55ce1817a3259fb11320d095afa6f85b68b49a6e7d3b92375b703ccabe6778c" gracePeriod=30
Jul 29 19:29:01.312335 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:01.312281    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f0bee1-6fbbdf9548-bm4zq"]
Jul 29 19:29:01.312809 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:01.312609    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0bee1-6fbbdf9548-bm4zq" podUID="ffc38b90-3e01-4eaa-a612-9aeb014ab1b5" containerName="payload-pre-processing" containerID="cri-o://c60fde680391cc7bfb839aed823a49a53a07c749152159882c4eb33c567928d0" gracePeriod=30
Jul 29 19:29:01.444414 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:01.444365    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws"
Jul 29 19:29:01.534083 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:01.534050    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa40169f-e600-4800-a452-afefaeabff92-openshift-service-ca\") pod \"aa40169f-e600-4800-a452-afefaeabff92\" (UID: \"aa40169f-e600-4800-a452-afefaeabff92\") "
Jul 29 19:29:01.534241 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:01.534160    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa40169f-e600-4800-a452-afefaeabff92-metrics-certs\") pod \"aa40169f-e600-4800-a452-afefaeabff92\" (UID: \"aa40169f-e600-4800-a452-afefaeabff92\") "
Jul 29 19:29:01.534241 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:01.534182    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9n7qt\" (UniqueName: \"kubernetes.io/projected/aa40169f-e600-4800-a452-afefaeabff92-kube-api-access-9n7qt\") pod \"aa40169f-e600-4800-a452-afefaeabff92\" (UID: \"aa40169f-e600-4800-a452-afefaeabff92\") "
Jul 29 19:29:01.534346 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:01.534245    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa40169f-e600-4800-a452-afefaeabff92-maas-api-tls\") pod \"aa40169f-e600-4800-a452-afefaeabff92\" (UID: \"aa40169f-e600-4800-a452-afefaeabff92\") "
Jul 29 19:29:01.534469 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:01.534446    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa40169f-e600-4800-a452-afefaeabff92-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "aa40169f-e600-4800-a452-afefaeabff92" (UID: "aa40169f-e600-4800-a452-afefaeabff92"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:29:01.534606 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:01.534584    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa40169f-e600-4800-a452-afefaeabff92-openshift-service-ca\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:29:01.537092 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:01.537056    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa40169f-e600-4800-a452-afefaeabff92-kube-api-access-9n7qt" (OuterVolumeSpecName: "kube-api-access-9n7qt") pod "aa40169f-e600-4800-a452-afefaeabff92" (UID: "aa40169f-e600-4800-a452-afefaeabff92"). InnerVolumeSpecName "kube-api-access-9n7qt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:29:01.537203 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:01.537061    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa40169f-e600-4800-a452-afefaeabff92-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "aa40169f-e600-4800-a452-afefaeabff92" (UID: "aa40169f-e600-4800-a452-afefaeabff92"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:29:01.537203 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:01.537165    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa40169f-e600-4800-a452-afefaeabff92-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "aa40169f-e600-4800-a452-afefaeabff92" (UID: "aa40169f-e600-4800-a452-afefaeabff92"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:29:01.555545 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:01.555510    2559 generic.go:356] "Generic (PLEG): container finished" podID="ffc38b90-3e01-4eaa-a612-9aeb014ab1b5" containerID="c60fde680391cc7bfb839aed823a49a53a07c749152159882c4eb33c567928d0" exitCode=0
Jul 29 19:29:01.555712 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:01.555552    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0bee1-6fbbdf9548-bm4zq" event={"ID":"ffc38b90-3e01-4eaa-a612-9aeb014ab1b5","Type":"ContainerDied","Data":"c60fde680391cc7bfb839aed823a49a53a07c749152159882c4eb33c567928d0"}
Jul 29 19:29:01.556955 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:01.556920    2559 generic.go:356] "Generic (PLEG): container finished" podID="aa40169f-e600-4800-a452-afefaeabff92" containerID="d55ce1817a3259fb11320d095afa6f85b68b49a6e7d3b92375b703ccabe6778c" exitCode=0
Jul 29 19:29:01.557082 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:01.556972    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws" event={"ID":"aa40169f-e600-4800-a452-afefaeabff92","Type":"ContainerDied","Data":"d55ce1817a3259fb11320d095afa6f85b68b49a6e7d3b92375b703ccabe6778c"}
Jul 29 19:29:01.557082 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:01.557010    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws"
Jul 29 19:29:01.557082 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:01.557052    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws" event={"ID":"aa40169f-e600-4800-a452-afefaeabff92","Type":"ContainerDied","Data":"afb67ac5d01879a6b05fa8285e0b86b58d02a01330d9abda7e3e4a26559862b8"}
Jul 29 19:29:01.557189 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:01.557082    2559 scope.go:117] "RemoveContainer" containerID="d55ce1817a3259fb11320d095afa6f85b68b49a6e7d3b92375b703ccabe6778c"
Jul 29 19:29:01.566977 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:01.566696    2559 scope.go:117] "RemoveContainer" containerID="d55ce1817a3259fb11320d095afa6f85b68b49a6e7d3b92375b703ccabe6778c"
Jul 29 19:29:01.567069 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:29:01.567048    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d55ce1817a3259fb11320d095afa6f85b68b49a6e7d3b92375b703ccabe6778c\": container with ID starting with d55ce1817a3259fb11320d095afa6f85b68b49a6e7d3b92375b703ccabe6778c not found: ID does not exist" containerID="d55ce1817a3259fb11320d095afa6f85b68b49a6e7d3b92375b703ccabe6778c"
Jul 29 19:29:01.567133 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:01.567079    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d55ce1817a3259fb11320d095afa6f85b68b49a6e7d3b92375b703ccabe6778c"} err="failed to get container status \"d55ce1817a3259fb11320d095afa6f85b68b49a6e7d3b92375b703ccabe6778c\": rpc error: code = NotFound desc = could not find container \"d55ce1817a3259fb11320d095afa6f85b68b49a6e7d3b92375b703ccabe6778c\": container with ID starting with d55ce1817a3259fb11320d095afa6f85b68b49a6e7d3b92375b703ccabe6778c not found: ID does not exist"
Jul 29 19:29:01.576448 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:01.576427    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0bee1-6fbbdf9548-bm4zq"
Jul 29 19:29:01.582189 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:01.582156    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws"]
Jul 29 19:29:01.584791 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:01.584769    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0bee1-74fd9b646f-7cwws"]
Jul 29 19:29:01.635467 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:01.635425    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa40169f-e600-4800-a452-afefaeabff92-metrics-certs\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:29:01.635467 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:01.635462    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n7qt\" (UniqueName: \"kubernetes.io/projected/aa40169f-e600-4800-a452-afefaeabff92-kube-api-access-9n7qt\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:29:01.635467 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:01.635472    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa40169f-e600-4800-a452-afefaeabff92-maas-api-tls\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:29:01.736798 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:01.736709    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvl44\" (UniqueName: \"kubernetes.io/projected/ffc38b90-3e01-4eaa-a612-9aeb014ab1b5-kube-api-access-xvl44\") pod \"ffc38b90-3e01-4eaa-a612-9aeb014ab1b5\" (UID: \"ffc38b90-3e01-4eaa-a612-9aeb014ab1b5\") "
Jul 29 19:29:01.736966 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:01.736820    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffc38b90-3e01-4eaa-a612-9aeb014ab1b5-plugins-config-volume\") pod \"ffc38b90-3e01-4eaa-a612-9aeb014ab1b5\" (UID: \"ffc38b90-3e01-4eaa-a612-9aeb014ab1b5\") "
Jul 29 19:29:01.737251 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:01.737229    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffc38b90-3e01-4eaa-a612-9aeb014ab1b5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ffc38b90-3e01-4eaa-a612-9aeb014ab1b5" (UID: "ffc38b90-3e01-4eaa-a612-9aeb014ab1b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:29:01.739094 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:01.739072    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffc38b90-3e01-4eaa-a612-9aeb014ab1b5-kube-api-access-xvl44" (OuterVolumeSpecName: "kube-api-access-xvl44") pod "ffc38b90-3e01-4eaa-a612-9aeb014ab1b5" (UID: "ffc38b90-3e01-4eaa-a612-9aeb014ab1b5"). InnerVolumeSpecName "kube-api-access-xvl44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:29:01.837960 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:01.837921    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvl44\" (UniqueName: \"kubernetes.io/projected/ffc38b90-3e01-4eaa-a612-9aeb014ab1b5-kube-api-access-xvl44\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:29:01.837960 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:01.837953    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffc38b90-3e01-4eaa-a612-9aeb014ab1b5-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:29:02.150493 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:02.150460    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa40169f-e600-4800-a452-afefaeabff92" path="/var/lib/kubelet/pods/aa40169f-e600-4800-a452-afefaeabff92/volumes"
Jul 29 19:29:02.564103 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:02.564002    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0bee1-6fbbdf9548-bm4zq" event={"ID":"ffc38b90-3e01-4eaa-a612-9aeb014ab1b5","Type":"ContainerDied","Data":"f86ac99a78f2144622470e6d71e4aab357c3c6aabad15cdeef37b9086c5272cd"}
Jul 29 19:29:02.564103 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:02.564041    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f0bee1-6fbbdf9548-bm4zq"
Jul 29 19:29:02.564103 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:02.564073    2559 scope.go:117] "RemoveContainer" containerID="c60fde680391cc7bfb839aed823a49a53a07c749152159882c4eb33c567928d0"
Jul 29 19:29:02.580726 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:02.580696    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f0bee1-6fbbdf9548-bm4zq"]
Jul 29 19:29:02.584652 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:02.584624    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f0bee1-6fbbdf9548-bm4zq"]
Jul 29 19:29:04.150773 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:04.150739    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffc38b90-3e01-4eaa-a612-9aeb014ab1b5" path="/var/lib/kubelet/pods/ffc38b90-3e01-4eaa-a612-9aeb014ab1b5/volumes"
Jul 29 19:29:04.347206 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:04.347176    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:29:18.855734 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:18.855643    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d04442-76994dcbf5-pzbdh"]
Jul 29 19:29:18.856280 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:18.855924    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d04442-76994dcbf5-pzbdh" podUID="08ef2ec9-453f-42c0-be3f-a801a7f2812d" containerName="maas-api" containerID="cri-o://fdf69b6b6abb0d1d8dc02d6975a15064b26c1acc71b5fad48426fabccb0ef8d4" gracePeriod=30
Jul 29 19:29:18.921186 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:18.921111    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-d04442-7c85966bb-l9q2w"]
Jul 29 19:29:18.922399 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:18.921448    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-d04442-7c85966bb-l9q2w" podUID="f137423f-ecf1-424b-adfc-da0e242b17a2" containerName="payload-processing" containerID="cri-o://bf9d25031b54ce3c73300ce60956ff1dea285a53cf2055dd6a55625fe01f44f2" gracePeriod=30
Jul 29 19:29:18.928290 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:18.928218    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-d04442-5598777f68-svhg8"]
Jul 29 19:29:18.928492 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:18.928460    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-d04442-5598777f68-svhg8" podUID="2c5bec84-2a80-466e-9ba7-6efcdea6a38e" containerName="payload-pre-processing" containerID="cri-o://8b9016cd495bd85963f655011f7a3e9925b42fa814d5ed040dfd48b7fbfb1399" gracePeriod=30
Jul 29 19:29:19.212640 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.212612    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-d04442-7c85966bb-l9q2w"
Jul 29 19:29:19.228920 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.228895    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d04442-76994dcbf5-pzbdh"
Jul 29 19:29:19.249152 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.249126    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-d04442-5598777f68-svhg8"
Jul 29 19:29:19.292352 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.292322    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08ef2ec9-453f-42c0-be3f-a801a7f2812d-maas-api-tls\") pod \"08ef2ec9-453f-42c0-be3f-a801a7f2812d\" (UID: \"08ef2ec9-453f-42c0-be3f-a801a7f2812d\") "
Jul 29 19:29:19.292536 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.292366    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08ef2ec9-453f-42c0-be3f-a801a7f2812d-openshift-service-ca\") pod \"08ef2ec9-453f-42c0-be3f-a801a7f2812d\" (UID: \"08ef2ec9-453f-42c0-be3f-a801a7f2812d\") "
Jul 29 19:29:19.292536 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.292418    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxddr\" (UniqueName: \"kubernetes.io/projected/08ef2ec9-453f-42c0-be3f-a801a7f2812d-kube-api-access-hxddr\") pod \"08ef2ec9-453f-42c0-be3f-a801a7f2812d\" (UID: \"08ef2ec9-453f-42c0-be3f-a801a7f2812d\") "
Jul 29 19:29:19.292536 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.292446    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c5bec84-2a80-466e-9ba7-6efcdea6a38e-plugins-config-volume\") pod \"2c5bec84-2a80-466e-9ba7-6efcdea6a38e\" (UID: \"2c5bec84-2a80-466e-9ba7-6efcdea6a38e\") "
Jul 29 19:29:19.292536 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.292473    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4jsm\" (UniqueName: \"kubernetes.io/projected/2c5bec84-2a80-466e-9ba7-6efcdea6a38e-kube-api-access-t4jsm\") pod \"2c5bec84-2a80-466e-9ba7-6efcdea6a38e\" (UID: \"2c5bec84-2a80-466e-9ba7-6efcdea6a38e\") "
Jul 29 19:29:19.292536 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.292495    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f137423f-ecf1-424b-adfc-da0e242b17a2-plugins-config-volume\") pod \"f137423f-ecf1-424b-adfc-da0e242b17a2\" (UID: \"f137423f-ecf1-424b-adfc-da0e242b17a2\") "
Jul 29 19:29:19.292775 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.292640    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5nsx\" (UniqueName: \"kubernetes.io/projected/f137423f-ecf1-424b-adfc-da0e242b17a2-kube-api-access-p5nsx\") pod \"f137423f-ecf1-424b-adfc-da0e242b17a2\" (UID: \"f137423f-ecf1-424b-adfc-da0e242b17a2\") "
Jul 29 19:29:19.292775 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.292694    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08ef2ec9-453f-42c0-be3f-a801a7f2812d-metrics-certs\") pod \"08ef2ec9-453f-42c0-be3f-a801a7f2812d\" (UID: \"08ef2ec9-453f-42c0-be3f-a801a7f2812d\") "
Jul 29 19:29:19.292871 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.292838    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08ef2ec9-453f-42c0-be3f-a801a7f2812d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "08ef2ec9-453f-42c0-be3f-a801a7f2812d" (UID: "08ef2ec9-453f-42c0-be3f-a801a7f2812d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:29:19.292871 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.292842    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f137423f-ecf1-424b-adfc-da0e242b17a2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f137423f-ecf1-424b-adfc-da0e242b17a2" (UID: "f137423f-ecf1-424b-adfc-da0e242b17a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:29:19.292871 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.292842    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c5bec84-2a80-466e-9ba7-6efcdea6a38e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2c5bec84-2a80-466e-9ba7-6efcdea6a38e" (UID: "2c5bec84-2a80-466e-9ba7-6efcdea6a38e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:29:19.293061 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.293038    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08ef2ec9-453f-42c0-be3f-a801a7f2812d-openshift-service-ca\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:29:19.293121 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.293060    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c5bec84-2a80-466e-9ba7-6efcdea6a38e-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:29:19.293121 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.293079    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f137423f-ecf1-424b-adfc-da0e242b17a2-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:29:19.294917 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.294882    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f137423f-ecf1-424b-adfc-da0e242b17a2-kube-api-access-p5nsx" (OuterVolumeSpecName: "kube-api-access-p5nsx") pod "f137423f-ecf1-424b-adfc-da0e242b17a2" (UID: "f137423f-ecf1-424b-adfc-da0e242b17a2"). InnerVolumeSpecName "kube-api-access-p5nsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:29:19.294917 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.294898    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c5bec84-2a80-466e-9ba7-6efcdea6a38e-kube-api-access-t4jsm" (OuterVolumeSpecName: "kube-api-access-t4jsm") pod "2c5bec84-2a80-466e-9ba7-6efcdea6a38e" (UID: "2c5bec84-2a80-466e-9ba7-6efcdea6a38e"). InnerVolumeSpecName "kube-api-access-t4jsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:29:19.295086 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.295046    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08ef2ec9-453f-42c0-be3f-a801a7f2812d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "08ef2ec9-453f-42c0-be3f-a801a7f2812d" (UID: "08ef2ec9-453f-42c0-be3f-a801a7f2812d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:29:19.295173 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.295158    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08ef2ec9-453f-42c0-be3f-a801a7f2812d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "08ef2ec9-453f-42c0-be3f-a801a7f2812d" (UID: "08ef2ec9-453f-42c0-be3f-a801a7f2812d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:29:19.295210 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.295175    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08ef2ec9-453f-42c0-be3f-a801a7f2812d-kube-api-access-hxddr" (OuterVolumeSpecName: "kube-api-access-hxddr") pod "08ef2ec9-453f-42c0-be3f-a801a7f2812d" (UID: "08ef2ec9-453f-42c0-be3f-a801a7f2812d"). InnerVolumeSpecName "kube-api-access-hxddr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:29:19.393821 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.393738    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4jsm\" (UniqueName: \"kubernetes.io/projected/2c5bec84-2a80-466e-9ba7-6efcdea6a38e-kube-api-access-t4jsm\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:29:19.393821 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.393769    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5nsx\" (UniqueName: \"kubernetes.io/projected/f137423f-ecf1-424b-adfc-da0e242b17a2-kube-api-access-p5nsx\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:29:19.393821 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.393779    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08ef2ec9-453f-42c0-be3f-a801a7f2812d-metrics-certs\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:29:19.393821 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.393789    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08ef2ec9-453f-42c0-be3f-a801a7f2812d-maas-api-tls\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:29:19.393821 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.393798    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxddr\" (UniqueName: \"kubernetes.io/projected/08ef2ec9-453f-42c0-be3f-a801a7f2812d-kube-api-access-hxddr\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:29:19.652737 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.652650    2559 generic.go:356] "Generic (PLEG): container finished" podID="f137423f-ecf1-424b-adfc-da0e242b17a2" containerID="bf9d25031b54ce3c73300ce60956ff1dea285a53cf2055dd6a55625fe01f44f2" exitCode=0
Jul 29 19:29:19.652737 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.652687    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-d04442-7c85966bb-l9q2w"
Jul 29 19:29:19.652737 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.652722    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-d04442-7c85966bb-l9q2w" event={"ID":"f137423f-ecf1-424b-adfc-da0e242b17a2","Type":"ContainerDied","Data":"bf9d25031b54ce3c73300ce60956ff1dea285a53cf2055dd6a55625fe01f44f2"}
Jul 29 19:29:19.653049 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.652769    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-d04442-7c85966bb-l9q2w" event={"ID":"f137423f-ecf1-424b-adfc-da0e242b17a2","Type":"ContainerDied","Data":"95895affd348422fe0f4b98719f0fca14cfb03388c94332773dc50f474eaec77"}
Jul 29 19:29:19.653049 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.652790    2559 scope.go:117] "RemoveContainer" containerID="bf9d25031b54ce3c73300ce60956ff1dea285a53cf2055dd6a55625fe01f44f2"
Jul 29 19:29:19.654293 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.654266    2559 generic.go:356] "Generic (PLEG): container finished" podID="08ef2ec9-453f-42c0-be3f-a801a7f2812d" containerID="fdf69b6b6abb0d1d8dc02d6975a15064b26c1acc71b5fad48426fabccb0ef8d4" exitCode=0
Jul 29 19:29:19.654433 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.654306    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d04442-76994dcbf5-pzbdh"
Jul 29 19:29:19.654433 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.654352    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d04442-76994dcbf5-pzbdh" event={"ID":"08ef2ec9-453f-42c0-be3f-a801a7f2812d","Type":"ContainerDied","Data":"fdf69b6b6abb0d1d8dc02d6975a15064b26c1acc71b5fad48426fabccb0ef8d4"}
Jul 29 19:29:19.654572 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.654490    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d04442-76994dcbf5-pzbdh" event={"ID":"08ef2ec9-453f-42c0-be3f-a801a7f2812d","Type":"ContainerDied","Data":"1f7bbc4ea92ffa3f8e4db55871a3272490f15e2dd223933cf831c9557e56103c"}
Jul 29 19:29:19.655884 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.655856    2559 generic.go:356] "Generic (PLEG): container finished" podID="2c5bec84-2a80-466e-9ba7-6efcdea6a38e" containerID="8b9016cd495bd85963f655011f7a3e9925b42fa814d5ed040dfd48b7fbfb1399" exitCode=0
Jul 29 19:29:19.655980 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.655902    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-d04442-5598777f68-svhg8"
Jul 29 19:29:19.656041 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.655976    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-d04442-5598777f68-svhg8" event={"ID":"2c5bec84-2a80-466e-9ba7-6efcdea6a38e","Type":"ContainerDied","Data":"8b9016cd495bd85963f655011f7a3e9925b42fa814d5ed040dfd48b7fbfb1399"}
Jul 29 19:29:19.656041 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.656028    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-d04442-5598777f68-svhg8" event={"ID":"2c5bec84-2a80-466e-9ba7-6efcdea6a38e","Type":"ContainerDied","Data":"185533def893b040c4d2688bcadbe9a3e3b9de6850a601bbebb331d0a961394a"}
Jul 29 19:29:19.664261 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.664241    2559 scope.go:117] "RemoveContainer" containerID="bf9d25031b54ce3c73300ce60956ff1dea285a53cf2055dd6a55625fe01f44f2"
Jul 29 19:29:19.666463 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:29:19.664587    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf9d25031b54ce3c73300ce60956ff1dea285a53cf2055dd6a55625fe01f44f2\": container with ID starting with bf9d25031b54ce3c73300ce60956ff1dea285a53cf2055dd6a55625fe01f44f2 not found: ID does not exist" containerID="bf9d25031b54ce3c73300ce60956ff1dea285a53cf2055dd6a55625fe01f44f2"
Jul 29 19:29:19.666463 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.664629    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf9d25031b54ce3c73300ce60956ff1dea285a53cf2055dd6a55625fe01f44f2"} err="failed to get container status \"bf9d25031b54ce3c73300ce60956ff1dea285a53cf2055dd6a55625fe01f44f2\": rpc error: code = NotFound desc = could not find container \"bf9d25031b54ce3c73300ce60956ff1dea285a53cf2055dd6a55625fe01f44f2\": container with ID starting with bf9d25031b54ce3c73300ce60956ff1dea285a53cf2055dd6a55625fe01f44f2 not found: ID does not exist"
Jul 29 19:29:19.666463 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.664653    2559 scope.go:117] "RemoveContainer" containerID="fdf69b6b6abb0d1d8dc02d6975a15064b26c1acc71b5fad48426fabccb0ef8d4"
Jul 29 19:29:19.677742 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.677721    2559 scope.go:117] "RemoveContainer" containerID="fdf69b6b6abb0d1d8dc02d6975a15064b26c1acc71b5fad48426fabccb0ef8d4"
Jul 29 19:29:19.678028 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:29:19.678010    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdf69b6b6abb0d1d8dc02d6975a15064b26c1acc71b5fad48426fabccb0ef8d4\": container with ID starting with fdf69b6b6abb0d1d8dc02d6975a15064b26c1acc71b5fad48426fabccb0ef8d4 not found: ID does not exist" containerID="fdf69b6b6abb0d1d8dc02d6975a15064b26c1acc71b5fad48426fabccb0ef8d4"
Jul 29 19:29:19.678072 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.678037    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdf69b6b6abb0d1d8dc02d6975a15064b26c1acc71b5fad48426fabccb0ef8d4"} err="failed to get container status \"fdf69b6b6abb0d1d8dc02d6975a15064b26c1acc71b5fad48426fabccb0ef8d4\": rpc error: code = NotFound desc = could not find container \"fdf69b6b6abb0d1d8dc02d6975a15064b26c1acc71b5fad48426fabccb0ef8d4\": container with ID starting with fdf69b6b6abb0d1d8dc02d6975a15064b26c1acc71b5fad48426fabccb0ef8d4 not found: ID does not exist"
Jul 29 19:29:19.678072 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.678060    2559 scope.go:117] "RemoveContainer" containerID="8b9016cd495bd85963f655011f7a3e9925b42fa814d5ed040dfd48b7fbfb1399"
Jul 29 19:29:19.681267 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.681244    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-d04442-7c85966bb-l9q2w"]
Jul 29 19:29:19.686531 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.686505    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-d04442-7c85966bb-l9q2w"]
Jul 29 19:29:19.688822 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.688803    2559 scope.go:117] "RemoveContainer" containerID="8b9016cd495bd85963f655011f7a3e9925b42fa814d5ed040dfd48b7fbfb1399"
Jul 29 19:29:19.690349 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:29:19.690330    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b9016cd495bd85963f655011f7a3e9925b42fa814d5ed040dfd48b7fbfb1399\": container with ID starting with 8b9016cd495bd85963f655011f7a3e9925b42fa814d5ed040dfd48b7fbfb1399 not found: ID does not exist" containerID="8b9016cd495bd85963f655011f7a3e9925b42fa814d5ed040dfd48b7fbfb1399"
Jul 29 19:29:19.690506 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.690359    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b9016cd495bd85963f655011f7a3e9925b42fa814d5ed040dfd48b7fbfb1399"} err="failed to get container status \"8b9016cd495bd85963f655011f7a3e9925b42fa814d5ed040dfd48b7fbfb1399\": rpc error: code = NotFound desc = could not find container \"8b9016cd495bd85963f655011f7a3e9925b42fa814d5ed040dfd48b7fbfb1399\": container with ID starting with 8b9016cd495bd85963f655011f7a3e9925b42fa814d5ed040dfd48b7fbfb1399 not found: ID does not exist"
Jul 29 19:29:19.696049 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.696030    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d04442-76994dcbf5-pzbdh"]
Jul 29 19:29:19.701922 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.701893    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d04442-76994dcbf5-pzbdh"]
Jul 29 19:29:19.712592 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.712568    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-d04442-5598777f68-svhg8"]
Jul 29 19:29:19.722844 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:19.722816    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-d04442-5598777f68-svhg8"]
Jul 29 19:29:20.150360 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:20.150320    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08ef2ec9-453f-42c0-be3f-a801a7f2812d" path="/var/lib/kubelet/pods/08ef2ec9-453f-42c0-be3f-a801a7f2812d/volumes"
Jul 29 19:29:20.150867 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:20.150838    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c5bec84-2a80-466e-9ba7-6efcdea6a38e" path="/var/lib/kubelet/pods/2c5bec84-2a80-466e-9ba7-6efcdea6a38e/volumes"
Jul 29 19:29:20.151205 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:20.151185    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f137423f-ecf1-424b-adfc-da0e242b17a2" path="/var/lib/kubelet/pods/f137423f-ecf1-424b-adfc-da0e242b17a2/volumes"
Jul 29 19:29:21.847683 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:21.847647    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:29:36.874959 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:36.874921    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-422d30-5886654bcc-8tg5t"]
Jul 29 19:29:36.876585 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:36.875408    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-422d30-5886654bcc-8tg5t" podUID="9e5ca931-aa77-48a6-a941-2d8e6c6483fe" containerName="maas-api" containerID="cri-o://eed0e4d05079e3e4bc20930d50af62600d3e7ec0f5d4e7764d94bbceaae8eef7" gracePeriod=30
Jul 29 19:29:36.933726 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:36.933692    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-422d30-68f8cc9484-mb5d9"]
Jul 29 19:29:36.933944 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:36.933916    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-422d30-68f8cc9484-mb5d9" podUID="888ca0f7-0649-4536-a2d5-8ff65029f5f1" containerName="payload-pre-processing" containerID="cri-o://09a6d04d8f14a2fbd049b48946ec6f0fcbdfa94f76ff2a6306e37b53061ee108" gracePeriod=30
Jul 29 19:29:37.182514 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:37.182486    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-422d30-5886654bcc-8tg5t"
Jul 29 19:29:37.216973 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:37.216948    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-422d30-68f8cc9484-mb5d9"
Jul 29 19:29:37.252340 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:37.252308    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e5ca931-aa77-48a6-a941-2d8e6c6483fe-maas-api-tls\") pod \"9e5ca931-aa77-48a6-a941-2d8e6c6483fe\" (UID: \"9e5ca931-aa77-48a6-a941-2d8e6c6483fe\") "
Jul 29 19:29:37.252534 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:37.252408    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e5ca931-aa77-48a6-a941-2d8e6c6483fe-metrics-certs\") pod \"9e5ca931-aa77-48a6-a941-2d8e6c6483fe\" (UID: \"9e5ca931-aa77-48a6-a941-2d8e6c6483fe\") "
Jul 29 19:29:37.252534 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:37.252448    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e5ca931-aa77-48a6-a941-2d8e6c6483fe-openshift-service-ca\") pod \"9e5ca931-aa77-48a6-a941-2d8e6c6483fe\" (UID: \"9e5ca931-aa77-48a6-a941-2d8e6c6483fe\") "
Jul 29 19:29:37.252534 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:37.252494    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdtmd\" (UniqueName: \"kubernetes.io/projected/9e5ca931-aa77-48a6-a941-2d8e6c6483fe-kube-api-access-xdtmd\") pod \"9e5ca931-aa77-48a6-a941-2d8e6c6483fe\" (UID: \"9e5ca931-aa77-48a6-a941-2d8e6c6483fe\") "
Jul 29 19:29:37.252826 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:37.252802    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e5ca931-aa77-48a6-a941-2d8e6c6483fe-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9e5ca931-aa77-48a6-a941-2d8e6c6483fe" (UID: "9e5ca931-aa77-48a6-a941-2d8e6c6483fe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:29:37.254678 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:37.254649    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e5ca931-aa77-48a6-a941-2d8e6c6483fe-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9e5ca931-aa77-48a6-a941-2d8e6c6483fe" (UID: "9e5ca931-aa77-48a6-a941-2d8e6c6483fe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:29:37.254778 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:37.254726    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e5ca931-aa77-48a6-a941-2d8e6c6483fe-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9e5ca931-aa77-48a6-a941-2d8e6c6483fe" (UID: "9e5ca931-aa77-48a6-a941-2d8e6c6483fe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:29:37.254778 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:37.254736    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e5ca931-aa77-48a6-a941-2d8e6c6483fe-kube-api-access-xdtmd" (OuterVolumeSpecName: "kube-api-access-xdtmd") pod "9e5ca931-aa77-48a6-a941-2d8e6c6483fe" (UID: "9e5ca931-aa77-48a6-a941-2d8e6c6483fe"). InnerVolumeSpecName "kube-api-access-xdtmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:29:37.353652 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:37.353610    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/888ca0f7-0649-4536-a2d5-8ff65029f5f1-plugins-config-volume\") pod \"888ca0f7-0649-4536-a2d5-8ff65029f5f1\" (UID: \"888ca0f7-0649-4536-a2d5-8ff65029f5f1\") "
Jul 29 19:29:37.353652 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:37.353650    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5nxv\" (UniqueName: \"kubernetes.io/projected/888ca0f7-0649-4536-a2d5-8ff65029f5f1-kube-api-access-t5nxv\") pod \"888ca0f7-0649-4536-a2d5-8ff65029f5f1\" (UID: \"888ca0f7-0649-4536-a2d5-8ff65029f5f1\") "
Jul 29 19:29:37.353922 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:37.353779    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e5ca931-aa77-48a6-a941-2d8e6c6483fe-metrics-certs\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:29:37.353922 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:37.353789    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e5ca931-aa77-48a6-a941-2d8e6c6483fe-openshift-service-ca\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:29:37.353922 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:37.353799    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdtmd\" (UniqueName: \"kubernetes.io/projected/9e5ca931-aa77-48a6-a941-2d8e6c6483fe-kube-api-access-xdtmd\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:29:37.353922 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:37.353808    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e5ca931-aa77-48a6-a941-2d8e6c6483fe-maas-api-tls\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:29:37.354170 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:37.353974    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/888ca0f7-0649-4536-a2d5-8ff65029f5f1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "888ca0f7-0649-4536-a2d5-8ff65029f5f1" (UID: "888ca0f7-0649-4536-a2d5-8ff65029f5f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:29:37.355999 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:37.355970    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/888ca0f7-0649-4536-a2d5-8ff65029f5f1-kube-api-access-t5nxv" (OuterVolumeSpecName: "kube-api-access-t5nxv") pod "888ca0f7-0649-4536-a2d5-8ff65029f5f1" (UID: "888ca0f7-0649-4536-a2d5-8ff65029f5f1"). InnerVolumeSpecName "kube-api-access-t5nxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:29:37.454673 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:37.454580    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/888ca0f7-0649-4536-a2d5-8ff65029f5f1-plugins-config-volume\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:29:37.454673 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:37.454613    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5nxv\" (UniqueName: \"kubernetes.io/projected/888ca0f7-0649-4536-a2d5-8ff65029f5f1-kube-api-access-t5nxv\") on node \"ip-10-0-138-152.ec2.internal\" DevicePath \"\""
Jul 29 19:29:37.749253 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:37.749160    2559 generic.go:356] "Generic (PLEG): container finished" podID="888ca0f7-0649-4536-a2d5-8ff65029f5f1" containerID="09a6d04d8f14a2fbd049b48946ec6f0fcbdfa94f76ff2a6306e37b53061ee108" exitCode=0
Jul 29 19:29:37.749253 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:37.749219    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-422d30-68f8cc9484-mb5d9"
Jul 29 19:29:37.749253 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:37.749243    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-422d30-68f8cc9484-mb5d9" event={"ID":"888ca0f7-0649-4536-a2d5-8ff65029f5f1","Type":"ContainerDied","Data":"09a6d04d8f14a2fbd049b48946ec6f0fcbdfa94f76ff2a6306e37b53061ee108"}
Jul 29 19:29:37.749563 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:37.749330    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-422d30-68f8cc9484-mb5d9" event={"ID":"888ca0f7-0649-4536-a2d5-8ff65029f5f1","Type":"ContainerDied","Data":"5fcbae0a22dfb065d202c79cb828ce23b12404c0aebb93e57e69d92b2a5aba53"}
Jul 29 19:29:37.749563 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:37.749357    2559 scope.go:117] "RemoveContainer" containerID="09a6d04d8f14a2fbd049b48946ec6f0fcbdfa94f76ff2a6306e37b53061ee108"
Jul 29 19:29:37.750747 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:37.750726    2559 generic.go:356] "Generic (PLEG): container finished" podID="9e5ca931-aa77-48a6-a941-2d8e6c6483fe" containerID="eed0e4d05079e3e4bc20930d50af62600d3e7ec0f5d4e7764d94bbceaae8eef7" exitCode=0
Jul 29 19:29:37.750865 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:37.750770    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-422d30-5886654bcc-8tg5t" event={"ID":"9e5ca931-aa77-48a6-a941-2d8e6c6483fe","Type":"ContainerDied","Data":"eed0e4d05079e3e4bc20930d50af62600d3e7ec0f5d4e7764d94bbceaae8eef7"}
Jul 29 19:29:37.750865 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:37.750794    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-422d30-5886654bcc-8tg5t"
Jul 29 19:29:37.750865 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:37.750807    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-422d30-5886654bcc-8tg5t" event={"ID":"9e5ca931-aa77-48a6-a941-2d8e6c6483fe","Type":"ContainerDied","Data":"369e0487739f2d096b6f1d1db8241523ce7ef2f918efafeefe4971d7d227cc93"}
Jul 29 19:29:37.759032 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:37.759013    2559 scope.go:117] "RemoveContainer" containerID="09a6d04d8f14a2fbd049b48946ec6f0fcbdfa94f76ff2a6306e37b53061ee108"
Jul 29 19:29:37.759319 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:29:37.759301    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09a6d04d8f14a2fbd049b48946ec6f0fcbdfa94f76ff2a6306e37b53061ee108\": container with ID starting with 09a6d04d8f14a2fbd049b48946ec6f0fcbdfa94f76ff2a6306e37b53061ee108 not found: ID does not exist" containerID="09a6d04d8f14a2fbd049b48946ec6f0fcbdfa94f76ff2a6306e37b53061ee108"
Jul 29 19:29:37.759366 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:37.759326    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09a6d04d8f14a2fbd049b48946ec6f0fcbdfa94f76ff2a6306e37b53061ee108"} err="failed to get container status \"09a6d04d8f14a2fbd049b48946ec6f0fcbdfa94f76ff2a6306e37b53061ee108\": rpc error: code = NotFound desc = could not find container \"09a6d04d8f14a2fbd049b48946ec6f0fcbdfa94f76ff2a6306e37b53061ee108\": container with ID starting with 09a6d04d8f14a2fbd049b48946ec6f0fcbdfa94f76ff2a6306e37b53061ee108 not found: ID does not exist"
Jul 29 19:29:37.759366 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:37.759341    2559 scope.go:117] "RemoveContainer" containerID="eed0e4d05079e3e4bc20930d50af62600d3e7ec0f5d4e7764d94bbceaae8eef7"
Jul 29 19:29:37.768137 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:37.768116    2559 scope.go:117] "RemoveContainer" containerID="eed0e4d05079e3e4bc20930d50af62600d3e7ec0f5d4e7764d94bbceaae8eef7"
Jul 29 19:29:37.768420 ip-10-0-138-152 kubenswrapper[2559]: E0729 19:29:37.768401    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eed0e4d05079e3e4bc20930d50af62600d3e7ec0f5d4e7764d94bbceaae8eef7\": container with ID starting with eed0e4d05079e3e4bc20930d50af62600d3e7ec0f5d4e7764d94bbceaae8eef7 not found: ID does not exist" containerID="eed0e4d05079e3e4bc20930d50af62600d3e7ec0f5d4e7764d94bbceaae8eef7"
Jul 29 19:29:37.768503 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:37.768428    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eed0e4d05079e3e4bc20930d50af62600d3e7ec0f5d4e7764d94bbceaae8eef7"} err="failed to get container status \"eed0e4d05079e3e4bc20930d50af62600d3e7ec0f5d4e7764d94bbceaae8eef7\": rpc error: code = NotFound desc = could not find container \"eed0e4d05079e3e4bc20930d50af62600d3e7ec0f5d4e7764d94bbceaae8eef7\": container with ID starting with eed0e4d05079e3e4bc20930d50af62600d3e7ec0f5d4e7764d94bbceaae8eef7 not found: ID does not exist"
Jul 29 19:29:37.772306 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:37.772282    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-422d30-68f8cc9484-mb5d9"]
Jul 29 19:29:37.779352 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:37.779329    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-422d30-68f8cc9484-mb5d9"]
Jul 29 19:29:37.790250 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:37.790227    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-422d30-5886654bcc-8tg5t"]
Jul 29 19:29:37.794861 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:37.794839    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-422d30-5886654bcc-8tg5t"]
Jul 29 19:29:38.150970 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:38.150933    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="888ca0f7-0649-4536-a2d5-8ff65029f5f1" path="/var/lib/kubelet/pods/888ca0f7-0649-4536-a2d5-8ff65029f5f1/volumes"
Jul 29 19:29:38.151345 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:38.151305    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e5ca931-aa77-48a6-a941-2d8e6c6483fe" path="/var/lib/kubelet/pods/9e5ca931-aa77-48a6-a941-2d8e6c6483fe/volumes"
Jul 29 19:29:39.741731 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:39.741697    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-268m5"]
Jul 29 19:29:50.131485 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:50.131444    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-b4kv2_7fbb1c34-1fcb-4b7b-bd0f-b2c2e0502bba/manager/0.log"
Jul 29 19:29:50.253742 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:50.253692    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-7gxpk_2b524f44-0875-4793-80e3-5886ee9bf148/manager/0.log"
Jul 29 19:29:51.032619 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:51.032593    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76c6f8f9b-qn29c_9e820933-bb87-4c58-b7aa-0cb939217b29/copy-manifests/0.log"
Jul 29 19:29:51.057271 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:51.057245    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76c6f8f9b-qn29c_9e820933-bb87-4c58-b7aa-0cb939217b29/manager/0.log"
Jul 29 19:29:51.680212 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:51.680171    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29755860-8rw6q_3d3fdb5f-b06c-40a6-84a2-9f47bbd0e735/cleanup/0.log"
Jul 29 19:29:51.793230 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:51.793203    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29755875-9r79b_d04bdf68-c320-46c7-83ae-d14e6fadbba8/cleanup/0.log"
Jul 29 19:29:53.979261 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:53.979227    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-268m5_463080e3-ad6e-4f21-a9b4-c30b44f452c6/limitador/0.log"
Jul 29 19:29:54.104037 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:54.104006    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-lhmrl_bf735020-ded2-4908-ae14-05c16e3c760b/manager/0.log"
Jul 29 19:29:55.326507 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:29:55.326478    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7688c5db95-gcqb5_66a293fb-92cb-4908-bf90-d97ed0d831a6/router/0.log"
Jul 29 19:30:00.141372 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:00.141339    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755890-mp4tm"]
Jul 29 19:30:00.141764 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:00.141706    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa40169f-e600-4800-a452-afefaeabff92" containerName="maas-api"
Jul 29 19:30:00.141764 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:00.141722    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa40169f-e600-4800-a452-afefaeabff92" containerName="maas-api"
Jul 29 19:30:00.141764 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:00.141736    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08ef2ec9-453f-42c0-be3f-a801a7f2812d" containerName="maas-api"
Jul 29 19:30:00.141764 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:00.141741    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="08ef2ec9-453f-42c0-be3f-a801a7f2812d" containerName="maas-api"
Jul 29 19:30:00.141764 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:00.141749    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f137423f-ecf1-424b-adfc-da0e242b17a2" containerName="payload-processing"
Jul 29 19:30:00.141764 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:00.141754    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f137423f-ecf1-424b-adfc-da0e242b17a2" containerName="payload-processing"
Jul 29 19:30:00.141764 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:00.141760    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a2b03a7-d196-4391-86b3-5ba0ce84031f" containerName="payload-processing"
Jul 29 19:30:00.141764 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:00.141765    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a2b03a7-d196-4391-86b3-5ba0ce84031f" containerName="payload-processing"
Jul 29 19:30:00.142016 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:00.141774    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6988a18-fdfb-484a-ba02-290a8fdc4e1a" containerName="payload-pre-processing"
Jul 29 19:30:00.142016 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:00.141781    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6988a18-fdfb-484a-ba02-290a8fdc4e1a" containerName="payload-pre-processing"
Jul 29 19:30:00.142016 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:00.141791    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="888ca0f7-0649-4536-a2d5-8ff65029f5f1" containerName="payload-pre-processing"
Jul 29 19:30:00.142016 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:00.141799    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="888ca0f7-0649-4536-a2d5-8ff65029f5f1" containerName="payload-pre-processing"
Jul 29 19:30:00.142016 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:00.141811    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffc38b90-3e01-4eaa-a612-9aeb014ab1b5" containerName="payload-pre-processing"
Jul 29 19:30:00.142016 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:00.141816    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffc38b90-3e01-4eaa-a612-9aeb014ab1b5" containerName="payload-pre-processing"
Jul 29 19:30:00.142016 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:00.141825    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2f550c1-979d-40ed-9dcc-627f2281ecb4" containerName="maas-api"
Jul 29 19:30:00.142016 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:00.141830    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2f550c1-979d-40ed-9dcc-627f2281ecb4" containerName="maas-api"
Jul 29 19:30:00.142016 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:00.141836    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c5bec84-2a80-466e-9ba7-6efcdea6a38e" containerName="payload-pre-processing"
Jul 29 19:30:00.142016 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:00.141841    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c5bec84-2a80-466e-9ba7-6efcdea6a38e" containerName="payload-pre-processing"
Jul 29 19:30:00.142016 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:00.141848    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29c4311a-1165-4afb-8ea3-4c1e72124021" containerName="maas-api"
Jul 29 19:30:00.142016 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:00.141852    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="29c4311a-1165-4afb-8ea3-4c1e72124021" containerName="maas-api"
Jul 29 19:30:00.142016 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:00.141860    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e5ca931-aa77-48a6-a941-2d8e6c6483fe" containerName="maas-api"
Jul 29 19:30:00.142016 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:00.141865    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e5ca931-aa77-48a6-a941-2d8e6c6483fe" containerName="maas-api"
Jul 29 19:30:00.142016 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:00.141922    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e5ca931-aa77-48a6-a941-2d8e6c6483fe" containerName="maas-api"
Jul 29 19:30:00.142016 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:00.141928    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a2b03a7-d196-4391-86b3-5ba0ce84031f" containerName="payload-processing"
Jul 29 19:30:00.142016 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:00.141936    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6988a18-fdfb-484a-ba02-290a8fdc4e1a" containerName="payload-pre-processing"
Jul 29 19:30:00.142016 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:00.141942    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="08ef2ec9-453f-42c0-be3f-a801a7f2812d" containerName="maas-api"
Jul 29 19:30:00.142016 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:00.141948    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f137423f-ecf1-424b-adfc-da0e242b17a2" containerName="payload-processing"
Jul 29 19:30:00.142016 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:00.141958    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="29c4311a-1165-4afb-8ea3-4c1e72124021" containerName="maas-api"
Jul 29 19:30:00.142016 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:00.141965    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c5bec84-2a80-466e-9ba7-6efcdea6a38e" containerName="payload-pre-processing"
Jul 29 19:30:00.142016 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:00.141970    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="888ca0f7-0649-4536-a2d5-8ff65029f5f1" containerName="payload-pre-processing"
Jul 29 19:30:00.142016 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:00.141975    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffc38b90-3e01-4eaa-a612-9aeb014ab1b5" containerName="payload-pre-processing"
Jul 29 19:30:00.142016 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:00.141981    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2f550c1-979d-40ed-9dcc-627f2281ecb4" containerName="maas-api"
Jul 29 19:30:00.142016 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:00.141985    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa40169f-e600-4800-a452-afefaeabff92" containerName="maas-api"
Jul 29 19:30:00.146868 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:00.146848    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755890-mp4tm"
Jul 29 19:30:00.149758 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:00.149583    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vgb9z\""
Jul 29 19:30:00.154275 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:00.154253    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755890-mp4tm"]
Jul 29 19:30:00.158526 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:00.158507    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755890-mp4tm"
Jul 29 19:30:00.292572 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:00.292551    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755890-mp4tm"]
Jul 29 19:30:00.294888 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:30:00.294853    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f8b52ec_3767_450e_a10d_6e8ca991b781.slice/crio-527ae51251695cdf7bea77b0842d593f956af898f5b588a012a6865878e83e60 WatchSource:0}: Error finding container 527ae51251695cdf7bea77b0842d593f956af898f5b588a012a6865878e83e60: Status 404 returned error can't find the container with id 527ae51251695cdf7bea77b0842d593f956af898f5b588a012a6865878e83e60
Jul 29 19:30:00.864785 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:00.864749    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755890-mp4tm" event={"ID":"0f8b52ec-3767-450e-a10d-6e8ca991b781","Type":"ContainerStarted","Data":"00acc319f34944e9ce99ea7dd413813d996780f9167949186a3444b3f351c8b4"}
Jul 29 19:30:00.864973 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:00.864810    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755890-mp4tm" event={"ID":"0f8b52ec-3767-450e-a10d-6e8ca991b781","Type":"ContainerStarted","Data":"527ae51251695cdf7bea77b0842d593f956af898f5b588a012a6865878e83e60"}
Jul 29 19:30:00.881575 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:00.881524    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755890-mp4tm" podStartSLOduration=0.881509612 podStartE2EDuration="881.509612ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:30:00.879710411 +0000 UTC m=+3365.149479499" watchObservedRunningTime="2026-07-29 19:30:00.881509612 +0000 UTC m=+3365.151278699"
Jul 29 19:30:01.870585 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:01.870548    2559 generic.go:356] "Generic (PLEG): container finished" podID="0f8b52ec-3767-450e-a10d-6e8ca991b781" containerID="00acc319f34944e9ce99ea7dd413813d996780f9167949186a3444b3f351c8b4" exitCode=0
Jul 29 19:30:01.870585 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:01.870589    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755890-mp4tm" event={"ID":"0f8b52ec-3767-450e-a10d-6e8ca991b781","Type":"ContainerDied","Data":"00acc319f34944e9ce99ea7dd413813d996780f9167949186a3444b3f351c8b4"}
Jul 29 19:30:02.317099 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:02.317019    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kmkl2/must-gather-j7f7n"]
Jul 29 19:30:02.320589 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:02.320571    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kmkl2/must-gather-j7f7n"
Jul 29 19:30:02.323670 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:02.323653    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kmkl2\"/\"kube-root-ca.crt\""
Jul 29 19:30:02.323806 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:02.323787    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kmkl2\"/\"openshift-service-ca.crt\""
Jul 29 19:30:02.332061 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:02.332034    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kmkl2/must-gather-j7f7n"]
Jul 29 19:30:02.362932 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:02.362895    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/59f50a3d-f140-449d-b832-67d6b69ba9a0-must-gather-output\") pod \"must-gather-j7f7n\" (UID: \"59f50a3d-f140-449d-b832-67d6b69ba9a0\") " pod="openshift-must-gather-kmkl2/must-gather-j7f7n"
Jul 29 19:30:02.363106 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:02.362950    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n6km\" (UniqueName: \"kubernetes.io/projected/59f50a3d-f140-449d-b832-67d6b69ba9a0-kube-api-access-5n6km\") pod \"must-gather-j7f7n\" (UID: \"59f50a3d-f140-449d-b832-67d6b69ba9a0\") " pod="openshift-must-gather-kmkl2/must-gather-j7f7n"
Jul 29 19:30:02.464367 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:02.464317    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/59f50a3d-f140-449d-b832-67d6b69ba9a0-must-gather-output\") pod \"must-gather-j7f7n\" (UID: \"59f50a3d-f140-449d-b832-67d6b69ba9a0\") " pod="openshift-must-gather-kmkl2/must-gather-j7f7n"
Jul 29 19:30:02.464566 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:02.464421    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n6km\" (UniqueName: \"kubernetes.io/projected/59f50a3d-f140-449d-b832-67d6b69ba9a0-kube-api-access-5n6km\") pod \"must-gather-j7f7n\" (UID: \"59f50a3d-f140-449d-b832-67d6b69ba9a0\") " pod="openshift-must-gather-kmkl2/must-gather-j7f7n"
Jul 29 19:30:02.464711 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:02.464692    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/59f50a3d-f140-449d-b832-67d6b69ba9a0-must-gather-output\") pod \"must-gather-j7f7n\" (UID: \"59f50a3d-f140-449d-b832-67d6b69ba9a0\") " pod="openshift-must-gather-kmkl2/must-gather-j7f7n"
Jul 29 19:30:02.473613 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:02.473585    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n6km\" (UniqueName: \"kubernetes.io/projected/59f50a3d-f140-449d-b832-67d6b69ba9a0-kube-api-access-5n6km\") pod \"must-gather-j7f7n\" (UID: \"59f50a3d-f140-449d-b832-67d6b69ba9a0\") " pod="openshift-must-gather-kmkl2/must-gather-j7f7n"
Jul 29 19:30:02.629497 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:02.629455    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kmkl2/must-gather-j7f7n"
Jul 29 19:30:02.774643 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:02.774611    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kmkl2/must-gather-j7f7n"]
Jul 29 19:30:02.777871 ip-10-0-138-152 kubenswrapper[2559]: W0729 19:30:02.777837    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59f50a3d_f140_449d_b832_67d6b69ba9a0.slice/crio-6b62eae2d0cdbdbde37ffb3394b4a453d8009e85661eefeb70f50d76abf30a51 WatchSource:0}: Error finding container 6b62eae2d0cdbdbde37ffb3394b4a453d8009e85661eefeb70f50d76abf30a51: Status 404 returned error can't find the container with id 6b62eae2d0cdbdbde37ffb3394b4a453d8009e85661eefeb70f50d76abf30a51
Jul 29 19:30:02.876664 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:02.876625    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kmkl2/must-gather-j7f7n" event={"ID":"59f50a3d-f140-449d-b832-67d6b69ba9a0","Type":"ContainerStarted","Data":"6b62eae2d0cdbdbde37ffb3394b4a453d8009e85661eefeb70f50d76abf30a51"}
Jul 29 19:30:03.006136 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:03.006096    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755890-mp4tm"
Jul 29 19:30:03.883756 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:03.883726    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755890-mp4tm"
Jul 29 19:30:03.884180 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:03.883759    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755890-mp4tm" event={"ID":"0f8b52ec-3767-450e-a10d-6e8ca991b781","Type":"ContainerDied","Data":"527ae51251695cdf7bea77b0842d593f956af898f5b588a012a6865878e83e60"}
Jul 29 19:30:03.884180 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:03.883812    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="527ae51251695cdf7bea77b0842d593f956af898f5b588a012a6865878e83e60"
Jul 29 19:30:04.891234 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:04.891193    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kmkl2/must-gather-j7f7n" event={"ID":"59f50a3d-f140-449d-b832-67d6b69ba9a0","Type":"ContainerStarted","Data":"b64b1d84f8986a02fe27a923d2d1a6e812b73ed712d52fa3f947430afc8745d7"}
Jul 29 19:30:04.891234 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:04.891239    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kmkl2/must-gather-j7f7n" event={"ID":"59f50a3d-f140-449d-b832-67d6b69ba9a0","Type":"ContainerStarted","Data":"e9fc70944c321759c12cc70fa295088c6489b27147016861814af95520d9eaae"}
Jul 29 19:30:04.910780 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:04.910708    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kmkl2/must-gather-j7f7n" podStartSLOduration=1.6009782609999998 podStartE2EDuration="2.910681765s" totalImagesPullingTime="1.309703504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:30:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:30:04.908485575 +0000 UTC m=+3369.178254664" watchObservedRunningTime="2026-07-29 19:30:04.910681765 +0000 UTC m=+3369.180450925"
Jul 29 19:30:05.672234 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:05.672202    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-f5w95_faa3a79b-75fb-4890-b169-09674f6d00db/global-pull-secret-syncer/0.log"
Jul 29 19:30:05.951763 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:05.951687    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bp28x_6a62d29d-5bf6-444f-bd28-bf2d1e54ac87/konnectivity-agent/0.log"
Jul 29 19:30:06.075471 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:06.075442    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-152.ec2.internal_8923e83396bb7ee7f5fbbb29dfb80487/haproxy/0.log"
Jul 29 19:30:11.625902 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:11.625869    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-268m5_463080e3-ad6e-4f21-a9b4-c30b44f452c6/limitador/0.log"
Jul 29 19:30:11.725955 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:11.725676    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-lhmrl_bf735020-ded2-4908-ae14-05c16e3c760b/manager/0.log"
Jul 29 19:30:13.600834 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:13.600796    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-jxmfh_c284fbd5-a70f-4085-948c-6a2392502e6f/cluster-monitoring-operator/0.log"
Jul 29 19:30:13.939097 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:13.939014    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v62d6_1d18cddd-00d7-4d19-a31d-574900bef72b/node-exporter/0.log"
Jul 29 19:30:13.964635 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:13.964599    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v62d6_1d18cddd-00d7-4d19-a31d-574900bef72b/kube-rbac-proxy/0.log"
Jul 29 19:30:13.988435 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:13.988409    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v62d6_1d18cddd-00d7-4d19-a31d-574900bef72b/init-textfile/0.log"
Jul 29 19:30:14.379123 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:14.379093    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-6vgm5_c307b2b5-1ce6-4f9f-9358-393654020bc9/prometheus-operator-admission-webhook/0.log"
Jul 29 19:30:15.588919 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:15.588877    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kmkl2/perf-node-gather-daemonset-jhwsn"]
Jul 29 19:30:15.590666 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:15.590631    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f8b52ec-3767-450e-a10d-6e8ca991b781" containerName="cleanup"
Jul 29 19:30:15.590666 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:15.590659    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f8b52ec-3767-450e-a10d-6e8ca991b781" containerName="cleanup"
Jul 29 19:30:15.590876 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:15.590762    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f8b52ec-3767-450e-a10d-6e8ca991b781" containerName="cleanup"
Jul 29 19:30:15.599058 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:15.599030    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-jhwsn"
Jul 29 19:30:15.601889 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:15.601775    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kmkl2\"/\"default-dockercfg-qtdw2\""
Jul 29 19:30:15.608579 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:15.608552    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kmkl2/perf-node-gather-daemonset-jhwsn"]
Jul 29 19:30:15.697005 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:15.696935    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/860e69ed-fddf-41c9-86d9-623e023ffa30-proc\") pod \"perf-node-gather-daemonset-jhwsn\" (UID: \"860e69ed-fddf-41c9-86d9-623e023ffa30\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-jhwsn"
Jul 29 19:30:15.697203 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:15.697152    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5flb\" (UniqueName: \"kubernetes.io/projected/860e69ed-fddf-41c9-86d9-623e023ffa30-kube-api-access-q5flb\") pod \"perf-node-gather-daemonset-jhwsn\" (UID: \"860e69ed-fddf-41c9-86d9-623e023ffa30\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-jhwsn"
Jul 29 19:30:15.697270 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:15.697212    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/860e69ed-fddf-41c9-86d9-623e023ffa30-lib-modules\") pod \"perf-node-gather-daemonset-jhwsn\" (UID: \"860e69ed-fddf-41c9-86d9-623e023ffa30\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-jhwsn"
Jul 29 19:30:15.697270 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:15.697247    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/860e69ed-fddf-41c9-86d9-623e023ffa30-podres\") pod \"perf-node-gather-daemonset-jhwsn\" (UID: \"860e69ed-fddf-41c9-86d9-623e023ffa30\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-jhwsn"
Jul 29 19:30:15.697422 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:15.697368    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/860e69ed-fddf-41c9-86d9-623e023ffa30-sys\") pod \"perf-node-gather-daemonset-jhwsn\" (UID: \"860e69ed-fddf-41c9-86d9-623e023ffa30\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-jhwsn"
Jul 29 19:30:15.792002 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:15.791941    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-rxng5_b2274e5f-4130-477a-900f-915070daeb14/networking-console-plugin/0.log"
Jul 29 19:30:15.797834 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:15.797803    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/860e69ed-fddf-41c9-86d9-623e023ffa30-proc\") pod \"perf-node-gather-daemonset-jhwsn\" (UID: \"860e69ed-fddf-41c9-86d9-623e023ffa30\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-jhwsn"
Jul 29 19:30:15.798004 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:15.797857    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5flb\" (UniqueName: \"kubernetes.io/projected/860e69ed-fddf-41c9-86d9-623e023ffa30-kube-api-access-q5flb\") pod \"perf-node-gather-daemonset-jhwsn\" (UID: \"860e69ed-fddf-41c9-86d9-623e023ffa30\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-jhwsn"
Jul 29 19:30:15.798004 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:15.797944    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/860e69ed-fddf-41c9-86d9-623e023ffa30-proc\") pod \"perf-node-gather-daemonset-jhwsn\" (UID: \"860e69ed-fddf-41c9-86d9-623e023ffa30\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-jhwsn"
Jul 29 19:30:15.798004 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:15.797972    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/860e69ed-fddf-41c9-86d9-623e023ffa30-lib-modules\") pod \"perf-node-gather-daemonset-jhwsn\" (UID: \"860e69ed-fddf-41c9-86d9-623e023ffa30\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-jhwsn"
Jul 29 19:30:15.798004 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:15.797999    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/860e69ed-fddf-41c9-86d9-623e023ffa30-podres\") pod \"perf-node-gather-daemonset-jhwsn\" (UID: \"860e69ed-fddf-41c9-86d9-623e023ffa30\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-jhwsn"
Jul 29 19:30:15.798238 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:15.798057    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/860e69ed-fddf-41c9-86d9-623e023ffa30-sys\") pod \"perf-node-gather-daemonset-jhwsn\" (UID: \"860e69ed-fddf-41c9-86d9-623e023ffa30\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-jhwsn"
Jul 29 19:30:15.798238 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:15.798159    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/860e69ed-fddf-41c9-86d9-623e023ffa30-lib-modules\") pod \"perf-node-gather-daemonset-jhwsn\" (UID: \"860e69ed-fddf-41c9-86d9-623e023ffa30\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-jhwsn"
Jul 29 19:30:15.798238 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:15.798166    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/860e69ed-fddf-41c9-86d9-623e023ffa30-sys\") pod \"perf-node-gather-daemonset-jhwsn\" (UID: \"860e69ed-fddf-41c9-86d9-623e023ffa30\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-jhwsn"
Jul 29 19:30:15.798238 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:15.798188    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/860e69ed-fddf-41c9-86d9-623e023ffa30-podres\") pod \"perf-node-gather-daemonset-jhwsn\" (UID: \"860e69ed-fddf-41c9-86d9-623e023ffa30\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-jhwsn"
Jul 29 19:30:15.806103 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:15.806065    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5flb\" (UniqueName: \"kubernetes.io/projected/860e69ed-fddf-41c9-86d9-623e023ffa30-kube-api-access-q5flb\") pod \"perf-node-gather-daemonset-jhwsn\" (UID: \"860e69ed-fddf-41c9-86d9-623e023ffa30\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-jhwsn"
Jul 29 19:30:15.913804 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:15.913712    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-jhwsn"
Jul 29 19:30:16.106926 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:16.105576    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kmkl2/perf-node-gather-daemonset-jhwsn"]
Jul 29 19:30:16.447023 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:16.446935    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-v9nsr_7f8b4737-d4c2-4598-bd40-208015542831/console-operator/1.log"
Jul 29 19:30:16.454308 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:16.454282    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-v9nsr_7f8b4737-d4c2-4598-bd40-208015542831/console-operator/2.log"
Jul 29 19:30:16.978825 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:16.978788    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-jhwsn" event={"ID":"860e69ed-fddf-41c9-86d9-623e023ffa30","Type":"ContainerStarted","Data":"3ffa8409609ea0fbab909d59751a144f8a574ba21e46efa2d74f20f841f7bc3e"}
Jul 29 19:30:16.979314 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:16.978849    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-jhwsn" event={"ID":"860e69ed-fddf-41c9-86d9-623e023ffa30","Type":"ContainerStarted","Data":"f51ce437cf07a1de6ae9a03f76692eaaeca8ecc7493995de48b89a40dbd63e55"}
Jul 29 19:30:16.979314 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:16.978883    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-jhwsn"
Jul 29 19:30:16.998039 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:16.997989    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-jhwsn" podStartSLOduration=1.997967873 podStartE2EDuration="1.997967873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:30:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:30:16.996627669 +0000 UTC m=+3381.266396780" watchObservedRunningTime="2026-07-29 19:30:16.997967873 +0000 UTC m=+3381.267736962"
Jul 29 19:30:16.998279 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:16.998018    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-24vj5_26035b26-9c55-49cf-a2e9-48634f8c5cd5/download-server/0.log"
Jul 29 19:30:17.625842 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:17.625815    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-ljn4r_4b692565-aef0-416f-8fad-2c74bca55d4d/volume-data-source-validator/0.log"
Jul 29 19:30:18.736168 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:18.736140    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-95vjs_c854cbbc-3a2c-46ca-adf3-86fc3028bd7c/dns/0.log"
Jul 29 19:30:18.762476 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:18.762447    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-95vjs_c854cbbc-3a2c-46ca-adf3-86fc3028bd7c/kube-rbac-proxy/0.log"
Jul 29 19:30:18.869065 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:18.869038    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t4wjl_b03a3682-95be-4852-b28a-8e19fee4cbe7/dns-node-resolver/0.log"
Jul 29 19:30:19.504616 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:19.504588    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ldlwt_2201b8bf-575c-4958-b909-29ff288ed75e/node-ca/0.log"
Jul 29 19:30:21.046282 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:21.046255    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7688c5db95-gcqb5_66a293fb-92cb-4908-bf90-d97ed0d831a6/router/0.log"
Jul 29 19:30:21.927899 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:21.927870    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qhj2h_cc87b3c5-0b72-4fa2-9f87-ca3d4d6da614/serve-healthcheck-canary/0.log"
Jul 29 19:30:22.425129 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:22.425101    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-j76ql_a84d718f-459d-4e2e-9c9e-d08d1500b145/insights-operator/0.log"
Jul 29 19:30:22.426660 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:22.426633    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-j76ql_a84d718f-459d-4e2e-9c9e-d08d1500b145/insights-operator/1.log"
Jul 29 19:30:22.648885 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:22.648859    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wpg7b_6c9d825b-217d-499b-b8a4-b0dbf396b283/kube-rbac-proxy/0.log"
Jul 29 19:30:22.675984 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:22.675908    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wpg7b_6c9d825b-217d-499b-b8a4-b0dbf396b283/exporter/0.log"
Jul 29 19:30:22.705709 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:22.705685    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wpg7b_6c9d825b-217d-499b-b8a4-b0dbf396b283/extractor/0.log"
Jul 29 19:30:23.000300 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:22.999191    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-jhwsn"
Jul 29 19:30:25.417084 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:25.417054    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-b4kv2_7fbb1c34-1fcb-4b7b-bd0f-b2c2e0502bba/manager/0.log"
Jul 29 19:30:25.724121 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:25.724048    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-7gxpk_2b524f44-0875-4793-80e3-5886ee9bf148/manager/0.log"
Jul 29 19:30:26.337314 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:26.337286    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76c6f8f9b-qn29c_9e820933-bb87-4c58-b7aa-0cb939217b29/manager/0.log"
Jul 29 19:30:26.383393 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:26.383356    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76c6f8f9b-qn29c_9e820933-bb87-4c58-b7aa-0cb939217b29/copy-manifests/0.log"
Jul 29 19:30:26.578243 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:26.578212    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7688c5db95-gcqb5_66a293fb-92cb-4908-bf90-d97ed0d831a6/router/0.log"
Jul 29 19:30:27.688533 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:27.688504    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5b7494b68d-8mvm4_2ebb787a-5d11-4a08-89dc-335737c2c9d5/manager/0.log"
Jul 29 19:30:28.406124 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:28.406083    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7688c5db95-gcqb5_66a293fb-92cb-4908-bf90-d97ed0d831a6/router/0.log"
Jul 29 19:30:32.454246 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:32.454214    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-j5wbh_32e5eb71-45e9-4586-81ea-f1d56935a083/migrator/0.log"
Jul 29 19:30:32.482942 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:32.482915    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-j5wbh_32e5eb71-45e9-4586-81ea-f1d56935a083/graceful-termination/0.log"
Jul 29 19:30:32.836963 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:32.836919    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-nj275_05f777ff-5b55-41e4-b440-1705c2c3842e/kube-storage-version-migrator-operator/1.log"
Jul 29 19:30:32.837913 ip-10-0-138-152 kubenswrapper[2559]: I0729 19:30:32.837893    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-nj275_05f777ff-5b55-41e4-b440-1705c2c3842e/kube-storage-version-migrator-operator/0.log"