Jul 24 15:06:24.649335 ip-10-0-140-160 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 24 15:06:24.649349 ip-10-0-140-160 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 24 15:06:24.649359 ip-10-0-140-160 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 24 15:06:24.649698 ip-10-0-140-160 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 24 15:06:34.898838 ip-10-0-140-160 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 24 15:06:34.898860 ip-10-0-140-160 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot a1598c16ef9d4ef5aaff920e6ea39491 --
Jul 24 15:08:27.075076 ip-10-0-140-160 systemd[1]: Starting Kubernetes Kubelet...
Jul 24 15:08:27.461701 ip-10-0-140-160 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 15:08:27.461701 ip-10-0-140-160 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 24 15:08:27.461701 ip-10-0-140-160 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 15:08:27.461701 ip-10-0-140-160 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 24 15:08:27.461701 ip-10-0-140-160 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 15:08:27.462889 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.462795    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 24 15:08:27.469584 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469562    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 15:08:27.469584 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469579    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 15:08:27.469584 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469586    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 15:08:27.469800 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469593    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 15:08:27.469800 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469598    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 15:08:27.469800 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469602    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 15:08:27.469800 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469606    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 15:08:27.469800 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469611    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 15:08:27.469800 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469614    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 15:08:27.469800 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469618    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 15:08:27.469800 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469622    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 15:08:27.469800 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469626    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 15:08:27.469800 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469630    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 15:08:27.469800 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469634    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 15:08:27.469800 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469638    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 15:08:27.469800 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469641    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 15:08:27.469800 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469645    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 15:08:27.469800 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469649    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 15:08:27.469800 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469653    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 15:08:27.469800 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469657    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 15:08:27.469800 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469661    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 15:08:27.469800 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469665    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 15:08:27.469800 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469669    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 15:08:27.470680 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469674    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 15:08:27.470680 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469678    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 15:08:27.470680 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469681    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 15:08:27.470680 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469691    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 15:08:27.470680 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469695    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 15:08:27.470680 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469698    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 15:08:27.470680 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469702    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 15:08:27.470680 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469706    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 15:08:27.470680 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469710    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 15:08:27.470680 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469714    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 15:08:27.470680 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469719    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 15:08:27.470680 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469723    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 15:08:27.470680 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469727    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 15:08:27.470680 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469733    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 15:08:27.470680 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469737    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 15:08:27.470680 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469742    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 15:08:27.470680 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469746    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 15:08:27.470680 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469751    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 15:08:27.470680 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469755    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 15:08:27.470680 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469759    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 15:08:27.471531 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469763    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 15:08:27.471531 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469768    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 15:08:27.471531 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469772    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 15:08:27.471531 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469777    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 15:08:27.471531 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469781    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 15:08:27.471531 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469785    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 15:08:27.471531 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469789    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 15:08:27.471531 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469796    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 15:08:27.471531 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469800    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 15:08:27.471531 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469804    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 15:08:27.471531 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469809    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 15:08:27.471531 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469829    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 15:08:27.471531 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469834    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 15:08:27.471531 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469838    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 15:08:27.471531 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469843    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 15:08:27.471531 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469847    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 15:08:27.471531 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469851    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 15:08:27.471531 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469855    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 15:08:27.471531 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469860    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 15:08:27.471531 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469864    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 15:08:27.472382 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469868    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 15:08:27.472382 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469872    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 15:08:27.472382 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469876    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 15:08:27.472382 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469881    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 15:08:27.472382 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469885    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 15:08:27.472382 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469890    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 15:08:27.472382 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469897    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 15:08:27.472382 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469904    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 15:08:27.472382 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469909    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 15:08:27.472382 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469914    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 15:08:27.472382 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469919    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 15:08:27.472382 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469923    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 15:08:27.472382 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469927    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 15:08:27.472382 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469931    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 15:08:27.472382 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469936    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 15:08:27.472382 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469940    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 15:08:27.472382 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469944    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 15:08:27.472382 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469948    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 15:08:27.472382 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469952    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 15:08:27.473222 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469957    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 15:08:27.473222 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469962    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 15:08:27.473222 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469971    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 15:08:27.473222 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469977    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 15:08:27.473222 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469982    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 15:08:27.473222 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469987    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 15:08:27.473222 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469992    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 15:08:27.473222 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.469996    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 15:08:27.473222 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.470000    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 15:08:27.473222 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.470004    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 15:08:27.473222 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.470008    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 15:08:27.473222 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.470012    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 15:08:27.473222 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.470017    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 15:08:27.473222 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.470021    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 15:08:27.473222 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.470026    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 15:08:27.473222 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.470030    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 15:08:27.473222 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.470035    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 15:08:27.473222 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471197    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 15:08:27.473222 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471209    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 15:08:27.474060 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471214    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 15:08:27.474060 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471218    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 15:08:27.474060 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471222    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 15:08:27.474060 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471227    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 15:08:27.474060 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471231    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 15:08:27.474060 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471235    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 15:08:27.474060 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471239    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 15:08:27.474060 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471244    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 15:08:27.474060 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471248    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 15:08:27.474060 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471253    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 15:08:27.474060 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471257    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 15:08:27.474060 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471261    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 15:08:27.474060 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471265    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 15:08:27.474060 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471269    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 15:08:27.474060 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471274    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 15:08:27.474060 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471280    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 15:08:27.474060 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471284    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 15:08:27.474060 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471289    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 15:08:27.474060 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471293    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 15:08:27.474060 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471298    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 15:08:27.474716 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471302    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 15:08:27.474716 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471306    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 15:08:27.474716 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471310    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 15:08:27.474716 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471314    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 15:08:27.474716 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471318    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 15:08:27.474716 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471323    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 15:08:27.474716 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471327    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 15:08:27.474716 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471331    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 15:08:27.474716 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471335    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 15:08:27.474716 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471340    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 15:08:27.474716 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471344    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 15:08:27.474716 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471351    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 15:08:27.474716 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471358    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 15:08:27.474716 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471363    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 15:08:27.474716 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471367    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 15:08:27.474716 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471372    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 15:08:27.474716 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471377    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 15:08:27.474716 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471381    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 15:08:27.474716 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471386    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 15:08:27.475216 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471390    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 15:08:27.475216 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471395    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 15:08:27.475216 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471400    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 15:08:27.475216 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471404    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 15:08:27.475216 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471409    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 15:08:27.475216 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471413    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 15:08:27.475216 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471418    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 15:08:27.475216 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471423    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 15:08:27.475216 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471427    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 15:08:27.475216 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471432    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 15:08:27.475216 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471438    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 15:08:27.475216 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471443    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 15:08:27.475216 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471448    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 15:08:27.475216 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471452    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 15:08:27.475216 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471456    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 15:08:27.475216 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471460    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 15:08:27.475216 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471464    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 15:08:27.475216 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471469    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 15:08:27.475216 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471473    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 15:08:27.475754 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471477    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 15:08:27.475754 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471481    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 15:08:27.475754 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471486    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 15:08:27.475754 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471491    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 15:08:27.475754 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471495    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 15:08:27.475754 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471501    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 15:08:27.475754 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471505    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 15:08:27.475754 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471510    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 15:08:27.475754 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471515    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 15:08:27.475754 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471519    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 15:08:27.475754 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471524    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 15:08:27.475754 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471527    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 15:08:27.475754 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471532    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 15:08:27.475754 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471538    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 15:08:27.475754 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471542    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 15:08:27.475754 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471547    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 15:08:27.475754 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471551    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 15:08:27.475754 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471555    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 15:08:27.475754 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471559    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 15:08:27.475754 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471563    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 15:08:27.476519 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471568    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 15:08:27.476519 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471572    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 15:08:27.476519 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471577    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 15:08:27.476519 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471582    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 15:08:27.476519 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471586    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 15:08:27.476519 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471593    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 15:08:27.476519 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471597    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 15:08:27.476519 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471601    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 15:08:27.476519 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471605    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 15:08:27.476519 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471609    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 15:08:27.476519 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471614    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 15:08:27.476519 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471618    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 15:08:27.476519 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471622    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 15:08:27.476519 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471627    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 15:08:27.476519 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471632    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 15:08:27.476519 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471636    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 15:08:27.476519 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471640    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 15:08:27.476519 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471644    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 15:08:27.476519 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.471649    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 15:08:27.476519 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472249    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 15:08:27.477044 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472258    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 15:08:27.477044 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472262    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 15:08:27.477044 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472266    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 15:08:27.477044 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472271    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 15:08:27.477044 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472276    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 15:08:27.477044 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472280    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 15:08:27.477044 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472284    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 15:08:27.477044 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472288    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 15:08:27.477044 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472292    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 15:08:27.477044 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472296    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 15:08:27.477044 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472301    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 15:08:27.477044 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472306    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 15:08:27.477044 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472311    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 15:08:27.477044 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472315    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 15:08:27.477044 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472319    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 15:08:27.477044 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472325    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 15:08:27.477044 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472330    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 15:08:27.477044 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472334    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 15:08:27.477044 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472338    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 15:08:27.477044 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472342    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 15:08:27.477558 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472346    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 15:08:27.477558 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472351    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 15:08:27.477558 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472355    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 15:08:27.477558 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472359    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 15:08:27.477558 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472363    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 15:08:27.477558 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472368    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 15:08:27.477558 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472372    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 15:08:27.477558 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472376    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 15:08:27.477558 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472380    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 15:08:27.477558 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472384    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 15:08:27.477558 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472388    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 15:08:27.477558 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472392    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 15:08:27.477558 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472396    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 15:08:27.477558 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472400    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 15:08:27.477558 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472404    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 15:08:27.477558 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472408    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 15:08:27.477558 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472412    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 15:08:27.477558 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472416    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 15:08:27.477558 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472421    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 15:08:27.477558 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472425    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 15:08:27.478184 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472429    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 15:08:27.478184 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472433    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 15:08:27.478184 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472438    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 15:08:27.478184 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472442    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 15:08:27.478184 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472446    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 15:08:27.478184 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472450    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 15:08:27.478184 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472454    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 15:08:27.478184 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472458    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 15:08:27.478184 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472463    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 15:08:27.478184 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472472    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 15:08:27.478184 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472477    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 15:08:27.478184 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472482    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 15:08:27.478184 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472486    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 15:08:27.478184 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472491    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 15:08:27.478184 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472496    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 15:08:27.478184 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472501    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 15:08:27.478184 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472506    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 15:08:27.478184 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472511    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 15:08:27.478184 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472516    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 15:08:27.478657 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472520    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 15:08:27.478657 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472525    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 15:08:27.478657 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472529    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 15:08:27.478657 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472534    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 15:08:27.478657 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472538    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 15:08:27.478657 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472543    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 15:08:27.478657 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472547    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 15:08:27.478657 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472551    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 15:08:27.478657 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472558    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 15:08:27.478657 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472563    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 15:08:27.478657 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472568    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 15:08:27.478657 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472572    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 15:08:27.478657 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472578    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 15:08:27.478657 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472583    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 15:08:27.478657 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472587    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 15:08:27.478657 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472591    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 15:08:27.478657 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472596    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 15:08:27.478657 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472600    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 15:08:27.478657 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472604    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 15:08:27.479131 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472609    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 15:08:27.479131 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472613    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 15:08:27.479131 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472616    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 15:08:27.479131 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472622    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 15:08:27.479131 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472638    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 15:08:27.479131 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472644    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 15:08:27.479131 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472648    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 15:08:27.479131 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472653    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 15:08:27.479131 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472657    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 15:08:27.479131 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472661    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 15:08:27.479131 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472665    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 15:08:27.479131 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472670    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 15:08:27.479131 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472674    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 15:08:27.479131 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472679    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 15:08:27.479131 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472683    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 15:08:27.479131 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472687    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 15:08:27.479131 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472691    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 15:08:27.479131 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472695    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 15:08:27.479131 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472699    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 15:08:27.479131 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.472703    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 15:08:27.479641 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.472799    2579 flags.go:64] FLAG: --address="0.0.0.0"
Jul 24 15:08:27.479641 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.472809    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 24 15:08:27.479641 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.472834    2579 flags.go:64] FLAG: --anonymous-auth="true"
Jul 24 15:08:27.479641 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.472841    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 24 15:08:27.479641 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.472847    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 24 15:08:27.479641 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.472852    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 24 15:08:27.479641 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.472858    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 24 15:08:27.479641 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.472864    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 24 15:08:27.479641 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.472869    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 24 15:08:27.479641 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.472874    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 24 15:08:27.479641 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.472879    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 24 15:08:27.479641 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.472885    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 24 15:08:27.479641 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.472890    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 24 15:08:27.479641 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.472894    2579 flags.go:64] FLAG: --cgroup-root=""
Jul 24 15:08:27.479641 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.472899    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 24 15:08:27.479641 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.472904    2579 flags.go:64] FLAG: --client-ca-file=""
Jul 24 15:08:27.479641 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.472910    2579 flags.go:64] FLAG: --cloud-provider="external"
Jul 24 15:08:27.479641 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.472915    2579 flags.go:64] FLAG: --cluster-dns="[]"
Jul 24 15:08:27.479641 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.472921    2579 flags.go:64] FLAG: --cluster-domain=""
Jul 24 15:08:27.479641 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.472925    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 24 15:08:27.479641 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.472930    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 24 15:08:27.479641 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.472935    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 24 15:08:27.479641 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.472941    2579 flags.go:64] FLAG: --container-log-max-files="5"
Jul 24 15:08:27.479641 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.472947    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 24 15:08:27.480243 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.472955    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 24 15:08:27.480243 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.472960    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 24 15:08:27.480243 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.472965    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 24 15:08:27.480243 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.472970    2579 flags.go:64] FLAG: --contention-profiling="false"
Jul 24 15:08:27.480243 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.472975    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 24 15:08:27.480243 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.472979    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 24 15:08:27.480243 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.472984    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 24 15:08:27.480243 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.472989    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 24 15:08:27.480243 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.472995    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 24 15:08:27.480243 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473000    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 24 15:08:27.480243 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473005    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 24 15:08:27.480243 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473009    2579 flags.go:64] FLAG: --enable-load-reader="false"
Jul 24 15:08:27.480243 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473014    2579 flags.go:64] FLAG: --enable-server="true"
Jul 24 15:08:27.480243 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473018    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 24 15:08:27.480243 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473025    2579 flags.go:64] FLAG: --event-burst="100"
Jul 24 15:08:27.480243 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473030    2579 flags.go:64] FLAG: --event-qps="50"
Jul 24 15:08:27.480243 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473034    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 24 15:08:27.480243 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473039    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 24 15:08:27.480243 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473044    2579 flags.go:64] FLAG: --eviction-hard=""
Jul 24 15:08:27.480243 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473050    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 24 15:08:27.480243 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473054    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 24 15:08:27.480243 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473059    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 24 15:08:27.480243 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473064    2579 flags.go:64] FLAG: --eviction-soft=""
Jul 24 15:08:27.480243 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473069    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 24 15:08:27.480243 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473074    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 24 15:08:27.480939 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473079    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 24 15:08:27.480939 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473084    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 24 15:08:27.480939 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473088    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 24 15:08:27.480939 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473100    2579 flags.go:64] FLAG: --fail-swap-on="true"
Jul 24 15:08:27.480939 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473105    2579 flags.go:64] FLAG: --feature-gates=""
Jul 24 15:08:27.480939 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473111    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 24 15:08:27.480939 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473116    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 24 15:08:27.480939 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473124    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 24 15:08:27.480939 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473130    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 24 15:08:27.480939 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473134    2579 flags.go:64] FLAG: --healthz-port="10248"
Jul 24 15:08:27.480939 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473139    2579 flags.go:64] FLAG: --help="false"
Jul 24 15:08:27.480939 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473144    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-140-160.ec2.internal"
Jul 24 15:08:27.480939 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473149    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 24 15:08:27.480939 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473154    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 24 15:08:27.480939 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473159    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 24 15:08:27.480939 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473164    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 24 15:08:27.480939 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473170    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 24 15:08:27.480939 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473175    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 24 15:08:27.480939 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473179    2579 flags.go:64] FLAG: --image-service-endpoint=""
Jul 24 15:08:27.480939 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473183    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 24 15:08:27.480939 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473188    2579 flags.go:64] FLAG: --kube-api-burst="100"
Jul 24 15:08:27.480939 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473193    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 24 15:08:27.480939 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473199    2579 flags.go:64] FLAG: --kube-api-qps="50"
Jul 24 15:08:27.480939 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473203    2579 flags.go:64] FLAG: --kube-reserved=""
Jul 24 15:08:27.481532 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473208    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 24 15:08:27.481532 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473214    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 24 15:08:27.481532 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473219    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 24 15:08:27.481532 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473223    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 24 15:08:27.481532 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473228    2579 flags.go:64] FLAG: --lock-file=""
Jul 24 15:08:27.481532 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473232    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 24 15:08:27.481532 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473237    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 24 15:08:27.481532 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473241    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 24 15:08:27.481532 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473251    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 24 15:08:27.481532 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473257    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 24 15:08:27.481532 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473262    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 24 15:08:27.481532 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473267    2579 flags.go:64] FLAG: --logging-format="text"
Jul 24 15:08:27.481532 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473271    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 24 15:08:27.481532 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473277    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 24 15:08:27.481532 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473281    2579 flags.go:64] FLAG: --manifest-url=""
Jul 24 15:08:27.481532 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473287    2579 flags.go:64] FLAG: --manifest-url-header=""
Jul 24 15:08:27.481532 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473294    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 24 15:08:27.481532 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473299    2579 flags.go:64] FLAG: --max-open-files="1000000"
Jul 24 15:08:27.481532 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473306    2579 flags.go:64] FLAG: --max-pods="110"
Jul 24 15:08:27.481532 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473310    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 24 15:08:27.481532 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473315    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 24 15:08:27.481532 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473320    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 24 15:08:27.481532 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473324    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 24 15:08:27.481532 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473329    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 24 15:08:27.481532 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473334    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 24 15:08:27.482138 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473339    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 24 15:08:27.482138 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473350    2579 flags.go:64] FLAG: --node-status-max-images="50"
Jul 24 15:08:27.482138 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473355    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 24 15:08:27.482138 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473359    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 24 15:08:27.482138 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473364    2579 flags.go:64] FLAG: --pod-cidr=""
Jul 24 15:08:27.482138 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473369    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 24 15:08:27.482138 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473376    2579 flags.go:64] FLAG: --pod-manifest-path=""
Jul 24 15:08:27.482138 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473381    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 24 15:08:27.482138 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473388    2579 flags.go:64] FLAG: --pods-per-core="0"
Jul 24 15:08:27.482138 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473393    2579 flags.go:64] FLAG: --port="10250"
Jul 24 15:08:27.482138 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473397    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 24 15:08:27.482138 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473402    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00a433920295a720c"
Jul 24 15:08:27.482138 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473408    2579 flags.go:64] FLAG: --qos-reserved=""
Jul 24 15:08:27.482138 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473412    2579 flags.go:64] FLAG: --read-only-port="10255"
Jul 24 15:08:27.482138 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473417    2579 flags.go:64] FLAG: --register-node="true"
Jul 24 15:08:27.482138 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473421    2579 flags.go:64] FLAG: --register-with-taints=""
Jul 24 15:08:27.482138 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473427    2579 flags.go:64] FLAG: --registry-burst="10"
Jul 24 15:08:27.482138 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473433    2579 flags.go:64] FLAG: --registry-qps="5"
Jul 24 15:08:27.482138 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473438    2579 flags.go:64] FLAG: --reserved-cpus=""
Jul 24 15:08:27.482138 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473442    2579 flags.go:64] FLAG: --reserved-memory=""
Jul 24 15:08:27.482138 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473448    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 24 15:08:27.482138 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473453    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 24 15:08:27.482138 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473458    2579 flags.go:64] FLAG: --rotate-certificates="false"
Jul 24 15:08:27.482138 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473464    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 24 15:08:27.482138 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473469    2579 flags.go:64] FLAG: --runonce="false"
Jul 24 15:08:27.482766 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473474    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 24 15:08:27.482766 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473479    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 24 15:08:27.482766 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473484    2579 flags.go:64] FLAG: --seccomp-default="false"
Jul 24 15:08:27.482766 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473488    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 24 15:08:27.482766 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473493    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 24 15:08:27.482766 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473498    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 24 15:08:27.482766 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473503    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 24 15:08:27.482766 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473507    2579 flags.go:64] FLAG: --storage-driver-password="root"
Jul 24 15:08:27.482766 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473512    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 24 15:08:27.482766 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473517    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 24 15:08:27.482766 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473521    2579 flags.go:64] FLAG: --storage-driver-user="root"
Jul 24 15:08:27.482766 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473526    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 24 15:08:27.482766 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473531    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 24 15:08:27.482766 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473536    2579 flags.go:64] FLAG: --system-cgroups=""
Jul 24 15:08:27.482766 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473541    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 24 15:08:27.482766 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473551    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 24 15:08:27.482766 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473556    2579 flags.go:64] FLAG: --tls-cert-file=""
Jul 24 15:08:27.482766 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473560    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 24 15:08:27.482766 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473567    2579 flags.go:64] FLAG: --tls-min-version=""
Jul 24 15:08:27.482766 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473571    2579 flags.go:64] FLAG: --tls-private-key-file=""
Jul 24 15:08:27.482766 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473576    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 24 15:08:27.482766 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473580    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 24 15:08:27.482766 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473585    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 24 15:08:27.482766 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473590    2579 flags.go:64] FLAG: --v="2"
Jul 24 15:08:27.482766 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473596    2579 flags.go:64] FLAG: --version="false"
Jul 24 15:08:27.483391 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473604    2579 flags.go:64] FLAG: --vmodule=""
Jul 24 15:08:27.483391 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473610    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 24 15:08:27.483391 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.473615    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 24 15:08:27.483391 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.473791    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 15:08:27.483391 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.473799    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 15:08:27.483391 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.473805    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 15:08:27.483391 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.473810    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 15:08:27.483391 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.473833    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 15:08:27.483391 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.473837    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 15:08:27.483391 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.473842    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 15:08:27.483391 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.473846    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 15:08:27.483391 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.473850    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 15:08:27.483391 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.473854    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 15:08:27.483391 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.473858    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 15:08:27.483391 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.473862    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 15:08:27.483391 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.473867    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 15:08:27.483391 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.473871    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 15:08:27.483391 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.473875    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 15:08:27.483391 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.473879    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 15:08:27.483391 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.473884    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 15:08:27.483391 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.473888    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 15:08:27.483926 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.473892    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 15:08:27.483926 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.473898    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 15:08:27.483926 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.473902    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 15:08:27.483926 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.473907    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 15:08:27.483926 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.473913    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 15:08:27.483926 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.473919    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 15:08:27.483926 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.473925    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 15:08:27.483926 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.473929    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 15:08:27.483926 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.473933    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 15:08:27.483926 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.473937    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 15:08:27.483926 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.473942    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 15:08:27.483926 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.473946    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 15:08:27.483926 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.473951    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 15:08:27.483926 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.473955    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 15:08:27.483926 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.473960    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 15:08:27.483926 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.473964    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 15:08:27.483926 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.473970    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 15:08:27.483926 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.473974    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 15:08:27.483926 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.473978    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 15:08:27.484393 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.473983    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 15:08:27.484393 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.473987    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 15:08:27.484393 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.473991    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 15:08:27.484393 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.473996    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 15:08:27.484393 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474000    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 15:08:27.484393 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474005    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 15:08:27.484393 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474009    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 15:08:27.484393 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474013    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 15:08:27.484393 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474017    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 15:08:27.484393 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474021    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 15:08:27.484393 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474025    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 15:08:27.484393 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474029    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 15:08:27.484393 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474034    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 15:08:27.484393 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474038    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 15:08:27.484393 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474044    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 15:08:27.484393 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474048    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 15:08:27.484393 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474054    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 15:08:27.484393 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474059    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 15:08:27.484393 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474063    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 15:08:27.484393 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474067    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 15:08:27.484904 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474071    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 15:08:27.484904 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474075    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 15:08:27.484904 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474079    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 15:08:27.484904 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474083    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 15:08:27.484904 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474087    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 15:08:27.484904 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474091    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 15:08:27.484904 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474096    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 15:08:27.484904 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474100    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 15:08:27.484904 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474105    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 15:08:27.484904 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474111    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 15:08:27.484904 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474115    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 15:08:27.484904 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474119    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 15:08:27.484904 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474123    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 15:08:27.484904 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474127    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 15:08:27.484904 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474131    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 15:08:27.484904 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474135    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 15:08:27.484904 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474139    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 15:08:27.484904 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474144    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 15:08:27.484904 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474148    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 15:08:27.485362 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474152    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 15:08:27.485362 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474156    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 15:08:27.485362 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474161    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 15:08:27.485362 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474165    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 15:08:27.485362 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474169    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 15:08:27.485362 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474173    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 15:08:27.485362 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474178    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 15:08:27.485362 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474184    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 15:08:27.485362 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474189    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 15:08:27.485362 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474193    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 15:08:27.485362 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474197    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 15:08:27.485362 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474202    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 15:08:27.485362 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474206    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 15:08:27.485362 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474210    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 15:08:27.485362 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474214    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 15:08:27.485362 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474218    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 15:08:27.485362 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474223    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 15:08:27.485362 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474227    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 15:08:27.485362 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474231    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 15:08:27.485362 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474235    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 15:08:27.485923 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474240    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 15:08:27.485923 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474246    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 15:08:27.485923 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.474252    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 15:08:27.485923 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.475311    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 15:08:27.485923 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.482328    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 24 15:08:27.485923 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.482343    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 24 15:08:27.485923 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.483194    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 24 15:08:27.485923 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.483206    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 24 15:08:27.485923 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483266    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 15:08:27.485923 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483272    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 15:08:27.485923 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483276    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 15:08:27.485923 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483279    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 15:08:27.485923 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483282    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 15:08:27.485923 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483286    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 15:08:27.486271 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483289    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 15:08:27.486271 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483292    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 15:08:27.486271 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483295    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 15:08:27.486271 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483298    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 15:08:27.486271 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483301    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 15:08:27.486271 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483304    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 15:08:27.486271 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483307    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 15:08:27.486271 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483309    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 15:08:27.486271 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483313    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 15:08:27.486271 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483315    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 15:08:27.486271 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483318    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 15:08:27.486271 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483320    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 15:08:27.486271 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483323    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 15:08:27.486271 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483326    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 15:08:27.486271 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483328    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 15:08:27.486271 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483331    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 15:08:27.486271 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483333    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 15:08:27.486271 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483336    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 15:08:27.486271 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483339    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 15:08:27.486271 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483342    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 15:08:27.486763 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483344    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 15:08:27.486763 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483347    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 15:08:27.486763 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483350    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 15:08:27.486763 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483354    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 15:08:27.486763 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483356    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 15:08:27.486763 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483359    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 15:08:27.486763 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483362    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 15:08:27.486763 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483365    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 15:08:27.486763 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483367    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 15:08:27.486763 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483370    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 15:08:27.486763 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483373    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 15:08:27.486763 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483375    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 15:08:27.486763 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483379    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 15:08:27.486763 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483382    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 15:08:27.486763 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483385    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 15:08:27.486763 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483388    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 15:08:27.486763 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483391    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 15:08:27.486763 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483393    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 15:08:27.486763 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483396    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 15:08:27.486763 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483399    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 15:08:27.487269 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483402    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 15:08:27.487269 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483404    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 15:08:27.487269 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483407    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 15:08:27.487269 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483409    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 15:08:27.487269 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483412    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 15:08:27.487269 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483415    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 15:08:27.487269 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483417    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 15:08:27.487269 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483420    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 15:08:27.487269 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483422    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 15:08:27.487269 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483425    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 15:08:27.487269 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483427    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 15:08:27.487269 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483430    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 15:08:27.487269 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483432    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 15:08:27.487269 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483435    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 15:08:27.487269 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483438    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 15:08:27.487269 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483442    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 15:08:27.487269 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483451    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 15:08:27.487269 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483454    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 15:08:27.487269 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483456    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 15:08:27.487269 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483459    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 15:08:27.487748 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483461    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 15:08:27.487748 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483464    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 15:08:27.487748 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483467    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 15:08:27.487748 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483469    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 15:08:27.487748 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483472    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 15:08:27.487748 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483474    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 15:08:27.487748 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483477    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 15:08:27.487748 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483479    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 15:08:27.487748 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483482    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 15:08:27.487748 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483485    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 15:08:27.487748 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483487    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 15:08:27.487748 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483490    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 15:08:27.487748 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483492    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 15:08:27.487748 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483495    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 15:08:27.487748 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483497    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 15:08:27.487748 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483500    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 15:08:27.487748 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483504    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 15:08:27.487748 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483507    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 15:08:27.487748 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483511    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 15:08:27.488224 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483513    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 15:08:27.488224 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483516    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 15:08:27.488224 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483519    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 15:08:27.488224 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483521    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 15:08:27.488224 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483524    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 15:08:27.488224 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483526    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 15:08:27.488224 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483529    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 15:08:27.488224 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483531    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 15:08:27.488224 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483534    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 15:08:27.488224 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483537    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 15:08:27.488224 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483546    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 15:08:27.488224 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483550    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 15:08:27.488224 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483553    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 15:08:27.488224 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483556    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 15:08:27.488567 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.483561    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 15:08:27.488567 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483695    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 15:08:27.488567 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483699    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 15:08:27.488567 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483702    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 15:08:27.488567 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483705    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 15:08:27.488567 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483708    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 15:08:27.488567 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483711    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 15:08:27.488567 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483714    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 15:08:27.488567 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483716    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 15:08:27.488567 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483719    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 15:08:27.488567 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483722    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 15:08:27.488567 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483725    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 15:08:27.488567 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483728    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 15:08:27.488567 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483730    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 15:08:27.488972 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483733    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 15:08:27.488972 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483736    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 15:08:27.488972 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483738    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 15:08:27.488972 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483741    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 15:08:27.488972 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483743    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 15:08:27.488972 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483746    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 15:08:27.488972 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483748    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 15:08:27.488972 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483751    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 15:08:27.488972 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483753    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 15:08:27.488972 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483756    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 15:08:27.488972 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483758    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 15:08:27.488972 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483761    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 15:08:27.488972 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483764    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 15:08:27.488972 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483766    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 15:08:27.488972 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483768    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 15:08:27.488972 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483776    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 15:08:27.488972 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483779    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 15:08:27.488972 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483781    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 15:08:27.488972 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483784    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 15:08:27.488972 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483786    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 15:08:27.489469 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483789    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 15:08:27.489469 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483791    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 15:08:27.489469 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483794    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 15:08:27.489469 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483796    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 15:08:27.489469 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483799    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 15:08:27.489469 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483802    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 15:08:27.489469 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483805    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 15:08:27.489469 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483808    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 15:08:27.489469 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483810    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 15:08:27.489469 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483826    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 15:08:27.489469 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483830    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 15:08:27.489469 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483836    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 15:08:27.489469 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483841    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 15:08:27.489469 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483843    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 15:08:27.489469 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483846    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 15:08:27.489469 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483849    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 15:08:27.489469 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483852    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 15:08:27.489469 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483854    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 15:08:27.489469 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483857    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 15:08:27.489944 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483859    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 15:08:27.489944 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483862    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 15:08:27.489944 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483864    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 15:08:27.489944 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483867    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 15:08:27.489944 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483870    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 15:08:27.489944 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483872    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 15:08:27.489944 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483875    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 15:08:27.489944 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483877    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 15:08:27.489944 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483880    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 15:08:27.489944 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483890    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 15:08:27.489944 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483893    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 15:08:27.489944 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483896    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 15:08:27.489944 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483898    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 15:08:27.489944 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483900    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 15:08:27.489944 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483903    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 15:08:27.489944 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483905    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 15:08:27.489944 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483908    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 15:08:27.489944 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483910    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 15:08:27.489944 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483913    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 15:08:27.489944 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483915    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 15:08:27.490425 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483918    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 15:08:27.490425 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483921    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 15:08:27.490425 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483923    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 15:08:27.490425 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483925    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 15:08:27.490425 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483928    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 15:08:27.490425 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483930    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 15:08:27.490425 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483933    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 15:08:27.490425 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483936    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 15:08:27.490425 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483938    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 15:08:27.490425 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483941    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 15:08:27.490425 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483943    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 15:08:27.490425 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483946    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 15:08:27.490425 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483948    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 15:08:27.490425 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483951    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 15:08:27.490425 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483953    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 15:08:27.490425 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483956    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 15:08:27.490425 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483959    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 15:08:27.490425 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483962    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 15:08:27.490425 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483964    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 15:08:27.490425 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483966    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 15:08:27.490929 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483969    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 15:08:27.490929 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483971    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 15:08:27.490929 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483979    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 15:08:27.490929 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483982    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 15:08:27.490929 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483986    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 15:08:27.490929 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483989    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 15:08:27.490929 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:27.483992    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 15:08:27.490929 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.483997    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 15:08:27.490929 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.484109    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 24 15:08:27.490929 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.486194    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 24 15:08:27.490929 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.487376    2579 server.go:1019] "Starting client certificate rotation"
Jul 24 15:08:27.490929 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.487475    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 15:08:27.490929 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.487505    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 15:08:27.491258 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.489807    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 15:08:27.491416 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.491381    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 15:08:27.503620 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.503574    2579 log.go:25] "Validated CRI v1 runtime API"
Jul 24 15:08:27.506024 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.506007    2579 log.go:25] "Validated CRI v1 image API"
Jul 24 15:08:27.506867 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.506851    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 24 15:08:27.510016 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.509997    2579 fs.go:135] Filesystem UUIDs: map[22a9721b-0b43-4b72-acb6-e1a5011895a2:/dev/nvme0n1p4 2b52df4e-cc8d-4ad7-b4a9-91f36a1d8b4b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 24 15:08:27.510090 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.510015    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 24 15:08:27.516059 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.515923    2579 manager.go:216] Machine: {Timestamp:2026-07-24 15:08:27.514554247 +0000 UTC m=+0.326906418 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100138 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29e77a22961d1fd4b59c694c5ce12c SystemUUID:ec29e77a-2296-1d1f-d4b5-9c694c5ce12c BootID:a1598c16-ef9d-4ef5-aaff-920e6ea39491 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:96:2f:65:40:b7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:96:2f:65:40:b7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:52:79:f6:76:f9:41 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 24 15:08:27.516059 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.516050    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 24 15:08:27.516190 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.516120    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 24 15:08:27.518966 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.518931    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 24 15:08:27.519107 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.518967    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-160.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 24 15:08:27.519640 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.519628    2579 topology_manager.go:138] "Creating topology manager with none policy"
Jul 24 15:08:27.519721 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.519642    2579 container_manager_linux.go:309] "Creating device plugin manager"
Jul 24 15:08:27.519721 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.519650    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 15:08:27.520416 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.520404    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 15:08:27.521329 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.521318    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 24 15:08:27.521618 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.521602    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 15:08:27.523149 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.523137    2579 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 15:08:27.523260 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.523250    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 24 15:08:27.523323 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.523283    2579 kubelet.go:486] "Attempting to sync node with API server"
Jul 24 15:08:27.523323 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.523301    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 24 15:08:27.523323 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.523320    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 24 15:08:27.523454 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.523329    2579 kubelet.go:398] "Adding apiserver pod source"
Jul 24 15:08:27.523454 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.523337    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 24 15:08:27.523667 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.523655    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 15:08:27.523696 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.523680    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 15:08:27.525964 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.525945    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 24 15:08:27.527887 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.527871    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 24 15:08:27.527942 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.527903    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 24 15:08:27.529722 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.529707    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 24 15:08:27.529806 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.529731    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 24 15:08:27.529806 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.529740    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 24 15:08:27.529806 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.529746    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 24 15:08:27.529806 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.529755    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 24 15:08:27.529806 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.529760    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 24 15:08:27.529806 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.529768    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 24 15:08:27.529806 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.529773    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 24 15:08:27.529806 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.529780    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 24 15:08:27.529806 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.529786    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 24 15:08:27.529806 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.529795    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 24 15:08:27.529806 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.529800    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 24 15:08:27.530631 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.530618    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 24 15:08:27.530631 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.530627    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 24 15:08:27.533664 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.533651    2579 server.go:1268] "Started kubelet"
Jul 24 15:08:27.533736 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.533711    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 24 15:08:27.533935 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.533886    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 24 15:08:27.534017 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.533960    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 24 15:08:27.535281 ip-10-0-140-160 systemd[1]: Started Kubernetes Kubelet.
Jul 24 15:08:27.535440 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.535424    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 24 15:08:27.537695 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.537679    2579 server.go:310] "Adding debug handlers to kubelet server"
Jul 24 15:08:27.538419 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:27.538399    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 24 15:08:27.539410 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:27.539388    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-160.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 24 15:08:27.539410 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:27.539388    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 24 15:08:27.539556 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.539544    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-160.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 24 15:08:27.540109 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:27.539359    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-160.ec2.internal.18c54248bf1be9a2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-160.ec2.internal,UID:ip-10-0-140-160.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-160.ec2.internal,},FirstTimestamp:2026-07-24 15:08:27.533625762 +0000 UTC m=+0.345977931,LastTimestamp:2026-07-24 15:08:27.533625762 +0000 UTC m=+0.345977931,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-160.ec2.internal,}"
Jul 24 15:08:27.542105 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.542091    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 24 15:08:27.542163 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.542104    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 24 15:08:27.542264 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:27.542210    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-160.ec2.internal\" not found"
Jul 24 15:08:27.542938 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.542836    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 24 15:08:27.543024 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.542942    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 24 15:08:27.543074 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.542807    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 24 15:08:27.543127 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.543111    2579 reconstruct.go:97] "Volume reconstruction finished"
Jul 24 15:08:27.543127 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.543118    2579 reconciler.go:29] "Reconciler: start to sync state"
Jul 24 15:08:27.544187 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.544169    2579 factory.go:55] Registering systemd factory
Jul 24 15:08:27.544187 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.544189    2579 factory.go:223] Registration of the systemd container factory successfully
Jul 24 15:08:27.544445 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.544427    2579 factory.go:153] Registering CRI-O factory
Jul 24 15:08:27.544510 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.544449    2579 factory.go:223] Registration of the crio container factory successfully
Jul 24 15:08:27.544510 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.544499    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 24 15:08:27.544608 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.544523    2579 factory.go:103] Registering Raw factory
Jul 24 15:08:27.544608 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.544538    2579 manager.go:1196] Started watching for new ooms in manager
Jul 24 15:08:27.545188 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.545174    2579 manager.go:349] Starting recovery of all containers
Jul 24 15:08:27.552342 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:27.552304    2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-160.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 24 15:08:27.552583 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:27.552558    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 24 15:08:27.553889 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.553718    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qtwld"
Jul 24 15:08:27.555612 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.555596    2579 manager.go:354] Recovery completed
Jul 24 15:08:27.560074 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.560057    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 15:08:27.561795 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.561777    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-160.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 15:08:27.561879 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.561809    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-160.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 15:08:27.561879 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.561839    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-160.ec2.internal" event="NodeHasSufficientPID"
Jul 24 15:08:27.562368 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.562355    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 24 15:08:27.562368 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.562367    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 24 15:08:27.562368 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.562370    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 15:08:27.562497 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.562381    2579 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 15:08:27.562857 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.562843    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qtwld"
Jul 24 15:08:27.563079 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.563056    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-160.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 15:08:27.563123 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.563082    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-160.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 15:08:27.563123 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.563092    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-160.ec2.internal" event="NodeHasSufficientPID"
Jul 24 15:08:27.564251 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.564240    2579 policy_none.go:49] "None policy: Start"
Jul 24 15:08:27.564296 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.564255    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 24 15:08:27.564296 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.564266    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 24 15:08:27.565323 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.565313    2579 policy_none.go:47] "Start"
Jul 24 15:08:27.610395 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.610377    2579 manager.go:337] "Starting Device Plugin manager"
Jul 24 15:08:27.610479 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:27.610411    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 24 15:08:27.610479 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.610421    2579 server.go:87] "Starting device plugin registration server"
Jul 24 15:08:27.610665 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.610651    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 24 15:08:27.610712 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.610668    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 24 15:08:27.610760 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.610740    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 24 15:08:27.610843 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.610827    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 24 15:08:27.610915 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.610867    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 24 15:08:27.611327 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:27.611310    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 24 15:08:27.611401 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:27.611356    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-160.ec2.internal\" not found"
Jul 24 15:08:27.709302 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.709271    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 24 15:08:27.710626 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.710611    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 24 15:08:27.710695 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.710653    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 24 15:08:27.710695 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.710674    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 24 15:08:27.710787 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:27.710713    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 24 15:08:27.710787 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.710733    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 15:08:27.711550 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.711532    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-160.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 15:08:27.711627 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.711562    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-160.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 15:08:27.711627 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.711576    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-160.ec2.internal" event="NodeHasSufficientPID"
Jul 24 15:08:27.711627 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.711600    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-160.ec2.internal"
Jul 24 15:08:27.713308 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.713269    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 15:08:27.721144 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.721130    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-160.ec2.internal"
Jul 24 15:08:27.721212 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:27.721151    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-160.ec2.internal\": node \"ip-10-0-140-160.ec2.internal\" not found"
Jul 24 15:08:27.737573 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:27.737513    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-160.ec2.internal\" not found"
Jul 24 15:08:27.811203 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.811149    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-160.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-160.ec2.internal"]
Jul 24 15:08:27.811304 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.811237    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 15:08:27.812139 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.812120    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-160.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 15:08:27.812199 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.812151    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-160.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 15:08:27.812199 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.812161    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-160.ec2.internal" event="NodeHasSufficientPID"
Jul 24 15:08:27.813463 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.813448    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 15:08:27.813589 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.813575    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-160.ec2.internal"
Jul 24 15:08:27.813644 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.813608    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 15:08:27.814103 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.814085    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-160.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 15:08:27.814201 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.814111    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-160.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 15:08:27.814201 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.814122    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-160.ec2.internal" event="NodeHasSufficientPID"
Jul 24 15:08:27.814201 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.814172    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-160.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 15:08:27.814201 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.814193    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-160.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 15:08:27.814336 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.814205    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-160.ec2.internal" event="NodeHasSufficientPID"
Jul 24 15:08:27.815198 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.815182    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-160.ec2.internal"
Jul 24 15:08:27.815253 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.815209    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 15:08:27.815879 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.815861    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-160.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 15:08:27.815939 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.815896    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-160.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 15:08:27.815939 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.815912    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-160.ec2.internal" event="NodeHasSufficientPID"
Jul 24 15:08:27.836473 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:27.836452    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-160.ec2.internal\" not found" node="ip-10-0-140-160.ec2.internal"
Jul 24 15:08:27.838514 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:27.838496    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-160.ec2.internal\" not found"
Jul 24 15:08:27.840867 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:27.840854    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-160.ec2.internal\" not found" node="ip-10-0-140-160.ec2.internal"
Jul 24 15:08:27.845042 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.845028    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/22fca178f17fa43f468a8cc5fc9fd939-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-160.ec2.internal\" (UID: \"22fca178f17fa43f468a8cc5fc9fd939\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-160.ec2.internal"
Jul 24 15:08:27.845097 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.845049    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22fca178f17fa43f468a8cc5fc9fd939-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-160.ec2.internal\" (UID: \"22fca178f17fa43f468a8cc5fc9fd939\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-160.ec2.internal"
Jul 24 15:08:27.845097 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.845066    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6aa67c3d9e8eb538931b1417f3d45520-config\") pod \"kube-apiserver-proxy-ip-10-0-140-160.ec2.internal\" (UID: \"6aa67c3d9e8eb538931b1417f3d45520\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-160.ec2.internal"
Jul 24 15:08:27.939135 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:27.939107    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-160.ec2.internal\" not found"
Jul 24 15:08:27.945977 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.945957    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/22fca178f17fa43f468a8cc5fc9fd939-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-160.ec2.internal\" (UID: \"22fca178f17fa43f468a8cc5fc9fd939\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-160.ec2.internal"
Jul 24 15:08:27.946042 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.945981    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/22fca178f17fa43f468a8cc5fc9fd939-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-160.ec2.internal\" (UID: \"22fca178f17fa43f468a8cc5fc9fd939\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-160.ec2.internal"
Jul 24 15:08:27.946042 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.945997    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22fca178f17fa43f468a8cc5fc9fd939-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-160.ec2.internal\" (UID: \"22fca178f17fa43f468a8cc5fc9fd939\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-160.ec2.internal"
Jul 24 15:08:27.946042 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.946026    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6aa67c3d9e8eb538931b1417f3d45520-config\") pod \"kube-apiserver-proxy-ip-10-0-140-160.ec2.internal\" (UID: \"6aa67c3d9e8eb538931b1417f3d45520\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-160.ec2.internal"
Jul 24 15:08:27.946156 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.946056    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6aa67c3d9e8eb538931b1417f3d45520-config\") pod \"kube-apiserver-proxy-ip-10-0-140-160.ec2.internal\" (UID: \"6aa67c3d9e8eb538931b1417f3d45520\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-160.ec2.internal"
Jul 24 15:08:27.946156 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:27.946098    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22fca178f17fa43f468a8cc5fc9fd939-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-160.ec2.internal\" (UID: \"22fca178f17fa43f468a8cc5fc9fd939\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-160.ec2.internal"
Jul 24 15:08:28.040294 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:28.040232    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-160.ec2.internal\" not found"
Jul 24 15:08:28.139179 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:28.139150    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-160.ec2.internal"
Jul 24 15:08:28.140312 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:28.140295    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-160.ec2.internal\" not found"
Jul 24 15:08:28.144605 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:28.144585    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-160.ec2.internal"
Jul 24 15:08:28.241360 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:28.241322    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-160.ec2.internal\" not found"
Jul 24 15:08:28.344978 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:28.344893    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-160.ec2.internal\" not found"
Jul 24 15:08:28.429922 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:28.429888    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 15:08:28.445580 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:28.445555    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-160.ec2.internal\" not found"
Jul 24 15:08:28.481471 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:28.481445    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 15:08:28.487301 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:28.487288    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 24 15:08:28.487407 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:28.487390    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 24 15:08:28.487454 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:28.487439    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 24 15:08:28.487500 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:28.487439    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 24 15:08:28.546053 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:28.546026    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-160.ec2.internal\" not found"
Jul 24 15:08:28.567526 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:28.567486    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 15:03:27 +0000 UTC" deadline="2028-01-29 02:34:09.266009589 +0000 UTC"
Jul 24 15:08:28.567526 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:28.567516    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13283h25m40.698496797s"
Jul 24 15:08:28.646561 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:28.646485    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-160.ec2.internal\" not found"
Jul 24 15:08:28.670855 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:28.670811    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22fca178f17fa43f468a8cc5fc9fd939.slice/crio-2c01fb93879d616dcf329ca449d8e919796a0e28077dbdccb000250992f6bf0c WatchSource:0}: Error finding container 2c01fb93879d616dcf329ca449d8e919796a0e28077dbdccb000250992f6bf0c: Status 404 returned error can't find the container with id 2c01fb93879d616dcf329ca449d8e919796a0e28077dbdccb000250992f6bf0c
Jul 24 15:08:28.676197 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:28.676175    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 15:08:28.695820 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:28.695794    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aa67c3d9e8eb538931b1417f3d45520.slice/crio-664f62d131349541384d431e6af0251896bc766c6469981b4aea28538e6061ef WatchSource:0}: Error finding container 664f62d131349541384d431e6af0251896bc766c6469981b4aea28538e6061ef: Status 404 returned error can't find the container with id 664f62d131349541384d431e6af0251896bc766c6469981b4aea28538e6061ef
Jul 24 15:08:28.714308 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:28.714252    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-160.ec2.internal" event={"ID":"6aa67c3d9e8eb538931b1417f3d45520","Type":"ContainerStarted","Data":"664f62d131349541384d431e6af0251896bc766c6469981b4aea28538e6061ef"}
Jul 24 15:08:28.715311 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:28.715288    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-160.ec2.internal" event={"ID":"22fca178f17fa43f468a8cc5fc9fd939","Type":"ContainerStarted","Data":"2c01fb93879d616dcf329ca449d8e919796a0e28077dbdccb000250992f6bf0c"}
Jul 24 15:08:28.747224 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:28.747194    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-160.ec2.internal\" not found"
Jul 24 15:08:28.849081 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:28.849037    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-160.ec2.internal\" not found"
Jul 24 15:08:28.921366 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:28.921302    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 15:08:28.942499 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:28.942471    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-160.ec2.internal"
Jul 24 15:08:28.959850 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:28.959812    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 15:08:28.961077 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:28.961062    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-160.ec2.internal"
Jul 24 15:08:28.977286 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:28.977267    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 15:08:29.526063 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.526029    2579 apiserver.go:52] "Watching apiserver"
Jul 24 15:08:29.532990 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.532965    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 24 15:08:29.536410 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.536378    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-7qc8n","kube-system/kube-apiserver-proxy-ip-10-0-140-160.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9t4rt","openshift-dns/node-resolver-s5nzq","openshift-multus/multus-97xpk","openshift-multus/multus-additional-cni-plugins-g4ts4","openshift-multus/network-metrics-daemon-qgzfw","openshift-ovn-kubernetes/ovnkube-node-52fwk","kube-system/konnectivity-agent-d7d7z","openshift-cluster-node-tuning-operator/tuned-gm95d","openshift-image-registry/node-ca-n9snp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-160.ec2.internal","openshift-network-diagnostics/network-check-target-z654w","openshift-network-operator/iptables-alerter-k6zwd"]
Jul 24 15:08:29.537988 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.537960    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgzfw"
Jul 24 15:08:29.538108 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:29.538053    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgzfw" podUID="e0b197de-537f-457b-8d3f-4dc75d38c402"
Jul 24 15:08:29.539308 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.539275    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z654w"
Jul 24 15:08:29.539414 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:29.539367    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z654w" podUID="a817b528-08b2-49a7-bbef-22838f3ca22d"
Jul 24 15:08:29.540849 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.540770    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9t4rt"
Jul 24 15:08:29.543438 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.542417    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s5nzq"
Jul 24 15:08:29.543438 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.542581    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 24 15:08:29.544202 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.544182    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 24 15:08:29.544604 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.544575    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 24 15:08:29.544785 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.544768    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 24 15:08:29.544922 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.544768    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.547610 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.547581    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g4ts4"
Jul 24 15:08:29.549339 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.549312    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 24 15:08:29.549473 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.549455    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-485hn\""
Jul 24 15:08:29.549956 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.549326    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-74l4r\""
Jul 24 15:08:29.550119 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.550102    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-m5zkn\""
Jul 24 15:08:29.550537 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.550521    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 24 15:08:29.551138 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.550875    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d7d7z"
Jul 24 15:08:29.551975 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.551954    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 24 15:08:29.552223 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.552205    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 24 15:08:29.552420 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.552402    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7qc8n"
Jul 24 15:08:29.552522 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:29.552492    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7qc8n" podUID="5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70"
Jul 24 15:08:29.553565 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.553498    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 24 15:08:29.554183 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.554103    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4bzx\" (UniqueName: \"kubernetes.io/projected/e0b197de-537f-457b-8d3f-4dc75d38c402-kube-api-access-f4bzx\") pod \"network-metrics-daemon-qgzfw\" (UID: \"e0b197de-537f-457b-8d3f-4dc75d38c402\") " pod="openshift-multus/network-metrics-daemon-qgzfw"
Jul 24 15:08:29.554183 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.554139    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca98516e-79dc-4972-86de-8ff390bc0de2-tmp-dir\") pod \"node-resolver-s5nzq\" (UID: \"ca98516e-79dc-4972-86de-8ff390bc0de2\") " pod="openshift-dns/node-resolver-s5nzq"
Jul 24 15:08:29.554352 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.554194    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e3a67ea4-031f-45a0-860f-1771acbbb3be-multus-cni-dir\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.554352 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.554220    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c94nk\" (UniqueName: \"kubernetes.io/projected/e3a67ea4-031f-45a0-860f-1771acbbb3be-kube-api-access-c94nk\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.554352 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.554243    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e3a67ea4-031f-45a0-860f-1771acbbb3be-host-run-netns\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.554352 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.554269    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e3a67ea4-031f-45a0-860f-1771acbbb3be-host-var-lib-cni-multus\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.554352 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.554295    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0b197de-537f-457b-8d3f-4dc75d38c402-metrics-certs\") pod \"network-metrics-daemon-qgzfw\" (UID: \"e0b197de-537f-457b-8d3f-4dc75d38c402\") " pod="openshift-multus/network-metrics-daemon-qgzfw"
Jul 24 15:08:29.554352 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.554319    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/699558f6-ffac-4afd-b1b0-ca6f0611c47e-device-dir\") pod \"aws-ebs-csi-driver-node-9t4rt\" (UID: \"699558f6-ffac-4afd-b1b0-ca6f0611c47e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9t4rt"
Jul 24 15:08:29.554352 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.554346    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/699558f6-ffac-4afd-b1b0-ca6f0611c47e-sys-fs\") pod \"aws-ebs-csi-driver-node-9t4rt\" (UID: \"699558f6-ffac-4afd-b1b0-ca6f0611c47e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9t4rt"
Jul 24 15:08:29.554723 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.554372    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e3a67ea4-031f-45a0-860f-1771acbbb3be-multus-socket-dir-parent\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.554723 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.554397    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e3a67ea4-031f-45a0-860f-1771acbbb3be-multus-daemon-config\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.554723 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.554423    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppp76\" (UniqueName: \"kubernetes.io/projected/a817b528-08b2-49a7-bbef-22838f3ca22d-kube-api-access-ppp76\") pod \"network-check-target-z654w\" (UID: \"a817b528-08b2-49a7-bbef-22838f3ca22d\") " pod="openshift-network-diagnostics/network-check-target-z654w"
Jul 24 15:08:29.554723 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.554449    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/699558f6-ffac-4afd-b1b0-ca6f0611c47e-socket-dir\") pod \"aws-ebs-csi-driver-node-9t4rt\" (UID: \"699558f6-ffac-4afd-b1b0-ca6f0611c47e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9t4rt"
Jul 24 15:08:29.554723 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.554473    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4vzq\" (UniqueName: \"kubernetes.io/projected/ca98516e-79dc-4972-86de-8ff390bc0de2-kube-api-access-r4vzq\") pod \"node-resolver-s5nzq\" (UID: \"ca98516e-79dc-4972-86de-8ff390bc0de2\") " pod="openshift-dns/node-resolver-s5nzq"
Jul 24 15:08:29.554723 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.554498    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e3a67ea4-031f-45a0-860f-1771acbbb3be-cnibin\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.554723 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.554559    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/699558f6-ffac-4afd-b1b0-ca6f0611c47e-etc-selinux\") pod \"aws-ebs-csi-driver-node-9t4rt\" (UID: \"699558f6-ffac-4afd-b1b0-ca6f0611c47e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9t4rt"
Jul 24 15:08:29.554723 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.554609    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e3a67ea4-031f-45a0-860f-1771acbbb3be-system-cni-dir\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.554723 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.554634    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e3a67ea4-031f-45a0-860f-1771acbbb3be-os-release\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.554723 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.554658    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e3a67ea4-031f-45a0-860f-1771acbbb3be-cni-binary-copy\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.554723 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.554701    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e3a67ea4-031f-45a0-860f-1771acbbb3be-multus-conf-dir\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.554723 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.554725    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3a67ea4-031f-45a0-860f-1771acbbb3be-etc-kubernetes\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.555737 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.554753    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/699558f6-ffac-4afd-b1b0-ca6f0611c47e-registration-dir\") pod \"aws-ebs-csi-driver-node-9t4rt\" (UID: \"699558f6-ffac-4afd-b1b0-ca6f0611c47e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9t4rt"
Jul 24 15:08:29.555737 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.554779    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e3a67ea4-031f-45a0-860f-1771acbbb3be-host-var-lib-cni-bin\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.555737 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.554812    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e3a67ea4-031f-45a0-860f-1771acbbb3be-host-run-multus-certs\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.555737 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.554859    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/699558f6-ffac-4afd-b1b0-ca6f0611c47e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9t4rt\" (UID: \"699558f6-ffac-4afd-b1b0-ca6f0611c47e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9t4rt"
Jul 24 15:08:29.555737 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.554891    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l72f\" (UniqueName: \"kubernetes.io/projected/699558f6-ffac-4afd-b1b0-ca6f0611c47e-kube-api-access-2l72f\") pod \"aws-ebs-csi-driver-node-9t4rt\" (UID: \"699558f6-ffac-4afd-b1b0-ca6f0611c47e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9t4rt"
Jul 24 15:08:29.555737 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.554931    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k6zwd"
Jul 24 15:08:29.555737 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.554939    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e3a67ea4-031f-45a0-860f-1771acbbb3be-host-run-k8s-cni-cncf-io\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.555737 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.554966    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3a67ea4-031f-45a0-860f-1771acbbb3be-host-var-lib-kubelet\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.555737 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.554994    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ca98516e-79dc-4972-86de-8ff390bc0de2-hosts-file\") pod \"node-resolver-s5nzq\" (UID: \"ca98516e-79dc-4972-86de-8ff390bc0de2\") " pod="openshift-dns/node-resolver-s5nzq"
Jul 24 15:08:29.555737 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.555019    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e3a67ea4-031f-45a0-860f-1771acbbb3be-hostroot\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.556322 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.555890    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 24 15:08:29.556322 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.556216    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gm95d"
Jul 24 15:08:29.556643 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.556622    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 24 15:08:29.556781 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.556673    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 24 15:08:29.556781 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.556677    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pmcj4\""
Jul 24 15:08:29.556781 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.556624    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-bvn5n\""
Jul 24 15:08:29.557095 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.557049    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 24 15:08:29.557496 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.557322    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 24 15:08:29.557583 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.557545    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n9snp"
Jul 24 15:08:29.559171 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.559143    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.560491 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.560409    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 24 15:08:29.560600 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.560508    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 24 15:08:29.562562 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.562523    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 24 15:08:29.562893 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.562877    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 24 15:08:29.563190 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.563171    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 24 15:08:29.563710 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.563673    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 24 15:08:29.567304 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.567074    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j4kmn\""
Jul 24 15:08:29.568486 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.568239    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4kpwh\""
Jul 24 15:08:29.569623 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.569465    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 24 15:08:29.570044 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.569817    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 24 15:08:29.571175 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.570546    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 24 15:08:29.571175 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.570967    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 24 15:08:29.571347 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.571203    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 24 15:08:29.571787 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.571591    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 24 15:08:29.572142 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.572124    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 24 15:08:29.575056 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.574788    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-85mrc\""
Jul 24 15:08:29.577102 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.577083    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 24 15:08:29.577913 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.577855    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ffhpv\""
Jul 24 15:08:29.580484 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.580462    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 15:08:29.603652 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.603625    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-lswfp"
Jul 24 15:08:29.614875 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.614854    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-lswfp"
Jul 24 15:08:29.644630 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.644604    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 24 15:08:29.656245 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.656209    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1b97ec16-7040-4aa3-b2a8-0827c3698c7a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g4ts4\" (UID: \"1b97ec16-7040-4aa3-b2a8-0827c3698c7a\") " pod="openshift-multus/multus-additional-cni-plugins-g4ts4"
Jul 24 15:08:29.656369 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.656254    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1fe7cd76-ff78-43b1-b33c-96b4591a61db-etc-systemd\") pod \"tuned-gm95d\" (UID: \"1fe7cd76-ff78-43b1-b33c-96b4591a61db\") " pod="openshift-cluster-node-tuning-operator/tuned-gm95d"
Jul 24 15:08:29.656369 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.656283    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-ovnkube-config\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.656369 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.656311    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/699558f6-ffac-4afd-b1b0-ca6f0611c47e-etc-selinux\") pod \"aws-ebs-csi-driver-node-9t4rt\" (UID: \"699558f6-ffac-4afd-b1b0-ca6f0611c47e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9t4rt"
Jul 24 15:08:29.656369 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.656336    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e3a67ea4-031f-45a0-860f-1771acbbb3be-system-cni-dir\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.656369 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.656354    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e3a67ea4-031f-45a0-860f-1771acbbb3be-os-release\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.656369 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.656369    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e3a67ea4-031f-45a0-860f-1771acbbb3be-cni-binary-copy\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.656666 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.656385    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3a67ea4-031f-45a0-860f-1771acbbb3be-etc-kubernetes\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.656666 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.656402    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-host-run-netns\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.656666 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.656417    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-ovn-node-metrics-cert\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.656666 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.656437    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/699558f6-ffac-4afd-b1b0-ca6f0611c47e-registration-dir\") pod \"aws-ebs-csi-driver-node-9t4rt\" (UID: \"699558f6-ffac-4afd-b1b0-ca6f0611c47e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9t4rt"
Jul 24 15:08:29.656666 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.656453    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e3a67ea4-031f-45a0-860f-1771acbbb3be-host-var-lib-cni-bin\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.656666 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.656469    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e3a67ea4-031f-45a0-860f-1771acbbb3be-host-run-multus-certs\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.656666 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.656493    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b97ec16-7040-4aa3-b2a8-0827c3698c7a-system-cni-dir\") pod \"multus-additional-cni-plugins-g4ts4\" (UID: \"1b97ec16-7040-4aa3-b2a8-0827c3698c7a\") " pod="openshift-multus/multus-additional-cni-plugins-g4ts4"
Jul 24 15:08:29.656666 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.656515    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1fe7cd76-ff78-43b1-b33c-96b4591a61db-etc-sysctl-d\") pod \"tuned-gm95d\" (UID: \"1fe7cd76-ff78-43b1-b33c-96b4591a61db\") " pod="openshift-cluster-node-tuning-operator/tuned-gm95d"
Jul 24 15:08:29.656666 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.656580    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3a67ea4-031f-45a0-860f-1771acbbb3be-etc-kubernetes\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.656666 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.656612    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/699558f6-ffac-4afd-b1b0-ca6f0611c47e-registration-dir\") pod \"aws-ebs-csi-driver-node-9t4rt\" (UID: \"699558f6-ffac-4afd-b1b0-ca6f0611c47e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9t4rt"
Jul 24 15:08:29.656666 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.656622    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e3a67ea4-031f-45a0-860f-1771acbbb3be-system-cni-dir\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.657137 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.656680    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e3a67ea4-031f-45a0-860f-1771acbbb3be-host-run-multus-certs\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.657137 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.656685    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e3a67ea4-031f-45a0-860f-1771acbbb3be-host-var-lib-cni-bin\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.657137 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.656707    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1fe7cd76-ff78-43b1-b33c-96b4591a61db-lib-modules\") pod \"tuned-gm95d\" (UID: \"1fe7cd76-ff78-43b1-b33c-96b4591a61db\") " pod="openshift-cluster-node-tuning-operator/tuned-gm95d"
Jul 24 15:08:29.657137 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.656737    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/699558f6-ffac-4afd-b1b0-ca6f0611c47e-etc-selinux\") pod \"aws-ebs-csi-driver-node-9t4rt\" (UID: \"699558f6-ffac-4afd-b1b0-ca6f0611c47e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9t4rt"
Jul 24 15:08:29.657137 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.656742    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e3a67ea4-031f-45a0-860f-1771acbbb3be-os-release\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.657137 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.656750    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtnfj\" (UniqueName: \"kubernetes.io/projected/2216188a-b1f1-4ddf-8f9e-346e93860df2-kube-api-access-gtnfj\") pod \"node-ca-n9snp\" (UID: \"2216188a-b1f1-4ddf-8f9e-346e93860df2\") " pod="openshift-image-registry/node-ca-n9snp"
Jul 24 15:08:29.657359 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.657246    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e3a67ea4-031f-45a0-860f-1771acbbb3be-cni-binary-copy\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.657359 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.656790    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l72f\" (UniqueName: \"kubernetes.io/projected/699558f6-ffac-4afd-b1b0-ca6f0611c47e-kube-api-access-2l72f\") pod \"aws-ebs-csi-driver-node-9t4rt\" (UID: \"699558f6-ffac-4afd-b1b0-ca6f0611c47e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9t4rt"
Jul 24 15:08:29.657359 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.657329    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e3a67ea4-031f-45a0-860f-1771acbbb3be-host-run-k8s-cni-cncf-io\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.657508 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.657359    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b97ec16-7040-4aa3-b2a8-0827c3698c7a-cni-binary-copy\") pod \"multus-additional-cni-plugins-g4ts4\" (UID: \"1b97ec16-7040-4aa3-b2a8-0827c3698c7a\") " pod="openshift-multus/multus-additional-cni-plugins-g4ts4"
Jul 24 15:08:29.657508 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.657388    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1fe7cd76-ff78-43b1-b33c-96b4591a61db-etc-modprobe-d\") pod \"tuned-gm95d\" (UID: \"1fe7cd76-ff78-43b1-b33c-96b4591a61db\") " pod="openshift-cluster-node-tuning-operator/tuned-gm95d"
Jul 24 15:08:29.657508 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.657429    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1fe7cd76-ff78-43b1-b33c-96b4591a61db-run\") pod \"tuned-gm95d\" (UID: \"1fe7cd76-ff78-43b1-b33c-96b4591a61db\") " pod="openshift-cluster-node-tuning-operator/tuned-gm95d"
Jul 24 15:08:29.657508 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.657439    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e3a67ea4-031f-45a0-860f-1771acbbb3be-host-run-k8s-cni-cncf-io\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.657508 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.657453    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2216188a-b1f1-4ddf-8f9e-346e93860df2-serviceca\") pod \"node-ca-n9snp\" (UID: \"2216188a-b1f1-4ddf-8f9e-346e93860df2\") " pod="openshift-image-registry/node-ca-n9snp"
Jul 24 15:08:29.657508 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.657480    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-var-lib-openvswitch\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.657753 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.657513    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-run-openvswitch\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.657753 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.657560    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b97ec16-7040-4aa3-b2a8-0827c3698c7a-cnibin\") pod \"multus-additional-cni-plugins-g4ts4\" (UID: \"1b97ec16-7040-4aa3-b2a8-0827c3698c7a\") " pod="openshift-multus/multus-additional-cni-plugins-g4ts4"
Jul 24 15:08:29.657753 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.657589    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1b97ec16-7040-4aa3-b2a8-0827c3698c7a-os-release\") pod \"multus-additional-cni-plugins-g4ts4\" (UID: \"1b97ec16-7040-4aa3-b2a8-0827c3698c7a\") " pod="openshift-multus/multus-additional-cni-plugins-g4ts4"
Jul 24 15:08:29.657753 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.657620    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1fe7cd76-ff78-43b1-b33c-96b4591a61db-var-lib-kubelet\") pod \"tuned-gm95d\" (UID: \"1fe7cd76-ff78-43b1-b33c-96b4591a61db\") " pod="openshift-cluster-node-tuning-operator/tuned-gm95d"
Jul 24 15:08:29.657753 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.657650    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2216188a-b1f1-4ddf-8f9e-346e93860df2-host\") pod \"node-ca-n9snp\" (UID: \"2216188a-b1f1-4ddf-8f9e-346e93860df2\") " pod="openshift-image-registry/node-ca-n9snp"
Jul 24 15:08:29.657753 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.657679    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4bzx\" (UniqueName: \"kubernetes.io/projected/e0b197de-537f-457b-8d3f-4dc75d38c402-kube-api-access-f4bzx\") pod \"network-metrics-daemon-qgzfw\" (UID: \"e0b197de-537f-457b-8d3f-4dc75d38c402\") " pod="openshift-multus/network-metrics-daemon-qgzfw"
Jul 24 15:08:29.657753 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.657706    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca98516e-79dc-4972-86de-8ff390bc0de2-tmp-dir\") pod \"node-resolver-s5nzq\" (UID: \"ca98516e-79dc-4972-86de-8ff390bc0de2\") " pod="openshift-dns/node-resolver-s5nzq"
Jul 24 15:08:29.657753 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.657731    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e4dfc4ee-1d7a-4e52-a295-efc820c560bd-konnectivity-ca\") pod \"konnectivity-agent-d7d7z\" (UID: \"e4dfc4ee-1d7a-4e52-a295-efc820c560bd\") " pod="kube-system/konnectivity-agent-d7d7z"
Jul 24 15:08:29.658161 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.657756    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70-original-pull-secret\") pod \"global-pull-secret-syncer-7qc8n\" (UID: \"5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70\") " pod="kube-system/global-pull-secret-syncer-7qc8n"
Jul 24 15:08:29.658161 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.657783    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/842eecfb-c927-4ef0-84fc-edba5a668fb2-host-slash\") pod \"iptables-alerter-k6zwd\" (UID: \"842eecfb-c927-4ef0-84fc-edba5a668fb2\") " pod="openshift-network-operator/iptables-alerter-k6zwd"
Jul 24 15:08:29.658161 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.657806    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1fe7cd76-ff78-43b1-b33c-96b4591a61db-etc-kubernetes\") pod \"tuned-gm95d\" (UID: \"1fe7cd76-ff78-43b1-b33c-96b4591a61db\") " pod="openshift-cluster-node-tuning-operator/tuned-gm95d"
Jul 24 15:08:29.658161 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.657829    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lwtw\" (UniqueName: \"kubernetes.io/projected/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-kube-api-access-5lwtw\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.658161 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.657868    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e3a67ea4-031f-45a0-860f-1771acbbb3be-host-run-netns\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.658161 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.657890    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70-dbus\") pod \"global-pull-secret-syncer-7qc8n\" (UID: \"5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70\") " pod="kube-system/global-pull-secret-syncer-7qc8n"
Jul 24 15:08:29.658161 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.657914    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1fe7cd76-ff78-43b1-b33c-96b4591a61db-etc-tuned\") pod \"tuned-gm95d\" (UID: \"1fe7cd76-ff78-43b1-b33c-96b4591a61db\") " pod="openshift-cluster-node-tuning-operator/tuned-gm95d"
Jul 24 15:08:29.658161 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.657936    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-node-log\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.658161 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.657961    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/699558f6-ffac-4afd-b1b0-ca6f0611c47e-sys-fs\") pod \"aws-ebs-csi-driver-node-9t4rt\" (UID: \"699558f6-ffac-4afd-b1b0-ca6f0611c47e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9t4rt"
Jul 24 15:08:29.658161 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.657975    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e3a67ea4-031f-45a0-860f-1771acbbb3be-multus-socket-dir-parent\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.658161 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.657990    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e3a67ea4-031f-45a0-860f-1771acbbb3be-multus-daemon-config\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.658161 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.658003    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1fe7cd76-ff78-43b1-b33c-96b4591a61db-sys\") pod \"tuned-gm95d\" (UID: \"1fe7cd76-ff78-43b1-b33c-96b4591a61db\") " pod="openshift-cluster-node-tuning-operator/tuned-gm95d"
Jul 24 15:08:29.658161 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.658017    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1fe7cd76-ff78-43b1-b33c-96b4591a61db-host\") pod \"tuned-gm95d\" (UID: \"1fe7cd76-ff78-43b1-b33c-96b4591a61db\") " pod="openshift-cluster-node-tuning-operator/tuned-gm95d"
Jul 24 15:08:29.658161 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.658032    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppp76\" (UniqueName: \"kubernetes.io/projected/a817b528-08b2-49a7-bbef-22838f3ca22d-kube-api-access-ppp76\") pod \"network-check-target-z654w\" (UID: \"a817b528-08b2-49a7-bbef-22838f3ca22d\") " pod="openshift-network-diagnostics/network-check-target-z654w"
Jul 24 15:08:29.658161 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.658055    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/699558f6-ffac-4afd-b1b0-ca6f0611c47e-socket-dir\") pod \"aws-ebs-csi-driver-node-9t4rt\" (UID: \"699558f6-ffac-4afd-b1b0-ca6f0611c47e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9t4rt"
Jul 24 15:08:29.658161 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.658093    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4vzq\" (UniqueName: \"kubernetes.io/projected/ca98516e-79dc-4972-86de-8ff390bc0de2-kube-api-access-r4vzq\") pod \"node-resolver-s5nzq\" (UID: \"ca98516e-79dc-4972-86de-8ff390bc0de2\") " pod="openshift-dns/node-resolver-s5nzq"
Jul 24 15:08:29.658161 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.658117    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e3a67ea4-031f-45a0-860f-1771acbbb3be-cnibin\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.659027 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.658142    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1b97ec16-7040-4aa3-b2a8-0827c3698c7a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g4ts4\" (UID: \"1b97ec16-7040-4aa3-b2a8-0827c3698c7a\") " pod="openshift-multus/multus-additional-cni-plugins-g4ts4"
Jul 24 15:08:29.659027 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.658166    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1fe7cd76-ff78-43b1-b33c-96b4591a61db-etc-sysconfig\") pod \"tuned-gm95d\" (UID: \"1fe7cd76-ff78-43b1-b33c-96b4591a61db\") " pod="openshift-cluster-node-tuning-operator/tuned-gm95d"
Jul 24 15:08:29.659027 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.658189    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx2rl\" (UniqueName: \"kubernetes.io/projected/1fe7cd76-ff78-43b1-b33c-96b4591a61db-kube-api-access-mx2rl\") pod \"tuned-gm95d\" (UID: \"1fe7cd76-ff78-43b1-b33c-96b4591a61db\") " pod="openshift-cluster-node-tuning-operator/tuned-gm95d"
Jul 24 15:08:29.659027 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.658212    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-host-cni-netd\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.659027 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.658236    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e3a67ea4-031f-45a0-860f-1771acbbb3be-multus-conf-dir\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.659027 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.658260    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/842eecfb-c927-4ef0-84fc-edba5a668fb2-iptables-alerter-script\") pod \"iptables-alerter-k6zwd\" (UID: \"842eecfb-c927-4ef0-84fc-edba5a668fb2\") " pod="openshift-network-operator/iptables-alerter-k6zwd"
Jul 24 15:08:29.659027 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.658284    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-etc-openvswitch\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.659027 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.658314    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-host-cni-bin\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.659027 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.658320    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca98516e-79dc-4972-86de-8ff390bc0de2-tmp-dir\") pod \"node-resolver-s5nzq\" (UID: \"ca98516e-79dc-4972-86de-8ff390bc0de2\") " pod="openshift-dns/node-resolver-s5nzq"
Jul 24 15:08:29.659027 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.658338    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kjmt\" (UniqueName: \"kubernetes.io/projected/842eecfb-c927-4ef0-84fc-edba5a668fb2-kube-api-access-9kjmt\") pod \"iptables-alerter-k6zwd\" (UID: \"842eecfb-c927-4ef0-84fc-edba5a668fb2\") " pod="openshift-network-operator/iptables-alerter-k6zwd"
Jul 24 15:08:29.659027 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.658364    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e3a67ea4-031f-45a0-860f-1771acbbb3be-cnibin\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.659027 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.658387    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1fe7cd76-ff78-43b1-b33c-96b4591a61db-etc-sysctl-conf\") pod \"tuned-gm95d\" (UID: \"1fe7cd76-ff78-43b1-b33c-96b4591a61db\") " pod="openshift-cluster-node-tuning-operator/tuned-gm95d"
Jul 24 15:08:29.659027 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.658462    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e3a67ea4-031f-45a0-860f-1771acbbb3be-multus-conf-dir\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.659027 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.658515    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/699558f6-ffac-4afd-b1b0-ca6f0611c47e-socket-dir\") pod \"aws-ebs-csi-driver-node-9t4rt\" (UID: \"699558f6-ffac-4afd-b1b0-ca6f0611c47e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9t4rt"
Jul 24 15:08:29.659027 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.658529    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/699558f6-ffac-4afd-b1b0-ca6f0611c47e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9t4rt\" (UID: \"699558f6-ffac-4afd-b1b0-ca6f0611c47e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9t4rt"
Jul 24 15:08:29.659027 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.658567    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3a67ea4-031f-45a0-860f-1771acbbb3be-host-var-lib-kubelet\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.659027 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.658606    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e3a67ea4-031f-45a0-860f-1771acbbb3be-multus-daemon-config\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.659841 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.658617    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70-kubelet-config\") pod \"global-pull-secret-syncer-7qc8n\" (UID: \"5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70\") " pod="kube-system/global-pull-secret-syncer-7qc8n"
Jul 24 15:08:29.659841 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.658877    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-host-kubelet\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.659841 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.658946    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-systemd-units\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.659841 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.658972    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-run-systemd\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.659841 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.659005    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ca98516e-79dc-4972-86de-8ff390bc0de2-hosts-file\") pod \"node-resolver-s5nzq\" (UID: \"ca98516e-79dc-4972-86de-8ff390bc0de2\") " pod="openshift-dns/node-resolver-s5nzq"
Jul 24 15:08:29.659841 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.659042    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e3a67ea4-031f-45a0-860f-1771acbbb3be-hostroot\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.659841 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.659359    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b97ec16-7040-4aa3-b2a8-0827c3698c7a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g4ts4\" (UID: \"1b97ec16-7040-4aa3-b2a8-0827c3698c7a\") " pod="openshift-multus/multus-additional-cni-plugins-g4ts4"
Jul 24 15:08:29.659841 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.659399    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwv6m\" (UniqueName: \"kubernetes.io/projected/1b97ec16-7040-4aa3-b2a8-0827c3698c7a-kube-api-access-qwv6m\") pod \"multus-additional-cni-plugins-g4ts4\" (UID: \"1b97ec16-7040-4aa3-b2a8-0827c3698c7a\") " pod="openshift-multus/multus-additional-cni-plugins-g4ts4"
Jul 24 15:08:29.659841 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.659435    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-run-ovn\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.659841 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.659466    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-log-socket\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.659841 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.659494    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-host-run-ovn-kubernetes\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.659841 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.659563    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e3a67ea4-031f-45a0-860f-1771acbbb3be-hostroot\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.659841 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.659630    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ca98516e-79dc-4972-86de-8ff390bc0de2-hosts-file\") pod \"node-resolver-s5nzq\" (UID: \"ca98516e-79dc-4972-86de-8ff390bc0de2\") " pod="openshift-dns/node-resolver-s5nzq"
Jul 24 15:08:29.660525 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.659841    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3a67ea4-031f-45a0-860f-1771acbbb3be-host-var-lib-kubelet\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.660525 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.660136    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/699558f6-ffac-4afd-b1b0-ca6f0611c47e-sys-fs\") pod \"aws-ebs-csi-driver-node-9t4rt\" (UID: \"699558f6-ffac-4afd-b1b0-ca6f0611c47e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9t4rt"
Jul 24 15:08:29.660821 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.660798    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e3a67ea4-031f-45a0-860f-1771acbbb3be-multus-socket-dir-parent\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.661035 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.661008    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/699558f6-ffac-4afd-b1b0-ca6f0611c47e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9t4rt\" (UID: \"699558f6-ffac-4afd-b1b0-ca6f0611c47e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9t4rt"
Jul 24 15:08:29.661193 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.661173    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e3a67ea4-031f-45a0-860f-1771acbbb3be-host-run-netns\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.661641 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.661621    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e3a67ea4-031f-45a0-860f-1771acbbb3be-multus-cni-dir\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.661709 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.661666    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c94nk\" (UniqueName: \"kubernetes.io/projected/e3a67ea4-031f-45a0-860f-1771acbbb3be-kube-api-access-c94nk\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.661760 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.661704    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e4dfc4ee-1d7a-4e52-a295-efc820c560bd-agent-certs\") pod \"konnectivity-agent-d7d7z\" (UID: \"e4dfc4ee-1d7a-4e52-a295-efc820c560bd\") " pod="kube-system/konnectivity-agent-d7d7z"
Jul 24 15:08:29.661760 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.661746    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1fe7cd76-ff78-43b1-b33c-96b4591a61db-tmp\") pod \"tuned-gm95d\" (UID: \"1fe7cd76-ff78-43b1-b33c-96b4591a61db\") " pod="openshift-cluster-node-tuning-operator/tuned-gm95d"
Jul 24 15:08:29.661858 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.661788    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-env-overrides\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.661923 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.661864    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e3a67ea4-031f-45a0-860f-1771acbbb3be-host-var-lib-cni-multus\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.661923 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.661916    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-ovnkube-script-lib\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.662026 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.661963    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0b197de-537f-457b-8d3f-4dc75d38c402-metrics-certs\") pod \"network-metrics-daemon-qgzfw\" (UID: \"e0b197de-537f-457b-8d3f-4dc75d38c402\") " pod="openshift-multus/network-metrics-daemon-qgzfw"
Jul 24 15:08:29.662123 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:29.662103    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 15:08:29.662176 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.662154    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e3a67ea4-031f-45a0-860f-1771acbbb3be-multus-cni-dir\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.662223 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:29.662199    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0b197de-537f-457b-8d3f-4dc75d38c402-metrics-certs podName:e0b197de-537f-457b-8d3f-4dc75d38c402 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:30.162168475 +0000 UTC m=+2.974520649 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0b197de-537f-457b-8d3f-4dc75d38c402-metrics-certs") pod "network-metrics-daemon-qgzfw" (UID: "e0b197de-537f-457b-8d3f-4dc75d38c402") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 15:08:29.662223 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.662203    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e3a67ea4-031f-45a0-860f-1771acbbb3be-host-var-lib-cni-multus\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.662330 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.662239    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/699558f6-ffac-4afd-b1b0-ca6f0611c47e-device-dir\") pod \"aws-ebs-csi-driver-node-9t4rt\" (UID: \"699558f6-ffac-4afd-b1b0-ca6f0611c47e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9t4rt"
Jul 24 15:08:29.662330 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.662270    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-host-slash\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.662330 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.662304    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.662479 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.662346    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/699558f6-ffac-4afd-b1b0-ca6f0611c47e-device-dir\") pod \"aws-ebs-csi-driver-node-9t4rt\" (UID: \"699558f6-ffac-4afd-b1b0-ca6f0611c47e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9t4rt"
Jul 24 15:08:29.674861 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.674830    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 24 15:08:29.676638 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:29.676507    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 15:08:29.676638 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:29.676534    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 15:08:29.676638 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:29.676548    2579 projected.go:196] Error preparing data for projected volume kube-api-access-ppp76 for pod openshift-network-diagnostics/network-check-target-z654w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 15:08:29.676638 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:29.676608    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a817b528-08b2-49a7-bbef-22838f3ca22d-kube-api-access-ppp76 podName:a817b528-08b2-49a7-bbef-22838f3ca22d nodeName:}" failed. No retries permitted until 2026-07-24 15:08:30.176589129 +0000 UTC m=+2.988941291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ppp76" (UniqueName: "kubernetes.io/projected/a817b528-08b2-49a7-bbef-22838f3ca22d-kube-api-access-ppp76") pod "network-check-target-z654w" (UID: "a817b528-08b2-49a7-bbef-22838f3ca22d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 15:08:29.678947 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.678924    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l72f\" (UniqueName: \"kubernetes.io/projected/699558f6-ffac-4afd-b1b0-ca6f0611c47e-kube-api-access-2l72f\") pod \"aws-ebs-csi-driver-node-9t4rt\" (UID: \"699558f6-ffac-4afd-b1b0-ca6f0611c47e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9t4rt"
Jul 24 15:08:29.678947 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.678939    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4bzx\" (UniqueName: \"kubernetes.io/projected/e0b197de-537f-457b-8d3f-4dc75d38c402-kube-api-access-f4bzx\") pod \"network-metrics-daemon-qgzfw\" (UID: \"e0b197de-537f-457b-8d3f-4dc75d38c402\") " pod="openshift-multus/network-metrics-daemon-qgzfw"
Jul 24 15:08:29.682798 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.682773    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c94nk\" (UniqueName: \"kubernetes.io/projected/e3a67ea4-031f-45a0-860f-1771acbbb3be-kube-api-access-c94nk\") pod \"multus-97xpk\" (UID: \"e3a67ea4-031f-45a0-860f-1771acbbb3be\") " pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.682798 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.682776    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4vzq\" (UniqueName: \"kubernetes.io/projected/ca98516e-79dc-4972-86de-8ff390bc0de2-kube-api-access-r4vzq\") pod \"node-resolver-s5nzq\" (UID: \"ca98516e-79dc-4972-86de-8ff390bc0de2\") " pod="openshift-dns/node-resolver-s5nzq"
Jul 24 15:08:29.763335 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.763298    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2216188a-b1f1-4ddf-8f9e-346e93860df2-serviceca\") pod \"node-ca-n9snp\" (UID: \"2216188a-b1f1-4ddf-8f9e-346e93860df2\") " pod="openshift-image-registry/node-ca-n9snp"
Jul 24 15:08:29.763525 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.763351    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-var-lib-openvswitch\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.763525 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.763378    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-run-openvswitch\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.763525 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.763400    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b97ec16-7040-4aa3-b2a8-0827c3698c7a-cnibin\") pod \"multus-additional-cni-plugins-g4ts4\" (UID: \"1b97ec16-7040-4aa3-b2a8-0827c3698c7a\") " pod="openshift-multus/multus-additional-cni-plugins-g4ts4"
Jul 24 15:08:29.763525 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.763426    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1b97ec16-7040-4aa3-b2a8-0827c3698c7a-os-release\") pod \"multus-additional-cni-plugins-g4ts4\" (UID: \"1b97ec16-7040-4aa3-b2a8-0827c3698c7a\") " pod="openshift-multus/multus-additional-cni-plugins-g4ts4"
Jul 24 15:08:29.763525 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.763449    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1fe7cd76-ff78-43b1-b33c-96b4591a61db-var-lib-kubelet\") pod \"tuned-gm95d\" (UID: \"1fe7cd76-ff78-43b1-b33c-96b4591a61db\") " pod="openshift-cluster-node-tuning-operator/tuned-gm95d"
Jul 24 15:08:29.763525 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.763471    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2216188a-b1f1-4ddf-8f9e-346e93860df2-host\") pod \"node-ca-n9snp\" (UID: \"2216188a-b1f1-4ddf-8f9e-346e93860df2\") " pod="openshift-image-registry/node-ca-n9snp"
Jul 24 15:08:29.763525 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.763497    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e4dfc4ee-1d7a-4e52-a295-efc820c560bd-konnectivity-ca\") pod \"konnectivity-agent-d7d7z\" (UID: \"e4dfc4ee-1d7a-4e52-a295-efc820c560bd\") " pod="kube-system/konnectivity-agent-d7d7z"
Jul 24 15:08:29.763525 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.763525    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70-original-pull-secret\") pod \"global-pull-secret-syncer-7qc8n\" (UID: \"5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70\") " pod="kube-system/global-pull-secret-syncer-7qc8n"
Jul 24 15:08:29.763923 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.763534    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-run-openvswitch\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.763923 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.763592    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1fe7cd76-ff78-43b1-b33c-96b4591a61db-var-lib-kubelet\") pod \"tuned-gm95d\" (UID: \"1fe7cd76-ff78-43b1-b33c-96b4591a61db\") " pod="openshift-cluster-node-tuning-operator/tuned-gm95d"
Jul 24 15:08:29.763923 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.763606    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2216188a-b1f1-4ddf-8f9e-346e93860df2-host\") pod \"node-ca-n9snp\" (UID: \"2216188a-b1f1-4ddf-8f9e-346e93860df2\") " pod="openshift-image-registry/node-ca-n9snp"
Jul 24 15:08:29.763923 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.763606    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-var-lib-openvswitch\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.763923 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.763633    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/842eecfb-c927-4ef0-84fc-edba5a668fb2-host-slash\") pod \"iptables-alerter-k6zwd\" (UID: \"842eecfb-c927-4ef0-84fc-edba5a668fb2\") " pod="openshift-network-operator/iptables-alerter-k6zwd"
Jul 24 15:08:29.763923 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.763664    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1fe7cd76-ff78-43b1-b33c-96b4591a61db-etc-kubernetes\") pod \"tuned-gm95d\" (UID: \"1fe7cd76-ff78-43b1-b33c-96b4591a61db\") " pod="openshift-cluster-node-tuning-operator/tuned-gm95d"
Jul 24 15:08:29.763923 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.763689    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lwtw\" (UniqueName: \"kubernetes.io/projected/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-kube-api-access-5lwtw\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.763923 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.763714    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70-dbus\") pod \"global-pull-secret-syncer-7qc8n\" (UID: \"5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70\") " pod="kube-system/global-pull-secret-syncer-7qc8n"
Jul 24 15:08:29.763923 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.763739    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1fe7cd76-ff78-43b1-b33c-96b4591a61db-etc-tuned\") pod \"tuned-gm95d\" (UID: \"1fe7cd76-ff78-43b1-b33c-96b4591a61db\") " pod="openshift-cluster-node-tuning-operator/tuned-gm95d"
Jul 24 15:08:29.763923 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.763740    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b97ec16-7040-4aa3-b2a8-0827c3698c7a-cnibin\") pod \"multus-additional-cni-plugins-g4ts4\" (UID: \"1b97ec16-7040-4aa3-b2a8-0827c3698c7a\") " pod="openshift-multus/multus-additional-cni-plugins-g4ts4"
Jul 24 15:08:29.763923 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.763777    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-node-log\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.763923 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.763795    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2216188a-b1f1-4ddf-8f9e-346e93860df2-serviceca\") pod \"node-ca-n9snp\" (UID: \"2216188a-b1f1-4ddf-8f9e-346e93860df2\") " pod="openshift-image-registry/node-ca-n9snp"
Jul 24 15:08:29.763923 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.763807    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1fe7cd76-ff78-43b1-b33c-96b4591a61db-sys\") pod \"tuned-gm95d\" (UID: \"1fe7cd76-ff78-43b1-b33c-96b4591a61db\") " pod="openshift-cluster-node-tuning-operator/tuned-gm95d"
Jul 24 15:08:29.763923 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.763848    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1fe7cd76-ff78-43b1-b33c-96b4591a61db-host\") pod \"tuned-gm95d\" (UID: \"1fe7cd76-ff78-43b1-b33c-96b4591a61db\") " pod="openshift-cluster-node-tuning-operator/tuned-gm95d"
Jul 24 15:08:29.763923 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.763890    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1b97ec16-7040-4aa3-b2a8-0827c3698c7a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g4ts4\" (UID: \"1b97ec16-7040-4aa3-b2a8-0827c3698c7a\") " pod="openshift-multus/multus-additional-cni-plugins-g4ts4"
Jul 24 15:08:29.763923 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:29.763912    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 15:08:29.764667 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.763958    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1fe7cd76-ff78-43b1-b33c-96b4591a61db-etc-sysconfig\") pod \"tuned-gm95d\" (UID: \"1fe7cd76-ff78-43b1-b33c-96b4591a61db\") " pod="openshift-cluster-node-tuning-operator/tuned-gm95d"
Jul 24 15:08:29.764667 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:29.763971    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70-original-pull-secret podName:5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:30.263953742 +0000 UTC m=+3.076305916 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70-original-pull-secret") pod "global-pull-secret-syncer-7qc8n" (UID: "5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70") : object "kube-system"/"original-pull-secret" not registered
Jul 24 15:08:29.764667 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.763998    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-node-log\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.764667 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.764035    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1b97ec16-7040-4aa3-b2a8-0827c3698c7a-os-release\") pod \"multus-additional-cni-plugins-g4ts4\" (UID: \"1b97ec16-7040-4aa3-b2a8-0827c3698c7a\") " pod="openshift-multus/multus-additional-cni-plugins-g4ts4"
Jul 24 15:08:29.764667 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.763916    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1fe7cd76-ff78-43b1-b33c-96b4591a61db-etc-sysconfig\") pod \"tuned-gm95d\" (UID: \"1fe7cd76-ff78-43b1-b33c-96b4591a61db\") " pod="openshift-cluster-node-tuning-operator/tuned-gm95d"
Jul 24 15:08:29.764667 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.764050    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1fe7cd76-ff78-43b1-b33c-96b4591a61db-sys\") pod \"tuned-gm95d\" (UID: \"1fe7cd76-ff78-43b1-b33c-96b4591a61db\") " pod="openshift-cluster-node-tuning-operator/tuned-gm95d"
Jul 24 15:08:29.764667 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.764077    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx2rl\" (UniqueName: \"kubernetes.io/projected/1fe7cd76-ff78-43b1-b33c-96b4591a61db-kube-api-access-mx2rl\") pod \"tuned-gm95d\" (UID: \"1fe7cd76-ff78-43b1-b33c-96b4591a61db\") " pod="openshift-cluster-node-tuning-operator/tuned-gm95d"
Jul 24 15:08:29.764667 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.764098    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1fe7cd76-ff78-43b1-b33c-96b4591a61db-host\") pod \"tuned-gm95d\" (UID: \"1fe7cd76-ff78-43b1-b33c-96b4591a61db\") " pod="openshift-cluster-node-tuning-operator/tuned-gm95d"
Jul 24 15:08:29.764667 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.764104    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-host-cni-netd\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.764667 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.764137    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-host-cni-netd\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.764667 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.764165    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/842eecfb-c927-4ef0-84fc-edba5a668fb2-iptables-alerter-script\") pod \"iptables-alerter-k6zwd\" (UID: \"842eecfb-c927-4ef0-84fc-edba5a668fb2\") " pod="openshift-network-operator/iptables-alerter-k6zwd"
Jul 24 15:08:29.764667 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.764198    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-etc-openvswitch\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.764667 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.764228    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-host-cni-bin\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.764667 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.764254    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kjmt\" (UniqueName: \"kubernetes.io/projected/842eecfb-c927-4ef0-84fc-edba5a668fb2-kube-api-access-9kjmt\") pod \"iptables-alerter-k6zwd\" (UID: \"842eecfb-c927-4ef0-84fc-edba5a668fb2\") " pod="openshift-network-operator/iptables-alerter-k6zwd"
Jul 24 15:08:29.764667 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.764273    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e4dfc4ee-1d7a-4e52-a295-efc820c560bd-konnectivity-ca\") pod \"konnectivity-agent-d7d7z\" (UID: \"e4dfc4ee-1d7a-4e52-a295-efc820c560bd\") " pod="kube-system/konnectivity-agent-d7d7z"
Jul 24 15:08:29.764667 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.764280    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1fe7cd76-ff78-43b1-b33c-96b4591a61db-etc-sysctl-conf\") pod \"tuned-gm95d\" (UID: \"1fe7cd76-ff78-43b1-b33c-96b4591a61db\") " pod="openshift-cluster-node-tuning-operator/tuned-gm95d"
Jul 24 15:08:29.764667 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.764309    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70-kubelet-config\") pod \"global-pull-secret-syncer-7qc8n\" (UID: \"5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70\") " pod="kube-system/global-pull-secret-syncer-7qc8n"
Jul 24 15:08:29.765701 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.764331    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-etc-openvswitch\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.765701 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.764334    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-host-kubelet\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.765701 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.764376    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-systemd-units\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.765701 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.764404    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-run-systemd\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.765701 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.764432    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1fe7cd76-ff78-43b1-b33c-96b4591a61db-etc-kubernetes\") pod \"tuned-gm95d\" (UID: \"1fe7cd76-ff78-43b1-b33c-96b4591a61db\") " pod="openshift-cluster-node-tuning-operator/tuned-gm95d"
Jul 24 15:08:29.765701 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.764434    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b97ec16-7040-4aa3-b2a8-0827c3698c7a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g4ts4\" (UID: \"1b97ec16-7040-4aa3-b2a8-0827c3698c7a\") " pod="openshift-multus/multus-additional-cni-plugins-g4ts4"
Jul 24 15:08:29.765701 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.764438    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-host-cni-bin\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.765701 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.764469    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwv6m\" (UniqueName: \"kubernetes.io/projected/1b97ec16-7040-4aa3-b2a8-0827c3698c7a-kube-api-access-qwv6m\") pod \"multus-additional-cni-plugins-g4ts4\" (UID: \"1b97ec16-7040-4aa3-b2a8-0827c3698c7a\") " pod="openshift-multus/multus-additional-cni-plugins-g4ts4"
Jul 24 15:08:29.765701 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.764494    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-run-ovn\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.765701 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.764517    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-log-socket\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.765701 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.764543    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-host-run-ovn-kubernetes\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.765701 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.764569    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e4dfc4ee-1d7a-4e52-a295-efc820c560bd-agent-certs\") pod \"konnectivity-agent-d7d7z\" (UID: \"e4dfc4ee-1d7a-4e52-a295-efc820c560bd\") " pod="kube-system/konnectivity-agent-d7d7z"
Jul 24 15:08:29.765701 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.764587    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1fe7cd76-ff78-43b1-b33c-96b4591a61db-tmp\") pod \"tuned-gm95d\" (UID: \"1fe7cd76-ff78-43b1-b33c-96b4591a61db\") " pod="openshift-cluster-node-tuning-operator/tuned-gm95d"
Jul 24 15:08:29.765701 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.764590    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1b97ec16-7040-4aa3-b2a8-0827c3698c7a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g4ts4\" (UID: \"1b97ec16-7040-4aa3-b2a8-0827c3698c7a\") " pod="openshift-multus/multus-additional-cni-plugins-g4ts4"
Jul 24 15:08:29.765701 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.764612    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-env-overrides\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.765701 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.764378    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-host-kubelet\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.765701 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.764658    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-log-socket\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.766676 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.764704    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-systemd-units\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.766676 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.764720    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1fe7cd76-ff78-43b1-b33c-96b4591a61db-etc-sysctl-conf\") pod \"tuned-gm95d\" (UID: \"1fe7cd76-ff78-43b1-b33c-96b4591a61db\") " pod="openshift-cluster-node-tuning-operator/tuned-gm95d"
Jul 24 15:08:29.766676 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.764746    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/842eecfb-c927-4ef0-84fc-edba5a668fb2-host-slash\") pod \"iptables-alerter-k6zwd\" (UID: \"842eecfb-c927-4ef0-84fc-edba5a668fb2\") " pod="openshift-network-operator/iptables-alerter-k6zwd"
Jul 24 15:08:29.766676 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.764803    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70-kubelet-config\") pod \"global-pull-secret-syncer-7qc8n\" (UID: \"5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70\") " pod="kube-system/global-pull-secret-syncer-7qc8n"
Jul 24 15:08:29.766676 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.764805    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-run-ovn\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.766676 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.764806    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/842eecfb-c927-4ef0-84fc-edba5a668fb2-iptables-alerter-script\") pod \"iptables-alerter-k6zwd\" (UID: \"842eecfb-c927-4ef0-84fc-edba5a668fb2\") " pod="openshift-network-operator/iptables-alerter-k6zwd"
Jul 24 15:08:29.766676 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.764834    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-ovnkube-script-lib\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.766676 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.764886    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-run-systemd\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.766676 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.764990    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70-dbus\") pod \"global-pull-secret-syncer-7qc8n\" (UID: \"5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70\") " pod="kube-system/global-pull-secret-syncer-7qc8n"
Jul 24 15:08:29.766676 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.764587    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b97ec16-7040-4aa3-b2a8-0827c3698c7a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g4ts4\" (UID: \"1b97ec16-7040-4aa3-b2a8-0827c3698c7a\") " pod="openshift-multus/multus-additional-cni-plugins-g4ts4"
Jul 24 15:08:29.766676 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.765101    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-host-slash\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.766676 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.765109    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-env-overrides\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.766676 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.765130    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.766676 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.765161    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-host-slash\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.766676 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.765162    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-host-run-ovn-kubernetes\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.766676 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.765162    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1b97ec16-7040-4aa3-b2a8-0827c3698c7a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g4ts4\" (UID: \"1b97ec16-7040-4aa3-b2a8-0827c3698c7a\") " pod="openshift-multus/multus-additional-cni-plugins-g4ts4"
Jul 24 15:08:29.766676 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.765214    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.767643 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.765215    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1fe7cd76-ff78-43b1-b33c-96b4591a61db-etc-systemd\") pod \"tuned-gm95d\" (UID: \"1fe7cd76-ff78-43b1-b33c-96b4591a61db\") " pod="openshift-cluster-node-tuning-operator/tuned-gm95d"
Jul 24 15:08:29.767643 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.765246    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-ovnkube-config\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.767643 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.765277    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-host-run-netns\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.767643 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.765304    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-ovn-node-metrics-cert\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.767643 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.765337    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b97ec16-7040-4aa3-b2a8-0827c3698c7a-system-cni-dir\") pod \"multus-additional-cni-plugins-g4ts4\" (UID: \"1b97ec16-7040-4aa3-b2a8-0827c3698c7a\") " pod="openshift-multus/multus-additional-cni-plugins-g4ts4"
Jul 24 15:08:29.767643 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.765362    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1fe7cd76-ff78-43b1-b33c-96b4591a61db-etc-sysctl-d\") pod \"tuned-gm95d\" (UID: \"1fe7cd76-ff78-43b1-b33c-96b4591a61db\") " pod="openshift-cluster-node-tuning-operator/tuned-gm95d"
Jul 24 15:08:29.767643 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.765385    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1fe7cd76-ff78-43b1-b33c-96b4591a61db-lib-modules\") pod \"tuned-gm95d\" (UID: \"1fe7cd76-ff78-43b1-b33c-96b4591a61db\") " pod="openshift-cluster-node-tuning-operator/tuned-gm95d"
Jul 24 15:08:29.767643 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.765409    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtnfj\" (UniqueName: \"kubernetes.io/projected/2216188a-b1f1-4ddf-8f9e-346e93860df2-kube-api-access-gtnfj\") pod \"node-ca-n9snp\" (UID: \"2216188a-b1f1-4ddf-8f9e-346e93860df2\") " pod="openshift-image-registry/node-ca-n9snp"
Jul 24 15:08:29.767643 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.765437    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b97ec16-7040-4aa3-b2a8-0827c3698c7a-cni-binary-copy\") pod \"multus-additional-cni-plugins-g4ts4\" (UID: \"1b97ec16-7040-4aa3-b2a8-0827c3698c7a\") " pod="openshift-multus/multus-additional-cni-plugins-g4ts4"
Jul 24 15:08:29.767643 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.765461    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1fe7cd76-ff78-43b1-b33c-96b4591a61db-etc-modprobe-d\") pod \"tuned-gm95d\" (UID: \"1fe7cd76-ff78-43b1-b33c-96b4591a61db\") " pod="openshift-cluster-node-tuning-operator/tuned-gm95d"
Jul 24 15:08:29.767643 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.765486    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1fe7cd76-ff78-43b1-b33c-96b4591a61db-run\") pod \"tuned-gm95d\" (UID: \"1fe7cd76-ff78-43b1-b33c-96b4591a61db\") " pod="openshift-cluster-node-tuning-operator/tuned-gm95d"
Jul 24 15:08:29.767643 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.765536    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1fe7cd76-ff78-43b1-b33c-96b4591a61db-etc-systemd\") pod \"tuned-gm95d\" (UID: \"1fe7cd76-ff78-43b1-b33c-96b4591a61db\") " pod="openshift-cluster-node-tuning-operator/tuned-gm95d"
Jul 24 15:08:29.767643 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.765572    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1fe7cd76-ff78-43b1-b33c-96b4591a61db-run\") pod \"tuned-gm95d\" (UID: \"1fe7cd76-ff78-43b1-b33c-96b4591a61db\") " pod="openshift-cluster-node-tuning-operator/tuned-gm95d"
Jul 24 15:08:29.767643 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.765622    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1b97ec16-7040-4aa3-b2a8-0827c3698c7a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g4ts4\" (UID: \"1b97ec16-7040-4aa3-b2a8-0827c3698c7a\") " pod="openshift-multus/multus-additional-cni-plugins-g4ts4"
Jul 24 15:08:29.767643 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.765612    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-host-run-netns\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.767643 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.765798    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1fe7cd76-ff78-43b1-b33c-96b4591a61db-lib-modules\") pod \"tuned-gm95d\" (UID: \"1fe7cd76-ff78-43b1-b33c-96b4591a61db\") " pod="openshift-cluster-node-tuning-operator/tuned-gm95d"
Jul 24 15:08:29.767643 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.765850    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b97ec16-7040-4aa3-b2a8-0827c3698c7a-system-cni-dir\") pod \"multus-additional-cni-plugins-g4ts4\" (UID: \"1b97ec16-7040-4aa3-b2a8-0827c3698c7a\") " pod="openshift-multus/multus-additional-cni-plugins-g4ts4"
Jul 24 15:08:29.768492 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.765979    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1fe7cd76-ff78-43b1-b33c-96b4591a61db-etc-sysctl-d\") pod \"tuned-gm95d\" (UID: \"1fe7cd76-ff78-43b1-b33c-96b4591a61db\") " pod="openshift-cluster-node-tuning-operator/tuned-gm95d"
Jul 24 15:08:29.768492 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.766052    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-ovnkube-script-lib\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.768492 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.766083    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1fe7cd76-ff78-43b1-b33c-96b4591a61db-etc-modprobe-d\") pod \"tuned-gm95d\" (UID: \"1fe7cd76-ff78-43b1-b33c-96b4591a61db\") " pod="openshift-cluster-node-tuning-operator/tuned-gm95d"
Jul 24 15:08:29.768492 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.766146    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-ovnkube-config\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.768492 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.766371    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b97ec16-7040-4aa3-b2a8-0827c3698c7a-cni-binary-copy\") pod \"multus-additional-cni-plugins-g4ts4\" (UID: \"1b97ec16-7040-4aa3-b2a8-0827c3698c7a\") " pod="openshift-multus/multus-additional-cni-plugins-g4ts4"
Jul 24 15:08:29.768492 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.767401    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1fe7cd76-ff78-43b1-b33c-96b4591a61db-etc-tuned\") pod \"tuned-gm95d\" (UID: \"1fe7cd76-ff78-43b1-b33c-96b4591a61db\") " pod="openshift-cluster-node-tuning-operator/tuned-gm95d"
Jul 24 15:08:29.768492 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.767621    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e4dfc4ee-1d7a-4e52-a295-efc820c560bd-agent-certs\") pod \"konnectivity-agent-d7d7z\" (UID: \"e4dfc4ee-1d7a-4e52-a295-efc820c560bd\") " pod="kube-system/konnectivity-agent-d7d7z"
Jul 24 15:08:29.769337 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.769228    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-ovn-node-metrics-cert\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.769337 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.769297    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1fe7cd76-ff78-43b1-b33c-96b4591a61db-tmp\") pod \"tuned-gm95d\" (UID: \"1fe7cd76-ff78-43b1-b33c-96b4591a61db\") " pod="openshift-cluster-node-tuning-operator/tuned-gm95d"
Jul 24 15:08:29.774985 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.774934    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kjmt\" (UniqueName: \"kubernetes.io/projected/842eecfb-c927-4ef0-84fc-edba5a668fb2-kube-api-access-9kjmt\") pod \"iptables-alerter-k6zwd\" (UID: \"842eecfb-c927-4ef0-84fc-edba5a668fb2\") " pod="openshift-network-operator/iptables-alerter-k6zwd"
Jul 24 15:08:29.775732 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.775704    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx2rl\" (UniqueName: \"kubernetes.io/projected/1fe7cd76-ff78-43b1-b33c-96b4591a61db-kube-api-access-mx2rl\") pod \"tuned-gm95d\" (UID: \"1fe7cd76-ff78-43b1-b33c-96b4591a61db\") " pod="openshift-cluster-node-tuning-operator/tuned-gm95d"
Jul 24 15:08:29.776452 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.775737    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwv6m\" (UniqueName: \"kubernetes.io/projected/1b97ec16-7040-4aa3-b2a8-0827c3698c7a-kube-api-access-qwv6m\") pod \"multus-additional-cni-plugins-g4ts4\" (UID: \"1b97ec16-7040-4aa3-b2a8-0827c3698c7a\") " pod="openshift-multus/multus-additional-cni-plugins-g4ts4"
Jul 24 15:08:29.776805 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.776776    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lwtw\" (UniqueName: \"kubernetes.io/projected/f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e-kube-api-access-5lwtw\") pod \"ovnkube-node-52fwk\" (UID: \"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:29.777634 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.777611    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtnfj\" (UniqueName: \"kubernetes.io/projected/2216188a-b1f1-4ddf-8f9e-346e93860df2-kube-api-access-gtnfj\") pod \"node-ca-n9snp\" (UID: \"2216188a-b1f1-4ddf-8f9e-346e93860df2\") " pod="openshift-image-registry/node-ca-n9snp"
Jul 24 15:08:29.860155 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.860122    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9t4rt"
Jul 24 15:08:29.878936 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.878889    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s5nzq"
Jul 24 15:08:29.888880 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.888853    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-97xpk"
Jul 24 15:08:29.895786 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.895757    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g4ts4"
Jul 24 15:08:29.902589 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.902565    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d7d7z"
Jul 24 15:08:29.911289 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.911270    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k6zwd"
Jul 24 15:08:29.920163 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.920141    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gm95d"
Jul 24 15:08:29.927059 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.927038    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n9snp"
Jul 24 15:08:29.933788 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:29.933768    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:30.003906 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:30.003871    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 15:08:30.139922 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:30.139813    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 15:08:30.168043 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:30.168012    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0b197de-537f-457b-8d3f-4dc75d38c402-metrics-certs\") pod \"network-metrics-daemon-qgzfw\" (UID: \"e0b197de-537f-457b-8d3f-4dc75d38c402\") " pod="openshift-multus/network-metrics-daemon-qgzfw"
Jul 24 15:08:30.168228 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:30.168189    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 15:08:30.168284 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:30.168269    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0b197de-537f-457b-8d3f-4dc75d38c402-metrics-certs podName:e0b197de-537f-457b-8d3f-4dc75d38c402 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:31.168250682 +0000 UTC m=+3.980602867 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0b197de-537f-457b-8d3f-4dc75d38c402-metrics-certs") pod "network-metrics-daemon-qgzfw" (UID: "e0b197de-537f-457b-8d3f-4dc75d38c402") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 15:08:30.269202 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:30.269161    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70-original-pull-secret\") pod \"global-pull-secret-syncer-7qc8n\" (UID: \"5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70\") " pod="kube-system/global-pull-secret-syncer-7qc8n"
Jul 24 15:08:30.269400 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:30.269220    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppp76\" (UniqueName: \"kubernetes.io/projected/a817b528-08b2-49a7-bbef-22838f3ca22d-kube-api-access-ppp76\") pod \"network-check-target-z654w\" (UID: \"a817b528-08b2-49a7-bbef-22838f3ca22d\") " pod="openshift-network-diagnostics/network-check-target-z654w"
Jul 24 15:08:30.269400 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:30.269345    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 15:08:30.269400 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:30.269373    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 15:08:30.269400 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:30.269390    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 15:08:30.269400 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:30.269402    2579 projected.go:196] Error preparing data for projected volume kube-api-access-ppp76 for pod openshift-network-diagnostics/network-check-target-z654w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 15:08:30.269609 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:30.269424    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70-original-pull-secret podName:5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:31.269409013 +0000 UTC m=+4.081761171 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70-original-pull-secret") pod "global-pull-secret-syncer-7qc8n" (UID: "5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70") : object "kube-system"/"original-pull-secret" not registered
Jul 24 15:08:30.269609 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:30.269456    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a817b528-08b2-49a7-bbef-22838f3ca22d-kube-api-access-ppp76 podName:a817b528-08b2-49a7-bbef-22838f3ca22d nodeName:}" failed. No retries permitted until 2026-07-24 15:08:31.269440055 +0000 UTC m=+4.081792229 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ppp76" (UniqueName: "kubernetes.io/projected/a817b528-08b2-49a7-bbef-22838f3ca22d-kube-api-access-ppp76") pod "network-check-target-z654w" (UID: "a817b528-08b2-49a7-bbef-22838f3ca22d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 15:08:30.617551 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:30.617417    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 15:03:29 +0000 UTC" deadline="2028-03-23 13:44:08.64828649 +0000 UTC"
Jul 24 15:08:30.617551 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:30.617487    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14590h35m38.030804133s"
Jul 24 15:08:30.711416 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:30.711378    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7qc8n"
Jul 24 15:08:30.711600 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:30.711498    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7qc8n" podUID="5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70"
Jul 24 15:08:30.768373 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:30.768318    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b97ec16_7040_4aa3_b2a8_0827c3698c7a.slice/crio-7d485629dd8cdad50e5613b794fa32f791dd195f93bfbb89aed3414a6c195d64 WatchSource:0}: Error finding container 7d485629dd8cdad50e5613b794fa32f791dd195f93bfbb89aed3414a6c195d64: Status 404 returned error can't find the container with id 7d485629dd8cdad50e5613b794fa32f791dd195f93bfbb89aed3414a6c195d64
Jul 24 15:08:30.772450 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:30.772421    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod842eecfb_c927_4ef0_84fc_edba5a668fb2.slice/crio-1525a2f5aa2802aa485c3d41d5fe71ee4ad56063381bffc59a6255f7d1af801c WatchSource:0}: Error finding container 1525a2f5aa2802aa485c3d41d5fe71ee4ad56063381bffc59a6255f7d1af801c: Status 404 returned error can't find the container with id 1525a2f5aa2802aa485c3d41d5fe71ee4ad56063381bffc59a6255f7d1af801c
Jul 24 15:08:30.773249 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:30.773224    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2216188a_b1f1_4ddf_8f9e_346e93860df2.slice/crio-dff941e902efa77500dbe0b7b755c462255bdeaaca905a6ba663c8556a640dd4 WatchSource:0}: Error finding container dff941e902efa77500dbe0b7b755c462255bdeaaca905a6ba663c8556a640dd4: Status 404 returned error can't find the container with id dff941e902efa77500dbe0b7b755c462255bdeaaca905a6ba663c8556a640dd4
Jul 24 15:08:30.774400 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:30.774240    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4dfc4ee_1d7a_4e52_a295_efc820c560bd.slice/crio-6f01a7f5e0b053ac4b87cbb5a4a2c30546323a454dc957fe2eba37941157caa1 WatchSource:0}: Error finding container 6f01a7f5e0b053ac4b87cbb5a4a2c30546323a454dc957fe2eba37941157caa1: Status 404 returned error can't find the container with id 6f01a7f5e0b053ac4b87cbb5a4a2c30546323a454dc957fe2eba37941157caa1
Jul 24 15:08:30.776587 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:30.776552    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8e012d6_1f9f_4d7f_9dda_d4ee365d5b4e.slice/crio-92fa068a199ebe97020ea2c044c1dfbec3bf5343ff9737c0e8f91b5107c33290 WatchSource:0}: Error finding container 92fa068a199ebe97020ea2c044c1dfbec3bf5343ff9737c0e8f91b5107c33290: Status 404 returned error can't find the container with id 92fa068a199ebe97020ea2c044c1dfbec3bf5343ff9737c0e8f91b5107c33290
Jul 24 15:08:30.777279 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:30.777172    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca98516e_79dc_4972_86de_8ff390bc0de2.slice/crio-0128119627a7d950cad61d0e874d39fde876b4fcfa066f45ede834b3df7f8d35 WatchSource:0}: Error finding container 0128119627a7d950cad61d0e874d39fde876b4fcfa066f45ede834b3df7f8d35: Status 404 returned error can't find the container with id 0128119627a7d950cad61d0e874d39fde876b4fcfa066f45ede834b3df7f8d35
Jul 24 15:08:30.778255 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:30.778113    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3a67ea4_031f_45a0_860f_1771acbbb3be.slice/crio-10c0cf980a332a3b87700919fe8602707c0be05a2362e6f78d6f71ec68adb1aa WatchSource:0}: Error finding container 10c0cf980a332a3b87700919fe8602707c0be05a2362e6f78d6f71ec68adb1aa: Status 404 returned error can't find the container with id 10c0cf980a332a3b87700919fe8602707c0be05a2362e6f78d6f71ec68adb1aa
Jul 24 15:08:30.779288 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:08:30.779254    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod699558f6_ffac_4afd_b1b0_ca6f0611c47e.slice/crio-8c3a6cef868d213d2944840fc6f3cf7908086d088b6b5f49c770773c70ff1ff8 WatchSource:0}: Error finding container 8c3a6cef868d213d2944840fc6f3cf7908086d088b6b5f49c770773c70ff1ff8: Status 404 returned error can't find the container with id 8c3a6cef868d213d2944840fc6f3cf7908086d088b6b5f49c770773c70ff1ff8
Jul 24 15:08:30.837279 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:30.837255    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 15:08:31.178482 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:31.178443    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0b197de-537f-457b-8d3f-4dc75d38c402-metrics-certs\") pod \"network-metrics-daemon-qgzfw\" (UID: \"e0b197de-537f-457b-8d3f-4dc75d38c402\") " pod="openshift-multus/network-metrics-daemon-qgzfw"
Jul 24 15:08:31.178667 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:31.178579    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 15:08:31.178667 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:31.178635    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0b197de-537f-457b-8d3f-4dc75d38c402-metrics-certs podName:e0b197de-537f-457b-8d3f-4dc75d38c402 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:33.178620639 +0000 UTC m=+5.990972797 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0b197de-537f-457b-8d3f-4dc75d38c402-metrics-certs") pod "network-metrics-daemon-qgzfw" (UID: "e0b197de-537f-457b-8d3f-4dc75d38c402") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 15:08:31.279560 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:31.279478    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70-original-pull-secret\") pod \"global-pull-secret-syncer-7qc8n\" (UID: \"5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70\") " pod="kube-system/global-pull-secret-syncer-7qc8n"
Jul 24 15:08:31.279560 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:31.279526    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppp76\" (UniqueName: \"kubernetes.io/projected/a817b528-08b2-49a7-bbef-22838f3ca22d-kube-api-access-ppp76\") pod \"network-check-target-z654w\" (UID: \"a817b528-08b2-49a7-bbef-22838f3ca22d\") " pod="openshift-network-diagnostics/network-check-target-z654w"
Jul 24 15:08:31.279758 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:31.279637    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 15:08:31.279758 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:31.279649    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 15:08:31.279758 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:31.279668    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 15:08:31.279758 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:31.279682    2579 projected.go:196] Error preparing data for projected volume kube-api-access-ppp76 for pod openshift-network-diagnostics/network-check-target-z654w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 15:08:31.279758 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:31.279722    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70-original-pull-secret podName:5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:33.279700579 +0000 UTC m=+6.092052756 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70-original-pull-secret") pod "global-pull-secret-syncer-7qc8n" (UID: "5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70") : object "kube-system"/"original-pull-secret" not registered
Jul 24 15:08:31.279758 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:31.279738    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a817b528-08b2-49a7-bbef-22838f3ca22d-kube-api-access-ppp76 podName:a817b528-08b2-49a7-bbef-22838f3ca22d nodeName:}" failed. No retries permitted until 2026-07-24 15:08:33.279730926 +0000 UTC m=+6.092083084 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ppp76" (UniqueName: "kubernetes.io/projected/a817b528-08b2-49a7-bbef-22838f3ca22d-kube-api-access-ppp76") pod "network-check-target-z654w" (UID: "a817b528-08b2-49a7-bbef-22838f3ca22d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 15:08:31.618763 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:31.618660    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 15:03:29 +0000 UTC" deadline="2028-02-24 22:03:05.532895029 +0000 UTC"
Jul 24 15:08:31.618763 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:31.618706    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13926h54m33.914194201s"
Jul 24 15:08:31.713455 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:31.713424    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgzfw"
Jul 24 15:08:31.713618 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:31.713572    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgzfw" podUID="e0b197de-537f-457b-8d3f-4dc75d38c402"
Jul 24 15:08:31.714149 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:31.714124    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z654w"
Jul 24 15:08:31.714253 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:31.714235    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z654w" podUID="a817b528-08b2-49a7-bbef-22838f3ca22d"
Jul 24 15:08:31.730253 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:31.730186    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9t4rt" event={"ID":"699558f6-ffac-4afd-b1b0-ca6f0611c47e","Type":"ContainerStarted","Data":"8c3a6cef868d213d2944840fc6f3cf7908086d088b6b5f49c770773c70ff1ff8"}
Jul 24 15:08:31.742776 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:31.742735    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n9snp" event={"ID":"2216188a-b1f1-4ddf-8f9e-346e93860df2","Type":"ContainerStarted","Data":"dff941e902efa77500dbe0b7b755c462255bdeaaca905a6ba663c8556a640dd4"}
Jul 24 15:08:31.749539 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:31.749507    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-97xpk" event={"ID":"e3a67ea4-031f-45a0-860f-1771acbbb3be","Type":"ContainerStarted","Data":"10c0cf980a332a3b87700919fe8602707c0be05a2362e6f78d6f71ec68adb1aa"}
Jul 24 15:08:31.754216 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:31.754170    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s5nzq" event={"ID":"ca98516e-79dc-4972-86de-8ff390bc0de2","Type":"ContainerStarted","Data":"0128119627a7d950cad61d0e874d39fde876b4fcfa066f45ede834b3df7f8d35"}
Jul 24 15:08:31.759087 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:31.758999    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-52fwk" event={"ID":"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e","Type":"ContainerStarted","Data":"92fa068a199ebe97020ea2c044c1dfbec3bf5343ff9737c0e8f91b5107c33290"}
Jul 24 15:08:31.762376 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:31.762313    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d7d7z" event={"ID":"e4dfc4ee-1d7a-4e52-a295-efc820c560bd","Type":"ContainerStarted","Data":"6f01a7f5e0b053ac4b87cbb5a4a2c30546323a454dc957fe2eba37941157caa1"}
Jul 24 15:08:31.774791 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:31.774765    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g4ts4" event={"ID":"1b97ec16-7040-4aa3-b2a8-0827c3698c7a","Type":"ContainerStarted","Data":"7d485629dd8cdad50e5613b794fa32f791dd195f93bfbb89aed3414a6c195d64"}
Jul 24 15:08:31.782832 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:31.782800    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-160.ec2.internal" event={"ID":"6aa67c3d9e8eb538931b1417f3d45520","Type":"ContainerStarted","Data":"aee768214b399952fab5b426a10c040e00e7be538aec34bb4537156648c63792"}
Jul 24 15:08:31.806012 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:31.805165    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-160.ec2.internal" podStartSLOduration=2.805143059 podStartE2EDuration="2.805143059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:08:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:08:31.804763312 +0000 UTC m=+4.617115495" watchObservedRunningTime="2026-07-24 15:08:31.805143059 +0000 UTC m=+4.617495242"
Jul 24 15:08:31.808414 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:31.808310    2579 generic.go:356] "Generic (PLEG): container finished" podID="22fca178f17fa43f468a8cc5fc9fd939" containerID="141eb1fec366d73456ce2d728bcc8e729d3043a8d72c86360f0a226b2e25a029" exitCode=0
Jul 24 15:08:31.808571 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:31.808417    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-160.ec2.internal" event={"ID":"22fca178f17fa43f468a8cc5fc9fd939","Type":"ContainerDied","Data":"141eb1fec366d73456ce2d728bcc8e729d3043a8d72c86360f0a226b2e25a029"}
Jul 24 15:08:31.820731 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:31.819319    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gm95d" event={"ID":"1fe7cd76-ff78-43b1-b33c-96b4591a61db","Type":"ContainerStarted","Data":"7a7a1fe0865dbe3445a8adbade9f7f1eea8f99f3c1a51f121e1696f68c4d4cc1"}
Jul 24 15:08:31.828911 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:31.823195    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k6zwd" event={"ID":"842eecfb-c927-4ef0-84fc-edba5a668fb2","Type":"ContainerStarted","Data":"1525a2f5aa2802aa485c3d41d5fe71ee4ad56063381bffc59a6255f7d1af801c"}
Jul 24 15:08:32.711656 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:32.711143    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7qc8n"
Jul 24 15:08:32.711656 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:32.711287    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7qc8n" podUID="5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70"
Jul 24 15:08:32.840541 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:32.840403    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-160.ec2.internal" event={"ID":"22fca178f17fa43f468a8cc5fc9fd939","Type":"ContainerStarted","Data":"0f3fc6215b88804368befd132c043b32ef0a45cbf115f6d09950af729d7006c5"}
Jul 24 15:08:33.197678 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:33.197590    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0b197de-537f-457b-8d3f-4dc75d38c402-metrics-certs\") pod \"network-metrics-daemon-qgzfw\" (UID: \"e0b197de-537f-457b-8d3f-4dc75d38c402\") " pod="openshift-multus/network-metrics-daemon-qgzfw"
Jul 24 15:08:33.197889 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:33.197758    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 15:08:33.197889 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:33.197837    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0b197de-537f-457b-8d3f-4dc75d38c402-metrics-certs podName:e0b197de-537f-457b-8d3f-4dc75d38c402 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:37.197804224 +0000 UTC m=+10.010156388 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0b197de-537f-457b-8d3f-4dc75d38c402-metrics-certs") pod "network-metrics-daemon-qgzfw" (UID: "e0b197de-537f-457b-8d3f-4dc75d38c402") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 15:08:33.298465 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:33.298428    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppp76\" (UniqueName: \"kubernetes.io/projected/a817b528-08b2-49a7-bbef-22838f3ca22d-kube-api-access-ppp76\") pod \"network-check-target-z654w\" (UID: \"a817b528-08b2-49a7-bbef-22838f3ca22d\") " pod="openshift-network-diagnostics/network-check-target-z654w"
Jul 24 15:08:33.298633 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:33.298525    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70-original-pull-secret\") pod \"global-pull-secret-syncer-7qc8n\" (UID: \"5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70\") " pod="kube-system/global-pull-secret-syncer-7qc8n"
Jul 24 15:08:33.298698 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:33.298648    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 15:08:33.298753 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:33.298708    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70-original-pull-secret podName:5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:37.298689926 +0000 UTC m=+10.111042101 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70-original-pull-secret") pod "global-pull-secret-syncer-7qc8n" (UID: "5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70") : object "kube-system"/"original-pull-secret" not registered
Jul 24 15:08:33.299147 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:33.299125    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 15:08:33.299147 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:33.299149    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 15:08:33.299314 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:33.299162    2579 projected.go:196] Error preparing data for projected volume kube-api-access-ppp76 for pod openshift-network-diagnostics/network-check-target-z654w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 15:08:33.299314 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:33.299219    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a817b528-08b2-49a7-bbef-22838f3ca22d-kube-api-access-ppp76 podName:a817b528-08b2-49a7-bbef-22838f3ca22d nodeName:}" failed. No retries permitted until 2026-07-24 15:08:37.299205431 +0000 UTC m=+10.111557602 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ppp76" (UniqueName: "kubernetes.io/projected/a817b528-08b2-49a7-bbef-22838f3ca22d-kube-api-access-ppp76") pod "network-check-target-z654w" (UID: "a817b528-08b2-49a7-bbef-22838f3ca22d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 15:08:33.711742 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:33.711098    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgzfw"
Jul 24 15:08:33.711742 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:33.711245    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgzfw" podUID="e0b197de-537f-457b-8d3f-4dc75d38c402"
Jul 24 15:08:33.711742 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:33.711349    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z654w"
Jul 24 15:08:33.711742 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:33.711415    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z654w" podUID="a817b528-08b2-49a7-bbef-22838f3ca22d"
Jul 24 15:08:34.712115 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:34.711336    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7qc8n"
Jul 24 15:08:34.712115 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:34.711482    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7qc8n" podUID="5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70"
Jul 24 15:08:35.713885 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:35.713857    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgzfw"
Jul 24 15:08:35.714266 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:35.713989    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgzfw" podUID="e0b197de-537f-457b-8d3f-4dc75d38c402"
Jul 24 15:08:35.714266 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:35.714080    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z654w"
Jul 24 15:08:35.714266 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:35.714178    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z654w" podUID="a817b528-08b2-49a7-bbef-22838f3ca22d"
Jul 24 15:08:36.711476 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:36.711420    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7qc8n"
Jul 24 15:08:36.711663 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:36.711557    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7qc8n" podUID="5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70"
Jul 24 15:08:37.234843 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:37.234378    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0b197de-537f-457b-8d3f-4dc75d38c402-metrics-certs\") pod \"network-metrics-daemon-qgzfw\" (UID: \"e0b197de-537f-457b-8d3f-4dc75d38c402\") " pod="openshift-multus/network-metrics-daemon-qgzfw"
Jul 24 15:08:37.234843 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:37.234554    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 15:08:37.234843 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:37.234618    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0b197de-537f-457b-8d3f-4dc75d38c402-metrics-certs podName:e0b197de-537f-457b-8d3f-4dc75d38c402 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:45.234600279 +0000 UTC m=+18.046952441 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0b197de-537f-457b-8d3f-4dc75d38c402-metrics-certs") pod "network-metrics-daemon-qgzfw" (UID: "e0b197de-537f-457b-8d3f-4dc75d38c402") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 15:08:37.334784 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:37.334736    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70-original-pull-secret\") pod \"global-pull-secret-syncer-7qc8n\" (UID: \"5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70\") " pod="kube-system/global-pull-secret-syncer-7qc8n"
Jul 24 15:08:37.335034 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:37.334794    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppp76\" (UniqueName: \"kubernetes.io/projected/a817b528-08b2-49a7-bbef-22838f3ca22d-kube-api-access-ppp76\") pod \"network-check-target-z654w\" (UID: \"a817b528-08b2-49a7-bbef-22838f3ca22d\") " pod="openshift-network-diagnostics/network-check-target-z654w"
Jul 24 15:08:37.335034 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:37.334924    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 15:08:37.335034 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:37.334957    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 15:08:37.335034 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:37.334976    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 15:08:37.335034 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:37.334989    2579 projected.go:196] Error preparing data for projected volume kube-api-access-ppp76 for pod openshift-network-diagnostics/network-check-target-z654w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 15:08:37.335034 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:37.335001    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70-original-pull-secret podName:5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:45.33498024 +0000 UTC m=+18.147332416 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70-original-pull-secret") pod "global-pull-secret-syncer-7qc8n" (UID: "5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70") : object "kube-system"/"original-pull-secret" not registered
Jul 24 15:08:37.335034 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:37.335035    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a817b528-08b2-49a7-bbef-22838f3ca22d-kube-api-access-ppp76 podName:a817b528-08b2-49a7-bbef-22838f3ca22d nodeName:}" failed. No retries permitted until 2026-07-24 15:08:45.335020282 +0000 UTC m=+18.147372439 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ppp76" (UniqueName: "kubernetes.io/projected/a817b528-08b2-49a7-bbef-22838f3ca22d-kube-api-access-ppp76") pod "network-check-target-z654w" (UID: "a817b528-08b2-49a7-bbef-22838f3ca22d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 15:08:37.711880 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:37.711792    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgzfw"
Jul 24 15:08:37.712043 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:37.711931    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgzfw" podUID="e0b197de-537f-457b-8d3f-4dc75d38c402"
Jul 24 15:08:37.712274 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:37.712254    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z654w"
Jul 24 15:08:37.712369 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:37.712352    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z654w" podUID="a817b528-08b2-49a7-bbef-22838f3ca22d"
Jul 24 15:08:38.710866 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:38.710833    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7qc8n"
Jul 24 15:08:38.711333 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:38.710971    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7qc8n" podUID="5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70"
Jul 24 15:08:39.719289 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:39.719071    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z654w"
Jul 24 15:08:39.719289 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:39.719110    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgzfw"
Jul 24 15:08:39.719289 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:39.719191    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z654w" podUID="a817b528-08b2-49a7-bbef-22838f3ca22d"
Jul 24 15:08:39.719896 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:39.719448    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgzfw" podUID="e0b197de-537f-457b-8d3f-4dc75d38c402"
Jul 24 15:08:40.711468 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:40.711434    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7qc8n"
Jul 24 15:08:40.711664 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:40.711563    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7qc8n" podUID="5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70"
Jul 24 15:08:41.711119 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:41.711029    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z654w"
Jul 24 15:08:41.711553 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:41.711029    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgzfw"
Jul 24 15:08:41.711553 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:41.711176    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z654w" podUID="a817b528-08b2-49a7-bbef-22838f3ca22d"
Jul 24 15:08:41.711553 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:41.711287    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgzfw" podUID="e0b197de-537f-457b-8d3f-4dc75d38c402"
Jul 24 15:08:42.711564 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:42.711527    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7qc8n"
Jul 24 15:08:42.712010 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:42.711675    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7qc8n" podUID="5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70"
Jul 24 15:08:43.725117 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:43.725086    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgzfw"
Jul 24 15:08:43.725529 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:43.725204    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgzfw" podUID="e0b197de-537f-457b-8d3f-4dc75d38c402"
Jul 24 15:08:43.725529 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:43.725252    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z654w"
Jul 24 15:08:43.725529 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:43.725337    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z654w" podUID="a817b528-08b2-49a7-bbef-22838f3ca22d"
Jul 24 15:08:44.711629 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:44.711593    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7qc8n"
Jul 24 15:08:44.711929 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:44.711734    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7qc8n" podUID="5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70"
Jul 24 15:08:45.295954 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:45.295921    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0b197de-537f-457b-8d3f-4dc75d38c402-metrics-certs\") pod \"network-metrics-daemon-qgzfw\" (UID: \"e0b197de-537f-457b-8d3f-4dc75d38c402\") " pod="openshift-multus/network-metrics-daemon-qgzfw"
Jul 24 15:08:45.296342 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:45.296065    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 15:08:45.296342 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:45.296125    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0b197de-537f-457b-8d3f-4dc75d38c402-metrics-certs podName:e0b197de-537f-457b-8d3f-4dc75d38c402 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:01.296109313 +0000 UTC m=+34.108461469 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0b197de-537f-457b-8d3f-4dc75d38c402-metrics-certs") pod "network-metrics-daemon-qgzfw" (UID: "e0b197de-537f-457b-8d3f-4dc75d38c402") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 15:08:45.396576 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:45.396551    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70-original-pull-secret\") pod \"global-pull-secret-syncer-7qc8n\" (UID: \"5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70\") " pod="kube-system/global-pull-secret-syncer-7qc8n"
Jul 24 15:08:45.396685 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:45.396595    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppp76\" (UniqueName: \"kubernetes.io/projected/a817b528-08b2-49a7-bbef-22838f3ca22d-kube-api-access-ppp76\") pod \"network-check-target-z654w\" (UID: \"a817b528-08b2-49a7-bbef-22838f3ca22d\") " pod="openshift-network-diagnostics/network-check-target-z654w"
Jul 24 15:08:45.396751 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:45.396695    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 15:08:45.396808 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:45.396756    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70-original-pull-secret podName:5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:01.396737478 +0000 UTC m=+34.209089636 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70-original-pull-secret") pod "global-pull-secret-syncer-7qc8n" (UID: "5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70") : object "kube-system"/"original-pull-secret" not registered
Jul 24 15:08:45.396808 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:45.396759    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 15:08:45.396808 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:45.396781    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 15:08:45.396808 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:45.396794    2579 projected.go:196] Error preparing data for projected volume kube-api-access-ppp76 for pod openshift-network-diagnostics/network-check-target-z654w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 15:08:45.396983 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:45.396863    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a817b528-08b2-49a7-bbef-22838f3ca22d-kube-api-access-ppp76 podName:a817b528-08b2-49a7-bbef-22838f3ca22d nodeName:}" failed. No retries permitted until 2026-07-24 15:09:01.396848166 +0000 UTC m=+34.209200323 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ppp76" (UniqueName: "kubernetes.io/projected/a817b528-08b2-49a7-bbef-22838f3ca22d-kube-api-access-ppp76") pod "network-check-target-z654w" (UID: "a817b528-08b2-49a7-bbef-22838f3ca22d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 15:08:45.711519 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:45.711487    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgzfw"
Jul 24 15:08:45.711519 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:45.711513    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z654w"
Jul 24 15:08:45.711752 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:45.711635    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgzfw" podUID="e0b197de-537f-457b-8d3f-4dc75d38c402"
Jul 24 15:08:45.711833 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:45.711745    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z654w" podUID="a817b528-08b2-49a7-bbef-22838f3ca22d"
Jul 24 15:08:46.711187 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:46.711153    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7qc8n"
Jul 24 15:08:46.711660 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:46.711258    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7qc8n" podUID="5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70"
Jul 24 15:08:47.712068 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:47.712031    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgzfw"
Jul 24 15:08:47.712513 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:47.712153    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgzfw" podUID="e0b197de-537f-457b-8d3f-4dc75d38c402"
Jul 24 15:08:47.712513 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:47.712236    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z654w"
Jul 24 15:08:47.712513 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:47.712343    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z654w" podUID="a817b528-08b2-49a7-bbef-22838f3ca22d"
Jul 24 15:08:48.711282 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:48.711243    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7qc8n"
Jul 24 15:08:48.711468 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:48.711375    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7qc8n" podUID="5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70"
Jul 24 15:08:49.711365 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:49.711075    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgzfw"
Jul 24 15:08:49.712134 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:49.711462    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgzfw" podUID="e0b197de-537f-457b-8d3f-4dc75d38c402"
Jul 24 15:08:49.712134 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:49.711143    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z654w"
Jul 24 15:08:49.712134 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:49.711537    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z654w" podUID="a817b528-08b2-49a7-bbef-22838f3ca22d"
Jul 24 15:08:49.878468 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:49.878442    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-52fwk" event={"ID":"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e","Type":"ContainerStarted","Data":"b172f7e23c7990e044f69ad4ea4ba1dd8dc15ea4f7d605b27503412481ac52ee"}
Jul 24 15:08:49.878578 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:49.878522    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-52fwk" event={"ID":"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e","Type":"ContainerStarted","Data":"d60c43cb8b775b30c974f757742d6e8c79400bc690458f85492a54c1da6b2c4e"}
Jul 24 15:08:49.879873 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:49.879847    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d7d7z" event={"ID":"e4dfc4ee-1d7a-4e52-a295-efc820c560bd","Type":"ContainerStarted","Data":"f0c823524361975b9e11f096a44461e7bc6ad909f907fc5c96964c8dce9b9acf"}
Jul 24 15:08:49.881392 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:49.881370    2579 generic.go:356] "Generic (PLEG): container finished" podID="1b97ec16-7040-4aa3-b2a8-0827c3698c7a" containerID="dfe1ab4bce9d27f669aa0ee253bf898389739e741f79433abec6c35436116d01" exitCode=0
Jul 24 15:08:49.881491 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:49.881443    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g4ts4" event={"ID":"1b97ec16-7040-4aa3-b2a8-0827c3698c7a","Type":"ContainerDied","Data":"dfe1ab4bce9d27f669aa0ee253bf898389739e741f79433abec6c35436116d01"}
Jul 24 15:08:49.882870 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:49.882846    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gm95d" event={"ID":"1fe7cd76-ff78-43b1-b33c-96b4591a61db","Type":"ContainerStarted","Data":"3490ccfcbcf120101bf562a98890a7bf3c9dbe57947d58226fdbadfd85fc437b"}
Jul 24 15:08:49.884459 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:49.884422    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9t4rt" event={"ID":"699558f6-ffac-4afd-b1b0-ca6f0611c47e","Type":"ContainerStarted","Data":"0d47ae9703a79d4a216a6ae165f7cd440e79b2a920dc38ce0921736f87ac7fb6"}
Jul 24 15:08:49.889713 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:49.889686    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n9snp" event={"ID":"2216188a-b1f1-4ddf-8f9e-346e93860df2","Type":"ContainerStarted","Data":"0ce8dc7bec0c04ee38a4a3215054e7c3bc890488549a0ffcfd2e58c57854152c"}
Jul 24 15:08:49.891252 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:49.891233    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-97xpk" event={"ID":"e3a67ea4-031f-45a0-860f-1771acbbb3be","Type":"ContainerStarted","Data":"dd76b3cb816453b13f0416efd209595178014a20437a6e9964feaba3a50935d5"}
Jul 24 15:08:49.892482 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:49.892464    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s5nzq" event={"ID":"ca98516e-79dc-4972-86de-8ff390bc0de2","Type":"ContainerStarted","Data":"f856a57d562815daede3b6f074508c7df3870dbb87940009a995130cc79b5263"}
Jul 24 15:08:49.895890 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:49.895856    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-d7d7z" podStartSLOduration=3.340172991 podStartE2EDuration="21.895845106s" totalImagesPullingTime="18.555672115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:08:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:08:49.895140926 +0000 UTC m=+22.707493105" watchObservedRunningTime="2026-07-24 15:08:49.895845106 +0000 UTC m=+22.708197283"
Jul 24 15:08:49.896018 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:49.896001    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-160.ec2.internal" podStartSLOduration=20.895995522 podStartE2EDuration="20.895995522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:08:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:08:32.85983878 +0000 UTC m=+5.672190965" watchObservedRunningTime="2026-07-24 15:08:49.895995522 +0000 UTC m=+22.708347700"
Jul 24 15:08:49.936761 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:49.936713    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gm95d" podStartSLOduration=3.311559393 podStartE2EDuration="21.936698595s" totalImagesPullingTime="18.625139202s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:08:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:08:49.935809483 +0000 UTC m=+22.748161662" watchObservedRunningTime="2026-07-24 15:08:49.936698595 +0000 UTC m=+22.749050773"
Jul 24 15:08:49.957684 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:49.957634    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-n9snp" podStartSLOduration=3.325837063 podStartE2EDuration="21.957613542s" totalImagesPullingTime="18.631776479s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:08:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:08:49.957101292 +0000 UTC m=+22.769453473" watchObservedRunningTime="2026-07-24 15:08:49.957613542 +0000 UTC m=+22.769965728"
Jul 24 15:08:49.973089 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:49.973053    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-s5nzq" podStartSLOduration=3.340153989 podStartE2EDuration="21.973039019s" totalImagesPullingTime="18.63288503s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:08:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:08:49.972839919 +0000 UTC m=+22.785192098" watchObservedRunningTime="2026-07-24 15:08:49.973039019 +0000 UTC m=+22.785391199"
Jul 24 15:08:49.996973 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:49.996931    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-97xpk" podStartSLOduration=3.363145288 podStartE2EDuration="21.996917094s" totalImagesPullingTime="18.633771806s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:08:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:08:49.995299037 +0000 UTC m=+22.807651220" watchObservedRunningTime="2026-07-24 15:08:49.996917094 +0000 UTC m=+22.809269307"
Jul 24 15:08:50.711675 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:50.711643    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7qc8n"
Jul 24 15:08:50.712314 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:50.711800    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7qc8n" podUID="5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70"
Jul 24 15:08:50.897389 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:50.897356    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-52fwk" event={"ID":"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e","Type":"ContainerStarted","Data":"2109cbefd5201c380bf888728dab5c7e03dcb4b55c7ced8815eee06df848ab2e"}
Jul 24 15:08:50.897549 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:50.897451    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-52fwk" event={"ID":"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e","Type":"ContainerStarted","Data":"77f6c6d0f5af5d83a62d4358d847793c3911601043de5ebf782e2111def341b4"}
Jul 24 15:08:50.897549 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:50.897503    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-52fwk" event={"ID":"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e","Type":"ContainerStarted","Data":"dc2273b4b803dd1675f23bd00d947007112aedfaf5bb44b2748d23dca43dc06b"}
Jul 24 15:08:50.897655 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:50.897550    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-52fwk" event={"ID":"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e","Type":"ContainerStarted","Data":"79df353093cc43bc8199ff596eb55a7dde9b3957ec9c0f952183154174346346"}
Jul 24 15:08:50.898886 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:50.898854    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k6zwd" event={"ID":"842eecfb-c927-4ef0-84fc-edba5a668fb2","Type":"ContainerStarted","Data":"610aac7732a992b123eab2d85465b9b7a7cdfec1b497d4c10717c2717033981a"}
Jul 24 15:08:50.915593 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:50.915546    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-k6zwd" podStartSLOduration=4.357868388 podStartE2EDuration="22.915525417s" totalImagesPullingTime="18.557657029s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:08:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:08:50.915331732 +0000 UTC m=+23.727683911" watchObservedRunningTime="2026-07-24 15:08:50.915525417 +0000 UTC m=+23.727877597"
Jul 24 15:08:50.958174 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:50.958142    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 24 15:08:51.623767 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:51.623639    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T15:08:50.9581698Z","UUID":"7022da1d-0a55-4739-8daf-34d62880eb3b","Handler":null,"Name":"","Endpoint":""}
Jul 24 15:08:51.625353 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:51.625329    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 24 15:08:51.625491 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:51.625370    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 24 15:08:51.711932 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:51.711899    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgzfw"
Jul 24 15:08:51.712345 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:51.712017    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgzfw" podUID="e0b197de-537f-457b-8d3f-4dc75d38c402"
Jul 24 15:08:51.712345 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:51.712076    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z654w"
Jul 24 15:08:51.712345 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:51.712187    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z654w" podUID="a817b528-08b2-49a7-bbef-22838f3ca22d"
Jul 24 15:08:51.905461 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:51.905375    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9t4rt" event={"ID":"699558f6-ffac-4afd-b1b0-ca6f0611c47e","Type":"ContainerStarted","Data":"b381b176465316816127aecb2720bcb5b75b7b2470ec12c9347a1543657198f2"}
Jul 24 15:08:52.711649 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:52.711444    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7qc8n"
Jul 24 15:08:52.711889 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:52.711749    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7qc8n" podUID="5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70"
Jul 24 15:08:52.909746 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:52.909709    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9t4rt" event={"ID":"699558f6-ffac-4afd-b1b0-ca6f0611c47e","Type":"ContainerStarted","Data":"386745959c9b96262225515b03f1573fb0eb9f21482faa1636b126c4b7462d26"}
Jul 24 15:08:52.913292 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:52.913264    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-52fwk" event={"ID":"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e","Type":"ContainerStarted","Data":"1100a0373216c2362021c1eb319198d31818a4fa5b9267aa83c32d2e552094ee"}
Jul 24 15:08:52.931689 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:52.931637    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9t4rt" podStartSLOduration=4.008712277 podStartE2EDuration="24.931617949s" totalImagesPullingTime="20.922905672s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:08:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:08:52.930023073 +0000 UTC m=+25.742375276" watchObservedRunningTime="2026-07-24 15:08:52.931617949 +0000 UTC m=+25.743970130"
Jul 24 15:08:53.416026 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:53.415991    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-d7d7z"
Jul 24 15:08:53.416713 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:53.416694    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-d7d7z"
Jul 24 15:08:53.711388 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:53.711308    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z654w"
Jul 24 15:08:53.711388 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:53.711341    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgzfw"
Jul 24 15:08:53.711608 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:53.711439    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z654w" podUID="a817b528-08b2-49a7-bbef-22838f3ca22d"
Jul 24 15:08:53.711799 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:53.711767    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgzfw" podUID="e0b197de-537f-457b-8d3f-4dc75d38c402"
Jul 24 15:08:53.916059 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:53.916026    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-d7d7z"
Jul 24 15:08:53.916609 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:53.916498    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-d7d7z"
Jul 24 15:08:54.711512 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:54.711364    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7qc8n"
Jul 24 15:08:54.711711 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:54.711506    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7qc8n" podUID="5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70"
Jul 24 15:08:54.921293 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:54.921145    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-52fwk" event={"ID":"f8e012d6-1f9f-4d7f-9dda-d4ee365d5b4e","Type":"ContainerStarted","Data":"4581c76d801ee8fe7e985cc0ac1d5d3ebb2fbb1a8b1bf11bbb4c983b4f808acd"}
Jul 24 15:08:54.949633 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:54.949592    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-52fwk" podStartSLOduration=8.214903508999999 podStartE2EDuration="26.949577049s" totalImagesPullingTime="18.73467354s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:08:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:08:54.949242412 +0000 UTC m=+27.761594603" watchObservedRunningTime="2026-07-24 15:08:54.949577049 +0000 UTC m=+27.761929227"
Jul 24 15:08:55.711273 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:55.711243    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgzfw"
Jul 24 15:08:55.711273 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:55.711261    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z654w"
Jul 24 15:08:55.711518 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:55.711357    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgzfw" podUID="e0b197de-537f-457b-8d3f-4dc75d38c402"
Jul 24 15:08:55.711518 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:55.711462    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z654w" podUID="a817b528-08b2-49a7-bbef-22838f3ca22d"
Jul 24 15:08:55.928472 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:55.925748    2579 generic.go:356] "Generic (PLEG): container finished" podID="1b97ec16-7040-4aa3-b2a8-0827c3698c7a" containerID="38b10a1318a7a8228cb0d6f57867667c364c2fbf705374ae2d9e214705e226b7" exitCode=0
Jul 24 15:08:55.928472 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:55.927535    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g4ts4" event={"ID":"1b97ec16-7040-4aa3-b2a8-0827c3698c7a","Type":"ContainerDied","Data":"38b10a1318a7a8228cb0d6f57867667c364c2fbf705374ae2d9e214705e226b7"}
Jul 24 15:08:55.928472 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:55.927587    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:55.928472 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:55.928099    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:55.928472 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:55.928172    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:55.956054 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:55.955949    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:55.957908 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:55.957885    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:08:56.652022 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:56.651989    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7qc8n"]
Jul 24 15:08:56.652210 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:56.652130    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7qc8n"
Jul 24 15:08:56.652282 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:56.652213    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7qc8n" podUID="5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70"
Jul 24 15:08:56.655017 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:56.654992    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z654w"]
Jul 24 15:08:56.655017 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:56.655090    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z654w"
Jul 24 15:08:56.655017 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:56.655181    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z654w" podUID="a817b528-08b2-49a7-bbef-22838f3ca22d"
Jul 24 15:08:56.655617 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:56.655591    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qgzfw"]
Jul 24 15:08:56.655685 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:56.655671    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgzfw"
Jul 24 15:08:56.655784 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:56.655762    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgzfw" podUID="e0b197de-537f-457b-8d3f-4dc75d38c402"
Jul 24 15:08:57.932829 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:57.932562    2579 generic.go:356] "Generic (PLEG): container finished" podID="1b97ec16-7040-4aa3-b2a8-0827c3698c7a" containerID="4533098d8a767facd7e5132af1b8d490c42efe6510b0ded5af88ad56aa8f181c" exitCode=0
Jul 24 15:08:57.933249 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:57.932644    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g4ts4" event={"ID":"1b97ec16-7040-4aa3-b2a8-0827c3698c7a","Type":"ContainerDied","Data":"4533098d8a767facd7e5132af1b8d490c42efe6510b0ded5af88ad56aa8f181c"}
Jul 24 15:08:58.711513 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:58.711421    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z654w"
Jul 24 15:08:58.711513 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:58.711450    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgzfw"
Jul 24 15:08:58.711513 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:58.711466    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7qc8n"
Jul 24 15:08:58.711900 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:58.711566    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgzfw" podUID="e0b197de-537f-457b-8d3f-4dc75d38c402"
Jul 24 15:08:58.711900 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:58.711605    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7qc8n" podUID="5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70"
Jul 24 15:08:58.711900 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:08:58.711662    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z654w" podUID="a817b528-08b2-49a7-bbef-22838f3ca22d"
Jul 24 15:08:59.939616 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:59.939581    2579 generic.go:356] "Generic (PLEG): container finished" podID="1b97ec16-7040-4aa3-b2a8-0827c3698c7a" containerID="80eae2d73c596cb6a8d1f5b92b5963d31875f7a8aec7a8bc5fe7369f0a7674fa" exitCode=0
Jul 24 15:08:59.939616 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:08:59.939621    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g4ts4" event={"ID":"1b97ec16-7040-4aa3-b2a8-0827c3698c7a","Type":"ContainerDied","Data":"80eae2d73c596cb6a8d1f5b92b5963d31875f7a8aec7a8bc5fe7369f0a7674fa"}
Jul 24 15:09:00.711135 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:00.711103    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z654w"
Jul 24 15:09:00.711333 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:00.711107    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7qc8n"
Jul 24 15:09:00.711333 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:00.711208    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z654w" podUID="a817b528-08b2-49a7-bbef-22838f3ca22d"
Jul 24 15:09:00.711333 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:00.711114    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgzfw"
Jul 24 15:09:00.711333 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:00.711293    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7qc8n" podUID="5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70"
Jul 24 15:09:00.711498 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:00.711374    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgzfw" podUID="e0b197de-537f-457b-8d3f-4dc75d38c402"
Jul 24 15:09:01.314262 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:01.314227    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0b197de-537f-457b-8d3f-4dc75d38c402-metrics-certs\") pod \"network-metrics-daemon-qgzfw\" (UID: \"e0b197de-537f-457b-8d3f-4dc75d38c402\") " pod="openshift-multus/network-metrics-daemon-qgzfw"
Jul 24 15:09:01.314744 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:01.314372    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 15:09:01.314744 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:01.314434    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0b197de-537f-457b-8d3f-4dc75d38c402-metrics-certs podName:e0b197de-537f-457b-8d3f-4dc75d38c402 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:33.314418933 +0000 UTC m=+66.126771091 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0b197de-537f-457b-8d3f-4dc75d38c402-metrics-certs") pod "network-metrics-daemon-qgzfw" (UID: "e0b197de-537f-457b-8d3f-4dc75d38c402") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 15:09:01.415416 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:01.415379    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70-original-pull-secret\") pod \"global-pull-secret-syncer-7qc8n\" (UID: \"5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70\") " pod="kube-system/global-pull-secret-syncer-7qc8n"
Jul 24 15:09:01.415606 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:01.415435    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppp76\" (UniqueName: \"kubernetes.io/projected/a817b528-08b2-49a7-bbef-22838f3ca22d-kube-api-access-ppp76\") pod \"network-check-target-z654w\" (UID: \"a817b528-08b2-49a7-bbef-22838f3ca22d\") " pod="openshift-network-diagnostics/network-check-target-z654w"
Jul 24 15:09:01.415606 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:01.415558    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 15:09:01.415606 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:01.415587    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 15:09:01.415606 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:01.415607    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 15:09:01.415781 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:01.415621    2579 projected.go:196] Error preparing data for projected volume kube-api-access-ppp76 for pod openshift-network-diagnostics/network-check-target-z654w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 15:09:01.415781 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:01.415645    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70-original-pull-secret podName:5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:33.415624428 +0000 UTC m=+66.227976609 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70-original-pull-secret") pod "global-pull-secret-syncer-7qc8n" (UID: "5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70") : object "kube-system"/"original-pull-secret" not registered
Jul 24 15:09:01.415781 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:01.415690    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a817b528-08b2-49a7-bbef-22838f3ca22d-kube-api-access-ppp76 podName:a817b528-08b2-49a7-bbef-22838f3ca22d nodeName:}" failed. No retries permitted until 2026-07-24 15:09:33.415673435 +0000 UTC m=+66.228025609 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ppp76" (UniqueName: "kubernetes.io/projected/a817b528-08b2-49a7-bbef-22838f3ca22d-kube-api-access-ppp76") pod "network-check-target-z654w" (UID: "a817b528-08b2-49a7-bbef-22838f3ca22d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 15:09:02.045960 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.045925    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-160.ec2.internal" event="NodeReady"
Jul 24 15:09:02.046126 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.046066    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 24 15:09:02.101427 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.101394    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bks2n"]
Jul 24 15:09:02.103295 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.103276    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bks2n"
Jul 24 15:09:02.104744 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.104722    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4vmc8"]
Jul 24 15:09:02.106634 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.106509    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 24 15:09:02.106634 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.106509    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-79w6c\""
Jul 24 15:09:02.106634 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.106543    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 24 15:09:02.106634 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.106576    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4vmc8"
Jul 24 15:09:02.107431 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.107416    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 24 15:09:02.109177 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.109160    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kkfwv\""
Jul 24 15:09:02.109279 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.109162    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 24 15:09:02.109463 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.109445    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 24 15:09:02.117809 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.117347    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bks2n"]
Jul 24 15:09:02.121532 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.121505    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4vmc8"]
Jul 24 15:09:02.221386 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.221301    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r48cl\" (UniqueName: \"kubernetes.io/projected/53659e71-d188-4315-81ab-7b53e4701a4f-kube-api-access-r48cl\") pod \"ingress-canary-bks2n\" (UID: \"53659e71-d188-4315-81ab-7b53e4701a4f\") " pod="openshift-ingress-canary/ingress-canary-bks2n"
Jul 24 15:09:02.221386 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.221364    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4427fb4-a65e-4505-9329-88c2c1c4e812-metrics-tls\") pod \"dns-default-4vmc8\" (UID: \"d4427fb4-a65e-4505-9329-88c2c1c4e812\") " pod="openshift-dns/dns-default-4vmc8"
Jul 24 15:09:02.221600 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.221399    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/53659e71-d188-4315-81ab-7b53e4701a4f-cert\") pod \"ingress-canary-bks2n\" (UID: \"53659e71-d188-4315-81ab-7b53e4701a4f\") " pod="openshift-ingress-canary/ingress-canary-bks2n"
Jul 24 15:09:02.221600 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.221479    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d4427fb4-a65e-4505-9329-88c2c1c4e812-config-volume\") pod \"dns-default-4vmc8\" (UID: \"d4427fb4-a65e-4505-9329-88c2c1c4e812\") " pod="openshift-dns/dns-default-4vmc8"
Jul 24 15:09:02.221600 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.221517    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4427fb4-a65e-4505-9329-88c2c1c4e812-tmp-dir\") pod \"dns-default-4vmc8\" (UID: \"d4427fb4-a65e-4505-9329-88c2c1c4e812\") " pod="openshift-dns/dns-default-4vmc8"
Jul 24 15:09:02.221702 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.221602    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfwb4\" (UniqueName: \"kubernetes.io/projected/d4427fb4-a65e-4505-9329-88c2c1c4e812-kube-api-access-lfwb4\") pod \"dns-default-4vmc8\" (UID: \"d4427fb4-a65e-4505-9329-88c2c1c4e812\") " pod="openshift-dns/dns-default-4vmc8"
Jul 24 15:09:02.322710 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.322656    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r48cl\" (UniqueName: \"kubernetes.io/projected/53659e71-d188-4315-81ab-7b53e4701a4f-kube-api-access-r48cl\") pod \"ingress-canary-bks2n\" (UID: \"53659e71-d188-4315-81ab-7b53e4701a4f\") " pod="openshift-ingress-canary/ingress-canary-bks2n"
Jul 24 15:09:02.322710 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.322727    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4427fb4-a65e-4505-9329-88c2c1c4e812-metrics-tls\") pod \"dns-default-4vmc8\" (UID: \"d4427fb4-a65e-4505-9329-88c2c1c4e812\") " pod="openshift-dns/dns-default-4vmc8"
Jul 24 15:09:02.323294 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.322764    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/53659e71-d188-4315-81ab-7b53e4701a4f-cert\") pod \"ingress-canary-bks2n\" (UID: \"53659e71-d188-4315-81ab-7b53e4701a4f\") " pod="openshift-ingress-canary/ingress-canary-bks2n"
Jul 24 15:09:02.323294 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.322804    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d4427fb4-a65e-4505-9329-88c2c1c4e812-config-volume\") pod \"dns-default-4vmc8\" (UID: \"d4427fb4-a65e-4505-9329-88c2c1c4e812\") " pod="openshift-dns/dns-default-4vmc8"
Jul 24 15:09:02.323294 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:02.322860    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 15:09:02.323294 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:02.322922    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4427fb4-a65e-4505-9329-88c2c1c4e812-metrics-tls podName:d4427fb4-a65e-4505-9329-88c2c1c4e812 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:02.822902113 +0000 UTC m=+35.635254289 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4427fb4-a65e-4505-9329-88c2c1c4e812-metrics-tls") pod "dns-default-4vmc8" (UID: "d4427fb4-a65e-4505-9329-88c2c1c4e812") : secret "dns-default-metrics-tls" not found
Jul 24 15:09:02.323294 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.322950    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4427fb4-a65e-4505-9329-88c2c1c4e812-tmp-dir\") pod \"dns-default-4vmc8\" (UID: \"d4427fb4-a65e-4505-9329-88c2c1c4e812\") " pod="openshift-dns/dns-default-4vmc8"
Jul 24 15:09:02.323294 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.323004    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfwb4\" (UniqueName: \"kubernetes.io/projected/d4427fb4-a65e-4505-9329-88c2c1c4e812-kube-api-access-lfwb4\") pod \"dns-default-4vmc8\" (UID: \"d4427fb4-a65e-4505-9329-88c2c1c4e812\") " pod="openshift-dns/dns-default-4vmc8"
Jul 24 15:09:02.323486 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.323302    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4427fb4-a65e-4505-9329-88c2c1c4e812-tmp-dir\") pod \"dns-default-4vmc8\" (UID: \"d4427fb4-a65e-4505-9329-88c2c1c4e812\") " pod="openshift-dns/dns-default-4vmc8"
Jul 24 15:09:02.323486 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.323389    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d4427fb4-a65e-4505-9329-88c2c1c4e812-config-volume\") pod \"dns-default-4vmc8\" (UID: \"d4427fb4-a65e-4505-9329-88c2c1c4e812\") " pod="openshift-dns/dns-default-4vmc8"
Jul 24 15:09:02.326971 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.326948    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/53659e71-d188-4315-81ab-7b53e4701a4f-cert\") pod \"ingress-canary-bks2n\" (UID: \"53659e71-d188-4315-81ab-7b53e4701a4f\") " pod="openshift-ingress-canary/ingress-canary-bks2n"
Jul 24 15:09:02.333256 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.333230    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r48cl\" (UniqueName: \"kubernetes.io/projected/53659e71-d188-4315-81ab-7b53e4701a4f-kube-api-access-r48cl\") pod \"ingress-canary-bks2n\" (UID: \"53659e71-d188-4315-81ab-7b53e4701a4f\") " pod="openshift-ingress-canary/ingress-canary-bks2n"
Jul 24 15:09:02.333900 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.333880    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfwb4\" (UniqueName: \"kubernetes.io/projected/d4427fb4-a65e-4505-9329-88c2c1c4e812-kube-api-access-lfwb4\") pod \"dns-default-4vmc8\" (UID: \"d4427fb4-a65e-4505-9329-88c2c1c4e812\") " pod="openshift-dns/dns-default-4vmc8"
Jul 24 15:09:02.415581 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.415533    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bks2n"
Jul 24 15:09:02.564389 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.564357    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bks2n"]
Jul 24 15:09:02.568179 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:09:02.568149    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53659e71_d188_4315_81ab_7b53e4701a4f.slice/crio-bb3521d9baf196ae300dfed957c5f1732166b85a46afd14e80279bcbd688f27c WatchSource:0}: Error finding container bb3521d9baf196ae300dfed957c5f1732166b85a46afd14e80279bcbd688f27c: Status 404 returned error can't find the container with id bb3521d9baf196ae300dfed957c5f1732166b85a46afd14e80279bcbd688f27c
Jul 24 15:09:02.711137 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.711103    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgzfw"
Jul 24 15:09:02.711324 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.711147    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z654w"
Jul 24 15:09:02.711324 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.711103    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7qc8n"
Jul 24 15:09:02.714009 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.713986    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 24 15:09:02.714009 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.713995    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 15:09:02.714463 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.714434    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-d22wg\""
Jul 24 15:09:02.714463 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.714464    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 15:09:02.714866 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.714849    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 24 15:09:02.714866 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.714859    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-trkb8\""
Jul 24 15:09:02.826825 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.826783    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4427fb4-a65e-4505-9329-88c2c1c4e812-metrics-tls\") pod \"dns-default-4vmc8\" (UID: \"d4427fb4-a65e-4505-9329-88c2c1c4e812\") " pod="openshift-dns/dns-default-4vmc8"
Jul 24 15:09:02.827010 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:02.826945    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 15:09:02.827070 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:02.827018    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4427fb4-a65e-4505-9329-88c2c1c4e812-metrics-tls podName:d4427fb4-a65e-4505-9329-88c2c1c4e812 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:03.827000937 +0000 UTC m=+36.639353093 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4427fb4-a65e-4505-9329-88c2c1c4e812-metrics-tls") pod "dns-default-4vmc8" (UID: "d4427fb4-a65e-4505-9329-88c2c1c4e812") : secret "dns-default-metrics-tls" not found
Jul 24 15:09:02.948727 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:02.948693    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bks2n" event={"ID":"53659e71-d188-4315-81ab-7b53e4701a4f","Type":"ContainerStarted","Data":"bb3521d9baf196ae300dfed957c5f1732166b85a46afd14e80279bcbd688f27c"}
Jul 24 15:09:03.835404 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:03.835361    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4427fb4-a65e-4505-9329-88c2c1c4e812-metrics-tls\") pod \"dns-default-4vmc8\" (UID: \"d4427fb4-a65e-4505-9329-88c2c1c4e812\") " pod="openshift-dns/dns-default-4vmc8"
Jul 24 15:09:03.835961 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:03.835501    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 15:09:03.835961 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:03.835580    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4427fb4-a65e-4505-9329-88c2c1c4e812-metrics-tls podName:d4427fb4-a65e-4505-9329-88c2c1c4e812 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:05.835558785 +0000 UTC m=+38.647910944 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4427fb4-a65e-4505-9329-88c2c1c4e812-metrics-tls") pod "dns-default-4vmc8" (UID: "d4427fb4-a65e-4505-9329-88c2c1c4e812") : secret "dns-default-metrics-tls" not found
Jul 24 15:09:04.957230 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:04.957187    2579 generic.go:356] "Generic (PLEG): container finished" podID="1b97ec16-7040-4aa3-b2a8-0827c3698c7a" containerID="1461ae7abca9b49662c4f5bb24665ab9a8d1438905cf5cde7d98c4b60bf6f9c7" exitCode=0
Jul 24 15:09:04.957644 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:04.957247    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g4ts4" event={"ID":"1b97ec16-7040-4aa3-b2a8-0827c3698c7a","Type":"ContainerDied","Data":"1461ae7abca9b49662c4f5bb24665ab9a8d1438905cf5cde7d98c4b60bf6f9c7"}
Jul 24 15:09:05.852608 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:05.852570    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4427fb4-a65e-4505-9329-88c2c1c4e812-metrics-tls\") pod \"dns-default-4vmc8\" (UID: \"d4427fb4-a65e-4505-9329-88c2c1c4e812\") " pod="openshift-dns/dns-default-4vmc8"
Jul 24 15:09:05.852797 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:05.852735    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 15:09:05.852871 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:05.852800    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4427fb4-a65e-4505-9329-88c2c1c4e812-metrics-tls podName:d4427fb4-a65e-4505-9329-88c2c1c4e812 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:09.852784349 +0000 UTC m=+42.665136506 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4427fb4-a65e-4505-9329-88c2c1c4e812-metrics-tls") pod "dns-default-4vmc8" (UID: "d4427fb4-a65e-4505-9329-88c2c1c4e812") : secret "dns-default-metrics-tls" not found
Jul 24 15:09:05.963599 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:05.963567    2579 generic.go:356] "Generic (PLEG): container finished" podID="1b97ec16-7040-4aa3-b2a8-0827c3698c7a" containerID="742b009286fb2a23134090c3dcaebdc74dfe6d67daf4473377b4c723f2c20046" exitCode=0
Jul 24 15:09:05.964064 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:05.963642    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g4ts4" event={"ID":"1b97ec16-7040-4aa3-b2a8-0827c3698c7a","Type":"ContainerDied","Data":"742b009286fb2a23134090c3dcaebdc74dfe6d67daf4473377b4c723f2c20046"}
Jul 24 15:09:06.970585 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:06.970324    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g4ts4" event={"ID":"1b97ec16-7040-4aa3-b2a8-0827c3698c7a","Type":"ContainerStarted","Data":"299139ed48a349c83b412d84a779a503e5614083c0f06536a1f59d4715c2abe6"}
Jul 24 15:09:06.971941 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:06.971900    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bks2n" event={"ID":"53659e71-d188-4315-81ab-7b53e4701a4f","Type":"ContainerStarted","Data":"26eb1ea6c9ee452812d20df56f24bd2d7008e6943ad9b91d6c0bfcf25c334eb1"}
Jul 24 15:09:06.998102 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:06.998052    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-g4ts4" podStartSLOduration=8.979506825 podStartE2EDuration="38.998035676s" totalImagesPullingTime="30.018528851s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:08:28 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:09:06.996697721 +0000 UTC m=+39.809049901" watchObservedRunningTime="2026-07-24 15:09:06.998035676 +0000 UTC m=+39.810387855"
Jul 24 15:09:09.880708 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:09.880668    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4427fb4-a65e-4505-9329-88c2c1c4e812-metrics-tls\") pod \"dns-default-4vmc8\" (UID: \"d4427fb4-a65e-4505-9329-88c2c1c4e812\") " pod="openshift-dns/dns-default-4vmc8"
Jul 24 15:09:09.881166 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:09.880841    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 15:09:09.881166 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:09.880907    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4427fb4-a65e-4505-9329-88c2c1c4e812-metrics-tls podName:d4427fb4-a65e-4505-9329-88c2c1c4e812 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:17.880891922 +0000 UTC m=+50.693244080 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4427fb4-a65e-4505-9329-88c2c1c4e812-metrics-tls") pod "dns-default-4vmc8" (UID: "d4427fb4-a65e-4505-9329-88c2c1c4e812") : secret "dns-default-metrics-tls" not found
Jul 24 15:09:17.925536 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:17.925490    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4427fb4-a65e-4505-9329-88c2c1c4e812-metrics-tls\") pod \"dns-default-4vmc8\" (UID: \"d4427fb4-a65e-4505-9329-88c2c1c4e812\") " pod="openshift-dns/dns-default-4vmc8"
Jul 24 15:09:17.926101 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:17.925634    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 15:09:17.926101 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:17.925709    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4427fb4-a65e-4505-9329-88c2c1c4e812-metrics-tls podName:d4427fb4-a65e-4505-9329-88c2c1c4e812 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:33.925692398 +0000 UTC m=+66.738044554 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4427fb4-a65e-4505-9329-88c2c1c4e812-metrics-tls") pod "dns-default-4vmc8" (UID: "d4427fb4-a65e-4505-9329-88c2c1c4e812") : secret "dns-default-metrics-tls" not found
Jul 24 15:09:27.954352 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:27.954322    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-52fwk"
Jul 24 15:09:28.001575 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:28.001524    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bks2n" podStartSLOduration=21.795641053 podStartE2EDuration="26.001510057s" totalImagesPullingTime="4.205869004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:09:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:09:07.01844558 +0000 UTC m=+39.830797760" watchObservedRunningTime="2026-07-24 15:09:28.001510057 +0000 UTC m=+60.813862274"
Jul 24 15:09:33.327561 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:33.327517    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0b197de-537f-457b-8d3f-4dc75d38c402-metrics-certs\") pod \"network-metrics-daemon-qgzfw\" (UID: \"e0b197de-537f-457b-8d3f-4dc75d38c402\") " pod="openshift-multus/network-metrics-daemon-qgzfw"
Jul 24 15:09:33.330296 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:33.330276    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 15:09:33.338261 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:33.338245    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 24 15:09:33.338330 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:33.338308    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0b197de-537f-457b-8d3f-4dc75d38c402-metrics-certs podName:e0b197de-537f-457b-8d3f-4dc75d38c402 nodeName:}" failed. No retries permitted until 2026-07-24 15:10:37.338293905 +0000 UTC m=+130.150646063 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0b197de-537f-457b-8d3f-4dc75d38c402-metrics-certs") pod "network-metrics-daemon-qgzfw" (UID: "e0b197de-537f-457b-8d3f-4dc75d38c402") : secret "metrics-daemon-secret" not found
Jul 24 15:09:33.428854 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:33.428802    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70-original-pull-secret\") pod \"global-pull-secret-syncer-7qc8n\" (UID: \"5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70\") " pod="kube-system/global-pull-secret-syncer-7qc8n"
Jul 24 15:09:33.428922 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:33.428865    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppp76\" (UniqueName: \"kubernetes.io/projected/a817b528-08b2-49a7-bbef-22838f3ca22d-kube-api-access-ppp76\") pod \"network-check-target-z654w\" (UID: \"a817b528-08b2-49a7-bbef-22838f3ca22d\") " pod="openshift-network-diagnostics/network-check-target-z654w"
Jul 24 15:09:33.432056 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:33.432041    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 15:09:33.432101 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:33.432043    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 24 15:09:33.441462 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:33.441429    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 24 15:09:33.442790 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:33.442764    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70-original-pull-secret\") pod \"global-pull-secret-syncer-7qc8n\" (UID: \"5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70\") " pod="kube-system/global-pull-secret-syncer-7qc8n"
Jul 24 15:09:33.451539 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:33.451513    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppp76\" (UniqueName: \"kubernetes.io/projected/a817b528-08b2-49a7-bbef-22838f3ca22d-kube-api-access-ppp76\") pod \"network-check-target-z654w\" (UID: \"a817b528-08b2-49a7-bbef-22838f3ca22d\") " pod="openshift-network-diagnostics/network-check-target-z654w"
Jul 24 15:09:33.630366 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:33.630296    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7qc8n"
Jul 24 15:09:33.639739 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:33.639720    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-d22wg\""
Jul 24 15:09:33.647066 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:33.647045    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z654w"
Jul 24 15:09:33.783045 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:33.782976    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7qc8n"]
Jul 24 15:09:33.786930 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:09:33.786904    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a7d4e75_bfc5_4fe8_8ced_eb1ba5ba2a70.slice/crio-88b8f8011e8ca23a9cb885687aea1cdd63868f1090d1b398efe8c3eb2a091732 WatchSource:0}: Error finding container 88b8f8011e8ca23a9cb885687aea1cdd63868f1090d1b398efe8c3eb2a091732: Status 404 returned error can't find the container with id 88b8f8011e8ca23a9cb885687aea1cdd63868f1090d1b398efe8c3eb2a091732
Jul 24 15:09:33.797096 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:33.797071    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z654w"]
Jul 24 15:09:33.799066 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:09:33.799044    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda817b528_08b2_49a7_bbef_22838f3ca22d.slice/crio-f6531dccac87f088217eb7853ffba10ffda90ed072e10b1fd2923b89098351d5 WatchSource:0}: Error finding container f6531dccac87f088217eb7853ffba10ffda90ed072e10b1fd2923b89098351d5: Status 404 returned error can't find the container with id f6531dccac87f088217eb7853ffba10ffda90ed072e10b1fd2923b89098351d5
Jul 24 15:09:33.932705 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:33.932604    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4427fb4-a65e-4505-9329-88c2c1c4e812-metrics-tls\") pod \"dns-default-4vmc8\" (UID: \"d4427fb4-a65e-4505-9329-88c2c1c4e812\") " pod="openshift-dns/dns-default-4vmc8"
Jul 24 15:09:33.932874 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:33.932721    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 15:09:33.932874 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:33.932773    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4427fb4-a65e-4505-9329-88c2c1c4e812-metrics-tls podName:d4427fb4-a65e-4505-9329-88c2c1c4e812 nodeName:}" failed. No retries permitted until 2026-07-24 15:10:05.932759798 +0000 UTC m=+98.745111955 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4427fb4-a65e-4505-9329-88c2c1c4e812-metrics-tls") pod "dns-default-4vmc8" (UID: "d4427fb4-a65e-4505-9329-88c2c1c4e812") : secret "dns-default-metrics-tls" not found
Jul 24 15:09:34.036237 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:34.036201    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7qc8n" event={"ID":"5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70","Type":"ContainerStarted","Data":"88b8f8011e8ca23a9cb885687aea1cdd63868f1090d1b398efe8c3eb2a091732"}
Jul 24 15:09:34.037244 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:34.037216    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z654w" event={"ID":"a817b528-08b2-49a7-bbef-22838f3ca22d","Type":"ContainerStarted","Data":"f6531dccac87f088217eb7853ffba10ffda90ed072e10b1fd2923b89098351d5"}
Jul 24 15:09:39.051544 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:39.051475    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7qc8n" event={"ID":"5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70","Type":"ContainerStarted","Data":"10863830483f461d071d39c355376ccb7bc906fda4dd75933881fbba123eff75"}
Jul 24 15:09:39.052944 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:39.052921    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z654w" event={"ID":"a817b528-08b2-49a7-bbef-22838f3ca22d","Type":"ContainerStarted","Data":"56573b50dbac3f9c14bf83c430bb3a617aa0492435311dc8a30dde104ef1466d"}
Jul 24 15:09:39.053023 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:39.052992    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-z654w"
Jul 24 15:09:39.068539 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:39.068495    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-7qc8n" podStartSLOduration=66.47016798 podStartE2EDuration="1m11.068482251s" totalImagesPullingTime="4.598314271s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:08:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:09:39.067219678 +0000 UTC m=+71.879571857" watchObservedRunningTime="2026-07-24 15:09:39.068482251 +0000 UTC m=+71.880834430"
Jul 24 15:09:39.083044 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:39.082998    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-z654w" podStartSLOduration=66.500152733 podStartE2EDuration="1m11.082982312s" totalImagesPullingTime="4.582829579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:08:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:09:39.082892678 +0000 UTC m=+71.895244860" watchObservedRunningTime="2026-07-24 15:09:39.082982312 +0000 UTC m=+71.895334491"
Jul 24 15:09:52.744410 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:52.744290    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-2vhjm"]
Jul 24 15:09:52.746607 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:52.746591    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-2vhjm"
Jul 24 15:09:52.748779 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:52.748757    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-2ftcj\""
Jul 24 15:09:52.748922 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:52.748903    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 24 15:09:52.749675 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:52.749656    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 24 15:09:52.761864 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:52.761842    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-2vhjm"]
Jul 24 15:09:52.863389 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:52.863354    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kd8p\" (UniqueName: \"kubernetes.io/projected/443116d2-ada8-4261-abcb-113fa99e38ae-kube-api-access-6kd8p\") pod \"volume-data-source-validator-549c6d54cf-2vhjm\" (UID: \"443116d2-ada8-4261-abcb-113fa99e38ae\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-2vhjm"
Jul 24 15:09:52.964697 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:52.964651    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kd8p\" (UniqueName: \"kubernetes.io/projected/443116d2-ada8-4261-abcb-113fa99e38ae-kube-api-access-6kd8p\") pod \"volume-data-source-validator-549c6d54cf-2vhjm\" (UID: \"443116d2-ada8-4261-abcb-113fa99e38ae\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-2vhjm"
Jul 24 15:09:52.976046 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:52.976013    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kd8p\" (UniqueName: \"kubernetes.io/projected/443116d2-ada8-4261-abcb-113fa99e38ae-kube-api-access-6kd8p\") pod \"volume-data-source-validator-549c6d54cf-2vhjm\" (UID: \"443116d2-ada8-4261-abcb-113fa99e38ae\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-2vhjm"
Jul 24 15:09:53.054751 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:53.054719    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-2vhjm"
Jul 24 15:09:53.063550 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:53.063531    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-55db6d66b7-2bssd"]
Jul 24 15:09:53.065330 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:53.065309    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-55db6d66b7-2bssd"
Jul 24 15:09:53.071503 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:53.071485    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-z5j8v\""
Jul 24 15:09:53.071901 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:53.071882    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 24 15:09:53.072144 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:53.072130    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 24 15:09:53.072220 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:53.072147    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 24 15:09:53.072392 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:53.072376    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 24 15:09:53.074042 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:53.074022    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 24 15:09:53.074745 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:53.074635    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 24 15:09:53.087923 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:53.087893    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-55db6d66b7-2bssd"]
Jul 24 15:09:53.166207 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:53.166165    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w4p6\" (UniqueName: \"kubernetes.io/projected/7eb671c8-eae8-409d-943a-c8f76ddede9f-kube-api-access-2w4p6\") pod \"router-default-55db6d66b7-2bssd\" (UID: \"7eb671c8-eae8-409d-943a-c8f76ddede9f\") " pod="openshift-ingress/router-default-55db6d66b7-2bssd"
Jul 24 15:09:53.166371 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:53.166243    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7eb671c8-eae8-409d-943a-c8f76ddede9f-default-certificate\") pod \"router-default-55db6d66b7-2bssd\" (UID: \"7eb671c8-eae8-409d-943a-c8f76ddede9f\") " pod="openshift-ingress/router-default-55db6d66b7-2bssd"
Jul 24 15:09:53.166371 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:53.166265    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eb671c8-eae8-409d-943a-c8f76ddede9f-metrics-certs\") pod \"router-default-55db6d66b7-2bssd\" (UID: \"7eb671c8-eae8-409d-943a-c8f76ddede9f\") " pod="openshift-ingress/router-default-55db6d66b7-2bssd"
Jul 24 15:09:53.166371 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:53.166339    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7eb671c8-eae8-409d-943a-c8f76ddede9f-service-ca-bundle\") pod \"router-default-55db6d66b7-2bssd\" (UID: \"7eb671c8-eae8-409d-943a-c8f76ddede9f\") " pod="openshift-ingress/router-default-55db6d66b7-2bssd"
Jul 24 15:09:53.166471 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:53.166386    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7eb671c8-eae8-409d-943a-c8f76ddede9f-stats-auth\") pod \"router-default-55db6d66b7-2bssd\" (UID: \"7eb671c8-eae8-409d-943a-c8f76ddede9f\") " pod="openshift-ingress/router-default-55db6d66b7-2bssd"
Jul 24 15:09:53.182079 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:53.182040    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-2vhjm"]
Jul 24 15:09:53.185612 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:09:53.185587    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod443116d2_ada8_4261_abcb_113fa99e38ae.slice/crio-b9141f761224a4ca3061818a9b82b952b584ec07ceedcf46f55b7d8b920b306e WatchSource:0}: Error finding container b9141f761224a4ca3061818a9b82b952b584ec07ceedcf46f55b7d8b920b306e: Status 404 returned error can't find the container with id b9141f761224a4ca3061818a9b82b952b584ec07ceedcf46f55b7d8b920b306e
Jul 24 15:09:53.267492 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:53.267450    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7eb671c8-eae8-409d-943a-c8f76ddede9f-service-ca-bundle\") pod \"router-default-55db6d66b7-2bssd\" (UID: \"7eb671c8-eae8-409d-943a-c8f76ddede9f\") " pod="openshift-ingress/router-default-55db6d66b7-2bssd"
Jul 24 15:09:53.267683 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:53.267511    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7eb671c8-eae8-409d-943a-c8f76ddede9f-stats-auth\") pod \"router-default-55db6d66b7-2bssd\" (UID: \"7eb671c8-eae8-409d-943a-c8f76ddede9f\") " pod="openshift-ingress/router-default-55db6d66b7-2bssd"
Jul 24 15:09:53.267683 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:53.267544    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w4p6\" (UniqueName: \"kubernetes.io/projected/7eb671c8-eae8-409d-943a-c8f76ddede9f-kube-api-access-2w4p6\") pod \"router-default-55db6d66b7-2bssd\" (UID: \"7eb671c8-eae8-409d-943a-c8f76ddede9f\") " pod="openshift-ingress/router-default-55db6d66b7-2bssd"
Jul 24 15:09:53.267683 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:53.267599    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7eb671c8-eae8-409d-943a-c8f76ddede9f-default-certificate\") pod \"router-default-55db6d66b7-2bssd\" (UID: \"7eb671c8-eae8-409d-943a-c8f76ddede9f\") " pod="openshift-ingress/router-default-55db6d66b7-2bssd"
Jul 24 15:09:53.267683 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:53.267632    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7eb671c8-eae8-409d-943a-c8f76ddede9f-service-ca-bundle podName:7eb671c8-eae8-409d-943a-c8f76ddede9f nodeName:}" failed. No retries permitted until 2026-07-24 15:09:53.767612718 +0000 UTC m=+86.579964897 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7eb671c8-eae8-409d-943a-c8f76ddede9f-service-ca-bundle") pod "router-default-55db6d66b7-2bssd" (UID: "7eb671c8-eae8-409d-943a-c8f76ddede9f") : configmap references non-existent config key: service-ca.crt
Jul 24 15:09:53.267683 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:53.267658    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eb671c8-eae8-409d-943a-c8f76ddede9f-metrics-certs\") pod \"router-default-55db6d66b7-2bssd\" (UID: \"7eb671c8-eae8-409d-943a-c8f76ddede9f\") " pod="openshift-ingress/router-default-55db6d66b7-2bssd"
Jul 24 15:09:53.270777 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:53.268144    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 24 15:09:53.270777 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:53.269072    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eb671c8-eae8-409d-943a-c8f76ddede9f-metrics-certs podName:7eb671c8-eae8-409d-943a-c8f76ddede9f nodeName:}" failed. No retries permitted until 2026-07-24 15:09:53.769045876 +0000 UTC m=+86.581398046 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7eb671c8-eae8-409d-943a-c8f76ddede9f-metrics-certs") pod "router-default-55db6d66b7-2bssd" (UID: "7eb671c8-eae8-409d-943a-c8f76ddede9f") : secret "router-metrics-certs-default" not found
Jul 24 15:09:53.271766 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:53.271126    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7eb671c8-eae8-409d-943a-c8f76ddede9f-stats-auth\") pod \"router-default-55db6d66b7-2bssd\" (UID: \"7eb671c8-eae8-409d-943a-c8f76ddede9f\") " pod="openshift-ingress/router-default-55db6d66b7-2bssd"
Jul 24 15:09:53.271766 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:53.271365    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7eb671c8-eae8-409d-943a-c8f76ddede9f-default-certificate\") pod \"router-default-55db6d66b7-2bssd\" (UID: \"7eb671c8-eae8-409d-943a-c8f76ddede9f\") " pod="openshift-ingress/router-default-55db6d66b7-2bssd"
Jul 24 15:09:53.279593 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:53.279567    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w4p6\" (UniqueName: \"kubernetes.io/projected/7eb671c8-eae8-409d-943a-c8f76ddede9f-kube-api-access-2w4p6\") pod \"router-default-55db6d66b7-2bssd\" (UID: \"7eb671c8-eae8-409d-943a-c8f76ddede9f\") " pod="openshift-ingress/router-default-55db6d66b7-2bssd"
Jul 24 15:09:53.771847 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:53.771776    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eb671c8-eae8-409d-943a-c8f76ddede9f-metrics-certs\") pod \"router-default-55db6d66b7-2bssd\" (UID: \"7eb671c8-eae8-409d-943a-c8f76ddede9f\") " pod="openshift-ingress/router-default-55db6d66b7-2bssd"
Jul 24 15:09:53.772224 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:53.771866    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7eb671c8-eae8-409d-943a-c8f76ddede9f-service-ca-bundle\") pod \"router-default-55db6d66b7-2bssd\" (UID: \"7eb671c8-eae8-409d-943a-c8f76ddede9f\") " pod="openshift-ingress/router-default-55db6d66b7-2bssd"
Jul 24 15:09:53.772224 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:53.771925    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 24 15:09:53.772224 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:53.771983    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7eb671c8-eae8-409d-943a-c8f76ddede9f-service-ca-bundle podName:7eb671c8-eae8-409d-943a-c8f76ddede9f nodeName:}" failed. No retries permitted until 2026-07-24 15:09:54.771969509 +0000 UTC m=+87.584321665 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7eb671c8-eae8-409d-943a-c8f76ddede9f-service-ca-bundle") pod "router-default-55db6d66b7-2bssd" (UID: "7eb671c8-eae8-409d-943a-c8f76ddede9f") : configmap references non-existent config key: service-ca.crt
Jul 24 15:09:53.772224 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:53.772000    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eb671c8-eae8-409d-943a-c8f76ddede9f-metrics-certs podName:7eb671c8-eae8-409d-943a-c8f76ddede9f nodeName:}" failed. No retries permitted until 2026-07-24 15:09:54.771992599 +0000 UTC m=+87.584344755 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7eb671c8-eae8-409d-943a-c8f76ddede9f-metrics-certs") pod "router-default-55db6d66b7-2bssd" (UID: "7eb671c8-eae8-409d-943a-c8f76ddede9f") : secret "router-metrics-certs-default" not found
Jul 24 15:09:54.088425 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:54.088343    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-2vhjm" event={"ID":"443116d2-ada8-4261-abcb-113fa99e38ae","Type":"ContainerStarted","Data":"b9141f761224a4ca3061818a9b82b952b584ec07ceedcf46f55b7d8b920b306e"}
Jul 24 15:09:54.779479 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:54.779446    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eb671c8-eae8-409d-943a-c8f76ddede9f-metrics-certs\") pod \"router-default-55db6d66b7-2bssd\" (UID: \"7eb671c8-eae8-409d-943a-c8f76ddede9f\") " pod="openshift-ingress/router-default-55db6d66b7-2bssd"
Jul 24 15:09:54.779894 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:54.779522    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7eb671c8-eae8-409d-943a-c8f76ddede9f-service-ca-bundle\") pod \"router-default-55db6d66b7-2bssd\" (UID: \"7eb671c8-eae8-409d-943a-c8f76ddede9f\") " pod="openshift-ingress/router-default-55db6d66b7-2bssd"
Jul 24 15:09:54.779894 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:54.779601    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 24 15:09:54.779894 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:54.779621    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7eb671c8-eae8-409d-943a-c8f76ddede9f-service-ca-bundle podName:7eb671c8-eae8-409d-943a-c8f76ddede9f nodeName:}" failed. No retries permitted until 2026-07-24 15:09:56.779604193 +0000 UTC m=+89.591956366 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7eb671c8-eae8-409d-943a-c8f76ddede9f-service-ca-bundle") pod "router-default-55db6d66b7-2bssd" (UID: "7eb671c8-eae8-409d-943a-c8f76ddede9f") : configmap references non-existent config key: service-ca.crt
Jul 24 15:09:54.779894 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:54.779645    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eb671c8-eae8-409d-943a-c8f76ddede9f-metrics-certs podName:7eb671c8-eae8-409d-943a-c8f76ddede9f nodeName:}" failed. No retries permitted until 2026-07-24 15:09:56.779633398 +0000 UTC m=+89.591985568 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7eb671c8-eae8-409d-943a-c8f76ddede9f-metrics-certs") pod "router-default-55db6d66b7-2bssd" (UID: "7eb671c8-eae8-409d-943a-c8f76ddede9f") : secret "router-metrics-certs-default" not found
Jul 24 15:09:55.091881 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:55.091794    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-2vhjm" event={"ID":"443116d2-ada8-4261-abcb-113fa99e38ae","Type":"ContainerStarted","Data":"14b8601d5f4a3c81d8e8fc2dcd7b506ff52cd008c571b358df06a22893bf59d2"}
Jul 24 15:09:55.109785 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:55.109736    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-2vhjm" podStartSLOduration=1.9065760090000001 podStartE2EDuration="3.109720073s" totalImagesPullingTime="1.203144064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:09:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:09:55.109364547 +0000 UTC m=+87.921716726" watchObservedRunningTime="2026-07-24 15:09:55.109720073 +0000 UTC m=+87.922072252"
Jul 24 15:09:56.611592 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:56.611556    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ncf4z"]
Jul 24 15:09:56.613560 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:56.613545    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ncf4z"
Jul 24 15:09:56.626124 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:56.626089    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 24 15:09:56.626257 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:56.626150    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nvtjc\""
Jul 24 15:09:56.626257 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:56.626209    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 24 15:09:56.626352 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:56.626287    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 24 15:09:56.626401 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:56.626387    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 24 15:09:56.644363 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:56.644335    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ncf4z"]
Jul 24 15:09:56.691497 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:56.691464    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8910cead-2436-44b7-b425-d42a7063d4b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ncf4z\" (UID: \"8910cead-2436-44b7-b425-d42a7063d4b8\") " pod="openshift-insights/insights-runtime-extractor-ncf4z"
Jul 24 15:09:56.691631 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:56.691503    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8910cead-2436-44b7-b425-d42a7063d4b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ncf4z\" (UID: \"8910cead-2436-44b7-b425-d42a7063d4b8\") " pod="openshift-insights/insights-runtime-extractor-ncf4z"
Jul 24 15:09:56.691631 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:56.691590    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8910cead-2436-44b7-b425-d42a7063d4b8-data-volume\") pod \"insights-runtime-extractor-ncf4z\" (UID: \"8910cead-2436-44b7-b425-d42a7063d4b8\") " pod="openshift-insights/insights-runtime-extractor-ncf4z"
Jul 24 15:09:56.691725 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:56.691628    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8910cead-2436-44b7-b425-d42a7063d4b8-crio-socket\") pod \"insights-runtime-extractor-ncf4z\" (UID: \"8910cead-2436-44b7-b425-d42a7063d4b8\") " pod="openshift-insights/insights-runtime-extractor-ncf4z"
Jul 24 15:09:56.691725 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:56.691651    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh9qx\" (UniqueName: \"kubernetes.io/projected/8910cead-2436-44b7-b425-d42a7063d4b8-kube-api-access-gh9qx\") pod \"insights-runtime-extractor-ncf4z\" (UID: \"8910cead-2436-44b7-b425-d42a7063d4b8\") " pod="openshift-insights/insights-runtime-extractor-ncf4z"
Jul 24 15:09:56.792679 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:56.792648    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8910cead-2436-44b7-b425-d42a7063d4b8-crio-socket\") pod \"insights-runtime-extractor-ncf4z\" (UID: \"8910cead-2436-44b7-b425-d42a7063d4b8\") " pod="openshift-insights/insights-runtime-extractor-ncf4z"
Jul 24 15:09:56.792807 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:56.792682    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh9qx\" (UniqueName: \"kubernetes.io/projected/8910cead-2436-44b7-b425-d42a7063d4b8-kube-api-access-gh9qx\") pod \"insights-runtime-extractor-ncf4z\" (UID: \"8910cead-2436-44b7-b425-d42a7063d4b8\") " pod="openshift-insights/insights-runtime-extractor-ncf4z"
Jul 24 15:09:56.792807 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:56.792707    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eb671c8-eae8-409d-943a-c8f76ddede9f-metrics-certs\") pod \"router-default-55db6d66b7-2bssd\" (UID: \"7eb671c8-eae8-409d-943a-c8f76ddede9f\") " pod="openshift-ingress/router-default-55db6d66b7-2bssd"
Jul 24 15:09:56.792807 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:56.792792    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8910cead-2436-44b7-b425-d42a7063d4b8-crio-socket\") pod \"insights-runtime-extractor-ncf4z\" (UID: \"8910cead-2436-44b7-b425-d42a7063d4b8\") " pod="openshift-insights/insights-runtime-extractor-ncf4z"
Jul 24 15:09:56.792937 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:56.792805    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 24 15:09:56.792937 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:56.792855    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8910cead-2436-44b7-b425-d42a7063d4b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ncf4z\" (UID: \"8910cead-2436-44b7-b425-d42a7063d4b8\") " pod="openshift-insights/insights-runtime-extractor-ncf4z"
Jul 24 15:09:56.792937 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:56.792883    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eb671c8-eae8-409d-943a-c8f76ddede9f-metrics-certs podName:7eb671c8-eae8-409d-943a-c8f76ddede9f nodeName:}" failed. No retries permitted until 2026-07-24 15:10:00.792866088 +0000 UTC m=+93.605218270 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7eb671c8-eae8-409d-943a-c8f76ddede9f-metrics-certs") pod "router-default-55db6d66b7-2bssd" (UID: "7eb671c8-eae8-409d-943a-c8f76ddede9f") : secret "router-metrics-certs-default" not found
Jul 24 15:09:56.792937 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:56.792913    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8910cead-2436-44b7-b425-d42a7063d4b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ncf4z\" (UID: \"8910cead-2436-44b7-b425-d42a7063d4b8\") " pod="openshift-insights/insights-runtime-extractor-ncf4z"
Jul 24 15:09:56.792937 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:56.792924    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 15:09:56.793115 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:56.792949    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7eb671c8-eae8-409d-943a-c8f76ddede9f-service-ca-bundle\") pod \"router-default-55db6d66b7-2bssd\" (UID: \"7eb671c8-eae8-409d-943a-c8f76ddede9f\") " pod="openshift-ingress/router-default-55db6d66b7-2bssd"
Jul 24 15:09:56.793115 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:56.792961    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8910cead-2436-44b7-b425-d42a7063d4b8-kube-rbac-proxy-cm podName:8910cead-2436-44b7-b425-d42a7063d4b8 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:57.292949771 +0000 UTC m=+90.105301948 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8910cead-2436-44b7-b425-d42a7063d4b8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ncf4z" (UID: "8910cead-2436-44b7-b425-d42a7063d4b8") : configmap "kube-rbac-proxy" not found
Jul 24 15:09:56.793115 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:56.793022    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7eb671c8-eae8-409d-943a-c8f76ddede9f-service-ca-bundle podName:7eb671c8-eae8-409d-943a-c8f76ddede9f nodeName:}" failed. No retries permitted until 2026-07-24 15:10:00.793012883 +0000 UTC m=+93.605365059 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7eb671c8-eae8-409d-943a-c8f76ddede9f-service-ca-bundle") pod "router-default-55db6d66b7-2bssd" (UID: "7eb671c8-eae8-409d-943a-c8f76ddede9f") : configmap references non-existent config key: service-ca.crt
Jul 24 15:09:56.793115 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:56.793036    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8910cead-2436-44b7-b425-d42a7063d4b8-data-volume\") pod \"insights-runtime-extractor-ncf4z\" (UID: \"8910cead-2436-44b7-b425-d42a7063d4b8\") " pod="openshift-insights/insights-runtime-extractor-ncf4z"
Jul 24 15:09:56.793115 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:56.793065    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 15:09:56.793265 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:56.793122    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8910cead-2436-44b7-b425-d42a7063d4b8-insights-runtime-extractor-tls podName:8910cead-2436-44b7-b425-d42a7063d4b8 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:57.293107628 +0000 UTC m=+90.105459786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8910cead-2436-44b7-b425-d42a7063d4b8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ncf4z" (UID: "8910cead-2436-44b7-b425-d42a7063d4b8") : secret "insights-runtime-extractor-tls" not found
Jul 24 15:09:56.793303 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:56.793288    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8910cead-2436-44b7-b425-d42a7063d4b8-data-volume\") pod \"insights-runtime-extractor-ncf4z\" (UID: \"8910cead-2436-44b7-b425-d42a7063d4b8\") " pod="openshift-insights/insights-runtime-extractor-ncf4z"
Jul 24 15:09:56.804045 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:56.804019    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh9qx\" (UniqueName: \"kubernetes.io/projected/8910cead-2436-44b7-b425-d42a7063d4b8-kube-api-access-gh9qx\") pod \"insights-runtime-extractor-ncf4z\" (UID: \"8910cead-2436-44b7-b425-d42a7063d4b8\") " pod="openshift-insights/insights-runtime-extractor-ncf4z"
Jul 24 15:09:57.295968 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:57.295928    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8910cead-2436-44b7-b425-d42a7063d4b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ncf4z\" (UID: \"8910cead-2436-44b7-b425-d42a7063d4b8\") " pod="openshift-insights/insights-runtime-extractor-ncf4z"
Jul 24 15:09:57.295968 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:57.295962    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8910cead-2436-44b7-b425-d42a7063d4b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ncf4z\" (UID: \"8910cead-2436-44b7-b425-d42a7063d4b8\") " pod="openshift-insights/insights-runtime-extractor-ncf4z"
Jul 24 15:09:57.296176 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:57.296051    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 15:09:57.296176 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:57.296069    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 15:09:57.296176 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:57.296126    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8910cead-2436-44b7-b425-d42a7063d4b8-kube-rbac-proxy-cm podName:8910cead-2436-44b7-b425-d42a7063d4b8 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:58.296107794 +0000 UTC m=+91.108459952 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8910cead-2436-44b7-b425-d42a7063d4b8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ncf4z" (UID: "8910cead-2436-44b7-b425-d42a7063d4b8") : configmap "kube-rbac-proxy" not found
Jul 24 15:09:57.296176 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:57.296145    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8910cead-2436-44b7-b425-d42a7063d4b8-insights-runtime-extractor-tls podName:8910cead-2436-44b7-b425-d42a7063d4b8 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:58.296137394 +0000 UTC m=+91.108489551 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8910cead-2436-44b7-b425-d42a7063d4b8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ncf4z" (UID: "8910cead-2436-44b7-b425-d42a7063d4b8") : secret "insights-runtime-extractor-tls" not found
Jul 24 15:09:57.508204 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:57.508171    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-xxl8c"]
Jul 24 15:09:57.509995 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:57.509980    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-xxl8c"
Jul 24 15:09:57.512608 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:57.512585    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 24 15:09:57.512608 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:57.512605    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 24 15:09:57.512743 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:57.512585    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-924h7\""
Jul 24 15:09:57.523380 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:57.523355    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-xxl8c"]
Jul 24 15:09:57.598272 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:57.598191    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq77w\" (UniqueName: \"kubernetes.io/projected/d9cecba9-741c-4afe-8dc3-9bc7790c6278-kube-api-access-xq77w\") pod \"migrator-6865c9967-xxl8c\" (UID: \"d9cecba9-741c-4afe-8dc3-9bc7790c6278\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-xxl8c"
Jul 24 15:09:57.699043 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:57.699006    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq77w\" (UniqueName: \"kubernetes.io/projected/d9cecba9-741c-4afe-8dc3-9bc7790c6278-kube-api-access-xq77w\") pod \"migrator-6865c9967-xxl8c\" (UID: \"d9cecba9-741c-4afe-8dc3-9bc7790c6278\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-xxl8c"
Jul 24 15:09:57.707465 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:57.707443    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq77w\" (UniqueName: \"kubernetes.io/projected/d9cecba9-741c-4afe-8dc3-9bc7790c6278-kube-api-access-xq77w\") pod \"migrator-6865c9967-xxl8c\" (UID: \"d9cecba9-741c-4afe-8dc3-9bc7790c6278\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-xxl8c"
Jul 24 15:09:57.818261 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:57.818225    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-xxl8c"
Jul 24 15:09:57.940359 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:57.940327    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-xxl8c"]
Jul 24 15:09:57.943122 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:09:57.943092    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9cecba9_741c_4afe_8dc3_9bc7790c6278.slice/crio-ab99eaed0a5c4e38ba5c49aa4e7e034f0c4f30b471d319cb17edc39fa17a0b0a WatchSource:0}: Error finding container ab99eaed0a5c4e38ba5c49aa4e7e034f0c4f30b471d319cb17edc39fa17a0b0a: Status 404 returned error can't find the container with id ab99eaed0a5c4e38ba5c49aa4e7e034f0c4f30b471d319cb17edc39fa17a0b0a
Jul 24 15:09:58.100501 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:58.100466    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-xxl8c" event={"ID":"d9cecba9-741c-4afe-8dc3-9bc7790c6278","Type":"ContainerStarted","Data":"ab99eaed0a5c4e38ba5c49aa4e7e034f0c4f30b471d319cb17edc39fa17a0b0a"}
Jul 24 15:09:58.304251 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:58.304212    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8910cead-2436-44b7-b425-d42a7063d4b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ncf4z\" (UID: \"8910cead-2436-44b7-b425-d42a7063d4b8\") " pod="openshift-insights/insights-runtime-extractor-ncf4z"
Jul 24 15:09:58.304423 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:58.304324    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8910cead-2436-44b7-b425-d42a7063d4b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ncf4z\" (UID: \"8910cead-2436-44b7-b425-d42a7063d4b8\") " pod="openshift-insights/insights-runtime-extractor-ncf4z"
Jul 24 15:09:58.304423 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:58.304372    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 15:09:58.304423 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:58.304392    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 15:09:58.304528 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:58.304444    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8910cead-2436-44b7-b425-d42a7063d4b8-kube-rbac-proxy-cm podName:8910cead-2436-44b7-b425-d42a7063d4b8 nodeName:}" failed. No retries permitted until 2026-07-24 15:10:00.304428939 +0000 UTC m=+93.116781099 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8910cead-2436-44b7-b425-d42a7063d4b8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ncf4z" (UID: "8910cead-2436-44b7-b425-d42a7063d4b8") : configmap "kube-rbac-proxy" not found
Jul 24 15:09:58.304528 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:09:58.304458    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8910cead-2436-44b7-b425-d42a7063d4b8-insights-runtime-extractor-tls podName:8910cead-2436-44b7-b425-d42a7063d4b8 nodeName:}" failed. No retries permitted until 2026-07-24 15:10:00.30445248 +0000 UTC m=+93.116804636 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8910cead-2436-44b7-b425-d42a7063d4b8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ncf4z" (UID: "8910cead-2436-44b7-b425-d42a7063d4b8") : secret "insights-runtime-extractor-tls" not found
Jul 24 15:09:58.776705 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:58.776673    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-s5nzq_ca98516e-79dc-4972-86de-8ff390bc0de2/dns-node-resolver/0.log"
Jul 24 15:09:59.494060 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:59.493981    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-c2pxl"]
Jul 24 15:09:59.495948 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:59.495933    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-c2pxl"
Jul 24 15:09:59.498192 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:59.498171    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 24 15:09:59.498298 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:59.498285    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 24 15:09:59.499034 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:59.499018    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 24 15:09:59.499122 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:59.499018    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 24 15:09:59.499122 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:59.499018    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-bt2dk\""
Jul 24 15:09:59.506946 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:59.506925    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-c2pxl"]
Jul 24 15:09:59.613780 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:59.613735    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b36fceee-a13b-4a28-9fe3-cc925e8f9398-signing-key\") pod \"service-ca-658d7b7c78-c2pxl\" (UID: \"b36fceee-a13b-4a28-9fe3-cc925e8f9398\") " pod="openshift-service-ca/service-ca-658d7b7c78-c2pxl"
Jul 24 15:09:59.613780 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:59.613786    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b36fceee-a13b-4a28-9fe3-cc925e8f9398-signing-cabundle\") pod \"service-ca-658d7b7c78-c2pxl\" (UID: \"b36fceee-a13b-4a28-9fe3-cc925e8f9398\") " pod="openshift-service-ca/service-ca-658d7b7c78-c2pxl"
Jul 24 15:09:59.614016 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:59.613885    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jwt2\" (UniqueName: \"kubernetes.io/projected/b36fceee-a13b-4a28-9fe3-cc925e8f9398-kube-api-access-8jwt2\") pod \"service-ca-658d7b7c78-c2pxl\" (UID: \"b36fceee-a13b-4a28-9fe3-cc925e8f9398\") " pod="openshift-service-ca/service-ca-658d7b7c78-c2pxl"
Jul 24 15:09:59.714403 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:59.714365    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jwt2\" (UniqueName: \"kubernetes.io/projected/b36fceee-a13b-4a28-9fe3-cc925e8f9398-kube-api-access-8jwt2\") pod \"service-ca-658d7b7c78-c2pxl\" (UID: \"b36fceee-a13b-4a28-9fe3-cc925e8f9398\") " pod="openshift-service-ca/service-ca-658d7b7c78-c2pxl"
Jul 24 15:09:59.714651 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:59.714481    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b36fceee-a13b-4a28-9fe3-cc925e8f9398-signing-key\") pod \"service-ca-658d7b7c78-c2pxl\" (UID: \"b36fceee-a13b-4a28-9fe3-cc925e8f9398\") " pod="openshift-service-ca/service-ca-658d7b7c78-c2pxl"
Jul 24 15:09:59.714651 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:59.714514    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b36fceee-a13b-4a28-9fe3-cc925e8f9398-signing-cabundle\") pod \"service-ca-658d7b7c78-c2pxl\" (UID: \"b36fceee-a13b-4a28-9fe3-cc925e8f9398\") " pod="openshift-service-ca/service-ca-658d7b7c78-c2pxl"
Jul 24 15:09:59.715169 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:59.715145    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b36fceee-a13b-4a28-9fe3-cc925e8f9398-signing-cabundle\") pod \"service-ca-658d7b7c78-c2pxl\" (UID: \"b36fceee-a13b-4a28-9fe3-cc925e8f9398\") " pod="openshift-service-ca/service-ca-658d7b7c78-c2pxl"
Jul 24 15:09:59.716967 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:59.716943    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b36fceee-a13b-4a28-9fe3-cc925e8f9398-signing-key\") pod \"service-ca-658d7b7c78-c2pxl\" (UID: \"b36fceee-a13b-4a28-9fe3-cc925e8f9398\") " pod="openshift-service-ca/service-ca-658d7b7c78-c2pxl"
Jul 24 15:09:59.731056 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:59.731029    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jwt2\" (UniqueName: \"kubernetes.io/projected/b36fceee-a13b-4a28-9fe3-cc925e8f9398-kube-api-access-8jwt2\") pod \"service-ca-658d7b7c78-c2pxl\" (UID: \"b36fceee-a13b-4a28-9fe3-cc925e8f9398\") " pod="openshift-service-ca/service-ca-658d7b7c78-c2pxl"
Jul 24 15:09:59.776643 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:59.776615    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n9snp_2216188a-b1f1-4ddf-8f9e-346e93860df2/node-ca/0.log"
Jul 24 15:09:59.804131 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:59.804106    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-c2pxl"
Jul 24 15:09:59.923642 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:09:59.923619    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-c2pxl"]
Jul 24 15:09:59.925897 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:09:59.925868    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb36fceee_a13b_4a28_9fe3_cc925e8f9398.slice/crio-1f1bc0cb0bc828af8780105baf7853d01f2ec9da011ea74b43d869353e5dce05 WatchSource:0}: Error finding container 1f1bc0cb0bc828af8780105baf7853d01f2ec9da011ea74b43d869353e5dce05: Status 404 returned error can't find the container with id 1f1bc0cb0bc828af8780105baf7853d01f2ec9da011ea74b43d869353e5dce05
Jul 24 15:10:00.109343 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:00.109242    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-c2pxl" event={"ID":"b36fceee-a13b-4a28-9fe3-cc925e8f9398","Type":"ContainerStarted","Data":"1f1bc0cb0bc828af8780105baf7853d01f2ec9da011ea74b43d869353e5dce05"}
Jul 24 15:10:00.110964 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:00.110932    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-xxl8c" event={"ID":"d9cecba9-741c-4afe-8dc3-9bc7790c6278","Type":"ContainerStarted","Data":"21bba21a5ccdb20ed043bd6a2dcd382a863268cfbc2c1ec2343aa1b754ed0caf"}
Jul 24 15:10:00.111108 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:00.111006    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-xxl8c" event={"ID":"d9cecba9-741c-4afe-8dc3-9bc7790c6278","Type":"ContainerStarted","Data":"dfb71009bb5365b4ab8189b49d9dccbc2e92750922fbe7ad13e99c37a1ae1f06"}
Jul 24 15:10:00.145930 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:00.145879    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-xxl8c" podStartSLOduration=1.984143403 podStartE2EDuration="3.145863173s" totalImagesPullingTime="1.16171977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:09:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:10:00.145249557 +0000 UTC m=+92.957601736" watchObservedRunningTime="2026-07-24 15:10:00.145863173 +0000 UTC m=+92.958215351"
Jul 24 15:10:00.179049 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:00.179028    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bks2n_53659e71-d188-4315-81ab-7b53e4701a4f/serve-healthcheck-canary/0.log"
Jul 24 15:10:00.318637 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:00.318607    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8910cead-2436-44b7-b425-d42a7063d4b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ncf4z\" (UID: \"8910cead-2436-44b7-b425-d42a7063d4b8\") " pod="openshift-insights/insights-runtime-extractor-ncf4z"
Jul 24 15:10:00.318799 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:00.318640    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8910cead-2436-44b7-b425-d42a7063d4b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ncf4z\" (UID: \"8910cead-2436-44b7-b425-d42a7063d4b8\") " pod="openshift-insights/insights-runtime-extractor-ncf4z"
Jul 24 15:10:00.318799 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:10:00.318731    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 15:10:00.318799 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:10:00.318794    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 15:10:00.318984 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:10:00.318806    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8910cead-2436-44b7-b425-d42a7063d4b8-kube-rbac-proxy-cm podName:8910cead-2436-44b7-b425-d42a7063d4b8 nodeName:}" failed. No retries permitted until 2026-07-24 15:10:04.318787518 +0000 UTC m=+97.131139679 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8910cead-2436-44b7-b425-d42a7063d4b8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ncf4z" (UID: "8910cead-2436-44b7-b425-d42a7063d4b8") : configmap "kube-rbac-proxy" not found
Jul 24 15:10:00.318984 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:10:00.318854    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8910cead-2436-44b7-b425-d42a7063d4b8-insights-runtime-extractor-tls podName:8910cead-2436-44b7-b425-d42a7063d4b8 nodeName:}" failed. No retries permitted until 2026-07-24 15:10:04.318842734 +0000 UTC m=+97.131194891 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8910cead-2436-44b7-b425-d42a7063d4b8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ncf4z" (UID: "8910cead-2436-44b7-b425-d42a7063d4b8") : secret "insights-runtime-extractor-tls" not found
Jul 24 15:10:00.822862 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:00.822811    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eb671c8-eae8-409d-943a-c8f76ddede9f-metrics-certs\") pod \"router-default-55db6d66b7-2bssd\" (UID: \"7eb671c8-eae8-409d-943a-c8f76ddede9f\") " pod="openshift-ingress/router-default-55db6d66b7-2bssd"
Jul 24 15:10:00.823296 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:10:00.822960    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 24 15:10:00.823296 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:10:00.823020    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eb671c8-eae8-409d-943a-c8f76ddede9f-metrics-certs podName:7eb671c8-eae8-409d-943a-c8f76ddede9f nodeName:}" failed. No retries permitted until 2026-07-24 15:10:08.823005667 +0000 UTC m=+101.635357823 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7eb671c8-eae8-409d-943a-c8f76ddede9f-metrics-certs") pod "router-default-55db6d66b7-2bssd" (UID: "7eb671c8-eae8-409d-943a-c8f76ddede9f") : secret "router-metrics-certs-default" not found
Jul 24 15:10:00.823296 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:00.823084    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7eb671c8-eae8-409d-943a-c8f76ddede9f-service-ca-bundle\") pod \"router-default-55db6d66b7-2bssd\" (UID: \"7eb671c8-eae8-409d-943a-c8f76ddede9f\") " pod="openshift-ingress/router-default-55db6d66b7-2bssd"
Jul 24 15:10:00.823296 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:10:00.823256    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7eb671c8-eae8-409d-943a-c8f76ddede9f-service-ca-bundle podName:7eb671c8-eae8-409d-943a-c8f76ddede9f nodeName:}" failed. No retries permitted until 2026-07-24 15:10:08.823238165 +0000 UTC m=+101.635590329 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7eb671c8-eae8-409d-943a-c8f76ddede9f-service-ca-bundle") pod "router-default-55db6d66b7-2bssd" (UID: "7eb671c8-eae8-409d-943a-c8f76ddede9f") : configmap references non-existent config key: service-ca.crt
Jul 24 15:10:02.119393 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:02.119357    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-c2pxl" event={"ID":"b36fceee-a13b-4a28-9fe3-cc925e8f9398","Type":"ContainerStarted","Data":"98daf1ac1e25803ab0033aaee36da0e910342a73be2a0f1fd1dcde2af927b2f8"}
Jul 24 15:10:02.140833 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:02.140756    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-c2pxl" podStartSLOduration=1.137834356 podStartE2EDuration="3.140737299s" totalImagesPullingTime="2.002902943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:09:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:10:02.140002629 +0000 UTC m=+94.952354809" watchObservedRunningTime="2026-07-24 15:10:02.140737299 +0000 UTC m=+94.953089478"
Jul 24 15:10:04.350985 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:04.350939    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8910cead-2436-44b7-b425-d42a7063d4b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ncf4z\" (UID: \"8910cead-2436-44b7-b425-d42a7063d4b8\") " pod="openshift-insights/insights-runtime-extractor-ncf4z"
Jul 24 15:10:04.351495 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:04.351000    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8910cead-2436-44b7-b425-d42a7063d4b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ncf4z\" (UID: \"8910cead-2436-44b7-b425-d42a7063d4b8\") " pod="openshift-insights/insights-runtime-extractor-ncf4z"
Jul 24 15:10:04.351495 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:10:04.351156    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 15:10:04.351495 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:10:04.351217    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8910cead-2436-44b7-b425-d42a7063d4b8-insights-runtime-extractor-tls podName:8910cead-2436-44b7-b425-d42a7063d4b8 nodeName:}" failed. No retries permitted until 2026-07-24 15:10:12.351198493 +0000 UTC m=+105.163550653 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8910cead-2436-44b7-b425-d42a7063d4b8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ncf4z" (UID: "8910cead-2436-44b7-b425-d42a7063d4b8") : secret "insights-runtime-extractor-tls" not found
Jul 24 15:10:04.351689 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:10:04.351594    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 15:10:04.351689 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:10:04.351637    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8910cead-2436-44b7-b425-d42a7063d4b8-kube-rbac-proxy-cm podName:8910cead-2436-44b7-b425-d42a7063d4b8 nodeName:}" failed. No retries permitted until 2026-07-24 15:10:12.351624209 +0000 UTC m=+105.163976373 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8910cead-2436-44b7-b425-d42a7063d4b8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ncf4z" (UID: "8910cead-2436-44b7-b425-d42a7063d4b8") : configmap "kube-rbac-proxy" not found
Jul 24 15:10:05.964959 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:05.964912    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4427fb4-a65e-4505-9329-88c2c1c4e812-metrics-tls\") pod \"dns-default-4vmc8\" (UID: \"d4427fb4-a65e-4505-9329-88c2c1c4e812\") " pod="openshift-dns/dns-default-4vmc8"
Jul 24 15:10:05.965382 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:10:05.965049    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 15:10:05.965382 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:10:05.965122    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4427fb4-a65e-4505-9329-88c2c1c4e812-metrics-tls podName:d4427fb4-a65e-4505-9329-88c2c1c4e812 nodeName:}" failed. No retries permitted until 2026-07-24 15:11:09.965107696 +0000 UTC m=+162.777459853 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4427fb4-a65e-4505-9329-88c2c1c4e812-metrics-tls") pod "dns-default-4vmc8" (UID: "d4427fb4-a65e-4505-9329-88c2c1c4e812") : secret "dns-default-metrics-tls" not found
Jul 24 15:10:08.890165 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:08.890122    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7eb671c8-eae8-409d-943a-c8f76ddede9f-service-ca-bundle\") pod \"router-default-55db6d66b7-2bssd\" (UID: \"7eb671c8-eae8-409d-943a-c8f76ddede9f\") " pod="openshift-ingress/router-default-55db6d66b7-2bssd"
Jul 24 15:10:08.890680 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:08.890211    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eb671c8-eae8-409d-943a-c8f76ddede9f-metrics-certs\") pod \"router-default-55db6d66b7-2bssd\" (UID: \"7eb671c8-eae8-409d-943a-c8f76ddede9f\") " pod="openshift-ingress/router-default-55db6d66b7-2bssd"
Jul 24 15:10:08.890680 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:10:08.890323    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7eb671c8-eae8-409d-943a-c8f76ddede9f-service-ca-bundle podName:7eb671c8-eae8-409d-943a-c8f76ddede9f nodeName:}" failed. No retries permitted until 2026-07-24 15:10:24.890295163 +0000 UTC m=+117.702647321 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7eb671c8-eae8-409d-943a-c8f76ddede9f-service-ca-bundle") pod "router-default-55db6d66b7-2bssd" (UID: "7eb671c8-eae8-409d-943a-c8f76ddede9f") : configmap references non-existent config key: service-ca.crt
Jul 24 15:10:08.892667 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:08.892644    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eb671c8-eae8-409d-943a-c8f76ddede9f-metrics-certs\") pod \"router-default-55db6d66b7-2bssd\" (UID: \"7eb671c8-eae8-409d-943a-c8f76ddede9f\") " pod="openshift-ingress/router-default-55db6d66b7-2bssd"
Jul 24 15:10:10.057829 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:10.057781    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-z654w"
Jul 24 15:10:12.419691 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:12.419655    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8910cead-2436-44b7-b425-d42a7063d4b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ncf4z\" (UID: \"8910cead-2436-44b7-b425-d42a7063d4b8\") " pod="openshift-insights/insights-runtime-extractor-ncf4z"
Jul 24 15:10:12.419691 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:12.419694    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8910cead-2436-44b7-b425-d42a7063d4b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ncf4z\" (UID: \"8910cead-2436-44b7-b425-d42a7063d4b8\") " pod="openshift-insights/insights-runtime-extractor-ncf4z"
Jul 24 15:10:12.420224 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:10:12.419888    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 15:10:12.420224 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:10:12.419944    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8910cead-2436-44b7-b425-d42a7063d4b8-insights-runtime-extractor-tls podName:8910cead-2436-44b7-b425-d42a7063d4b8 nodeName:}" failed. No retries permitted until 2026-07-24 15:10:28.419927856 +0000 UTC m=+121.232280016 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8910cead-2436-44b7-b425-d42a7063d4b8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ncf4z" (UID: "8910cead-2436-44b7-b425-d42a7063d4b8") : secret "insights-runtime-extractor-tls" not found
Jul 24 15:10:12.420224 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:12.420193    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8910cead-2436-44b7-b425-d42a7063d4b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ncf4z\" (UID: \"8910cead-2436-44b7-b425-d42a7063d4b8\") " pod="openshift-insights/insights-runtime-extractor-ncf4z"
Jul 24 15:10:24.913975 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:24.913918    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7eb671c8-eae8-409d-943a-c8f76ddede9f-service-ca-bundle\") pod \"router-default-55db6d66b7-2bssd\" (UID: \"7eb671c8-eae8-409d-943a-c8f76ddede9f\") " pod="openshift-ingress/router-default-55db6d66b7-2bssd"
Jul 24 15:10:24.914557 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:24.914534    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7eb671c8-eae8-409d-943a-c8f76ddede9f-service-ca-bundle\") pod \"router-default-55db6d66b7-2bssd\" (UID: \"7eb671c8-eae8-409d-943a-c8f76ddede9f\") " pod="openshift-ingress/router-default-55db6d66b7-2bssd"
Jul 24 15:10:25.177699 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:25.177600    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-55db6d66b7-2bssd"
Jul 24 15:10:25.305713 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:25.305682    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-55db6d66b7-2bssd"]
Jul 24 15:10:25.308346 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:10:25.308310    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eb671c8_eae8_409d_943a_c8f76ddede9f.slice/crio-770d7dfc135688481a9e158199bfa61359edd79f2d64964bdd4bf8bbe0c77b56 WatchSource:0}: Error finding container 770d7dfc135688481a9e158199bfa61359edd79f2d64964bdd4bf8bbe0c77b56: Status 404 returned error can't find the container with id 770d7dfc135688481a9e158199bfa61359edd79f2d64964bdd4bf8bbe0c77b56
Jul 24 15:10:25.932084 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:25.932051    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85c88588b4-nqzxl"]
Jul 24 15:10:25.935108 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:25.935078    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85c88588b4-nqzxl"
Jul 24 15:10:25.941206 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:25.941171    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 24 15:10:25.944659 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:25.944636    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 24 15:10:25.944659 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:25.944649    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 24 15:10:25.944832 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:25.944681    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-rp26s\""
Jul 24 15:10:25.944832 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:25.944649    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 24 15:10:25.960640 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:25.960615    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85c88588b4-nqzxl"]
Jul 24 15:10:26.122920 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:26.122881    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8x2l\" (UniqueName: \"kubernetes.io/projected/c860d066-ab93-428c-ac6e-09d01cd4fb48-kube-api-access-j8x2l\") pod \"managed-serviceaccount-addon-agent-85c88588b4-nqzxl\" (UID: \"c860d066-ab93-428c-ac6e-09d01cd4fb48\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85c88588b4-nqzxl"
Jul 24 15:10:26.123109 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:26.122960    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c860d066-ab93-428c-ac6e-09d01cd4fb48-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-85c88588b4-nqzxl\" (UID: \"c860d066-ab93-428c-ac6e-09d01cd4fb48\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85c88588b4-nqzxl"
Jul 24 15:10:26.192451 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:26.192351    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-55db6d66b7-2bssd" event={"ID":"7eb671c8-eae8-409d-943a-c8f76ddede9f","Type":"ContainerStarted","Data":"5875b2dd179569fd029adbdb99fee387734a22f3c0dfdfa521939c0e120823be"}
Jul 24 15:10:26.192451 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:26.192406    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-55db6d66b7-2bssd" event={"ID":"7eb671c8-eae8-409d-943a-c8f76ddede9f","Type":"ContainerStarted","Data":"770d7dfc135688481a9e158199bfa61359edd79f2d64964bdd4bf8bbe0c77b56"}
Jul 24 15:10:26.217526 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:26.217473    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-55db6d66b7-2bssd" podStartSLOduration=33.217457539 podStartE2EDuration="33.217457539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:09:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:10:26.216601158 +0000 UTC m=+119.028953334" watchObservedRunningTime="2026-07-24 15:10:26.217457539 +0000 UTC m=+119.029809717"
Jul 24 15:10:26.223919 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:26.223893    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8x2l\" (UniqueName: \"kubernetes.io/projected/c860d066-ab93-428c-ac6e-09d01cd4fb48-kube-api-access-j8x2l\") pod \"managed-serviceaccount-addon-agent-85c88588b4-nqzxl\" (UID: \"c860d066-ab93-428c-ac6e-09d01cd4fb48\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85c88588b4-nqzxl"
Jul 24 15:10:26.224007 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:26.223967    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c860d066-ab93-428c-ac6e-09d01cd4fb48-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-85c88588b4-nqzxl\" (UID: \"c860d066-ab93-428c-ac6e-09d01cd4fb48\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85c88588b4-nqzxl"
Jul 24 15:10:26.226478 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:26.226456    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c860d066-ab93-428c-ac6e-09d01cd4fb48-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-85c88588b4-nqzxl\" (UID: \"c860d066-ab93-428c-ac6e-09d01cd4fb48\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85c88588b4-nqzxl"
Jul 24 15:10:26.234835 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:26.234790    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8x2l\" (UniqueName: \"kubernetes.io/projected/c860d066-ab93-428c-ac6e-09d01cd4fb48-kube-api-access-j8x2l\") pod \"managed-serviceaccount-addon-agent-85c88588b4-nqzxl\" (UID: \"c860d066-ab93-428c-ac6e-09d01cd4fb48\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85c88588b4-nqzxl"
Jul 24 15:10:26.256922 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:26.256897    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85c88588b4-nqzxl"
Jul 24 15:10:26.384282 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:26.384251    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85c88588b4-nqzxl"]
Jul 24 15:10:26.387440 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:10:26.387409    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc860d066_ab93_428c_ac6e_09d01cd4fb48.slice/crio-7696974a208c57087847f82b04780cd9c30903bf3ed7a7a766a5974cb6749c06 WatchSource:0}: Error finding container 7696974a208c57087847f82b04780cd9c30903bf3ed7a7a766a5974cb6749c06: Status 404 returned error can't find the container with id 7696974a208c57087847f82b04780cd9c30903bf3ed7a7a766a5974cb6749c06
Jul 24 15:10:26.758121 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:26.758086    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-696dc74c75-r8lwz"]
Jul 24 15:10:26.760475 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:26.760458    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-696dc74c75-r8lwz"
Jul 24 15:10:26.764240 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:26.764220    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 24 15:10:26.764666 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:26.764647    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 24 15:10:26.764763 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:26.764714    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6fm9c\""
Jul 24 15:10:26.765138 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:26.765118    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 24 15:10:26.765221 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:26.765189    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 24 15:10:26.765221 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:26.765202    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 24 15:10:26.765436 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:26.765423    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 24 15:10:26.765479 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:26.765442    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 24 15:10:26.785152 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:26.785125    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-696dc74c75-r8lwz"]
Jul 24 15:10:26.930002 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:26.929962    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e959f71-f7c8-42b3-aad5-b613c6488b30-oauth-serving-cert\") pod \"console-696dc74c75-r8lwz\" (UID: \"0e959f71-f7c8-42b3-aad5-b613c6488b30\") " pod="openshift-console/console-696dc74c75-r8lwz"
Jul 24 15:10:26.930002 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:26.930004    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e959f71-f7c8-42b3-aad5-b613c6488b30-console-oauth-config\") pod \"console-696dc74c75-r8lwz\" (UID: \"0e959f71-f7c8-42b3-aad5-b613c6488b30\") " pod="openshift-console/console-696dc74c75-r8lwz"
Jul 24 15:10:26.930212 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:26.930081    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e959f71-f7c8-42b3-aad5-b613c6488b30-service-ca\") pod \"console-696dc74c75-r8lwz\" (UID: \"0e959f71-f7c8-42b3-aad5-b613c6488b30\") " pod="openshift-console/console-696dc74c75-r8lwz"
Jul 24 15:10:26.930212 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:26.930118    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e959f71-f7c8-42b3-aad5-b613c6488b30-console-config\") pod \"console-696dc74c75-r8lwz\" (UID: \"0e959f71-f7c8-42b3-aad5-b613c6488b30\") " pod="openshift-console/console-696dc74c75-r8lwz"
Jul 24 15:10:26.930212 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:26.930141    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e959f71-f7c8-42b3-aad5-b613c6488b30-console-serving-cert\") pod \"console-696dc74c75-r8lwz\" (UID: \"0e959f71-f7c8-42b3-aad5-b613c6488b30\") " pod="openshift-console/console-696dc74c75-r8lwz"
Jul 24 15:10:26.930212 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:26.930156    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtmsc\" (UniqueName: \"kubernetes.io/projected/0e959f71-f7c8-42b3-aad5-b613c6488b30-kube-api-access-rtmsc\") pod \"console-696dc74c75-r8lwz\" (UID: \"0e959f71-f7c8-42b3-aad5-b613c6488b30\") " pod="openshift-console/console-696dc74c75-r8lwz"
Jul 24 15:10:27.031340 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:27.031305    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e959f71-f7c8-42b3-aad5-b613c6488b30-service-ca\") pod \"console-696dc74c75-r8lwz\" (UID: \"0e959f71-f7c8-42b3-aad5-b613c6488b30\") " pod="openshift-console/console-696dc74c75-r8lwz"
Jul 24 15:10:27.031791 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:27.031354    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e959f71-f7c8-42b3-aad5-b613c6488b30-console-config\") pod \"console-696dc74c75-r8lwz\" (UID: \"0e959f71-f7c8-42b3-aad5-b613c6488b30\") " pod="openshift-console/console-696dc74c75-r8lwz"
Jul 24 15:10:27.031791 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:27.031443    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e959f71-f7c8-42b3-aad5-b613c6488b30-console-serving-cert\") pod \"console-696dc74c75-r8lwz\" (UID: \"0e959f71-f7c8-42b3-aad5-b613c6488b30\") " pod="openshift-console/console-696dc74c75-r8lwz"
Jul 24 15:10:27.031791 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:27.031460    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtmsc\" (UniqueName: \"kubernetes.io/projected/0e959f71-f7c8-42b3-aad5-b613c6488b30-kube-api-access-rtmsc\") pod \"console-696dc74c75-r8lwz\" (UID: \"0e959f71-f7c8-42b3-aad5-b613c6488b30\") " pod="openshift-console/console-696dc74c75-r8lwz"
Jul 24 15:10:27.031791 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:27.031495    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e959f71-f7c8-42b3-aad5-b613c6488b30-oauth-serving-cert\") pod \"console-696dc74c75-r8lwz\" (UID: \"0e959f71-f7c8-42b3-aad5-b613c6488b30\") " pod="openshift-console/console-696dc74c75-r8lwz"
Jul 24 15:10:27.031791 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:27.031521    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e959f71-f7c8-42b3-aad5-b613c6488b30-console-oauth-config\") pod \"console-696dc74c75-r8lwz\" (UID: \"0e959f71-f7c8-42b3-aad5-b613c6488b30\") " pod="openshift-console/console-696dc74c75-r8lwz"
Jul 24 15:10:27.032140 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:27.032116    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e959f71-f7c8-42b3-aad5-b613c6488b30-service-ca\") pod \"console-696dc74c75-r8lwz\" (UID: \"0e959f71-f7c8-42b3-aad5-b613c6488b30\") " pod="openshift-console/console-696dc74c75-r8lwz"
Jul 24 15:10:27.032208 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:27.032172    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e959f71-f7c8-42b3-aad5-b613c6488b30-oauth-serving-cert\") pod \"console-696dc74c75-r8lwz\" (UID: \"0e959f71-f7c8-42b3-aad5-b613c6488b30\") " pod="openshift-console/console-696dc74c75-r8lwz"
Jul 24 15:10:27.032331 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:27.032311    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e959f71-f7c8-42b3-aad5-b613c6488b30-console-config\") pod \"console-696dc74c75-r8lwz\" (UID: \"0e959f71-f7c8-42b3-aad5-b613c6488b30\") " pod="openshift-console/console-696dc74c75-r8lwz"
Jul 24 15:10:27.033887 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:27.033870    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e959f71-f7c8-42b3-aad5-b613c6488b30-console-oauth-config\") pod \"console-696dc74c75-r8lwz\" (UID: \"0e959f71-f7c8-42b3-aad5-b613c6488b30\") " pod="openshift-console/console-696dc74c75-r8lwz"
Jul 24 15:10:27.033944 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:27.033928    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e959f71-f7c8-42b3-aad5-b613c6488b30-console-serving-cert\") pod \"console-696dc74c75-r8lwz\" (UID: \"0e959f71-f7c8-42b3-aad5-b613c6488b30\") " pod="openshift-console/console-696dc74c75-r8lwz"
Jul 24 15:10:27.040351 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:27.040330    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtmsc\" (UniqueName: \"kubernetes.io/projected/0e959f71-f7c8-42b3-aad5-b613c6488b30-kube-api-access-rtmsc\") pod \"console-696dc74c75-r8lwz\" (UID: \"0e959f71-f7c8-42b3-aad5-b613c6488b30\") " pod="openshift-console/console-696dc74c75-r8lwz"
Jul 24 15:10:27.070447 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:27.070413    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-696dc74c75-r8lwz"
Jul 24 15:10:27.178836 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:27.178783    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-55db6d66b7-2bssd"
Jul 24 15:10:27.181860 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:27.181535    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-55db6d66b7-2bssd"
Jul 24 15:10:27.198268 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:27.198230    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85c88588b4-nqzxl" event={"ID":"c860d066-ab93-428c-ac6e-09d01cd4fb48","Type":"ContainerStarted","Data":"7696974a208c57087847f82b04780cd9c30903bf3ed7a7a766a5974cb6749c06"}
Jul 24 15:10:27.198556 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:27.198500    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-55db6d66b7-2bssd"
Jul 24 15:10:27.200077 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:27.200036    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-55db6d66b7-2bssd"
Jul 24 15:10:27.235088 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:27.235050    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-696dc74c75-r8lwz"]
Jul 24 15:10:27.238626 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:10:27.238597    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e959f71_f7c8_42b3_aad5_b613c6488b30.slice/crio-548d0b9fc2bc4d503c26c9548f8befcff06ec39a061a4276025eb3f5238bb51f WatchSource:0}: Error finding container 548d0b9fc2bc4d503c26c9548f8befcff06ec39a061a4276025eb3f5238bb51f: Status 404 returned error can't find the container with id 548d0b9fc2bc4d503c26c9548f8befcff06ec39a061a4276025eb3f5238bb51f
Jul 24 15:10:28.054575 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:28.054536    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2n2ms"]
Jul 24 15:10:28.069331 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:28.068628    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2n2ms"
Jul 24 15:10:28.069331 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:28.069043    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2n2ms"]
Jul 24 15:10:28.071484 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:28.071452    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 24 15:10:28.071746 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:28.071727    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-5cb47\""
Jul 24 15:10:28.202875 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:28.202835    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-696dc74c75-r8lwz" event={"ID":"0e959f71-f7c8-42b3-aad5-b613c6488b30","Type":"ContainerStarted","Data":"548d0b9fc2bc4d503c26c9548f8befcff06ec39a061a4276025eb3f5238bb51f"}
Jul 24 15:10:28.240509 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:28.240456    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/483059f3-b83b-4def-90d9-d42dd79d20e1-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-2n2ms\" (UID: \"483059f3-b83b-4def-90d9-d42dd79d20e1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2n2ms"
Jul 24 15:10:28.341794 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:28.341709    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/483059f3-b83b-4def-90d9-d42dd79d20e1-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-2n2ms\" (UID: \"483059f3-b83b-4def-90d9-d42dd79d20e1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2n2ms"
Jul 24 15:10:28.341988 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:10:28.341897    2579 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 24 15:10:28.341988 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:10:28.341966    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/483059f3-b83b-4def-90d9-d42dd79d20e1-tls-certificates podName:483059f3-b83b-4def-90d9-d42dd79d20e1 nodeName:}" failed. No retries permitted until 2026-07-24 15:10:28.841949605 +0000 UTC m=+121.654301782 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/483059f3-b83b-4def-90d9-d42dd79d20e1-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-2n2ms" (UID: "483059f3-b83b-4def-90d9-d42dd79d20e1") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 24 15:10:28.443244 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:28.443198    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8910cead-2436-44b7-b425-d42a7063d4b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ncf4z\" (UID: \"8910cead-2436-44b7-b425-d42a7063d4b8\") " pod="openshift-insights/insights-runtime-extractor-ncf4z"
Jul 24 15:10:28.445370 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:28.445340    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8910cead-2436-44b7-b425-d42a7063d4b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ncf4z\" (UID: \"8910cead-2436-44b7-b425-d42a7063d4b8\") " pod="openshift-insights/insights-runtime-extractor-ncf4z"
Jul 24 15:10:28.725064 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:28.724979    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nvtjc\""
Jul 24 15:10:28.732917 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:28.732888    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ncf4z"
Jul 24 15:10:28.847095 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:28.847065    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/483059f3-b83b-4def-90d9-d42dd79d20e1-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-2n2ms\" (UID: \"483059f3-b83b-4def-90d9-d42dd79d20e1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2n2ms"
Jul 24 15:10:28.849753 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:28.849727    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/483059f3-b83b-4def-90d9-d42dd79d20e1-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-2n2ms\" (UID: \"483059f3-b83b-4def-90d9-d42dd79d20e1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2n2ms"
Jul 24 15:10:28.924066 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:28.924040    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ncf4z"]
Jul 24 15:10:28.927121 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:10:28.927086    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8910cead_2436_44b7_b425_d42a7063d4b8.slice/crio-85ce2b4e838826333c635a1b1fd172e663bf905e61142a0984430c2decb2d77b WatchSource:0}: Error finding container 85ce2b4e838826333c635a1b1fd172e663bf905e61142a0984430c2decb2d77b: Status 404 returned error can't find the container with id 85ce2b4e838826333c635a1b1fd172e663bf905e61142a0984430c2decb2d77b
Jul 24 15:10:28.980542 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:28.980465    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2n2ms"
Jul 24 15:10:29.137209 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:29.137177    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2n2ms"]
Jul 24 15:10:29.140911 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:10:29.140882    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod483059f3_b83b_4def_90d9_d42dd79d20e1.slice/crio-43af9b5d0fadd6635b79375f011209d4546ba3b1f8d889f3a9aa9eff67599961 WatchSource:0}: Error finding container 43af9b5d0fadd6635b79375f011209d4546ba3b1f8d889f3a9aa9eff67599961: Status 404 returned error can't find the container with id 43af9b5d0fadd6635b79375f011209d4546ba3b1f8d889f3a9aa9eff67599961
Jul 24 15:10:29.210388 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:29.209618    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85c88588b4-nqzxl" event={"ID":"c860d066-ab93-428c-ac6e-09d01cd4fb48","Type":"ContainerStarted","Data":"1c8d1c29836c7f255b56c547e7f2920e7a708dbb3ca6a9a23175a8d49992d8b0"}
Jul 24 15:10:29.212446 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:29.212416    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2n2ms" event={"ID":"483059f3-b83b-4def-90d9-d42dd79d20e1","Type":"ContainerStarted","Data":"43af9b5d0fadd6635b79375f011209d4546ba3b1f8d889f3a9aa9eff67599961"}
Jul 24 15:10:29.216088 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:29.216005    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ncf4z" event={"ID":"8910cead-2436-44b7-b425-d42a7063d4b8","Type":"ContainerStarted","Data":"85ce2b4e838826333c635a1b1fd172e663bf905e61142a0984430c2decb2d77b"}
Jul 24 15:10:31.225943 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:31.225906    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2n2ms" event={"ID":"483059f3-b83b-4def-90d9-d42dd79d20e1","Type":"ContainerStarted","Data":"5c08ff4f20d104f4782630810745e76ca1caf311a1914a698340ff215ea584c6"}
Jul 24 15:10:31.226442 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:31.226159    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2n2ms"
Jul 24 15:10:31.232530 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:31.232484    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2n2ms"
Jul 24 15:10:31.245558 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:31.245500    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85c88588b4-nqzxl" podStartSLOduration=3.773110499 podStartE2EDuration="6.245481366s" totalImagesPullingTime="2.472370867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:10:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:10:29.232178562 +0000 UTC m=+122.044530745" watchObservedRunningTime="2026-07-24 15:10:31.245481366 +0000 UTC m=+124.057833547"
Jul 24 15:10:31.245727 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:31.245623    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2n2ms" podStartSLOduration=1.630474677 podStartE2EDuration="3.245612108s" totalImagesPullingTime="1.615137431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:10:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:10:31.244009843 +0000 UTC m=+124.056362023" watchObservedRunningTime="2026-07-24 15:10:31.245612108 +0000 UTC m=+124.057964288"
Jul 24 15:10:35.244076 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:35.244009    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-696dc74c75-r8lwz" event={"ID":"0e959f71-f7c8-42b3-aad5-b613c6488b30","Type":"ContainerStarted","Data":"35f9722f37b6efc827d65f37ecf90a9d2df451ae774a7c2aac9f5751e0445b60"}
Jul 24 15:10:35.245712 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:35.245680    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ncf4z" event={"ID":"8910cead-2436-44b7-b425-d42a7063d4b8","Type":"ContainerStarted","Data":"ac538e1307401a32a0c9e40f4a156592b61a62b89c1784cb9d2218ab31a2c9e2"}
Jul 24 15:10:35.269273 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:35.269210    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-696dc74c75-r8lwz" podStartSLOduration=1.883457562 podStartE2EDuration="9.269187597s" totalImagesPullingTime="7.385730035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:10:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:10:35.26806333 +0000 UTC m=+128.080415510" watchObservedRunningTime="2026-07-24 15:10:35.269187597 +0000 UTC m=+128.081539776"
Jul 24 15:10:37.071463 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:37.071410    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-696dc74c75-r8lwz"
Jul 24 15:10:37.071960 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:37.071488    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-696dc74c75-r8lwz"
Jul 24 15:10:37.077462 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:37.077432    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-696dc74c75-r8lwz"
Jul 24 15:10:37.257432 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:37.257395    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-696dc74c75-r8lwz"
Jul 24 15:10:37.421986 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:37.421902    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0b197de-537f-457b-8d3f-4dc75d38c402-metrics-certs\") pod \"network-metrics-daemon-qgzfw\" (UID: \"e0b197de-537f-457b-8d3f-4dc75d38c402\") " pod="openshift-multus/network-metrics-daemon-qgzfw"
Jul 24 15:10:37.424104 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:37.424071    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0b197de-537f-457b-8d3f-4dc75d38c402-metrics-certs\") pod \"network-metrics-daemon-qgzfw\" (UID: \"e0b197de-537f-457b-8d3f-4dc75d38c402\") " pod="openshift-multus/network-metrics-daemon-qgzfw"
Jul 24 15:10:37.525255 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:37.525225    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-trkb8\""
Jul 24 15:10:37.533151 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:37.533117    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgzfw"
Jul 24 15:10:38.163129 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:38.163100    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qgzfw"]
Jul 24 15:10:38.165867 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:10:38.165842    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b197de_537f_457b_8d3f_4dc75d38c402.slice/crio-aa8bb7c6c782f600bad751f03f6b32c70622c46c8f854c33a7526d23eeaf1f6c WatchSource:0}: Error finding container aa8bb7c6c782f600bad751f03f6b32c70622c46c8f854c33a7526d23eeaf1f6c: Status 404 returned error can't find the container with id aa8bb7c6c782f600bad751f03f6b32c70622c46c8f854c33a7526d23eeaf1f6c
Jul 24 15:10:38.259190 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:38.259146    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ncf4z" event={"ID":"8910cead-2436-44b7-b425-d42a7063d4b8","Type":"ContainerStarted","Data":"ff9e01b00e7db4d2287059ae8cf896e1c6fc4e59e17b35a431d166ca3895730e"}
Jul 24 15:10:38.260374 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:38.260343    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qgzfw" event={"ID":"e0b197de-537f-457b-8d3f-4dc75d38c402","Type":"ContainerStarted","Data":"aa8bb7c6c782f600bad751f03f6b32c70622c46c8f854c33a7526d23eeaf1f6c"}
Jul 24 15:10:40.022942 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.022329    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-hb5nq"]
Jul 24 15:10:40.032274 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.032231    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hb5nq"
Jul 24 15:10:40.036428 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.036393    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 24 15:10:40.036616 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.036598    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 24 15:10:40.036754 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.036741    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-6rzjp\""
Jul 24 15:10:40.037389 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.037367    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 24 15:10:40.037524 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.037367    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 24 15:10:40.037887 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.037870    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 24 15:10:40.049620 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.049577    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-q79n7"]
Jul 24 15:10:40.053139 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.053032    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-lwh8n"]
Jul 24 15:10:40.053273 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.053252    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q79n7"
Jul 24 15:10:40.056458 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.056435    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 24 15:10:40.056633 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.056617    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 24 15:10:40.057154 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.057138    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hs9mz\""
Jul 24 15:10:40.057462 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.057443    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-hb5nq"]
Jul 24 15:10:40.057584 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.057568    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lwh8n"
Jul 24 15:10:40.057738 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.057725    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 24 15:10:40.061975 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.061953    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 24 15:10:40.062684 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.062616    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-5v4w5\""
Jul 24 15:10:40.062684 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.062640    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 24 15:10:40.062860 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.062623    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 24 15:10:40.096330 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.096302    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-lwh8n"]
Jul 24 15:10:40.143896 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.143858    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/280a572b-d0e3-4074-9c59-51792e5adb32-metrics-client-ca\") pod \"node-exporter-q79n7\" (UID: \"280a572b-d0e3-4074-9c59-51792e5adb32\") " pod="openshift-monitoring/node-exporter-q79n7"
Jul 24 15:10:40.144079 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.143914    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/280a572b-d0e3-4074-9c59-51792e5adb32-root\") pod \"node-exporter-q79n7\" (UID: \"280a572b-d0e3-4074-9c59-51792e5adb32\") " pod="openshift-monitoring/node-exporter-q79n7"
Jul 24 15:10:40.144079 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.143949    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6wqr\" (UniqueName: \"kubernetes.io/projected/c6bc6e76-0a71-4834-9c9b-9f81846580ea-kube-api-access-b6wqr\") pod \"kube-state-metrics-56d9f944b4-lwh8n\" (UID: \"c6bc6e76-0a71-4834-9c9b-9f81846580ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lwh8n"
Jul 24 15:10:40.144079 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.143985    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7623b444-a307-4704-8ee5-0a53778a716f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-hb5nq\" (UID: \"7623b444-a307-4704-8ee5-0a53778a716f\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hb5nq"
Jul 24 15:10:40.144079 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.144033    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/280a572b-d0e3-4074-9c59-51792e5adb32-node-exporter-wtmp\") pod \"node-exporter-q79n7\" (UID: \"280a572b-d0e3-4074-9c59-51792e5adb32\") " pod="openshift-monitoring/node-exporter-q79n7"
Jul 24 15:10:40.144079 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.144059    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/280a572b-d0e3-4074-9c59-51792e5adb32-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q79n7\" (UID: \"280a572b-d0e3-4074-9c59-51792e5adb32\") " pod="openshift-monitoring/node-exporter-q79n7"
Jul 24 15:10:40.144328 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.144115    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/280a572b-d0e3-4074-9c59-51792e5adb32-node-exporter-textfile\") pod \"node-exporter-q79n7\" (UID: \"280a572b-d0e3-4074-9c59-51792e5adb32\") " pod="openshift-monitoring/node-exporter-q79n7"
Jul 24 15:10:40.144328 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.144161    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wthp8\" (UniqueName: \"kubernetes.io/projected/280a572b-d0e3-4074-9c59-51792e5adb32-kube-api-access-wthp8\") pod \"node-exporter-q79n7\" (UID: \"280a572b-d0e3-4074-9c59-51792e5adb32\") " pod="openshift-monitoring/node-exporter-q79n7"
Jul 24 15:10:40.144328 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.144202    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7623b444-a307-4704-8ee5-0a53778a716f-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-hb5nq\" (UID: \"7623b444-a307-4704-8ee5-0a53778a716f\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hb5nq"
Jul 24 15:10:40.144328 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.144239    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bk62\" (UniqueName: \"kubernetes.io/projected/7623b444-a307-4704-8ee5-0a53778a716f-kube-api-access-5bk62\") pod \"openshift-state-metrics-b8f49f5c6-hb5nq\" (UID: \"7623b444-a307-4704-8ee5-0a53778a716f\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hb5nq"
Jul 24 15:10:40.144328 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.144265    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c6bc6e76-0a71-4834-9c9b-9f81846580ea-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-lwh8n\" (UID: \"c6bc6e76-0a71-4834-9c9b-9f81846580ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lwh8n"
Jul 24 15:10:40.144328 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.144289    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c6bc6e76-0a71-4834-9c9b-9f81846580ea-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-lwh8n\" (UID: \"c6bc6e76-0a71-4834-9c9b-9f81846580ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lwh8n"
Jul 24 15:10:40.144328 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.144316    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/280a572b-d0e3-4074-9c59-51792e5adb32-node-exporter-tls\") pod \"node-exporter-q79n7\" (UID: \"280a572b-d0e3-4074-9c59-51792e5adb32\") " pod="openshift-monitoring/node-exporter-q79n7"
Jul 24 15:10:40.144656 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.144349    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6bc6e76-0a71-4834-9c9b-9f81846580ea-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-lwh8n\" (UID: \"c6bc6e76-0a71-4834-9c9b-9f81846580ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lwh8n"
Jul 24 15:10:40.144656 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.144365    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/280a572b-d0e3-4074-9c59-51792e5adb32-node-exporter-accelerators-collector-config\") pod \"node-exporter-q79n7\" (UID: \"280a572b-d0e3-4074-9c59-51792e5adb32\") " pod="openshift-monitoring/node-exporter-q79n7"
Jul 24 15:10:40.144656 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.144421    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7623b444-a307-4704-8ee5-0a53778a716f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-hb5nq\" (UID: \"7623b444-a307-4704-8ee5-0a53778a716f\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hb5nq"
Jul 24 15:10:40.144656 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.144466    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/280a572b-d0e3-4074-9c59-51792e5adb32-sys\") pod \"node-exporter-q79n7\" (UID: \"280a572b-d0e3-4074-9c59-51792e5adb32\") " pod="openshift-monitoring/node-exporter-q79n7"
Jul 24 15:10:40.144656 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.144579    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6bc6e76-0a71-4834-9c9b-9f81846580ea-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-lwh8n\" (UID: \"c6bc6e76-0a71-4834-9c9b-9f81846580ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lwh8n"
Jul 24 15:10:40.144656 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.144620    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c6bc6e76-0a71-4834-9c9b-9f81846580ea-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-lwh8n\" (UID: \"c6bc6e76-0a71-4834-9c9b-9f81846580ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lwh8n"
Jul 24 15:10:40.245633 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.245593    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/280a572b-d0e3-4074-9c59-51792e5adb32-sys\") pod \"node-exporter-q79n7\" (UID: \"280a572b-d0e3-4074-9c59-51792e5adb32\") " pod="openshift-monitoring/node-exporter-q79n7"
Jul 24 15:10:40.245844 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.245651    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6bc6e76-0a71-4834-9c9b-9f81846580ea-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-lwh8n\" (UID: \"c6bc6e76-0a71-4834-9c9b-9f81846580ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lwh8n"
Jul 24 15:10:40.245844 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.245682    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c6bc6e76-0a71-4834-9c9b-9f81846580ea-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-lwh8n\" (UID: \"c6bc6e76-0a71-4834-9c9b-9f81846580ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lwh8n"
Jul 24 15:10:40.245844 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.245712    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/280a572b-d0e3-4074-9c59-51792e5adb32-metrics-client-ca\") pod \"node-exporter-q79n7\" (UID: \"280a572b-d0e3-4074-9c59-51792e5adb32\") " pod="openshift-monitoring/node-exporter-q79n7"
Jul 24 15:10:40.245844 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.245744    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/280a572b-d0e3-4074-9c59-51792e5adb32-root\") pod \"node-exporter-q79n7\" (UID: \"280a572b-d0e3-4074-9c59-51792e5adb32\") " pod="openshift-monitoring/node-exporter-q79n7"
Jul 24 15:10:40.245844 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.245771    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6wqr\" (UniqueName: \"kubernetes.io/projected/c6bc6e76-0a71-4834-9c9b-9f81846580ea-kube-api-access-b6wqr\") pod \"kube-state-metrics-56d9f944b4-lwh8n\" (UID: \"c6bc6e76-0a71-4834-9c9b-9f81846580ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lwh8n"
Jul 24 15:10:40.245844 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.245808    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7623b444-a307-4704-8ee5-0a53778a716f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-hb5nq\" (UID: \"7623b444-a307-4704-8ee5-0a53778a716f\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hb5nq"
Jul 24 15:10:40.246086 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.245857    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/280a572b-d0e3-4074-9c59-51792e5adb32-node-exporter-wtmp\") pod \"node-exporter-q79n7\" (UID: \"280a572b-d0e3-4074-9c59-51792e5adb32\") " pod="openshift-monitoring/node-exporter-q79n7"
Jul 24 15:10:40.246086 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.245884    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/280a572b-d0e3-4074-9c59-51792e5adb32-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q79n7\" (UID: \"280a572b-d0e3-4074-9c59-51792e5adb32\") " pod="openshift-monitoring/node-exporter-q79n7"
Jul 24 15:10:40.246086 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.245912    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/280a572b-d0e3-4074-9c59-51792e5adb32-node-exporter-textfile\") pod \"node-exporter-q79n7\" (UID: \"280a572b-d0e3-4074-9c59-51792e5adb32\") " pod="openshift-monitoring/node-exporter-q79n7"
Jul 24 15:10:40.246086 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.245954    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wthp8\" (UniqueName: \"kubernetes.io/projected/280a572b-d0e3-4074-9c59-51792e5adb32-kube-api-access-wthp8\") pod \"node-exporter-q79n7\" (UID: \"280a572b-d0e3-4074-9c59-51792e5adb32\") " pod="openshift-monitoring/node-exporter-q79n7"
Jul 24 15:10:40.246086 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.245980    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7623b444-a307-4704-8ee5-0a53778a716f-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-hb5nq\" (UID: \"7623b444-a307-4704-8ee5-0a53778a716f\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hb5nq"
Jul 24 15:10:40.246086 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.246006    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bk62\" (UniqueName: \"kubernetes.io/projected/7623b444-a307-4704-8ee5-0a53778a716f-kube-api-access-5bk62\") pod \"openshift-state-metrics-b8f49f5c6-hb5nq\" (UID: \"7623b444-a307-4704-8ee5-0a53778a716f\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hb5nq"
Jul 24 15:10:40.246086 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.246033    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c6bc6e76-0a71-4834-9c9b-9f81846580ea-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-lwh8n\" (UID: \"c6bc6e76-0a71-4834-9c9b-9f81846580ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lwh8n"
Jul 24 15:10:40.246086 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.246058    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c6bc6e76-0a71-4834-9c9b-9f81846580ea-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-lwh8n\" (UID: \"c6bc6e76-0a71-4834-9c9b-9f81846580ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lwh8n"
Jul 24 15:10:40.246086 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.246083    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/280a572b-d0e3-4074-9c59-51792e5adb32-node-exporter-tls\") pod \"node-exporter-q79n7\" (UID: \"280a572b-d0e3-4074-9c59-51792e5adb32\") " pod="openshift-monitoring/node-exporter-q79n7"
Jul 24 15:10:40.246354 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.246111    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6bc6e76-0a71-4834-9c9b-9f81846580ea-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-lwh8n\" (UID: \"c6bc6e76-0a71-4834-9c9b-9f81846580ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lwh8n"
Jul 24 15:10:40.246354 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.246139    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/280a572b-d0e3-4074-9c59-51792e5adb32-node-exporter-accelerators-collector-config\") pod \"node-exporter-q79n7\" (UID: \"280a572b-d0e3-4074-9c59-51792e5adb32\") " pod="openshift-monitoring/node-exporter-q79n7"
Jul 24 15:10:40.246354 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.246178    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7623b444-a307-4704-8ee5-0a53778a716f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-hb5nq\" (UID: \"7623b444-a307-4704-8ee5-0a53778a716f\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hb5nq"
Jul 24 15:10:40.247011 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.246983    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c6bc6e76-0a71-4834-9c9b-9f81846580ea-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-lwh8n\" (UID: \"c6bc6e76-0a71-4834-9c9b-9f81846580ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lwh8n"
Jul 24 15:10:40.247279 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.247258    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/280a572b-d0e3-4074-9c59-51792e5adb32-node-exporter-textfile\") pod \"node-exporter-q79n7\" (UID: \"280a572b-d0e3-4074-9c59-51792e5adb32\") " pod="openshift-monitoring/node-exporter-q79n7"
Jul 24 15:10:40.248229 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.248207    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7623b444-a307-4704-8ee5-0a53778a716f-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-hb5nq\" (UID: \"7623b444-a307-4704-8ee5-0a53778a716f\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hb5nq"
Jul 24 15:10:40.249036 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.249015    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/280a572b-d0e3-4074-9c59-51792e5adb32-metrics-client-ca\") pod \"node-exporter-q79n7\" (UID: \"280a572b-d0e3-4074-9c59-51792e5adb32\") " pod="openshift-monitoring/node-exporter-q79n7"
Jul 24 15:10:40.249129 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.249069    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/280a572b-d0e3-4074-9c59-51792e5adb32-sys\") pod \"node-exporter-q79n7\" (UID: \"280a572b-d0e3-4074-9c59-51792e5adb32\") " pod="openshift-monitoring/node-exporter-q79n7"
Jul 24 15:10:40.249129 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.249066    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/280a572b-d0e3-4074-9c59-51792e5adb32-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q79n7\" (UID: \"280a572b-d0e3-4074-9c59-51792e5adb32\") " pod="openshift-monitoring/node-exporter-q79n7"
Jul 24 15:10:40.249129 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.249109    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/280a572b-d0e3-4074-9c59-51792e5adb32-root\") pod \"node-exporter-q79n7\" (UID: \"280a572b-d0e3-4074-9c59-51792e5adb32\") " pod="openshift-monitoring/node-exporter-q79n7"
Jul 24 15:10:40.249285 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:10:40.249131    2579 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 24 15:10:40.249285 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:10:40.249170    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6bc6e76-0a71-4834-9c9b-9f81846580ea-kube-state-metrics-tls podName:c6bc6e76-0a71-4834-9c9b-9f81846580ea nodeName:}" failed. No retries permitted until 2026-07-24 15:10:40.749158447 +0000 UTC m=+133.561510604 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/c6bc6e76-0a71-4834-9c9b-9f81846580ea-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-lwh8n" (UID: "c6bc6e76-0a71-4834-9c9b-9f81846580ea") : secret "kube-state-metrics-tls" not found
Jul 24 15:10:40.249390 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:10:40.249381    2579 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 24 15:10:40.249441 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:10:40.249423    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7623b444-a307-4704-8ee5-0a53778a716f-openshift-state-metrics-tls podName:7623b444-a307-4704-8ee5-0a53778a716f nodeName:}" failed. No retries permitted until 2026-07-24 15:10:40.749408539 +0000 UTC m=+133.561760703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/7623b444-a307-4704-8ee5-0a53778a716f-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-hb5nq" (UID: "7623b444-a307-4704-8ee5-0a53778a716f") : secret "openshift-state-metrics-tls" not found
Jul 24 15:10:40.249539 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.249511    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/280a572b-d0e3-4074-9c59-51792e5adb32-node-exporter-wtmp\") pod \"node-exporter-q79n7\" (UID: \"280a572b-d0e3-4074-9c59-51792e5adb32\") " pod="openshift-monitoring/node-exporter-q79n7"
Jul 24 15:10:40.251605 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.249763    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c6bc6e76-0a71-4834-9c9b-9f81846580ea-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-lwh8n\" (UID: \"c6bc6e76-0a71-4834-9c9b-9f81846580ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lwh8n"
Jul 24 15:10:40.251605 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.250328    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6bc6e76-0a71-4834-9c9b-9f81846580ea-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-lwh8n\" (UID: \"c6bc6e76-0a71-4834-9c9b-9f81846580ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lwh8n"
Jul 24 15:10:40.251605 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.251117    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/280a572b-d0e3-4074-9c59-51792e5adb32-node-exporter-accelerators-collector-config\") pod \"node-exporter-q79n7\" (UID: \"280a572b-d0e3-4074-9c59-51792e5adb32\") " pod="openshift-monitoring/node-exporter-q79n7"
Jul 24 15:10:40.251605 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.251564    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7623b444-a307-4704-8ee5-0a53778a716f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-hb5nq\" (UID: \"7623b444-a307-4704-8ee5-0a53778a716f\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hb5nq"
Jul 24 15:10:40.252213 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.252147    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/280a572b-d0e3-4074-9c59-51792e5adb32-node-exporter-tls\") pod \"node-exporter-q79n7\" (UID: \"280a572b-d0e3-4074-9c59-51792e5adb32\") " pod="openshift-monitoring/node-exporter-q79n7"
Jul 24 15:10:40.252951 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.252929    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c6bc6e76-0a71-4834-9c9b-9f81846580ea-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-lwh8n\" (UID: \"c6bc6e76-0a71-4834-9c9b-9f81846580ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lwh8n"
Jul 24 15:10:40.270629 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.270599    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6wqr\" (UniqueName: \"kubernetes.io/projected/c6bc6e76-0a71-4834-9c9b-9f81846580ea-kube-api-access-b6wqr\") pod \"kube-state-metrics-56d9f944b4-lwh8n\" (UID: \"c6bc6e76-0a71-4834-9c9b-9f81846580ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lwh8n"
Jul 24 15:10:40.271770 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.271744    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bk62\" (UniqueName: \"kubernetes.io/projected/7623b444-a307-4704-8ee5-0a53778a716f-kube-api-access-5bk62\") pod \"openshift-state-metrics-b8f49f5c6-hb5nq\" (UID: \"7623b444-a307-4704-8ee5-0a53778a716f\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hb5nq"
Jul 24 15:10:40.271952 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.271921    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qgzfw" event={"ID":"e0b197de-537f-457b-8d3f-4dc75d38c402","Type":"ContainerStarted","Data":"74ad1312637bb1c24b6d8aee2b818aae667e4b9108d4829facd9a6d2f54523be"}
Jul 24 15:10:40.272052 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.271983    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qgzfw" event={"ID":"e0b197de-537f-457b-8d3f-4dc75d38c402","Type":"ContainerStarted","Data":"b42605dabe46e64362637e90e4dd2594ebe8b12d195ca9d56a4742c65df45347"}
Jul 24 15:10:40.279699 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.279644    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wthp8\" (UniqueName: \"kubernetes.io/projected/280a572b-d0e3-4074-9c59-51792e5adb32-kube-api-access-wthp8\") pod \"node-exporter-q79n7\" (UID: \"280a572b-d0e3-4074-9c59-51792e5adb32\") " pod="openshift-monitoring/node-exporter-q79n7"
Jul 24 15:10:40.305414 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.305315    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qgzfw" podStartSLOduration=131.162010574 podStartE2EDuration="2m12.305296178s" totalImagesPullingTime="1.143285604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:08:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:10:40.304966637 +0000 UTC m=+133.117318823" watchObservedRunningTime="2026-07-24 15:10:40.305296178 +0000 UTC m=+133.117648357"
Jul 24 15:10:40.367658 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.367616    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q79n7"
Jul 24 15:10:40.750757 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.750677    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6bc6e76-0a71-4834-9c9b-9f81846580ea-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-lwh8n\" (UID: \"c6bc6e76-0a71-4834-9c9b-9f81846580ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lwh8n"
Jul 24 15:10:40.750757 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.750734    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7623b444-a307-4704-8ee5-0a53778a716f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-hb5nq\" (UID: \"7623b444-a307-4704-8ee5-0a53778a716f\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hb5nq"
Jul 24 15:10:40.753734 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.753707    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6bc6e76-0a71-4834-9c9b-9f81846580ea-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-lwh8n\" (UID: \"c6bc6e76-0a71-4834-9c9b-9f81846580ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lwh8n"
Jul 24 15:10:40.753877 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.753707    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7623b444-a307-4704-8ee5-0a53778a716f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-hb5nq\" (UID: \"7623b444-a307-4704-8ee5-0a53778a716f\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hb5nq"
Jul 24 15:10:40.944941 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.944689    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hb5nq"
Jul 24 15:10:40.974221 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:40.974187    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lwh8n"
Jul 24 15:10:41.169076 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.169037    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 15:10:41.174945 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.174917    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 24 15:10:41.186285 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.186253    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 24 15:10:41.186418 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.186386    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 24 15:10:41.186418 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.186409    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 24 15:10:41.186516 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.186388    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 24 15:10:41.186516 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.186511    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-v9wkj\""
Jul 24 15:10:41.186830 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.186794    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 24 15:10:41.189642 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.189622    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 24 15:10:41.191739 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.191721    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 24 15:10:41.192349 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.192327    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 24 15:10:41.195953 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.195938    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 24 15:10:41.218330 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.218292    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 15:10:41.254696 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.254658    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 15:10:41.254911 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.254719    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 15:10:41.254911 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.254759    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 15:10:41.254911 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.254809    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00-config-volume\") pod \"alertmanager-main-0\" (UID: \"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 15:10:41.254911 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.254861    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 15:10:41.255157 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.254926    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 15:10:41.255157 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.254953    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 15:10:41.255157 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.255026    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 15:10:41.255157 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.255072    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00-web-config\") pod \"alertmanager-main-0\" (UID: \"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 15:10:41.255157 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.255110    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhdgh\" (UniqueName: \"kubernetes.io/projected/b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00-kube-api-access-zhdgh\") pod \"alertmanager-main-0\" (UID: \"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 15:10:41.255157 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.255154    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00-config-out\") pod \"alertmanager-main-0\" (UID: \"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 15:10:41.255395 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.255179    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 15:10:41.255395 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.255209    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 15:10:41.356385 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.356344    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00-config-out\") pod \"alertmanager-main-0\" (UID: \"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 15:10:41.356567 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.356392    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 15:10:41.356567 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.356438    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 15:10:41.356567 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.356515    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 15:10:41.356734 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.356586    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 15:10:41.356734 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.356624    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 15:10:41.356734 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.356677    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00-config-volume\") pod \"alertmanager-main-0\" (UID: \"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 15:10:41.356734 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.356718    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 15:10:41.356973 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.356777    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 15:10:41.357185 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.357161    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 15:10:41.357259 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.357216    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 15:10:41.357321 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.357260    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 15:10:41.357321 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.357291    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00-web-config\") pod \"alertmanager-main-0\" (UID: \"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 15:10:41.357418 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.357340    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhdgh\" (UniqueName: \"kubernetes.io/projected/b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00-kube-api-access-zhdgh\") pod \"alertmanager-main-0\" (UID: \"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 15:10:41.357418 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.357390    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 15:10:41.357590 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.357570    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 15:10:41.361399 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:10:41.360130    2579 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 24 15:10:41.361399 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:10:41.360211    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00-secret-alertmanager-main-tls podName:b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00 nodeName:}" failed. No retries permitted until 2026-07-24 15:10:41.860183013 +0000 UTC m=+134.672535169 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00") : secret "alertmanager-main-tls" not found
Jul 24 15:10:41.361399 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.361134    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 15:10:41.363620 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.361643    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00-config-volume\") pod \"alertmanager-main-0\" (UID: \"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 15:10:41.363620 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.361838    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 15:10:41.363620 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.361869    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00-config-out\") pod \"alertmanager-main-0\" (UID: \"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 15:10:41.363620 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.362281    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00-web-config\") pod \"alertmanager-main-0\" (UID: \"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 15:10:41.363620 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.362946    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 15:10:41.363620 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.363022    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 15:10:41.366940 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.364295    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 15:10:41.374853 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.374805    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhdgh\" (UniqueName: \"kubernetes.io/projected/b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00-kube-api-access-zhdgh\") pod \"alertmanager-main-0\" (UID: \"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 15:10:41.746311 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:10:41.746276    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod280a572b_d0e3_4074_9c59_51792e5adb32.slice/crio-d81d64e3e482c1cb2da6973a92ffb66a911802d2f2712683998e62f9b91788a8 WatchSource:0}: Error finding container d81d64e3e482c1cb2da6973a92ffb66a911802d2f2712683998e62f9b91788a8: Status 404 returned error can't find the container with id d81d64e3e482c1cb2da6973a92ffb66a911802d2f2712683998e62f9b91788a8
Jul 24 15:10:41.861850 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.861795    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 15:10:41.864272 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.864243    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 15:10:41.874735 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.874708    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-hb5nq"]
Jul 24 15:10:41.877725 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:10:41.877694    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7623b444_a307_4704_8ee5_0a53778a716f.slice/crio-6abd892f0242e63625512061f259a15545f863c44bdc0e204180e9e497898c2f WatchSource:0}: Error finding container 6abd892f0242e63625512061f259a15545f863c44bdc0e204180e9e497898c2f: Status 404 returned error can't find the container with id 6abd892f0242e63625512061f259a15545f863c44bdc0e204180e9e497898c2f
Jul 24 15:10:41.893558 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:41.893479    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-lwh8n"]
Jul 24 15:10:41.904870 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:10:41.904845    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6bc6e76_0a71_4834_9c9b_9f81846580ea.slice/crio-49291cacf687285aad847f9d719eceea866503c6bcacb0fdd008bdfacb0ba89e WatchSource:0}: Error finding container 49291cacf687285aad847f9d719eceea866503c6bcacb0fdd008bdfacb0ba89e: Status 404 returned error can't find the container with id 49291cacf687285aad847f9d719eceea866503c6bcacb0fdd008bdfacb0ba89e
Jul 24 15:10:42.086476 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:42.086445    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 24 15:10:42.228075 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:42.228046    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 15:10:42.231489 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:10:42.231464    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb41ede1a_6b93_42d4_a4bd_3bf5f1a3ef00.slice/crio-c8a659b7b55c444117a358266af6d9ea4784275c3532f0038d7b2617e2cf5e3f WatchSource:0}: Error finding container c8a659b7b55c444117a358266af6d9ea4784275c3532f0038d7b2617e2cf5e3f: Status 404 returned error can't find the container with id c8a659b7b55c444117a358266af6d9ea4784275c3532f0038d7b2617e2cf5e3f
Jul 24 15:10:42.281191 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:42.281152    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ncf4z" event={"ID":"8910cead-2436-44b7-b425-d42a7063d4b8","Type":"ContainerStarted","Data":"c45dd2cde53cae65ee9cf5352ea31da34abeccd932e4cc1e59bd4a193f7304d3"}
Jul 24 15:10:42.282360 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:42.282331    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q79n7" event={"ID":"280a572b-d0e3-4074-9c59-51792e5adb32","Type":"ContainerStarted","Data":"d81d64e3e482c1cb2da6973a92ffb66a911802d2f2712683998e62f9b91788a8"}
Jul 24 15:10:42.283311 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:42.283284    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00","Type":"ContainerStarted","Data":"c8a659b7b55c444117a358266af6d9ea4784275c3532f0038d7b2617e2cf5e3f"}
Jul 24 15:10:42.284246 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:42.284223    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lwh8n" event={"ID":"c6bc6e76-0a71-4834-9c9b-9f81846580ea","Type":"ContainerStarted","Data":"49291cacf687285aad847f9d719eceea866503c6bcacb0fdd008bdfacb0ba89e"}
Jul 24 15:10:42.285778 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:42.285759    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hb5nq" event={"ID":"7623b444-a307-4704-8ee5-0a53778a716f","Type":"ContainerStarted","Data":"7ff06b414be208f2827d758fbdc4ebc137b5da9fe40bbb6394c6ada9d8e4c415"}
Jul 24 15:10:42.285879 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:42.285801    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hb5nq" event={"ID":"7623b444-a307-4704-8ee5-0a53778a716f","Type":"ContainerStarted","Data":"cf2d5ad57ed18da2f69349286cc3f924f6002310a6f886e9a35477a3af0632cd"}
Jul 24 15:10:42.285879 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:42.285841    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hb5nq" event={"ID":"7623b444-a307-4704-8ee5-0a53778a716f","Type":"ContainerStarted","Data":"6abd892f0242e63625512061f259a15545f863c44bdc0e204180e9e497898c2f"}
Jul 24 15:10:42.302579 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:42.302524    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ncf4z" podStartSLOduration=33.587723233 podStartE2EDuration="46.302505446s" totalImagesPullingTime="12.714782213s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:09:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:10:42.301744478 +0000 UTC m=+135.114096699" watchObservedRunningTime="2026-07-24 15:10:42.302505446 +0000 UTC m=+135.114857627"
Jul 24 15:10:42.991749 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:42.991712    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-67c4bd47f4-pt275"]
Jul 24 15:10:42.994984 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:42.994931    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-67c4bd47f4-pt275"
Jul 24 15:10:42.997622 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:42.997566    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 24 15:10:42.997787 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:42.997566    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 24 15:10:42.998161 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:42.998141    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-7dhdx\""
Jul 24 15:10:42.999286 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:42.999263    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 24 15:10:42.999385 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:42.999290    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5uvi0rjgtuacf\""
Jul 24 15:10:42.999385 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:42.999304    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 24 15:10:42.999385 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:42.999290    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 24 15:10:43.021136 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:43.021102    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-67c4bd47f4-pt275"]
Jul 24 15:10:43.072645 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:43.072444    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cfebcd8d-9ec5-400b-ad42-d5fc438d8e15-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67c4bd47f4-pt275\" (UID: \"cfebcd8d-9ec5-400b-ad42-d5fc438d8e15\") " pod="openshift-monitoring/thanos-querier-67c4bd47f4-pt275"
Jul 24 15:10:43.072645 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:43.072507    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cfebcd8d-9ec5-400b-ad42-d5fc438d8e15-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67c4bd47f4-pt275\" (UID: \"cfebcd8d-9ec5-400b-ad42-d5fc438d8e15\") " pod="openshift-monitoring/thanos-querier-67c4bd47f4-pt275"
Jul 24 15:10:43.072645 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:43.072546    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cfebcd8d-9ec5-400b-ad42-d5fc438d8e15-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67c4bd47f4-pt275\" (UID: \"cfebcd8d-9ec5-400b-ad42-d5fc438d8e15\") " pod="openshift-monitoring/thanos-querier-67c4bd47f4-pt275"
Jul 24 15:10:43.072645 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:43.072588    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sml4q\" (UniqueName: \"kubernetes.io/projected/cfebcd8d-9ec5-400b-ad42-d5fc438d8e15-kube-api-access-sml4q\") pod \"thanos-querier-67c4bd47f4-pt275\" (UID: \"cfebcd8d-9ec5-400b-ad42-d5fc438d8e15\") " pod="openshift-monitoring/thanos-querier-67c4bd47f4-pt275"
Jul 24 15:10:43.073003 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:43.072653    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cfebcd8d-9ec5-400b-ad42-d5fc438d8e15-secret-grpc-tls\") pod \"thanos-querier-67c4bd47f4-pt275\" (UID: \"cfebcd8d-9ec5-400b-ad42-d5fc438d8e15\") " pod="openshift-monitoring/thanos-querier-67c4bd47f4-pt275"
Jul 24 15:10:43.073003 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:43.072699    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cfebcd8d-9ec5-400b-ad42-d5fc438d8e15-metrics-client-ca\") pod \"thanos-querier-67c4bd47f4-pt275\" (UID: \"cfebcd8d-9ec5-400b-ad42-d5fc438d8e15\") " pod="openshift-monitoring/thanos-querier-67c4bd47f4-pt275"
Jul 24 15:10:43.073003 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:43.072803    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cfebcd8d-9ec5-400b-ad42-d5fc438d8e15-secret-thanos-querier-tls\") pod \"thanos-querier-67c4bd47f4-pt275\" (UID: \"cfebcd8d-9ec5-400b-ad42-d5fc438d8e15\") " pod="openshift-monitoring/thanos-querier-67c4bd47f4-pt275"
Jul 24 15:10:43.073003 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:43.072859    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cfebcd8d-9ec5-400b-ad42-d5fc438d8e15-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67c4bd47f4-pt275\" (UID: \"cfebcd8d-9ec5-400b-ad42-d5fc438d8e15\") " pod="openshift-monitoring/thanos-querier-67c4bd47f4-pt275"
Jul 24 15:10:43.173891 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:43.173846    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cfebcd8d-9ec5-400b-ad42-d5fc438d8e15-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67c4bd47f4-pt275\" (UID: \"cfebcd8d-9ec5-400b-ad42-d5fc438d8e15\") " pod="openshift-monitoring/thanos-querier-67c4bd47f4-pt275"
Jul 24 15:10:43.174077 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:43.173898    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cfebcd8d-9ec5-400b-ad42-d5fc438d8e15-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67c4bd47f4-pt275\" (UID: \"cfebcd8d-9ec5-400b-ad42-d5fc438d8e15\") " pod="openshift-monitoring/thanos-querier-67c4bd47f4-pt275"
Jul 24 15:10:43.174077 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:43.173931    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cfebcd8d-9ec5-400b-ad42-d5fc438d8e15-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67c4bd47f4-pt275\" (UID: \"cfebcd8d-9ec5-400b-ad42-d5fc438d8e15\") " pod="openshift-monitoring/thanos-querier-67c4bd47f4-pt275"
Jul 24 15:10:43.174077 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:43.173971    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sml4q\" (UniqueName: \"kubernetes.io/projected/cfebcd8d-9ec5-400b-ad42-d5fc438d8e15-kube-api-access-sml4q\") pod \"thanos-querier-67c4bd47f4-pt275\" (UID: \"cfebcd8d-9ec5-400b-ad42-d5fc438d8e15\") " pod="openshift-monitoring/thanos-querier-67c4bd47f4-pt275"
Jul 24 15:10:43.174077 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:43.174005    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cfebcd8d-9ec5-400b-ad42-d5fc438d8e15-secret-grpc-tls\") pod \"thanos-querier-67c4bd47f4-pt275\" (UID: \"cfebcd8d-9ec5-400b-ad42-d5fc438d8e15\") " pod="openshift-monitoring/thanos-querier-67c4bd47f4-pt275"
Jul 24 15:10:43.174077 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:43.174037    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cfebcd8d-9ec5-400b-ad42-d5fc438d8e15-metrics-client-ca\") pod \"thanos-querier-67c4bd47f4-pt275\" (UID: \"cfebcd8d-9ec5-400b-ad42-d5fc438d8e15\") " pod="openshift-monitoring/thanos-querier-67c4bd47f4-pt275"
Jul 24 15:10:43.174273 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:43.174118    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cfebcd8d-9ec5-400b-ad42-d5fc438d8e15-secret-thanos-querier-tls\") pod \"thanos-querier-67c4bd47f4-pt275\" (UID: \"cfebcd8d-9ec5-400b-ad42-d5fc438d8e15\") " pod="openshift-monitoring/thanos-querier-67c4bd47f4-pt275"
Jul 24 15:10:43.174273 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:43.174149    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cfebcd8d-9ec5-400b-ad42-d5fc438d8e15-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67c4bd47f4-pt275\" (UID: \"cfebcd8d-9ec5-400b-ad42-d5fc438d8e15\") " pod="openshift-monitoring/thanos-querier-67c4bd47f4-pt275"
Jul 24 15:10:43.175159 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:43.175129    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cfebcd8d-9ec5-400b-ad42-d5fc438d8e15-metrics-client-ca\") pod \"thanos-querier-67c4bd47f4-pt275\" (UID: \"cfebcd8d-9ec5-400b-ad42-d5fc438d8e15\") " pod="openshift-monitoring/thanos-querier-67c4bd47f4-pt275"
Jul 24 15:10:43.178354 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:43.178269    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cfebcd8d-9ec5-400b-ad42-d5fc438d8e15-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67c4bd47f4-pt275\" (UID: \"cfebcd8d-9ec5-400b-ad42-d5fc438d8e15\") " pod="openshift-monitoring/thanos-querier-67c4bd47f4-pt275"
Jul 24 15:10:43.178354 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:43.178277    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cfebcd8d-9ec5-400b-ad42-d5fc438d8e15-secret-thanos-querier-tls\") pod \"thanos-querier-67c4bd47f4-pt275\" (UID: \"cfebcd8d-9ec5-400b-ad42-d5fc438d8e15\") " pod="openshift-monitoring/thanos-querier-67c4bd47f4-pt275"
Jul 24 15:10:43.180171 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:43.180131    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cfebcd8d-9ec5-400b-ad42-d5fc438d8e15-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67c4bd47f4-pt275\" (UID: \"cfebcd8d-9ec5-400b-ad42-d5fc438d8e15\") " pod="openshift-monitoring/thanos-querier-67c4bd47f4-pt275"
Jul 24 15:10:43.180389 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:43.180359    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cfebcd8d-9ec5-400b-ad42-d5fc438d8e15-secret-grpc-tls\") pod \"thanos-querier-67c4bd47f4-pt275\" (UID: \"cfebcd8d-9ec5-400b-ad42-d5fc438d8e15\") " pod="openshift-monitoring/thanos-querier-67c4bd47f4-pt275"
Jul 24 15:10:43.181540 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:43.180787    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cfebcd8d-9ec5-400b-ad42-d5fc438d8e15-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67c4bd47f4-pt275\" (UID: \"cfebcd8d-9ec5-400b-ad42-d5fc438d8e15\") " pod="openshift-monitoring/thanos-querier-67c4bd47f4-pt275"
Jul 24 15:10:43.181540 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:43.181495    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cfebcd8d-9ec5-400b-ad42-d5fc438d8e15-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67c4bd47f4-pt275\" (UID: \"cfebcd8d-9ec5-400b-ad42-d5fc438d8e15\") " pod="openshift-monitoring/thanos-querier-67c4bd47f4-pt275"
Jul 24 15:10:43.187235 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:43.187208    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sml4q\" (UniqueName: \"kubernetes.io/projected/cfebcd8d-9ec5-400b-ad42-d5fc438d8e15-kube-api-access-sml4q\") pod \"thanos-querier-67c4bd47f4-pt275\" (UID: \"cfebcd8d-9ec5-400b-ad42-d5fc438d8e15\") " pod="openshift-monitoring/thanos-querier-67c4bd47f4-pt275"
Jul 24 15:10:43.308997 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:43.308867    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-67c4bd47f4-pt275"
Jul 24 15:10:44.000121 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:44.000076    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-67c4bd47f4-pt275"]
Jul 24 15:10:44.044612 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:10:44.044557    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfebcd8d_9ec5_400b_ad42_d5fc438d8e15.slice/crio-87e97be1768fa15b3bf794b45b091e597b7082031c926ad5e490a75c60795fda WatchSource:0}: Error finding container 87e97be1768fa15b3bf794b45b091e597b7082031c926ad5e490a75c60795fda: Status 404 returned error can't find the container with id 87e97be1768fa15b3bf794b45b091e597b7082031c926ad5e490a75c60795fda
Jul 24 15:10:44.296236 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:44.296202    2579 generic.go:356] "Generic (PLEG): container finished" podID="b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00" containerID="1fc3c704cb8c3662f1e6590d469de696bb1949c4b6d9f9f690e0f2f30cb2626b" exitCode=0
Jul 24 15:10:44.296408 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:44.296308    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00","Type":"ContainerDied","Data":"1fc3c704cb8c3662f1e6590d469de696bb1949c4b6d9f9f690e0f2f30cb2626b"}
Jul 24 15:10:44.299795 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:44.299765    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lwh8n" event={"ID":"c6bc6e76-0a71-4834-9c9b-9f81846580ea","Type":"ContainerStarted","Data":"c03cea23f7082e454c1ce33599b8a6f69172c0be88f1eaa6d836fe8af6e790b4"}
Jul 24 15:10:44.299937 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:44.299850    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lwh8n" event={"ID":"c6bc6e76-0a71-4834-9c9b-9f81846580ea","Type":"ContainerStarted","Data":"aaf457284b88ab222d6e9e8bfb4b2ba74d07e6f34468de28c3a19f9f7e6ab225"}
Jul 24 15:10:44.299937 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:44.299882    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lwh8n" event={"ID":"c6bc6e76-0a71-4834-9c9b-9f81846580ea","Type":"ContainerStarted","Data":"738474112b80f90b21e7b0d34bf6d2d10289012262fc235d6d9480d57fbf638d"}
Jul 24 15:10:44.301923 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:44.301609    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67c4bd47f4-pt275" event={"ID":"cfebcd8d-9ec5-400b-ad42-d5fc438d8e15","Type":"ContainerStarted","Data":"87e97be1768fa15b3bf794b45b091e597b7082031c926ad5e490a75c60795fda"}
Jul 24 15:10:44.304294 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:44.304212    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hb5nq" event={"ID":"7623b444-a307-4704-8ee5-0a53778a716f","Type":"ContainerStarted","Data":"75d58e50aff200d157cf2a30c3e497c0dc113071465fba722fde51bc66b53e30"}
Jul 24 15:10:44.306258 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:44.306234    2579 generic.go:356] "Generic (PLEG): container finished" podID="280a572b-d0e3-4074-9c59-51792e5adb32" containerID="818cc829c38870253e3d2b8b9622a086c3cbd52e5595411bdd3a9c0064e5d0a1" exitCode=0
Jul 24 15:10:44.306370 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:44.306303    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q79n7" event={"ID":"280a572b-d0e3-4074-9c59-51792e5adb32","Type":"ContainerDied","Data":"818cc829c38870253e3d2b8b9622a086c3cbd52e5595411bdd3a9c0064e5d0a1"}
Jul 24 15:10:44.413302 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:44.413253    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lwh8n" podStartSLOduration=3.474876214 podStartE2EDuration="5.413234907s" totalImagesPullingTime="1.938358693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:10:44.411965299 +0000 UTC m=+137.224317479" watchObservedRunningTime="2026-07-24 15:10:44.413234907 +0000 UTC m=+137.225587088"
Jul 24 15:10:44.432784 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:44.432752    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55fd4cb8d5-29wgz"]
Jul 24 15:10:44.436800 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:44.436779    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55fd4cb8d5-29wgz"
Jul 24 15:10:44.447668 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:44.447645    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 24 15:10:44.451735 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:44.451686    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hb5nq" podStartSLOduration=3.5957960939999998 podStartE2EDuration="5.451665844s" totalImagesPullingTime="1.85586975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:10:44.4491986 +0000 UTC m=+137.261550781" watchObservedRunningTime="2026-07-24 15:10:44.451665844 +0000 UTC m=+137.264018025"
Jul 24 15:10:44.455031 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:44.455006    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55fd4cb8d5-29wgz"]
Jul 24 15:10:44.486459 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:44.486406    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09921c51-0843-4384-8eb2-7840e85e0b7d-trusted-ca-bundle\") pod \"console-55fd4cb8d5-29wgz\" (UID: \"09921c51-0843-4384-8eb2-7840e85e0b7d\") " pod="openshift-console/console-55fd4cb8d5-29wgz"
Jul 24 15:10:44.486639 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:44.486516    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09921c51-0843-4384-8eb2-7840e85e0b7d-console-oauth-config\") pod \"console-55fd4cb8d5-29wgz\" (UID: \"09921c51-0843-4384-8eb2-7840e85e0b7d\") " pod="openshift-console/console-55fd4cb8d5-29wgz"
Jul 24 15:10:44.486639 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:44.486555    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09921c51-0843-4384-8eb2-7840e85e0b7d-service-ca\") pod \"console-55fd4cb8d5-29wgz\" (UID: \"09921c51-0843-4384-8eb2-7840e85e0b7d\") " pod="openshift-console/console-55fd4cb8d5-29wgz"
Jul 24 15:10:44.486639 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:44.486583    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09921c51-0843-4384-8eb2-7840e85e0b7d-console-serving-cert\") pod \"console-55fd4cb8d5-29wgz\" (UID: \"09921c51-0843-4384-8eb2-7840e85e0b7d\") " pod="openshift-console/console-55fd4cb8d5-29wgz"
Jul 24 15:10:44.486639 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:44.486603    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09921c51-0843-4384-8eb2-7840e85e0b7d-oauth-serving-cert\") pod \"console-55fd4cb8d5-29wgz\" (UID: \"09921c51-0843-4384-8eb2-7840e85e0b7d\") " pod="openshift-console/console-55fd4cb8d5-29wgz"
Jul 24 15:10:44.486639 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:44.486621    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09921c51-0843-4384-8eb2-7840e85e0b7d-console-config\") pod \"console-55fd4cb8d5-29wgz\" (UID: \"09921c51-0843-4384-8eb2-7840e85e0b7d\") " pod="openshift-console/console-55fd4cb8d5-29wgz"
Jul 24 15:10:44.486639 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:44.486639    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpl6s\" (UniqueName: \"kubernetes.io/projected/09921c51-0843-4384-8eb2-7840e85e0b7d-kube-api-access-dpl6s\") pod \"console-55fd4cb8d5-29wgz\" (UID: \"09921c51-0843-4384-8eb2-7840e85e0b7d\") " pod="openshift-console/console-55fd4cb8d5-29wgz"
Jul 24 15:10:44.587646 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:44.587612    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpl6s\" (UniqueName: \"kubernetes.io/projected/09921c51-0843-4384-8eb2-7840e85e0b7d-kube-api-access-dpl6s\") pod \"console-55fd4cb8d5-29wgz\" (UID: \"09921c51-0843-4384-8eb2-7840e85e0b7d\") " pod="openshift-console/console-55fd4cb8d5-29wgz"
Jul 24 15:10:44.587882 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:44.587674    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09921c51-0843-4384-8eb2-7840e85e0b7d-trusted-ca-bundle\") pod \"console-55fd4cb8d5-29wgz\" (UID: \"09921c51-0843-4384-8eb2-7840e85e0b7d\") " pod="openshift-console/console-55fd4cb8d5-29wgz"
Jul 24 15:10:44.587882 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:44.587710    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09921c51-0843-4384-8eb2-7840e85e0b7d-console-oauth-config\") pod \"console-55fd4cb8d5-29wgz\" (UID: \"09921c51-0843-4384-8eb2-7840e85e0b7d\") " pod="openshift-console/console-55fd4cb8d5-29wgz"
Jul 24 15:10:44.587882 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:44.587731    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09921c51-0843-4384-8eb2-7840e85e0b7d-service-ca\") pod \"console-55fd4cb8d5-29wgz\" (UID: \"09921c51-0843-4384-8eb2-7840e85e0b7d\") " pod="openshift-console/console-55fd4cb8d5-29wgz"
Jul 24 15:10:44.587882 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:44.587753    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09921c51-0843-4384-8eb2-7840e85e0b7d-console-serving-cert\") pod \"console-55fd4cb8d5-29wgz\" (UID: \"09921c51-0843-4384-8eb2-7840e85e0b7d\") " pod="openshift-console/console-55fd4cb8d5-29wgz"
Jul 24 15:10:44.587882 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:44.587771    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09921c51-0843-4384-8eb2-7840e85e0b7d-oauth-serving-cert\") pod \"console-55fd4cb8d5-29wgz\" (UID: \"09921c51-0843-4384-8eb2-7840e85e0b7d\") " pod="openshift-console/console-55fd4cb8d5-29wgz"
Jul 24 15:10:44.587882 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:44.587786    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09921c51-0843-4384-8eb2-7840e85e0b7d-console-config\") pod \"console-55fd4cb8d5-29wgz\" (UID: \"09921c51-0843-4384-8eb2-7840e85e0b7d\") " pod="openshift-console/console-55fd4cb8d5-29wgz"
Jul 24 15:10:44.588669 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:44.588635    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09921c51-0843-4384-8eb2-7840e85e0b7d-oauth-serving-cert\") pod \"console-55fd4cb8d5-29wgz\" (UID: \"09921c51-0843-4384-8eb2-7840e85e0b7d\") " pod="openshift-console/console-55fd4cb8d5-29wgz"
Jul 24 15:10:44.588770 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:44.588728    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09921c51-0843-4384-8eb2-7840e85e0b7d-console-config\") pod \"console-55fd4cb8d5-29wgz\" (UID: \"09921c51-0843-4384-8eb2-7840e85e0b7d\") " pod="openshift-console/console-55fd4cb8d5-29wgz"
Jul 24 15:10:44.588770 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:44.588747    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09921c51-0843-4384-8eb2-7840e85e0b7d-service-ca\") pod \"console-55fd4cb8d5-29wgz\" (UID: \"09921c51-0843-4384-8eb2-7840e85e0b7d\") " pod="openshift-console/console-55fd4cb8d5-29wgz"
Jul 24 15:10:44.588898 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:44.588766    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09921c51-0843-4384-8eb2-7840e85e0b7d-trusted-ca-bundle\") pod \"console-55fd4cb8d5-29wgz\" (UID: \"09921c51-0843-4384-8eb2-7840e85e0b7d\") " pod="openshift-console/console-55fd4cb8d5-29wgz"
Jul 24 15:10:44.590758 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:44.590737    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09921c51-0843-4384-8eb2-7840e85e0b7d-console-serving-cert\") pod \"console-55fd4cb8d5-29wgz\" (UID: \"09921c51-0843-4384-8eb2-7840e85e0b7d\") " pod="openshift-console/console-55fd4cb8d5-29wgz"
Jul 24 15:10:44.590857 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:44.590759    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09921c51-0843-4384-8eb2-7840e85e0b7d-console-oauth-config\") pod \"console-55fd4cb8d5-29wgz\" (UID: \"09921c51-0843-4384-8eb2-7840e85e0b7d\") " pod="openshift-console/console-55fd4cb8d5-29wgz"
Jul 24 15:10:44.603616 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:44.603593    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpl6s\" (UniqueName: \"kubernetes.io/projected/09921c51-0843-4384-8eb2-7840e85e0b7d-kube-api-access-dpl6s\") pod \"console-55fd4cb8d5-29wgz\" (UID: \"09921c51-0843-4384-8eb2-7840e85e0b7d\") " pod="openshift-console/console-55fd4cb8d5-29wgz"
Jul 24 15:10:44.746216 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:44.746126    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55fd4cb8d5-29wgz"
Jul 24 15:10:44.875140 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:44.875107    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55fd4cb8d5-29wgz"]
Jul 24 15:10:44.879332 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:10:44.879304    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09921c51_0843_4384_8eb2_7840e85e0b7d.slice/crio-7e6e3e8ad130c5d64cc11e4425937449fc6522636eb15598055c0102ffe33b37 WatchSource:0}: Error finding container 7e6e3e8ad130c5d64cc11e4425937449fc6522636eb15598055c0102ffe33b37: Status 404 returned error can't find the container with id 7e6e3e8ad130c5d64cc11e4425937449fc6522636eb15598055c0102ffe33b37
Jul 24 15:10:45.319840 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:45.319775    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q79n7" event={"ID":"280a572b-d0e3-4074-9c59-51792e5adb32","Type":"ContainerStarted","Data":"a4caf7ab4b2dace6e01f2cceabd893fbbce97a77c5511da4cb96a1f5809797b3"}
Jul 24 15:10:45.320013 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:45.319897    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q79n7" event={"ID":"280a572b-d0e3-4074-9c59-51792e5adb32","Type":"ContainerStarted","Data":"616044b57ce6484262e00553b54ecd5aeffc5a33c6a58bfd715051717072d185"}
Jul 24 15:10:45.321607 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:45.321576    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55fd4cb8d5-29wgz" event={"ID":"09921c51-0843-4384-8eb2-7840e85e0b7d","Type":"ContainerStarted","Data":"b6354b4d7f9e69cf80ef4ed79a56a92642b5a8b6d667ce453c03423e81a27f18"}
Jul 24 15:10:45.321755 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:45.321624    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55fd4cb8d5-29wgz" event={"ID":"09921c51-0843-4384-8eb2-7840e85e0b7d","Type":"ContainerStarted","Data":"7e6e3e8ad130c5d64cc11e4425937449fc6522636eb15598055c0102ffe33b37"}
Jul 24 15:10:45.387064 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:45.386996    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55fd4cb8d5-29wgz" podStartSLOduration=1.386973128 podStartE2EDuration="1.386973128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:10:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:10:45.386071597 +0000 UTC m=+138.198423823" watchObservedRunningTime="2026-07-24 15:10:45.386973128 +0000 UTC m=+138.199325309"
Jul 24 15:10:45.387620 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:45.387382    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-q79n7" podStartSLOduration=4.21207342 podStartE2EDuration="6.387371289s" totalImagesPullingTime="2.096417269s" totalInitContainerRuntime="78.8806ms" isStatefulPod=true podCreationTimestamp="2026-07-24 15:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:10:45.351950502 +0000 UTC m=+138.164302685" watchObservedRunningTime="2026-07-24 15:10:45.387371289 +0000 UTC m=+138.199723469"
Jul 24 15:10:46.202230 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.202185    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 15:10:46.206330 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.206302    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.210117 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.210091    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 24 15:10:46.210285 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.210094    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 24 15:10:46.210532 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.210510    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 24 15:10:46.210532 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.210525    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 24 15:10:46.210853 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.210806    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-dncfv\""
Jul 24 15:10:46.211001 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.210866    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 24 15:10:46.211001 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.210887    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 24 15:10:46.211001 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.210995    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 24 15:10:46.211187 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.211090    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 24 15:10:46.211187 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.211167    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 24 15:10:46.211426 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.211368    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 24 15:10:46.211426 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.211388    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 24 15:10:46.211426 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.211416    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 24 15:10:46.211589 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.211430    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9j7iitb8osdq4\""
Jul 24 15:10:46.211589 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.211535    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 24 15:10:46.215386 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.215366    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 24 15:10:46.217409 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.217389    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 24 15:10:46.225182 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.225122    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 15:10:46.305381 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.305340    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.305566 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.305433    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.305566 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.305473    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.305566 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.305492    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-config\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.305566 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.305508    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-config-out\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.305566 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.305528    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.305864 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.305584    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.305864 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.305618    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.305864 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.305651    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.305864 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.305705    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.305864 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.305730    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.305864 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.305755    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.305864 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.305784    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.305864 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.305839    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.306194 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.305867    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.306194 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.305897    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2bj4\" (UniqueName: \"kubernetes.io/projected/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-kube-api-access-t2bj4\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.306194 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.305936    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-web-config\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.306194 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.306003    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.306194 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.306038    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.306194 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.306142    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.406862 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.406800    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.407055 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.406875    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.407055 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.406901    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.407055 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.406930    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.407055 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.406965    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.407055 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.406991    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.407055 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.407019    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2bj4\" (UniqueName: \"kubernetes.io/projected/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-kube-api-access-t2bj4\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.407339 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.407067    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-web-config\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.407712 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.407679    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.409697 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.407872    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.409697 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.407933    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.409697 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.407988    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.409697 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.408041    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.409697 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.408118    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.409697 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.408151    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.409697 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.408175    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-config\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.409697 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.408203    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-config-out\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.409697 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.408227    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.409697 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.408260    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.409697 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.408285    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.409697 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.408333    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.409697 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.408935    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.409697 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.409350    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.409697 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.409536    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.410504 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.410148    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.410504 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.410175    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.410609 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.410550    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.410889 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.410860    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.412172 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.411036    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.412172 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.411606    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.412328 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.412230    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-config-out\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.412470 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.412445    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.413070 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.412530    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.413184 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.412970    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.413184 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.412979    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-web-config\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.413184 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.413176    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.413630 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.413589    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.413970 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.413952    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-config\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.414328 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.414307    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.419445 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.419427    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2bj4\" (UniqueName: \"kubernetes.io/projected/ff4c5089-e2d7-4345-9761-9d96a4c41a0f-kube-api-access-t2bj4\") pod \"prometheus-k8s-0\" (UID: \"ff4c5089-e2d7-4345-9761-9d96a4c41a0f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.518928 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.518886    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:10:46.794466 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:46.794441    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 15:10:46.796386 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:10:46.796360    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff4c5089_e2d7_4345_9761_9d96a4c41a0f.slice/crio-faaf3bd401d83a05fa11e4b5a44818e2c95645e31a1bf4386a1f3582110bafd9 WatchSource:0}: Error finding container faaf3bd401d83a05fa11e4b5a44818e2c95645e31a1bf4386a1f3582110bafd9: Status 404 returned error can't find the container with id faaf3bd401d83a05fa11e4b5a44818e2c95645e31a1bf4386a1f3582110bafd9
Jul 24 15:10:47.332474 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:47.332385    2579 generic.go:356] "Generic (PLEG): container finished" podID="ff4c5089-e2d7-4345-9761-9d96a4c41a0f" containerID="329516f6c6b8bd4a41ed68466608858dfea6a0e7388b19ddf90f4747a29ceced" exitCode=0
Jul 24 15:10:47.332952 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:47.332467    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff4c5089-e2d7-4345-9761-9d96a4c41a0f","Type":"ContainerDied","Data":"329516f6c6b8bd4a41ed68466608858dfea6a0e7388b19ddf90f4747a29ceced"}
Jul 24 15:10:47.332952 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:47.332562    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff4c5089-e2d7-4345-9761-9d96a4c41a0f","Type":"ContainerStarted","Data":"faaf3bd401d83a05fa11e4b5a44818e2c95645e31a1bf4386a1f3582110bafd9"}
Jul 24 15:10:47.335002 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:47.334976    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67c4bd47f4-pt275" event={"ID":"cfebcd8d-9ec5-400b-ad42-d5fc438d8e15","Type":"ContainerStarted","Data":"a08fa4ce9d88dd71b5049d214f3a70408e7e79206c6d49cc7e09b684bd2f120c"}
Jul 24 15:10:47.335099 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:47.335036    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67c4bd47f4-pt275" event={"ID":"cfebcd8d-9ec5-400b-ad42-d5fc438d8e15","Type":"ContainerStarted","Data":"5fa67dc04080d3a5eb3fa591e25542d4b9110a03bda679f5f18dc905be9f6a4e"}
Jul 24 15:10:47.335099 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:47.335076    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67c4bd47f4-pt275" event={"ID":"cfebcd8d-9ec5-400b-ad42-d5fc438d8e15","Type":"ContainerStarted","Data":"a0c5c1bbf14c60cdaf094d6b06011118d24c43f7b10b68035be0ef06f256ac72"}
Jul 24 15:10:47.338121 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:47.338101    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00","Type":"ContainerStarted","Data":"75b1e03a12de8027a4f960287e5fa9f08881b5675bf877e0d1eb616759ac08cb"}
Jul 24 15:10:47.338215 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:47.338150    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00","Type":"ContainerStarted","Data":"fa8f200ebb22fca8481bdbbf3a90207d35e59b07cfa1d93fa602f1c33426fd56"}
Jul 24 15:10:47.338215 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:47.338174    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00","Type":"ContainerStarted","Data":"18b60f344d213de98e4e37d5ce8e3e743cd36557aa2be73d3a73252a270fa090"}
Jul 24 15:10:47.338215 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:47.338200    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00","Type":"ContainerStarted","Data":"1930b9c1f6390d8d13c1c9acd0f10e036ce4edea4cb9a8cbd3921f05abf86d49"}
Jul 24 15:10:47.338313 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:47.338227    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00","Type":"ContainerStarted","Data":"36fdfb18b4d4b04c90c336399cf0500a9389cf8704c37152fcf464ae3eb510e7"}
Jul 24 15:10:48.345945 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:48.345907    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67c4bd47f4-pt275" event={"ID":"cfebcd8d-9ec5-400b-ad42-d5fc438d8e15","Type":"ContainerStarted","Data":"7f9fe5fea88114efaed10d8209b1aba21c77c0f21f70bd074ce05437e3e1d44f"}
Jul 24 15:10:48.346401 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:48.346034    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67c4bd47f4-pt275" event={"ID":"cfebcd8d-9ec5-400b-ad42-d5fc438d8e15","Type":"ContainerStarted","Data":"a5a6639cb9a106160da00d78f2f829513de74cf7c677b3150a54edefeaf0770a"}
Jul 24 15:10:48.346401 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:48.346069    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67c4bd47f4-pt275" event={"ID":"cfebcd8d-9ec5-400b-ad42-d5fc438d8e15","Type":"ContainerStarted","Data":"49c77cd4ed1dc6e2e2b4e2b30919ec3fb371ada4ea70e00ea027044d9123243d"}
Jul 24 15:10:48.346401 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:48.346108    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-67c4bd47f4-pt275"
Jul 24 15:10:48.349660 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:48.349621    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00","Type":"ContainerStarted","Data":"54a94bba0f39835287d64b575bf400cdb079c47955e83cc185aba3f1ec92fd24"}
Jul 24 15:10:48.373944 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:48.373891    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-67c4bd47f4-pt275" podStartSLOduration=2.6611893159999997 podStartE2EDuration="6.373876876s" totalImagesPullingTime="3.71268756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:10:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:10:48.373665731 +0000 UTC m=+141.186017921" watchObservedRunningTime="2026-07-24 15:10:48.373876876 +0000 UTC m=+141.186229055"
Jul 24 15:10:48.410622 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:48.410570    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.423305341 podStartE2EDuration="7.410551543s" totalImagesPullingTime="4.961760097s" totalInitContainerRuntime="25.486105ms" isStatefulPod=false podCreationTimestamp="2026-07-24 15:10:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:10:48.407915961 +0000 UTC m=+141.220268151" watchObservedRunningTime="2026-07-24 15:10:48.410551543 +0000 UTC m=+141.222903725"
Jul 24 15:10:52.376482 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:52.376446    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff4c5089-e2d7-4345-9761-9d96a4c41a0f","Type":"ContainerStarted","Data":"8e3fe5e041f480dfb3d6b4395d9305345c99bffa8a1093149ae49bac4dcd10a9"}
Jul 24 15:10:52.376921 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:52.376510    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff4c5089-e2d7-4345-9761-9d96a4c41a0f","Type":"ContainerStarted","Data":"70656a8da3512852cea30d83fbb5ed93431eae21bdf8268ff066fbfd7502dd49"}
Jul 24 15:10:52.376921 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:52.376538    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff4c5089-e2d7-4345-9761-9d96a4c41a0f","Type":"ContainerStarted","Data":"13d43ab7ccfcdf2deaf47d6ecec93fb3840ce9434e763b0254dca1ff73335363"}
Jul 24 15:10:52.376921 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:52.376561    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff4c5089-e2d7-4345-9761-9d96a4c41a0f","Type":"ContainerStarted","Data":"8ce956626288f478193d0904642ffc991b25bab60905e95dadb869989dd10862"}
Jul 24 15:10:52.376921 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:52.376584    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff4c5089-e2d7-4345-9761-9d96a4c41a0f","Type":"ContainerStarted","Data":"27b83458367c7cb8d3524dd64ff6f4248f60a0450697d17143f3931aba4796f1"}
Jul 24 15:10:52.376921 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:52.376607    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff4c5089-e2d7-4345-9761-9d96a4c41a0f","Type":"ContainerStarted","Data":"0aa08d08504881ab069daba8f3aeb1fe3a76733866a867a76017afe3aa45a449"}
Jul 24 15:10:52.410584 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:52.410527    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.218313805 podStartE2EDuration="6.410510109s" totalImagesPullingTime="4.164982177s" totalInitContainerRuntime="27.214127ms" isStatefulPod=false podCreationTimestamp="2026-07-24 15:10:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:10:52.409079543 +0000 UTC m=+145.221431724" watchObservedRunningTime="2026-07-24 15:10:52.410510109 +0000 UTC m=+145.222862332"
Jul 24 15:10:54.359298 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:54.359264    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-67c4bd47f4-pt275"
Jul 24 15:10:54.746841 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:54.746727    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55fd4cb8d5-29wgz"
Jul 24 15:10:54.746841 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:54.746801    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55fd4cb8d5-29wgz"
Jul 24 15:10:54.751723 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:54.751698    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-55fd4cb8d5-29wgz"
Jul 24 15:10:55.393718 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:55.393686    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-55fd4cb8d5-29wgz"
Jul 24 15:10:55.451264 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:55.451233    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-696dc74c75-r8lwz"]
Jul 24 15:10:56.520097 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:10:56.520058    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:11:04.546358 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:04.546323    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-s5nzq_ca98516e-79dc-4972-86de-8ff390bc0de2/dns-node-resolver/0.log"
Jul 24 15:11:04.747274 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:04.747234    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-55db6d66b7-2bssd_7eb671c8-eae8-409d-943a-c8f76ddede9f/router/0.log"
Jul 24 15:11:04.946469 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:04.946378    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bks2n_53659e71-d188-4315-81ab-7b53e4701a4f/serve-healthcheck-canary/0.log"
Jul 24 15:11:05.123089 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:11:05.123049    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-4vmc8" podUID="d4427fb4-a65e-4505-9329-88c2c1c4e812"
Jul 24 15:11:05.429099 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:05.429070    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4vmc8"
Jul 24 15:11:10.028670 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:10.028630    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4427fb4-a65e-4505-9329-88c2c1c4e812-metrics-tls\") pod \"dns-default-4vmc8\" (UID: \"d4427fb4-a65e-4505-9329-88c2c1c4e812\") " pod="openshift-dns/dns-default-4vmc8"
Jul 24 15:11:10.030949 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:10.030927    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4427fb4-a65e-4505-9329-88c2c1c4e812-metrics-tls\") pod \"dns-default-4vmc8\" (UID: \"d4427fb4-a65e-4505-9329-88c2c1c4e812\") " pod="openshift-dns/dns-default-4vmc8"
Jul 24 15:11:10.233833 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:10.233782    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kkfwv\""
Jul 24 15:11:10.239659 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:10.239631    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4vmc8"
Jul 24 15:11:10.785119 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:10.785091    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4vmc8"]
Jul 24 15:11:10.788251 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:11:10.788224    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4427fb4_a65e_4505_9329_88c2c1c4e812.slice/crio-528821bdae217047f1dc160e7cbfbd24437b3f7de55429968ff0503b3e74c32f WatchSource:0}: Error finding container 528821bdae217047f1dc160e7cbfbd24437b3f7de55429968ff0503b3e74c32f: Status 404 returned error can't find the container with id 528821bdae217047f1dc160e7cbfbd24437b3f7de55429968ff0503b3e74c32f
Jul 24 15:11:11.454722 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:11.454681    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4vmc8" event={"ID":"d4427fb4-a65e-4505-9329-88c2c1c4e812","Type":"ContainerStarted","Data":"528821bdae217047f1dc160e7cbfbd24437b3f7de55429968ff0503b3e74c32f"}
Jul 24 15:11:13.466726 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:13.466642    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4vmc8" event={"ID":"d4427fb4-a65e-4505-9329-88c2c1c4e812","Type":"ContainerStarted","Data":"9bda48797bda926aff6bb101d783d90c59560bc12b0f113c85dcf72668507f1a"}
Jul 24 15:11:13.467114 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:13.466733    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4vmc8" event={"ID":"d4427fb4-a65e-4505-9329-88c2c1c4e812","Type":"ContainerStarted","Data":"3d709187b78445475ca960cf2e465f401866df147d21d1c5fc2dd06485e4a7fa"}
Jul 24 15:11:13.467114 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:13.466808    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4vmc8"
Jul 24 15:11:13.488484 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:13.488425    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-4vmc8" podStartSLOduration=129.399693691 podStartE2EDuration="2m11.488402587s" totalImagesPullingTime="2.088708896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:09:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:11:13.487140178 +0000 UTC m=+166.299492358" watchObservedRunningTime="2026-07-24 15:11:13.488402587 +0000 UTC m=+166.300754771"
Jul 24 15:11:20.470462 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:20.470355    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-696dc74c75-r8lwz" podUID="0e959f71-f7c8-42b3-aad5-b613c6488b30" containerName="console" containerID="cri-o://35f9722f37b6efc827d65f37ecf90a9d2df451ae774a7c2aac9f5751e0445b60" gracePeriod=15
Jul 24 15:11:20.713284 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:20.713257    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-696dc74c75-r8lwz"
Jul 24 15:11:20.826626 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:20.826595    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e959f71-f7c8-42b3-aad5-b613c6488b30-service-ca\") pod \"0e959f71-f7c8-42b3-aad5-b613c6488b30\" (UID: \"0e959f71-f7c8-42b3-aad5-b613c6488b30\") "
Jul 24 15:11:20.826790 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:20.826631    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e959f71-f7c8-42b3-aad5-b613c6488b30-oauth-serving-cert\") pod \"0e959f71-f7c8-42b3-aad5-b613c6488b30\" (UID: \"0e959f71-f7c8-42b3-aad5-b613c6488b30\") "
Jul 24 15:11:20.826790 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:20.826681    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e959f71-f7c8-42b3-aad5-b613c6488b30-console-oauth-config\") pod \"0e959f71-f7c8-42b3-aad5-b613c6488b30\" (UID: \"0e959f71-f7c8-42b3-aad5-b613c6488b30\") "
Jul 24 15:11:20.826790 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:20.826707    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtmsc\" (UniqueName: \"kubernetes.io/projected/0e959f71-f7c8-42b3-aad5-b613c6488b30-kube-api-access-rtmsc\") pod \"0e959f71-f7c8-42b3-aad5-b613c6488b30\" (UID: \"0e959f71-f7c8-42b3-aad5-b613c6488b30\") "
Jul 24 15:11:20.826790 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:20.826768    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e959f71-f7c8-42b3-aad5-b613c6488b30-console-serving-cert\") pod \"0e959f71-f7c8-42b3-aad5-b613c6488b30\" (UID: \"0e959f71-f7c8-42b3-aad5-b613c6488b30\") "
Jul 24 15:11:20.827032 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:20.826799    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e959f71-f7c8-42b3-aad5-b613c6488b30-console-config\") pod \"0e959f71-f7c8-42b3-aad5-b613c6488b30\" (UID: \"0e959f71-f7c8-42b3-aad5-b613c6488b30\") "
Jul 24 15:11:20.827191 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:20.827152    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e959f71-f7c8-42b3-aad5-b613c6488b30-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0e959f71-f7c8-42b3-aad5-b613c6488b30" (UID: "0e959f71-f7c8-42b3-aad5-b613c6488b30"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:11:20.827267 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:20.827198    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e959f71-f7c8-42b3-aad5-b613c6488b30-service-ca" (OuterVolumeSpecName: "service-ca") pod "0e959f71-f7c8-42b3-aad5-b613c6488b30" (UID: "0e959f71-f7c8-42b3-aad5-b613c6488b30"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:11:20.827267 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:20.827250    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e959f71-f7c8-42b3-aad5-b613c6488b30-console-config" (OuterVolumeSpecName: "console-config") pod "0e959f71-f7c8-42b3-aad5-b613c6488b30" (UID: "0e959f71-f7c8-42b3-aad5-b613c6488b30"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:11:20.829096 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:20.829063    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e959f71-f7c8-42b3-aad5-b613c6488b30-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0e959f71-f7c8-42b3-aad5-b613c6488b30" (UID: "0e959f71-f7c8-42b3-aad5-b613c6488b30"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 15:11:20.829096 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:20.829085    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e959f71-f7c8-42b3-aad5-b613c6488b30-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0e959f71-f7c8-42b3-aad5-b613c6488b30" (UID: "0e959f71-f7c8-42b3-aad5-b613c6488b30"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 15:11:20.829334 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:20.829159    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e959f71-f7c8-42b3-aad5-b613c6488b30-kube-api-access-rtmsc" (OuterVolumeSpecName: "kube-api-access-rtmsc") pod "0e959f71-f7c8-42b3-aad5-b613c6488b30" (UID: "0e959f71-f7c8-42b3-aad5-b613c6488b30"). InnerVolumeSpecName "kube-api-access-rtmsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:11:20.927882 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:20.927839    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e959f71-f7c8-42b3-aad5-b613c6488b30-oauth-serving-cert\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:11:20.927882 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:20.927874    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e959f71-f7c8-42b3-aad5-b613c6488b30-console-oauth-config\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:11:20.927882 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:20.927889    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtmsc\" (UniqueName: \"kubernetes.io/projected/0e959f71-f7c8-42b3-aad5-b613c6488b30-kube-api-access-rtmsc\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:11:20.928112 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:20.927901    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e959f71-f7c8-42b3-aad5-b613c6488b30-console-serving-cert\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:11:20.928112 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:20.927914    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e959f71-f7c8-42b3-aad5-b613c6488b30-console-config\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:11:20.928112 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:20.927925    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e959f71-f7c8-42b3-aad5-b613c6488b30-service-ca\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:11:21.506218 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:21.506186    2579 generic.go:356] "Generic (PLEG): container finished" podID="0e959f71-f7c8-42b3-aad5-b613c6488b30" containerID="35f9722f37b6efc827d65f37ecf90a9d2df451ae774a7c2aac9f5751e0445b60" exitCode=0
Jul 24 15:11:21.506612 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:21.506231    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-696dc74c75-r8lwz"
Jul 24 15:11:21.506612 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:21.506267    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-696dc74c75-r8lwz" event={"ID":"0e959f71-f7c8-42b3-aad5-b613c6488b30","Type":"ContainerDied","Data":"35f9722f37b6efc827d65f37ecf90a9d2df451ae774a7c2aac9f5751e0445b60"}
Jul 24 15:11:21.506612 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:21.506315    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-696dc74c75-r8lwz" event={"ID":"0e959f71-f7c8-42b3-aad5-b613c6488b30","Type":"ContainerDied","Data":"548d0b9fc2bc4d503c26c9548f8befcff06ec39a061a4276025eb3f5238bb51f"}
Jul 24 15:11:21.506612 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:21.506336    2579 scope.go:117] "RemoveContainer" containerID="35f9722f37b6efc827d65f37ecf90a9d2df451ae774a7c2aac9f5751e0445b60"
Jul 24 15:11:21.515600 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:21.515583    2579 scope.go:117] "RemoveContainer" containerID="35f9722f37b6efc827d65f37ecf90a9d2df451ae774a7c2aac9f5751e0445b60"
Jul 24 15:11:21.515898 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:11:21.515878    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35f9722f37b6efc827d65f37ecf90a9d2df451ae774a7c2aac9f5751e0445b60\": container with ID starting with 35f9722f37b6efc827d65f37ecf90a9d2df451ae774a7c2aac9f5751e0445b60 not found: ID does not exist" containerID="35f9722f37b6efc827d65f37ecf90a9d2df451ae774a7c2aac9f5751e0445b60"
Jul 24 15:11:21.515976 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:21.515912    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35f9722f37b6efc827d65f37ecf90a9d2df451ae774a7c2aac9f5751e0445b60"} err="failed to get container status \"35f9722f37b6efc827d65f37ecf90a9d2df451ae774a7c2aac9f5751e0445b60\": rpc error: code = NotFound desc = could not find container \"35f9722f37b6efc827d65f37ecf90a9d2df451ae774a7c2aac9f5751e0445b60\": container with ID starting with 35f9722f37b6efc827d65f37ecf90a9d2df451ae774a7c2aac9f5751e0445b60 not found: ID does not exist"
Jul 24 15:11:21.539327 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:21.539297    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-696dc74c75-r8lwz"]
Jul 24 15:11:21.544690 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:21.544669    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-696dc74c75-r8lwz"]
Jul 24 15:11:21.716829 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:21.716791    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e959f71-f7c8-42b3-aad5-b613c6488b30" path="/var/lib/kubelet/pods/0e959f71-f7c8-42b3-aad5-b613c6488b30/volumes"
Jul 24 15:11:23.474566 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:23.474540    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4vmc8"
Jul 24 15:11:46.520073 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:46.520038    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:11:46.535716 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:46.535687    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:11:46.625217 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:11:46.625192    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 15:12:35.369932 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:12:35.369896    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55fd4cb8d5-29wgz"]
Jul 24 15:13:00.388412 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:13:00.388300    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-55fd4cb8d5-29wgz" podUID="09921c51-0843-4384-8eb2-7840e85e0b7d" containerName="console" containerID="cri-o://b6354b4d7f9e69cf80ef4ed79a56a92642b5a8b6d667ce453c03423e81a27f18" gracePeriod=15
Jul 24 15:13:00.627409 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:13:00.627382    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55fd4cb8d5-29wgz"
Jul 24 15:13:00.669562 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:13:00.669485    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09921c51-0843-4384-8eb2-7840e85e0b7d-trusted-ca-bundle\") pod \"09921c51-0843-4384-8eb2-7840e85e0b7d\" (UID: \"09921c51-0843-4384-8eb2-7840e85e0b7d\") "
Jul 24 15:13:00.669562 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:13:00.669539    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09921c51-0843-4384-8eb2-7840e85e0b7d-oauth-serving-cert\") pod \"09921c51-0843-4384-8eb2-7840e85e0b7d\" (UID: \"09921c51-0843-4384-8eb2-7840e85e0b7d\") "
Jul 24 15:13:00.669748 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:13:00.669595    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09921c51-0843-4384-8eb2-7840e85e0b7d-console-serving-cert\") pod \"09921c51-0843-4384-8eb2-7840e85e0b7d\" (UID: \"09921c51-0843-4384-8eb2-7840e85e0b7d\") "
Jul 24 15:13:00.669748 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:13:00.669622    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09921c51-0843-4384-8eb2-7840e85e0b7d-console-oauth-config\") pod \"09921c51-0843-4384-8eb2-7840e85e0b7d\" (UID: \"09921c51-0843-4384-8eb2-7840e85e0b7d\") "
Jul 24 15:13:00.669748 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:13:00.669641    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09921c51-0843-4384-8eb2-7840e85e0b7d-service-ca\") pod \"09921c51-0843-4384-8eb2-7840e85e0b7d\" (UID: \"09921c51-0843-4384-8eb2-7840e85e0b7d\") "
Jul 24 15:13:00.669748 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:13:00.669671    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09921c51-0843-4384-8eb2-7840e85e0b7d-console-config\") pod \"09921c51-0843-4384-8eb2-7840e85e0b7d\" (UID: \"09921c51-0843-4384-8eb2-7840e85e0b7d\") "
Jul 24 15:13:00.669748 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:13:00.669728    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpl6s\" (UniqueName: \"kubernetes.io/projected/09921c51-0843-4384-8eb2-7840e85e0b7d-kube-api-access-dpl6s\") pod \"09921c51-0843-4384-8eb2-7840e85e0b7d\" (UID: \"09921c51-0843-4384-8eb2-7840e85e0b7d\") "
Jul 24 15:13:00.670005 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:13:00.669881    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09921c51-0843-4384-8eb2-7840e85e0b7d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "09921c51-0843-4384-8eb2-7840e85e0b7d" (UID: "09921c51-0843-4384-8eb2-7840e85e0b7d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:13:00.670160 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:13:00.669935    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09921c51-0843-4384-8eb2-7840e85e0b7d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "09921c51-0843-4384-8eb2-7840e85e0b7d" (UID: "09921c51-0843-4384-8eb2-7840e85e0b7d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:13:00.670555 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:13:00.670525    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09921c51-0843-4384-8eb2-7840e85e0b7d-trusted-ca-bundle\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:13:00.670555 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:13:00.670530    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09921c51-0843-4384-8eb2-7840e85e0b7d-service-ca" (OuterVolumeSpecName: "service-ca") pod "09921c51-0843-4384-8eb2-7840e85e0b7d" (UID: "09921c51-0843-4384-8eb2-7840e85e0b7d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:13:00.670736 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:13:00.670672    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09921c51-0843-4384-8eb2-7840e85e0b7d-console-config" (OuterVolumeSpecName: "console-config") pod "09921c51-0843-4384-8eb2-7840e85e0b7d" (UID: "09921c51-0843-4384-8eb2-7840e85e0b7d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:13:00.672883 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:13:00.672849    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09921c51-0843-4384-8eb2-7840e85e0b7d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "09921c51-0843-4384-8eb2-7840e85e0b7d" (UID: "09921c51-0843-4384-8eb2-7840e85e0b7d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 15:13:00.676782 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:13:00.673566    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09921c51-0843-4384-8eb2-7840e85e0b7d-kube-api-access-dpl6s" (OuterVolumeSpecName: "kube-api-access-dpl6s") pod "09921c51-0843-4384-8eb2-7840e85e0b7d" (UID: "09921c51-0843-4384-8eb2-7840e85e0b7d"). InnerVolumeSpecName "kube-api-access-dpl6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:13:00.676782 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:13:00.676707    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09921c51-0843-4384-8eb2-7840e85e0b7d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "09921c51-0843-4384-8eb2-7840e85e0b7d" (UID: "09921c51-0843-4384-8eb2-7840e85e0b7d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 15:13:00.771534 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:13:00.771491    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpl6s\" (UniqueName: \"kubernetes.io/projected/09921c51-0843-4384-8eb2-7840e85e0b7d-kube-api-access-dpl6s\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:13:00.771534 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:13:00.771526    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09921c51-0843-4384-8eb2-7840e85e0b7d-oauth-serving-cert\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:13:00.771534 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:13:00.771537    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09921c51-0843-4384-8eb2-7840e85e0b7d-console-serving-cert\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:13:00.771534 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:13:00.771546    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09921c51-0843-4384-8eb2-7840e85e0b7d-console-oauth-config\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:13:00.771800 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:13:00.771555    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09921c51-0843-4384-8eb2-7840e85e0b7d-service-ca\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:13:00.771800 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:13:00.771565    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09921c51-0843-4384-8eb2-7840e85e0b7d-console-config\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:13:00.918407 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:13:00.918370    2579 generic.go:356] "Generic (PLEG): container finished" podID="09921c51-0843-4384-8eb2-7840e85e0b7d" containerID="b6354b4d7f9e69cf80ef4ed79a56a92642b5a8b6d667ce453c03423e81a27f18" exitCode=0
Jul 24 15:13:00.918596 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:13:00.918426    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55fd4cb8d5-29wgz"
Jul 24 15:13:00.918596 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:13:00.918462    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55fd4cb8d5-29wgz" event={"ID":"09921c51-0843-4384-8eb2-7840e85e0b7d","Type":"ContainerDied","Data":"b6354b4d7f9e69cf80ef4ed79a56a92642b5a8b6d667ce453c03423e81a27f18"}
Jul 24 15:13:00.918596 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:13:00.918522    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55fd4cb8d5-29wgz" event={"ID":"09921c51-0843-4384-8eb2-7840e85e0b7d","Type":"ContainerDied","Data":"7e6e3e8ad130c5d64cc11e4425937449fc6522636eb15598055c0102ffe33b37"}
Jul 24 15:13:00.918596 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:13:00.918550    2579 scope.go:117] "RemoveContainer" containerID="b6354b4d7f9e69cf80ef4ed79a56a92642b5a8b6d667ce453c03423e81a27f18"
Jul 24 15:13:00.927693 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:13:00.927675    2579 scope.go:117] "RemoveContainer" containerID="b6354b4d7f9e69cf80ef4ed79a56a92642b5a8b6d667ce453c03423e81a27f18"
Jul 24 15:13:00.928042 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:13:00.928023    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6354b4d7f9e69cf80ef4ed79a56a92642b5a8b6d667ce453c03423e81a27f18\": container with ID starting with b6354b4d7f9e69cf80ef4ed79a56a92642b5a8b6d667ce453c03423e81a27f18 not found: ID does not exist" containerID="b6354b4d7f9e69cf80ef4ed79a56a92642b5a8b6d667ce453c03423e81a27f18"
Jul 24 15:13:00.928079 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:13:00.928051    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6354b4d7f9e69cf80ef4ed79a56a92642b5a8b6d667ce453c03423e81a27f18"} err="failed to get container status \"b6354b4d7f9e69cf80ef4ed79a56a92642b5a8b6d667ce453c03423e81a27f18\": rpc error: code = NotFound desc = could not find container \"b6354b4d7f9e69cf80ef4ed79a56a92642b5a8b6d667ce453c03423e81a27f18\": container with ID starting with b6354b4d7f9e69cf80ef4ed79a56a92642b5a8b6d667ce453c03423e81a27f18 not found: ID does not exist"
Jul 24 15:13:00.940579 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:13:00.940557    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55fd4cb8d5-29wgz"]
Jul 24 15:13:00.944229 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:13:00.944208    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-55fd4cb8d5-29wgz"]
Jul 24 15:13:01.716140 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:13:01.716111    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09921c51-0843-4384-8eb2-7840e85e0b7d" path="/var/lib/kubelet/pods/09921c51-0843-4384-8eb2-7840e85e0b7d/volumes"
Jul 24 15:13:27.565860 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:13:27.565832    2579 kubelet.go:1648] "Image garbage collection succeeded"
Jul 24 15:14:27.424725 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:27.424645    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cff5b748-n772p"]
Jul 24 15:14:27.425141 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:27.425008    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e959f71-f7c8-42b3-aad5-b613c6488b30" containerName="console"
Jul 24 15:14:27.425141 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:27.425021    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e959f71-f7c8-42b3-aad5-b613c6488b30" containerName="console"
Jul 24 15:14:27.425141 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:27.425032    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09921c51-0843-4384-8eb2-7840e85e0b7d" containerName="console"
Jul 24 15:14:27.425141 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:27.425038    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="09921c51-0843-4384-8eb2-7840e85e0b7d" containerName="console"
Jul 24 15:14:27.425141 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:27.425086    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="09921c51-0843-4384-8eb2-7840e85e0b7d" containerName="console"
Jul 24 15:14:27.425141 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:27.425095    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e959f71-f7c8-42b3-aad5-b613c6488b30" containerName="console"
Jul 24 15:14:27.428187 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:27.428169    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-n772p"
Jul 24 15:14:27.430959 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:27.430939    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 24 15:14:27.431635 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:27.431612    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 24 15:14:27.431758 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:27.431695    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 24 15:14:27.431758 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:27.431700    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-vjk5s\""
Jul 24 15:14:27.432612 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:27.432597    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 24 15:14:27.453441 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:27.453417    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cff5b748-n772p"]
Jul 24 15:14:27.508043 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:27.508006    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5ffd42f8-a238-4b6f-b1fd-80cecfaac0f8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cff5b748-n772p\" (UID: \"5ffd42f8-a238-4b6f-b1fd-80cecfaac0f8\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-n772p"
Jul 24 15:14:27.508217 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:27.508065    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnjn8\" (UniqueName: \"kubernetes.io/projected/5ffd42f8-a238-4b6f-b1fd-80cecfaac0f8-kube-api-access-gnjn8\") pod \"opendatahub-operator-controller-manager-cff5b748-n772p\" (UID: \"5ffd42f8-a238-4b6f-b1fd-80cecfaac0f8\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-n772p"
Jul 24 15:14:27.508217 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:27.508089    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5ffd42f8-a238-4b6f-b1fd-80cecfaac0f8-manifests\") pod \"opendatahub-operator-controller-manager-cff5b748-n772p\" (UID: \"5ffd42f8-a238-4b6f-b1fd-80cecfaac0f8\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-n772p"
Jul 24 15:14:27.508217 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:27.508135    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5ffd42f8-a238-4b6f-b1fd-80cecfaac0f8-webhook-cert\") pod \"opendatahub-operator-controller-manager-cff5b748-n772p\" (UID: \"5ffd42f8-a238-4b6f-b1fd-80cecfaac0f8\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-n772p"
Jul 24 15:14:27.508217 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:27.508154    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ffd42f8-a238-4b6f-b1fd-80cecfaac0f8-tmp\") pod \"opendatahub-operator-controller-manager-cff5b748-n772p\" (UID: \"5ffd42f8-a238-4b6f-b1fd-80cecfaac0f8\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-n772p"
Jul 24 15:14:27.609009 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:27.608970    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5ffd42f8-a238-4b6f-b1fd-80cecfaac0f8-webhook-cert\") pod \"opendatahub-operator-controller-manager-cff5b748-n772p\" (UID: \"5ffd42f8-a238-4b6f-b1fd-80cecfaac0f8\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-n772p"
Jul 24 15:14:27.609009 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:27.609013    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ffd42f8-a238-4b6f-b1fd-80cecfaac0f8-tmp\") pod \"opendatahub-operator-controller-manager-cff5b748-n772p\" (UID: \"5ffd42f8-a238-4b6f-b1fd-80cecfaac0f8\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-n772p"
Jul 24 15:14:27.609253 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:27.609089    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5ffd42f8-a238-4b6f-b1fd-80cecfaac0f8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cff5b748-n772p\" (UID: \"5ffd42f8-a238-4b6f-b1fd-80cecfaac0f8\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-n772p"
Jul 24 15:14:27.609253 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:27.609125    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnjn8\" (UniqueName: \"kubernetes.io/projected/5ffd42f8-a238-4b6f-b1fd-80cecfaac0f8-kube-api-access-gnjn8\") pod \"opendatahub-operator-controller-manager-cff5b748-n772p\" (UID: \"5ffd42f8-a238-4b6f-b1fd-80cecfaac0f8\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-n772p"
Jul 24 15:14:27.609253 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:27.609161    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5ffd42f8-a238-4b6f-b1fd-80cecfaac0f8-manifests\") pod \"opendatahub-operator-controller-manager-cff5b748-n772p\" (UID: \"5ffd42f8-a238-4b6f-b1fd-80cecfaac0f8\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-n772p"
Jul 24 15:14:27.609559 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:27.609537    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5ffd42f8-a238-4b6f-b1fd-80cecfaac0f8-manifests\") pod \"opendatahub-operator-controller-manager-cff5b748-n772p\" (UID: \"5ffd42f8-a238-4b6f-b1fd-80cecfaac0f8\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-n772p"
Jul 24 15:14:27.611287 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:27.611257    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ffd42f8-a238-4b6f-b1fd-80cecfaac0f8-tmp\") pod \"opendatahub-operator-controller-manager-cff5b748-n772p\" (UID: \"5ffd42f8-a238-4b6f-b1fd-80cecfaac0f8\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-n772p"
Jul 24 15:14:27.612084 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:27.612068    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 24 15:14:27.621314 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:27.621294    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5ffd42f8-a238-4b6f-b1fd-80cecfaac0f8-webhook-cert\") pod \"opendatahub-operator-controller-manager-cff5b748-n772p\" (UID: \"5ffd42f8-a238-4b6f-b1fd-80cecfaac0f8\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-n772p"
Jul 24 15:14:27.621403 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:27.621356    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5ffd42f8-a238-4b6f-b1fd-80cecfaac0f8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cff5b748-n772p\" (UID: \"5ffd42f8-a238-4b6f-b1fd-80cecfaac0f8\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-n772p"
Jul 24 15:14:27.628910 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:27.628891    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 24 15:14:27.633000 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:27.632986    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 24 15:14:27.642316 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:27.642298    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnjn8\" (UniqueName: \"kubernetes.io/projected/5ffd42f8-a238-4b6f-b1fd-80cecfaac0f8-kube-api-access-gnjn8\") pod \"opendatahub-operator-controller-manager-cff5b748-n772p\" (UID: \"5ffd42f8-a238-4b6f-b1fd-80cecfaac0f8\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-n772p"
Jul 24 15:14:27.742072 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:27.741989    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-vjk5s\""
Jul 24 15:14:27.749930 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:27.749904    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-n772p"
Jul 24 15:14:27.900321 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:27.900298    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cff5b748-n772p"]
Jul 24 15:14:27.902910 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:14:27.902883    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ffd42f8_a238_4b6f_b1fd_80cecfaac0f8.slice/crio-6f9770261abfe0da5876b5e11931544c16b714ba739fb6354d9d852043fd0a35 WatchSource:0}: Error finding container 6f9770261abfe0da5876b5e11931544c16b714ba739fb6354d9d852043fd0a35: Status 404 returned error can't find the container with id 6f9770261abfe0da5876b5e11931544c16b714ba739fb6354d9d852043fd0a35
Jul 24 15:14:27.904595 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:27.904580    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 15:14:28.268251 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:28.268218    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-n772p" event={"ID":"5ffd42f8-a238-4b6f-b1fd-80cecfaac0f8","Type":"ContainerStarted","Data":"6f9770261abfe0da5876b5e11931544c16b714ba739fb6354d9d852043fd0a35"}
Jul 24 15:14:31.285633 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:31.285602    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-n772p" event={"ID":"5ffd42f8-a238-4b6f-b1fd-80cecfaac0f8","Type":"ContainerStarted","Data":"63004e612d05c580dd907ac277e39054d03aac3465685c05dd0f10f0690f27a3"}
Jul 24 15:14:32.291420 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:32.291381    2579 generic.go:356] "Generic (PLEG): container finished" podID="5ffd42f8-a238-4b6f-b1fd-80cecfaac0f8" containerID="63004e612d05c580dd907ac277e39054d03aac3465685c05dd0f10f0690f27a3" exitCode=0
Jul 24 15:14:32.291810 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:32.291424    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-n772p" event={"ID":"5ffd42f8-a238-4b6f-b1fd-80cecfaac0f8","Type":"ContainerDied","Data":"63004e612d05c580dd907ac277e39054d03aac3465685c05dd0f10f0690f27a3"}
Jul 24 15:14:33.297931 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:33.297889    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-n772p" event={"ID":"5ffd42f8-a238-4b6f-b1fd-80cecfaac0f8","Type":"ContainerStarted","Data":"61b7f1105ee5f3a33a701c81ec6dcb69388c91cf91ed73e7a38706184c471199"}
Jul 24 15:14:33.298367 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:33.297988    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-n772p"
Jul 24 15:14:33.321748 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:33.321692    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-n772p" podStartSLOduration=1.8094539790000002 podStartE2EDuration="6.321678659s" totalImagesPullingTime="3.447690923s" totalInitContainerRuntime="1.064533757s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:14:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:14:33.321399763 +0000 UTC m=+366.133751942" watchObservedRunningTime="2026-07-24 15:14:33.321678659 +0000 UTC m=+366.134030837"
Jul 24 15:14:44.305688 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:44.305646    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-n772p"
Jul 24 15:14:45.376800 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:45.376763    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-657dd8bf9c-lw5k2"]
Jul 24 15:14:45.386053 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:45.386027    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-657dd8bf9c-lw5k2"
Jul 24 15:14:45.388930 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:45.388906    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-rsdmg\""
Jul 24 15:14:45.389464 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:45.389448    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 24 15:14:45.390050 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:45.390027    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 24 15:14:45.390168 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:45.390056    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 24 15:14:45.390168 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:45.390076    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 24 15:14:45.390168 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:45.390143    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 24 15:14:45.398200 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:45.398178    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-657dd8bf9c-lw5k2"]
Jul 24 15:14:45.466851 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:45.466800    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47bce575-938f-44c6-9435-412e6a825e2c-cert\") pod \"lws-controller-manager-657dd8bf9c-lw5k2\" (UID: \"47bce575-938f-44c6-9435-412e6a825e2c\") " pod="openshift-lws-operator/lws-controller-manager-657dd8bf9c-lw5k2"
Jul 24 15:14:45.467036 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:45.466864    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/47bce575-938f-44c6-9435-412e6a825e2c-manager-config\") pod \"lws-controller-manager-657dd8bf9c-lw5k2\" (UID: \"47bce575-938f-44c6-9435-412e6a825e2c\") " pod="openshift-lws-operator/lws-controller-manager-657dd8bf9c-lw5k2"
Jul 24 15:14:45.467036 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:45.466887    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2xc4\" (UniqueName: \"kubernetes.io/projected/47bce575-938f-44c6-9435-412e6a825e2c-kube-api-access-c2xc4\") pod \"lws-controller-manager-657dd8bf9c-lw5k2\" (UID: \"47bce575-938f-44c6-9435-412e6a825e2c\") " pod="openshift-lws-operator/lws-controller-manager-657dd8bf9c-lw5k2"
Jul 24 15:14:45.467036 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:45.466927    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/47bce575-938f-44c6-9435-412e6a825e2c-metrics-cert\") pod \"lws-controller-manager-657dd8bf9c-lw5k2\" (UID: \"47bce575-938f-44c6-9435-412e6a825e2c\") " pod="openshift-lws-operator/lws-controller-manager-657dd8bf9c-lw5k2"
Jul 24 15:14:45.567648 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:45.567607    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/47bce575-938f-44c6-9435-412e6a825e2c-metrics-cert\") pod \"lws-controller-manager-657dd8bf9c-lw5k2\" (UID: \"47bce575-938f-44c6-9435-412e6a825e2c\") " pod="openshift-lws-operator/lws-controller-manager-657dd8bf9c-lw5k2"
Jul 24 15:14:45.567884 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:45.567669    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47bce575-938f-44c6-9435-412e6a825e2c-cert\") pod \"lws-controller-manager-657dd8bf9c-lw5k2\" (UID: \"47bce575-938f-44c6-9435-412e6a825e2c\") " pod="openshift-lws-operator/lws-controller-manager-657dd8bf9c-lw5k2"
Jul 24 15:14:45.567884 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:45.567729    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/47bce575-938f-44c6-9435-412e6a825e2c-manager-config\") pod \"lws-controller-manager-657dd8bf9c-lw5k2\" (UID: \"47bce575-938f-44c6-9435-412e6a825e2c\") " pod="openshift-lws-operator/lws-controller-manager-657dd8bf9c-lw5k2"
Jul 24 15:14:45.567884 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:45.567752    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2xc4\" (UniqueName: \"kubernetes.io/projected/47bce575-938f-44c6-9435-412e6a825e2c-kube-api-access-c2xc4\") pod \"lws-controller-manager-657dd8bf9c-lw5k2\" (UID: \"47bce575-938f-44c6-9435-412e6a825e2c\") " pod="openshift-lws-operator/lws-controller-manager-657dd8bf9c-lw5k2"
Jul 24 15:14:45.568394 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:45.568367    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/47bce575-938f-44c6-9435-412e6a825e2c-manager-config\") pod \"lws-controller-manager-657dd8bf9c-lw5k2\" (UID: \"47bce575-938f-44c6-9435-412e6a825e2c\") " pod="openshift-lws-operator/lws-controller-manager-657dd8bf9c-lw5k2"
Jul 24 15:14:45.570076 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:45.570057    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47bce575-938f-44c6-9435-412e6a825e2c-cert\") pod \"lws-controller-manager-657dd8bf9c-lw5k2\" (UID: \"47bce575-938f-44c6-9435-412e6a825e2c\") " pod="openshift-lws-operator/lws-controller-manager-657dd8bf9c-lw5k2"
Jul 24 15:14:45.570138 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:45.570083    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/47bce575-938f-44c6-9435-412e6a825e2c-metrics-cert\") pod \"lws-controller-manager-657dd8bf9c-lw5k2\" (UID: \"47bce575-938f-44c6-9435-412e6a825e2c\") " pod="openshift-lws-operator/lws-controller-manager-657dd8bf9c-lw5k2"
Jul 24 15:14:45.580025 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:45.580001    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2xc4\" (UniqueName: \"kubernetes.io/projected/47bce575-938f-44c6-9435-412e6a825e2c-kube-api-access-c2xc4\") pod \"lws-controller-manager-657dd8bf9c-lw5k2\" (UID: \"47bce575-938f-44c6-9435-412e6a825e2c\") " pod="openshift-lws-operator/lws-controller-manager-657dd8bf9c-lw5k2"
Jul 24 15:14:45.696194 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:45.696100    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-657dd8bf9c-lw5k2"
Jul 24 15:14:45.830701 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:45.830676    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-657dd8bf9c-lw5k2"]
Jul 24 15:14:45.832905 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:14:45.832877    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47bce575_938f_44c6_9435_412e6a825e2c.slice/crio-359a51069bb29ef111e72f0132858e12529b511989a903171ae3896fc49ca7bb WatchSource:0}: Error finding container 359a51069bb29ef111e72f0132858e12529b511989a903171ae3896fc49ca7bb: Status 404 returned error can't find the container with id 359a51069bb29ef111e72f0132858e12529b511989a903171ae3896fc49ca7bb
Jul 24 15:14:46.353713 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:46.353672    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-657dd8bf9c-lw5k2" event={"ID":"47bce575-938f-44c6-9435-412e6a825e2c","Type":"ContainerStarted","Data":"359a51069bb29ef111e72f0132858e12529b511989a903171ae3896fc49ca7bb"}
Jul 24 15:14:48.012990 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:48.012950    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-587457bbc9-tz554"]
Jul 24 15:14:48.017948 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:48.017898    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-587457bbc9-tz554"
Jul 24 15:14:48.025295 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:48.025089    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 24 15:14:48.025295 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:48.025235    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 24 15:14:48.025500 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:48.025479    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-bwhw2\""
Jul 24 15:14:48.043354 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:48.043293    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-587457bbc9-tz554"]
Jul 24 15:14:48.086900 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:48.086798    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwj4w\" (UniqueName: \"kubernetes.io/projected/b25451af-04f1-4aeb-88ad-ffc1d422c54d-kube-api-access-xwj4w\") pod \"kube-auth-proxy-587457bbc9-tz554\" (UID: \"b25451af-04f1-4aeb-88ad-ffc1d422c54d\") " pod="openshift-ingress/kube-auth-proxy-587457bbc9-tz554"
Jul 24 15:14:48.086900 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:48.086865    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b25451af-04f1-4aeb-88ad-ffc1d422c54d-tls-certs\") pod \"kube-auth-proxy-587457bbc9-tz554\" (UID: \"b25451af-04f1-4aeb-88ad-ffc1d422c54d\") " pod="openshift-ingress/kube-auth-proxy-587457bbc9-tz554"
Jul 24 15:14:48.087140 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:48.086964    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b25451af-04f1-4aeb-88ad-ffc1d422c54d-tmp\") pod \"kube-auth-proxy-587457bbc9-tz554\" (UID: \"b25451af-04f1-4aeb-88ad-ffc1d422c54d\") " pod="openshift-ingress/kube-auth-proxy-587457bbc9-tz554"
Jul 24 15:14:48.188159 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:48.188100    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwj4w\" (UniqueName: \"kubernetes.io/projected/b25451af-04f1-4aeb-88ad-ffc1d422c54d-kube-api-access-xwj4w\") pod \"kube-auth-proxy-587457bbc9-tz554\" (UID: \"b25451af-04f1-4aeb-88ad-ffc1d422c54d\") " pod="openshift-ingress/kube-auth-proxy-587457bbc9-tz554"
Jul 24 15:14:48.188353 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:48.188174    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b25451af-04f1-4aeb-88ad-ffc1d422c54d-tls-certs\") pod \"kube-auth-proxy-587457bbc9-tz554\" (UID: \"b25451af-04f1-4aeb-88ad-ffc1d422c54d\") " pod="openshift-ingress/kube-auth-proxy-587457bbc9-tz554"
Jul 24 15:14:48.188353 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:48.188274    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b25451af-04f1-4aeb-88ad-ffc1d422c54d-tmp\") pod \"kube-auth-proxy-587457bbc9-tz554\" (UID: \"b25451af-04f1-4aeb-88ad-ffc1d422c54d\") " pod="openshift-ingress/kube-auth-proxy-587457bbc9-tz554"
Jul 24 15:14:48.189394 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:14:48.189092    2579 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 24 15:14:48.189394 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:14:48.189165    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b25451af-04f1-4aeb-88ad-ffc1d422c54d-tls-certs podName:b25451af-04f1-4aeb-88ad-ffc1d422c54d nodeName:}" failed. No retries permitted until 2026-07-24 15:14:48.689145033 +0000 UTC m=+381.501497194 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b25451af-04f1-4aeb-88ad-ffc1d422c54d-tls-certs") pod "kube-auth-proxy-587457bbc9-tz554" (UID: "b25451af-04f1-4aeb-88ad-ffc1d422c54d") : secret "kube-auth-proxy-tls" not found
Jul 24 15:14:48.193716 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:48.193655    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b25451af-04f1-4aeb-88ad-ffc1d422c54d-tmp\") pod \"kube-auth-proxy-587457bbc9-tz554\" (UID: \"b25451af-04f1-4aeb-88ad-ffc1d422c54d\") " pod="openshift-ingress/kube-auth-proxy-587457bbc9-tz554"
Jul 24 15:14:48.223829 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:48.223792    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwj4w\" (UniqueName: \"kubernetes.io/projected/b25451af-04f1-4aeb-88ad-ffc1d422c54d-kube-api-access-xwj4w\") pod \"kube-auth-proxy-587457bbc9-tz554\" (UID: \"b25451af-04f1-4aeb-88ad-ffc1d422c54d\") " pod="openshift-ingress/kube-auth-proxy-587457bbc9-tz554"
Jul 24 15:14:48.366373 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:48.366343    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-657dd8bf9c-lw5k2" event={"ID":"47bce575-938f-44c6-9435-412e6a825e2c","Type":"ContainerStarted","Data":"ad60465322c6c13919564aa9e4dff262dc1bf72ffa8b85f56625cc44243d049b"}
Jul 24 15:14:48.693466 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:48.693380    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b25451af-04f1-4aeb-88ad-ffc1d422c54d-tls-certs\") pod \"kube-auth-proxy-587457bbc9-tz554\" (UID: \"b25451af-04f1-4aeb-88ad-ffc1d422c54d\") " pod="openshift-ingress/kube-auth-proxy-587457bbc9-tz554"
Jul 24 15:14:48.693619 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:14:48.693531    2579 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 24 15:14:48.693619 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:14:48.693593    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b25451af-04f1-4aeb-88ad-ffc1d422c54d-tls-certs podName:b25451af-04f1-4aeb-88ad-ffc1d422c54d nodeName:}" failed. No retries permitted until 2026-07-24 15:14:49.693577844 +0000 UTC m=+382.505930001 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b25451af-04f1-4aeb-88ad-ffc1d422c54d-tls-certs") pod "kube-auth-proxy-587457bbc9-tz554" (UID: "b25451af-04f1-4aeb-88ad-ffc1d422c54d") : secret "kube-auth-proxy-tls" not found
Jul 24 15:14:49.372248 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:49.372213    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-657dd8bf9c-lw5k2"
Jul 24 15:14:49.406409 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:49.406350    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-657dd8bf9c-lw5k2" podStartSLOduration=1.951692217 podStartE2EDuration="4.406330005s" totalImagesPullingTime="2.454637788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:14:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:14:49.402073775 +0000 UTC m=+382.214425957" watchObservedRunningTime="2026-07-24 15:14:49.406330005 +0000 UTC m=+382.218682185"
Jul 24 15:14:49.702124 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:49.702040    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b25451af-04f1-4aeb-88ad-ffc1d422c54d-tls-certs\") pod \"kube-auth-proxy-587457bbc9-tz554\" (UID: \"b25451af-04f1-4aeb-88ad-ffc1d422c54d\") " pod="openshift-ingress/kube-auth-proxy-587457bbc9-tz554"
Jul 24 15:14:49.704358 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:49.704329    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b25451af-04f1-4aeb-88ad-ffc1d422c54d-tls-certs\") pod \"kube-auth-proxy-587457bbc9-tz554\" (UID: \"b25451af-04f1-4aeb-88ad-ffc1d422c54d\") " pod="openshift-ingress/kube-auth-proxy-587457bbc9-tz554"
Jul 24 15:14:49.831200 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:49.831165    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-587457bbc9-tz554"
Jul 24 15:14:49.982901 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:49.982794    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-587457bbc9-tz554"]
Jul 24 15:14:49.986007 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:14:49.985970    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb25451af_04f1_4aeb_88ad_ffc1d422c54d.slice/crio-db636318bd411638f52ea199634f62c87cf048c382624008e970b1b57294837c WatchSource:0}: Error finding container db636318bd411638f52ea199634f62c87cf048c382624008e970b1b57294837c: Status 404 returned error can't find the container with id db636318bd411638f52ea199634f62c87cf048c382624008e970b1b57294837c
Jul 24 15:14:50.379129 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:50.379088    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-587457bbc9-tz554" event={"ID":"b25451af-04f1-4aeb-88ad-ffc1d422c54d","Type":"ContainerStarted","Data":"db636318bd411638f52ea199634f62c87cf048c382624008e970b1b57294837c"}
Jul 24 15:14:53.395882 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:53.395844    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-587457bbc9-tz554" event={"ID":"b25451af-04f1-4aeb-88ad-ffc1d422c54d","Type":"ContainerStarted","Data":"1bb8ab6c9835ea2e9c8015c73656f317b3b1615c507bfea2c9283705705ceb4d"}
Jul 24 15:14:53.419519 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:14:53.419468    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-587457bbc9-tz554" podStartSLOduration=3.812227209 podStartE2EDuration="6.419452149s" totalImagesPullingTime="2.60722494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:14:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:14:53.417192487 +0000 UTC m=+386.229544671" watchObservedRunningTime="2026-07-24 15:14:53.419452149 +0000 UTC m=+386.231804327"
Jul 24 15:15:00.381635 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:15:00.381601    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-657dd8bf9c-lw5k2"
Jul 24 15:16:51.430100 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:16:51.430059    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8n5rk"]
Jul 24 15:16:51.433304 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:16:51.433286    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8n5rk"
Jul 24 15:16:51.435729 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:16:51.435704    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 24 15:16:51.437073 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:16:51.437054    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 24 15:16:51.437298 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:16:51.437268    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-56vlm\""
Jul 24 15:16:51.455657 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:16:51.455626    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8n5rk"]
Jul 24 15:16:51.565688 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:16:51.565654    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtwk7\" (UniqueName: \"kubernetes.io/projected/db5d42dd-9cb5-4c0e-9b3e-cea29bda67b8-kube-api-access-jtwk7\") pod \"kuadrant-operator-controller-manager-55467d9f7-8n5rk\" (UID: \"db5d42dd-9cb5-4c0e-9b3e-cea29bda67b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8n5rk"
Jul 24 15:16:51.565874 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:16:51.565718    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/db5d42dd-9cb5-4c0e-9b3e-cea29bda67b8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-8n5rk\" (UID: \"db5d42dd-9cb5-4c0e-9b3e-cea29bda67b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8n5rk"
Jul 24 15:16:51.574001 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:16:51.573973    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8n5rk"]
Jul 24 15:16:51.574283 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:16:51.574259    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-jtwk7], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8n5rk" podUID="db5d42dd-9cb5-4c0e-9b3e-cea29bda67b8"
Jul 24 15:16:51.624120 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:16:51.624090    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8n5rk"]
Jul 24 15:16:51.637367 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:16:51.637339    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8n5rk"]
Jul 24 15:16:51.667051 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:16:51.667015    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/db5d42dd-9cb5-4c0e-9b3e-cea29bda67b8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-8n5rk\" (UID: \"db5d42dd-9cb5-4c0e-9b3e-cea29bda67b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8n5rk"
Jul 24 15:16:51.667205 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:16:51.667119    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtwk7\" (UniqueName: \"kubernetes.io/projected/db5d42dd-9cb5-4c0e-9b3e-cea29bda67b8-kube-api-access-jtwk7\") pod \"kuadrant-operator-controller-manager-55467d9f7-8n5rk\" (UID: \"db5d42dd-9cb5-4c0e-9b3e-cea29bda67b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8n5rk"
Jul 24 15:16:51.667427 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:16:51.667407    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/db5d42dd-9cb5-4c0e-9b3e-cea29bda67b8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-8n5rk\" (UID: \"db5d42dd-9cb5-4c0e-9b3e-cea29bda67b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8n5rk"
Jul 24 15:16:51.673173 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:16:51.673152    2579 projected.go:196] Error preparing data for projected volume kube-api-access-jtwk7 for pod kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8n5rk: failed to fetch token: serviceaccounts "kuadrant-operator-controller-manager" is forbidden: User "system:node:ip-10-0-140-160.ec2.internal" cannot create resource "serviceaccounts/token" in API group "" in the namespace "kuadrant-system": no relationship found between node 'ip-10-0-140-160.ec2.internal' and this object
Jul 24 15:16:51.673324 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:16:51.673227    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db5d42dd-9cb5-4c0e-9b3e-cea29bda67b8-kube-api-access-jtwk7 podName:db5d42dd-9cb5-4c0e-9b3e-cea29bda67b8 nodeName:}" failed. No retries permitted until 2026-07-24 15:16:52.173206941 +0000 UTC m=+504.985559098 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jtwk7" (UniqueName: "kubernetes.io/projected/db5d42dd-9cb5-4c0e-9b3e-cea29bda67b8-kube-api-access-jtwk7") pod "kuadrant-operator-controller-manager-55467d9f7-8n5rk" (UID: "db5d42dd-9cb5-4c0e-9b3e-cea29bda67b8") : failed to fetch token: serviceaccounts "kuadrant-operator-controller-manager" is forbidden: User "system:node:ip-10-0-140-160.ec2.internal" cannot create resource "serviceaccounts/token" in API group "" in the namespace "kuadrant-system": no relationship found between node 'ip-10-0-140-160.ec2.internal' and this object
Jul 24 15:16:51.931302 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:16:51.931271    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8n5rk"
Jul 24 15:16:51.937486 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:16:51.937464    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8n5rk"
Jul 24 15:16:51.945967 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:16:51.945936    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-8n5rk\" is forbidden: User \"system:node:ip-10-0-140-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-160.ec2.internal' and this object" podUID="db5d42dd-9cb5-4c0e-9b3e-cea29bda67b8" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8n5rk"
Jul 24 15:16:52.069655 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:16:52.069619    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/db5d42dd-9cb5-4c0e-9b3e-cea29bda67b8-extensions-socket-volume\") pod \"db5d42dd-9cb5-4c0e-9b3e-cea29bda67b8\" (UID: \"db5d42dd-9cb5-4c0e-9b3e-cea29bda67b8\") "
Jul 24 15:16:52.069903 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:16:52.069888    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtwk7\" (UniqueName: \"kubernetes.io/projected/db5d42dd-9cb5-4c0e-9b3e-cea29bda67b8-kube-api-access-jtwk7\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:16:52.069958 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:16:52.069883    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db5d42dd-9cb5-4c0e-9b3e-cea29bda67b8-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "db5d42dd-9cb5-4c0e-9b3e-cea29bda67b8" (UID: "db5d42dd-9cb5-4c0e-9b3e-cea29bda67b8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:16:52.171232 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:16:52.171201    2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/db5d42dd-9cb5-4c0e-9b3e-cea29bda67b8-extensions-socket-volume\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:16:52.935136 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:16:52.935102    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8n5rk"
Jul 24 15:16:52.939958 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:16:52.939927    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-8n5rk\" is forbidden: User \"system:node:ip-10-0-140-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-160.ec2.internal' and this object" podUID="db5d42dd-9cb5-4c0e-9b3e-cea29bda67b8" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8n5rk"
Jul 24 15:16:52.943428 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:16:52.943401    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-8n5rk\" is forbidden: User \"system:node:ip-10-0-140-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-160.ec2.internal' and this object" podUID="db5d42dd-9cb5-4c0e-9b3e-cea29bda67b8" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8n5rk"
Jul 24 15:16:53.716879 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:16:53.716847    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db5d42dd-9cb5-4c0e-9b3e-cea29bda67b8" path="/var/lib/kubelet/pods/db5d42dd-9cb5-4c0e-9b3e-cea29bda67b8/volumes"
Jul 24 15:17:30.714911 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:30.714833    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:17:30.718275 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:30.718258    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-dd79h"
Jul 24 15:17:30.720810 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:30.720787    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 24 15:17:30.721113 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:30.721097    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-h4mzg\""
Jul 24 15:17:30.721592 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:30.721577    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 24 15:17:30.721635 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:30.721593    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 24 15:17:30.736478 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:30.736449    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:17:30.764410 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:30.764377    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:17:30.820654 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:30.820619    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/20959482-70ef-4376-a727-48d94348946c-config-file\") pod \"limitador-limitador-54d6d988d5-dd79h\" (UID: \"20959482-70ef-4376-a727-48d94348946c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-dd79h"
Jul 24 15:17:30.820654 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:30.820653    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxlx8\" (UniqueName: \"kubernetes.io/projected/20959482-70ef-4376-a727-48d94348946c-kube-api-access-lxlx8\") pod \"limitador-limitador-54d6d988d5-dd79h\" (UID: \"20959482-70ef-4376-a727-48d94348946c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-dd79h"
Jul 24 15:17:30.921467 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:30.921427    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/20959482-70ef-4376-a727-48d94348946c-config-file\") pod \"limitador-limitador-54d6d988d5-dd79h\" (UID: \"20959482-70ef-4376-a727-48d94348946c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-dd79h"
Jul 24 15:17:30.921467 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:30.921469    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxlx8\" (UniqueName: \"kubernetes.io/projected/20959482-70ef-4376-a727-48d94348946c-kube-api-access-lxlx8\") pod \"limitador-limitador-54d6d988d5-dd79h\" (UID: \"20959482-70ef-4376-a727-48d94348946c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-dd79h"
Jul 24 15:17:30.922210 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:30.922184    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/20959482-70ef-4376-a727-48d94348946c-config-file\") pod \"limitador-limitador-54d6d988d5-dd79h\" (UID: \"20959482-70ef-4376-a727-48d94348946c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-dd79h"
Jul 24 15:17:30.930306 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:30.930282    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxlx8\" (UniqueName: \"kubernetes.io/projected/20959482-70ef-4376-a727-48d94348946c-kube-api-access-lxlx8\") pod \"limitador-limitador-54d6d988d5-dd79h\" (UID: \"20959482-70ef-4376-a727-48d94348946c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-dd79h"
Jul 24 15:17:31.028957 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:31.028920    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-dd79h"
Jul 24 15:17:31.068127 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:31.068093    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-nbg4d"]
Jul 24 15:17:31.071839 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:31.071802    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-nbg4d"
Jul 24 15:17:31.074656 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:31.074631    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vlnbj\""
Jul 24 15:17:31.086717 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:31.086689    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-nbg4d"]
Jul 24 15:17:31.123387 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:31.123359    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wvdf\" (UniqueName: \"kubernetes.io/projected/0eb45200-3d64-4d98-9b75-3909639da15d-kube-api-access-6wvdf\") pod \"authorino-58cd955675-nbg4d\" (UID: \"0eb45200-3d64-4d98-9b75-3909639da15d\") " pod="kuadrant-system/authorino-58cd955675-nbg4d"
Jul 24 15:17:31.174173 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:31.174148    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:17:31.176921 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:17:31.176890    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20959482_70ef_4376_a727_48d94348946c.slice/crio-17d34a68c25fadd86c96666937bbbaab9f40354b3df56d59395ee3834f13fccb WatchSource:0}: Error finding container 17d34a68c25fadd86c96666937bbbaab9f40354b3df56d59395ee3834f13fccb: Status 404 returned error can't find the container with id 17d34a68c25fadd86c96666937bbbaab9f40354b3df56d59395ee3834f13fccb
Jul 24 15:17:31.224528 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:31.224490    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wvdf\" (UniqueName: \"kubernetes.io/projected/0eb45200-3d64-4d98-9b75-3909639da15d-kube-api-access-6wvdf\") pod \"authorino-58cd955675-nbg4d\" (UID: \"0eb45200-3d64-4d98-9b75-3909639da15d\") " pod="kuadrant-system/authorino-58cd955675-nbg4d"
Jul 24 15:17:31.234757 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:31.234719    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wvdf\" (UniqueName: \"kubernetes.io/projected/0eb45200-3d64-4d98-9b75-3909639da15d-kube-api-access-6wvdf\") pod \"authorino-58cd955675-nbg4d\" (UID: \"0eb45200-3d64-4d98-9b75-3909639da15d\") " pod="kuadrant-system/authorino-58cd955675-nbg4d"
Jul 24 15:17:31.383277 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:31.383202    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-nbg4d"
Jul 24 15:17:31.511681 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:31.511653    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-nbg4d"]
Jul 24 15:17:31.512781 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:17:31.512756    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0eb45200_3d64_4d98_9b75_3909639da15d.slice/crio-2b24d74f6fca77ae96b8eb18f27d7cc862e1f41ee95946b67e6c39fbcb140c14 WatchSource:0}: Error finding container 2b24d74f6fca77ae96b8eb18f27d7cc862e1f41ee95946b67e6c39fbcb140c14: Status 404 returned error can't find the container with id 2b24d74f6fca77ae96b8eb18f27d7cc862e1f41ee95946b67e6c39fbcb140c14
Jul 24 15:17:32.106498 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:32.106458    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-nbg4d" event={"ID":"0eb45200-3d64-4d98-9b75-3909639da15d","Type":"ContainerStarted","Data":"2b24d74f6fca77ae96b8eb18f27d7cc862e1f41ee95946b67e6c39fbcb140c14"}
Jul 24 15:17:32.107929 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:32.107902    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-dd79h" event={"ID":"20959482-70ef-4376-a727-48d94348946c","Type":"ContainerStarted","Data":"17d34a68c25fadd86c96666937bbbaab9f40354b3df56d59395ee3834f13fccb"}
Jul 24 15:17:36.130512 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:36.130469    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-dd79h" event={"ID":"20959482-70ef-4376-a727-48d94348946c","Type":"ContainerStarted","Data":"d47a5a2c7ce9422325342d9a0655ddf0c2acc916a25b0457fdba0ad3853de582"}
Jul 24 15:17:36.131027 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:36.130549    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-dd79h"
Jul 24 15:17:36.131918 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:36.131896    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-nbg4d" event={"ID":"0eb45200-3d64-4d98-9b75-3909639da15d","Type":"ContainerStarted","Data":"8ba10d481961c7755e83e932ac6b5e281ee72f6d1d1812c54dc219e8bbd89ec7"}
Jul 24 15:17:36.152144 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:36.152098    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-dd79h" podStartSLOduration=1.84726384 podStartE2EDuration="6.152078245s" totalImagesPullingTime="4.304814405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:17:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:17:36.151410699 +0000 UTC m=+548.963762878" watchObservedRunningTime="2026-07-24 15:17:36.152078245 +0000 UTC m=+548.964430429"
Jul 24 15:17:36.172462 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:36.172432    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-nbg4d"]
Jul 24 15:17:36.190496 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:36.190445    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-nbg4d" podStartSLOduration=1.219810965 podStartE2EDuration="5.190429055s" totalImagesPullingTime="3.97061809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:17:36.18799198 +0000 UTC m=+549.000344159" watchObservedRunningTime="2026-07-24 15:17:36.190429055 +0000 UTC m=+549.002781234"
Jul 24 15:17:38.143428 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:38.143391    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-nbg4d" podUID="0eb45200-3d64-4d98-9b75-3909639da15d" containerName="authorino" containerID="cri-o://8ba10d481961c7755e83e932ac6b5e281ee72f6d1d1812c54dc219e8bbd89ec7" gracePeriod=30
Jul 24 15:17:38.390541 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:38.390514    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-nbg4d"
Jul 24 15:17:38.489385 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:38.489298    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wvdf\" (UniqueName: \"kubernetes.io/projected/0eb45200-3d64-4d98-9b75-3909639da15d-kube-api-access-6wvdf\") pod \"0eb45200-3d64-4d98-9b75-3909639da15d\" (UID: \"0eb45200-3d64-4d98-9b75-3909639da15d\") "
Jul 24 15:17:38.491415 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:38.491384    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eb45200-3d64-4d98-9b75-3909639da15d-kube-api-access-6wvdf" (OuterVolumeSpecName: "kube-api-access-6wvdf") pod "0eb45200-3d64-4d98-9b75-3909639da15d" (UID: "0eb45200-3d64-4d98-9b75-3909639da15d"). InnerVolumeSpecName "kube-api-access-6wvdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:17:38.589945 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:38.589906    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wvdf\" (UniqueName: \"kubernetes.io/projected/0eb45200-3d64-4d98-9b75-3909639da15d-kube-api-access-6wvdf\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:17:39.148888 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:39.148851    2579 generic.go:356] "Generic (PLEG): container finished" podID="0eb45200-3d64-4d98-9b75-3909639da15d" containerID="8ba10d481961c7755e83e932ac6b5e281ee72f6d1d1812c54dc219e8bbd89ec7" exitCode=0
Jul 24 15:17:39.149312 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:39.148895    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-nbg4d" event={"ID":"0eb45200-3d64-4d98-9b75-3909639da15d","Type":"ContainerDied","Data":"8ba10d481961c7755e83e932ac6b5e281ee72f6d1d1812c54dc219e8bbd89ec7"}
Jul 24 15:17:39.149312 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:39.148901    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-nbg4d"
Jul 24 15:17:39.149312 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:39.148928    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-nbg4d" event={"ID":"0eb45200-3d64-4d98-9b75-3909639da15d","Type":"ContainerDied","Data":"2b24d74f6fca77ae96b8eb18f27d7cc862e1f41ee95946b67e6c39fbcb140c14"}
Jul 24 15:17:39.149312 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:39.148944    2579 scope.go:117] "RemoveContainer" containerID="8ba10d481961c7755e83e932ac6b5e281ee72f6d1d1812c54dc219e8bbd89ec7"
Jul 24 15:17:39.157844 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:39.157804    2579 scope.go:117] "RemoveContainer" containerID="8ba10d481961c7755e83e932ac6b5e281ee72f6d1d1812c54dc219e8bbd89ec7"
Jul 24 15:17:39.158103 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:17:39.158082    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ba10d481961c7755e83e932ac6b5e281ee72f6d1d1812c54dc219e8bbd89ec7\": container with ID starting with 8ba10d481961c7755e83e932ac6b5e281ee72f6d1d1812c54dc219e8bbd89ec7 not found: ID does not exist" containerID="8ba10d481961c7755e83e932ac6b5e281ee72f6d1d1812c54dc219e8bbd89ec7"
Jul 24 15:17:39.158171 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:39.158111    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ba10d481961c7755e83e932ac6b5e281ee72f6d1d1812c54dc219e8bbd89ec7"} err="failed to get container status \"8ba10d481961c7755e83e932ac6b5e281ee72f6d1d1812c54dc219e8bbd89ec7\": rpc error: code = NotFound desc = could not find container \"8ba10d481961c7755e83e932ac6b5e281ee72f6d1d1812c54dc219e8bbd89ec7\": container with ID starting with 8ba10d481961c7755e83e932ac6b5e281ee72f6d1d1812c54dc219e8bbd89ec7 not found: ID does not exist"
Jul 24 15:17:39.170860 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:39.170835    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-nbg4d"]
Jul 24 15:17:39.178557 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:39.178532    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-nbg4d"]
Jul 24 15:17:39.717575 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:39.717539    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0eb45200-3d64-4d98-9b75-3909639da15d" path="/var/lib/kubelet/pods/0eb45200-3d64-4d98-9b75-3909639da15d/volumes"
Jul 24 15:17:47.141172 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:17:47.141139    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-dd79h"
Jul 24 15:18:05.091453 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:05.091407    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5965d688dc-h8lfw"]
Jul 24 15:18:05.092103 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:05.091944    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0eb45200-3d64-4d98-9b75-3909639da15d" containerName="authorino"
Jul 24 15:18:05.092103 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:05.091966    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0eb45200-3d64-4d98-9b75-3909639da15d" containerName="authorino"
Jul 24 15:18:05.092103 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:05.092086    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0eb45200-3d64-4d98-9b75-3909639da15d" containerName="authorino"
Jul 24 15:18:05.095186 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:05.095165    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5965d688dc-h8lfw"
Jul 24 15:18:05.097533 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:05.097514    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vlnbj\""
Jul 24 15:18:05.104806 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:05.104784    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5965d688dc-h8lfw"]
Jul 24 15:18:05.123720 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:05.123690    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh44f\" (UniqueName: \"kubernetes.io/projected/64c1bbb1-31f0-47d6-9607-c945289c48d3-kube-api-access-xh44f\") pod \"authorino-5965d688dc-h8lfw\" (UID: \"64c1bbb1-31f0-47d6-9607-c945289c48d3\") " pod="kuadrant-system/authorino-5965d688dc-h8lfw"
Jul 24 15:18:05.225078 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:05.225038    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh44f\" (UniqueName: \"kubernetes.io/projected/64c1bbb1-31f0-47d6-9607-c945289c48d3-kube-api-access-xh44f\") pod \"authorino-5965d688dc-h8lfw\" (UID: \"64c1bbb1-31f0-47d6-9607-c945289c48d3\") " pod="kuadrant-system/authorino-5965d688dc-h8lfw"
Jul 24 15:18:05.236889 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:05.236856    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh44f\" (UniqueName: \"kubernetes.io/projected/64c1bbb1-31f0-47d6-9607-c945289c48d3-kube-api-access-xh44f\") pod \"authorino-5965d688dc-h8lfw\" (UID: \"64c1bbb1-31f0-47d6-9607-c945289c48d3\") " pod="kuadrant-system/authorino-5965d688dc-h8lfw"
Jul 24 15:18:05.268647 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:05.268611    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5965d688dc-h8lfw"]
Jul 24 15:18:05.268904 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:05.268890    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5965d688dc-h8lfw"
Jul 24 15:18:05.306127 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:05.306087    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c48c595c9-sjvg5"]
Jul 24 15:18:05.310400 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:05.310372    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c48c595c9-sjvg5"
Jul 24 15:18:05.313063 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:05.313042    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 24 15:18:05.321445 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:05.321416    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c48c595c9-sjvg5"]
Jul 24 15:18:05.325746 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:05.325716    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5aa92f5f-3d33-4c43-9457-4a31917a9691-tls-cert\") pod \"authorino-c48c595c9-sjvg5\" (UID: \"5aa92f5f-3d33-4c43-9457-4a31917a9691\") " pod="kuadrant-system/authorino-c48c595c9-sjvg5"
Jul 24 15:18:05.325887 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:05.325760    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms5m2\" (UniqueName: \"kubernetes.io/projected/5aa92f5f-3d33-4c43-9457-4a31917a9691-kube-api-access-ms5m2\") pod \"authorino-c48c595c9-sjvg5\" (UID: \"5aa92f5f-3d33-4c43-9457-4a31917a9691\") " pod="kuadrant-system/authorino-c48c595c9-sjvg5"
Jul 24 15:18:05.419410 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:05.419368    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5965d688dc-h8lfw"]
Jul 24 15:18:05.422620 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:18:05.422587    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64c1bbb1_31f0_47d6_9607_c945289c48d3.slice/crio-77281f5cebe7631ca584be98d25ff857c5ff007beaf4b5f7edf356a3d2404e19 WatchSource:0}: Error finding container 77281f5cebe7631ca584be98d25ff857c5ff007beaf4b5f7edf356a3d2404e19: Status 404 returned error can't find the container with id 77281f5cebe7631ca584be98d25ff857c5ff007beaf4b5f7edf356a3d2404e19
Jul 24 15:18:05.426726 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:05.426703    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5aa92f5f-3d33-4c43-9457-4a31917a9691-tls-cert\") pod \"authorino-c48c595c9-sjvg5\" (UID: \"5aa92f5f-3d33-4c43-9457-4a31917a9691\") " pod="kuadrant-system/authorino-c48c595c9-sjvg5"
Jul 24 15:18:05.426854 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:05.426733    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms5m2\" (UniqueName: \"kubernetes.io/projected/5aa92f5f-3d33-4c43-9457-4a31917a9691-kube-api-access-ms5m2\") pod \"authorino-c48c595c9-sjvg5\" (UID: \"5aa92f5f-3d33-4c43-9457-4a31917a9691\") " pod="kuadrant-system/authorino-c48c595c9-sjvg5"
Jul 24 15:18:05.429378 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:05.429358    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5aa92f5f-3d33-4c43-9457-4a31917a9691-tls-cert\") pod \"authorino-c48c595c9-sjvg5\" (UID: \"5aa92f5f-3d33-4c43-9457-4a31917a9691\") " pod="kuadrant-system/authorino-c48c595c9-sjvg5"
Jul 24 15:18:05.437359 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:05.437337    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms5m2\" (UniqueName: \"kubernetes.io/projected/5aa92f5f-3d33-4c43-9457-4a31917a9691-kube-api-access-ms5m2\") pod \"authorino-c48c595c9-sjvg5\" (UID: \"5aa92f5f-3d33-4c43-9457-4a31917a9691\") " pod="kuadrant-system/authorino-c48c595c9-sjvg5"
Jul 24 15:18:05.622741 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:05.622648    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c48c595c9-sjvg5"
Jul 24 15:18:05.759128 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:05.759102    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c48c595c9-sjvg5"]
Jul 24 15:18:05.769054 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:18:05.769026    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5aa92f5f_3d33_4c43_9457_4a31917a9691.slice/crio-c89ff6993585f08473c040e9a05d162a8b86add17160f21f1a1a29b303542b1c WatchSource:0}: Error finding container c89ff6993585f08473c040e9a05d162a8b86add17160f21f1a1a29b303542b1c: Status 404 returned error can't find the container with id c89ff6993585f08473c040e9a05d162a8b86add17160f21f1a1a29b303542b1c
Jul 24 15:18:06.272251 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:06.272221    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5965d688dc-h8lfw" event={"ID":"64c1bbb1-31f0-47d6-9607-c945289c48d3","Type":"ContainerStarted","Data":"e6c0e4670a05ef82b616677afde119834cc0679142fcaf2dabe1a76ed92e201c"}
Jul 24 15:18:06.272653 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:06.272263    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5965d688dc-h8lfw" event={"ID":"64c1bbb1-31f0-47d6-9607-c945289c48d3","Type":"ContainerStarted","Data":"77281f5cebe7631ca584be98d25ff857c5ff007beaf4b5f7edf356a3d2404e19"}
Jul 24 15:18:06.272653 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:06.272265    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5965d688dc-h8lfw" podUID="64c1bbb1-31f0-47d6-9607-c945289c48d3" containerName="authorino" containerID="cri-o://e6c0e4670a05ef82b616677afde119834cc0679142fcaf2dabe1a76ed92e201c" gracePeriod=30
Jul 24 15:18:06.273541 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:06.273514    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c48c595c9-sjvg5" event={"ID":"5aa92f5f-3d33-4c43-9457-4a31917a9691","Type":"ContainerStarted","Data":"c89ff6993585f08473c040e9a05d162a8b86add17160f21f1a1a29b303542b1c"}
Jul 24 15:18:06.294171 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:06.294123    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5965d688dc-h8lfw" podStartSLOduration=0.98415129 podStartE2EDuration="1.294106101s" totalImagesPullingTime="309.954811ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:18:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:18:06.292703958 +0000 UTC m=+579.105056138" watchObservedRunningTime="2026-07-24 15:18:06.294106101 +0000 UTC m=+579.106458341"
Jul 24 15:18:06.518436 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:06.518404    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5965d688dc-h8lfw"
Jul 24 15:18:06.535941 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:06.535913    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xh44f\" (UniqueName: \"kubernetes.io/projected/64c1bbb1-31f0-47d6-9607-c945289c48d3-kube-api-access-xh44f\") pod \"64c1bbb1-31f0-47d6-9607-c945289c48d3\" (UID: \"64c1bbb1-31f0-47d6-9607-c945289c48d3\") "
Jul 24 15:18:06.538365 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:06.538331    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64c1bbb1-31f0-47d6-9607-c945289c48d3-kube-api-access-xh44f" (OuterVolumeSpecName: "kube-api-access-xh44f") pod "64c1bbb1-31f0-47d6-9607-c945289c48d3" (UID: "64c1bbb1-31f0-47d6-9607-c945289c48d3"). InnerVolumeSpecName "kube-api-access-xh44f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:18:06.637323 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:06.637284    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh44f\" (UniqueName: \"kubernetes.io/projected/64c1bbb1-31f0-47d6-9607-c945289c48d3-kube-api-access-xh44f\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:18:07.279811 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:07.279766    2579 generic.go:356] "Generic (PLEG): container finished" podID="64c1bbb1-31f0-47d6-9607-c945289c48d3" containerID="e6c0e4670a05ef82b616677afde119834cc0679142fcaf2dabe1a76ed92e201c" exitCode=0
Jul 24 15:18:07.279811 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:07.279831    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5965d688dc-h8lfw"
Jul 24 15:18:07.280387 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:07.279862    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5965d688dc-h8lfw" event={"ID":"64c1bbb1-31f0-47d6-9607-c945289c48d3","Type":"ContainerDied","Data":"e6c0e4670a05ef82b616677afde119834cc0679142fcaf2dabe1a76ed92e201c"}
Jul 24 15:18:07.280387 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:07.279907    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5965d688dc-h8lfw" event={"ID":"64c1bbb1-31f0-47d6-9607-c945289c48d3","Type":"ContainerDied","Data":"77281f5cebe7631ca584be98d25ff857c5ff007beaf4b5f7edf356a3d2404e19"}
Jul 24 15:18:07.280387 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:07.279931    2579 scope.go:117] "RemoveContainer" containerID="e6c0e4670a05ef82b616677afde119834cc0679142fcaf2dabe1a76ed92e201c"
Jul 24 15:18:07.281833 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:07.281798    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c48c595c9-sjvg5" event={"ID":"5aa92f5f-3d33-4c43-9457-4a31917a9691","Type":"ContainerStarted","Data":"5a51ed6a8016439cb1b67d8efd7e5c01b01237f8a58bc121b9ddeb2207d304ce"}
Jul 24 15:18:07.289072 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:07.289054    2579 scope.go:117] "RemoveContainer" containerID="e6c0e4670a05ef82b616677afde119834cc0679142fcaf2dabe1a76ed92e201c"
Jul 24 15:18:07.289319 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:18:07.289300    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6c0e4670a05ef82b616677afde119834cc0679142fcaf2dabe1a76ed92e201c\": container with ID starting with e6c0e4670a05ef82b616677afde119834cc0679142fcaf2dabe1a76ed92e201c not found: ID does not exist" containerID="e6c0e4670a05ef82b616677afde119834cc0679142fcaf2dabe1a76ed92e201c"
Jul 24 15:18:07.289367 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:07.289337    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6c0e4670a05ef82b616677afde119834cc0679142fcaf2dabe1a76ed92e201c"} err="failed to get container status \"e6c0e4670a05ef82b616677afde119834cc0679142fcaf2dabe1a76ed92e201c\": rpc error: code = NotFound desc = could not find container \"e6c0e4670a05ef82b616677afde119834cc0679142fcaf2dabe1a76ed92e201c\": container with ID starting with e6c0e4670a05ef82b616677afde119834cc0679142fcaf2dabe1a76ed92e201c not found: ID does not exist"
Jul 24 15:18:07.302095 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:07.302052    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c48c595c9-sjvg5" podStartSLOduration=1.859463578 podStartE2EDuration="2.30203712s" totalImagesPullingTime="442.573542ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:18:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:18:07.30025131 +0000 UTC m=+580.112603488" watchObservedRunningTime="2026-07-24 15:18:07.30203712 +0000 UTC m=+580.114389298"
Jul 24 15:18:07.315532 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:07.315503    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5965d688dc-h8lfw"]
Jul 24 15:18:07.321981 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:07.321944    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5965d688dc-h8lfw"]
Jul 24 15:18:07.721875 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:07.721777    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64c1bbb1-31f0-47d6-9607-c945289c48d3" path="/var/lib/kubelet/pods/64c1bbb1-31f0-47d6-9607-c945289c48d3/volumes"
Jul 24 15:18:48.084542 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:18:48.084501    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:19:18.125126 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:18.125085    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:19:25.031566 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:25.031531    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-89b5c"]
Jul 24 15:19:25.032179 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:25.031984    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64c1bbb1-31f0-47d6-9607-c945289c48d3" containerName="authorino"
Jul 24 15:19:25.032179 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:25.032005    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="64c1bbb1-31f0-47d6-9607-c945289c48d3" containerName="authorino"
Jul 24 15:19:25.032179 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:25.032136    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="64c1bbb1-31f0-47d6-9607-c945289c48d3" containerName="authorino"
Jul 24 15:19:25.035152 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:25.035129    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-89b5c"
Jul 24 15:19:25.037683 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:25.037663    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 24 15:19:25.038126 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:25.038108    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-t7tsm\""
Jul 24 15:19:25.038598 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:25.038581    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 24 15:19:25.038636 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:25.038581    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 24 15:19:25.057115 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:25.057084    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-89b5c"]
Jul 24 15:19:25.149568 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:25.149531    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b246993f-a343-4117-9ed9-7ef370e2e8df-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-89b5c\" (UID: \"b246993f-a343-4117-9ed9-7ef370e2e8df\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-89b5c"
Jul 24 15:19:25.149568 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:25.149572    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b246993f-a343-4117-9ed9-7ef370e2e8df-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-89b5c\" (UID: \"b246993f-a343-4117-9ed9-7ef370e2e8df\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-89b5c"
Jul 24 15:19:25.149794 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:25.149598    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b246993f-a343-4117-9ed9-7ef370e2e8df-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-89b5c\" (UID: \"b246993f-a343-4117-9ed9-7ef370e2e8df\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-89b5c"
Jul 24 15:19:25.149794 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:25.149708    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b246993f-a343-4117-9ed9-7ef370e2e8df-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-89b5c\" (UID: \"b246993f-a343-4117-9ed9-7ef370e2e8df\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-89b5c"
Jul 24 15:19:25.149794 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:25.149757    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b246993f-a343-4117-9ed9-7ef370e2e8df-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-89b5c\" (UID: \"b246993f-a343-4117-9ed9-7ef370e2e8df\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-89b5c"
Jul 24 15:19:25.149926 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:25.149840    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xts94\" (UniqueName: \"kubernetes.io/projected/b246993f-a343-4117-9ed9-7ef370e2e8df-kube-api-access-xts94\") pod \"e2e-distinct-simulated-kserve-59975d69f9-89b5c\" (UID: \"b246993f-a343-4117-9ed9-7ef370e2e8df\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-89b5c"
Jul 24 15:19:25.250498 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:25.250452    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b246993f-a343-4117-9ed9-7ef370e2e8df-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-89b5c\" (UID: \"b246993f-a343-4117-9ed9-7ef370e2e8df\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-89b5c"
Jul 24 15:19:25.250699 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:25.250517    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xts94\" (UniqueName: \"kubernetes.io/projected/b246993f-a343-4117-9ed9-7ef370e2e8df-kube-api-access-xts94\") pod \"e2e-distinct-simulated-kserve-59975d69f9-89b5c\" (UID: \"b246993f-a343-4117-9ed9-7ef370e2e8df\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-89b5c"
Jul 24 15:19:25.250699 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:25.250569    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b246993f-a343-4117-9ed9-7ef370e2e8df-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-89b5c\" (UID: \"b246993f-a343-4117-9ed9-7ef370e2e8df\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-89b5c"
Jul 24 15:19:25.250699 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:25.250590    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b246993f-a343-4117-9ed9-7ef370e2e8df-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-89b5c\" (UID: \"b246993f-a343-4117-9ed9-7ef370e2e8df\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-89b5c"
Jul 24 15:19:25.250699 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:25.250612    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b246993f-a343-4117-9ed9-7ef370e2e8df-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-89b5c\" (UID: \"b246993f-a343-4117-9ed9-7ef370e2e8df\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-89b5c"
Jul 24 15:19:25.250953 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:25.250762    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b246993f-a343-4117-9ed9-7ef370e2e8df-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-89b5c\" (UID: \"b246993f-a343-4117-9ed9-7ef370e2e8df\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-89b5c"
Jul 24 15:19:25.250953 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:25.250928    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b246993f-a343-4117-9ed9-7ef370e2e8df-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-89b5c\" (UID: \"b246993f-a343-4117-9ed9-7ef370e2e8df\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-89b5c"
Jul 24 15:19:25.251060 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:25.251009    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b246993f-a343-4117-9ed9-7ef370e2e8df-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-89b5c\" (UID: \"b246993f-a343-4117-9ed9-7ef370e2e8df\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-89b5c"
Jul 24 15:19:25.251125 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:25.251106    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b246993f-a343-4117-9ed9-7ef370e2e8df-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-89b5c\" (UID: \"b246993f-a343-4117-9ed9-7ef370e2e8df\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-89b5c"
Jul 24 15:19:25.252927 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:25.252904    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b246993f-a343-4117-9ed9-7ef370e2e8df-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-89b5c\" (UID: \"b246993f-a343-4117-9ed9-7ef370e2e8df\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-89b5c"
Jul 24 15:19:25.253142 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:25.253124    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b246993f-a343-4117-9ed9-7ef370e2e8df-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-89b5c\" (UID: \"b246993f-a343-4117-9ed9-7ef370e2e8df\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-89b5c"
Jul 24 15:19:25.260619 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:25.260596    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xts94\" (UniqueName: \"kubernetes.io/projected/b246993f-a343-4117-9ed9-7ef370e2e8df-kube-api-access-xts94\") pod \"e2e-distinct-simulated-kserve-59975d69f9-89b5c\" (UID: \"b246993f-a343-4117-9ed9-7ef370e2e8df\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-89b5c"
Jul 24 15:19:25.345556 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:25.345462    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-89b5c"
Jul 24 15:19:25.476908 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:25.476766    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-89b5c"]
Jul 24 15:19:25.479264 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:19:25.479238    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb246993f_a343_4117_9ed9_7ef370e2e8df.slice/crio-a8b89e0ee172be34282ad11a541f6db952af7322ebcf0d7fa87e4c259e5ed88a WatchSource:0}: Error finding container a8b89e0ee172be34282ad11a541f6db952af7322ebcf0d7fa87e4c259e5ed88a: Status 404 returned error can't find the container with id a8b89e0ee172be34282ad11a541f6db952af7322ebcf0d7fa87e4c259e5ed88a
Jul 24 15:19:25.646948 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:25.646855    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-89b5c" event={"ID":"b246993f-a343-4117-9ed9-7ef370e2e8df","Type":"ContainerStarted","Data":"a8b89e0ee172be34282ad11a541f6db952af7322ebcf0d7fa87e4c259e5ed88a"}
Jul 24 15:19:25.931857 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:25.931089    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:19:27.661239 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:27.661201    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-89b5c" event={"ID":"b246993f-a343-4117-9ed9-7ef370e2e8df","Type":"ContainerStarted","Data":"2b642f3944f0daaa8f92b7591d96b5ec36992fbcaa49f68a01c3e07befbdc277"}
Jul 24 15:19:27.682275 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:27.682221    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-89b5c" podStartSLOduration=2.573417247 podStartE2EDuration="3.682204297s" totalImagesPullingTime="1.10878705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:19:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:19:27.680590786 +0000 UTC m=+660.492942979" watchObservedRunningTime="2026-07-24 15:19:27.682204297 +0000 UTC m=+660.494556504"
Jul 24 15:19:35.030594 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:35.030557    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:19:35.345880 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:35.345746    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-89b5c"
Jul 24 15:19:35.346029 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:35.345880    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-89b5c"
Jul 24 15:19:35.359175 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:35.359147    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-89b5c"
Jul 24 15:19:35.709131 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:35.709048    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-89b5c"
Jul 24 15:19:39.633531 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:39.633489    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:19:51.532213 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:51.532181    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:19:58.033101 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:58.033067    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2"]
Jul 24 15:19:58.038281 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:58.038261    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2"
Jul 24 15:19:58.041070 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:58.041049    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 24 15:19:58.050921 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:58.050891    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2"]
Jul 24 15:19:58.133789 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:58.133749    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljn6q\" (UniqueName: \"kubernetes.io/projected/ad3fa403-b926-4005-bf62-95542d6c32c2-kube-api-access-ljn6q\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2\" (UID: \"ad3fa403-b926-4005-bf62-95542d6c32c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2"
Jul 24 15:19:58.133972 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:58.133844    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad3fa403-b926-4005-bf62-95542d6c32c2-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2\" (UID: \"ad3fa403-b926-4005-bf62-95542d6c32c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2"
Jul 24 15:19:58.133972 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:58.133873    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad3fa403-b926-4005-bf62-95542d6c32c2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2\" (UID: \"ad3fa403-b926-4005-bf62-95542d6c32c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2"
Jul 24 15:19:58.133972 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:58.133951    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad3fa403-b926-4005-bf62-95542d6c32c2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2\" (UID: \"ad3fa403-b926-4005-bf62-95542d6c32c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2"
Jul 24 15:19:58.134078 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:58.134019    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad3fa403-b926-4005-bf62-95542d6c32c2-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2\" (UID: \"ad3fa403-b926-4005-bf62-95542d6c32c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2"
Jul 24 15:19:58.134078 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:58.134063    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad3fa403-b926-4005-bf62-95542d6c32c2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2\" (UID: \"ad3fa403-b926-4005-bf62-95542d6c32c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2"
Jul 24 15:19:58.235554 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:58.235521    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad3fa403-b926-4005-bf62-95542d6c32c2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2\" (UID: \"ad3fa403-b926-4005-bf62-95542d6c32c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2"
Jul 24 15:19:58.235738 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:58.235576    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljn6q\" (UniqueName: \"kubernetes.io/projected/ad3fa403-b926-4005-bf62-95542d6c32c2-kube-api-access-ljn6q\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2\" (UID: \"ad3fa403-b926-4005-bf62-95542d6c32c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2"
Jul 24 15:19:58.235738 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:58.235620    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad3fa403-b926-4005-bf62-95542d6c32c2-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2\" (UID: \"ad3fa403-b926-4005-bf62-95542d6c32c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2"
Jul 24 15:19:58.235738 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:58.235638    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad3fa403-b926-4005-bf62-95542d6c32c2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2\" (UID: \"ad3fa403-b926-4005-bf62-95542d6c32c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2"
Jul 24 15:19:58.235738 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:58.235679    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad3fa403-b926-4005-bf62-95542d6c32c2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2\" (UID: \"ad3fa403-b926-4005-bf62-95542d6c32c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2"
Jul 24 15:19:58.235738 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:58.235708    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad3fa403-b926-4005-bf62-95542d6c32c2-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2\" (UID: \"ad3fa403-b926-4005-bf62-95542d6c32c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2"
Jul 24 15:19:58.236114 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:58.236035    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad3fa403-b926-4005-bf62-95542d6c32c2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2\" (UID: \"ad3fa403-b926-4005-bf62-95542d6c32c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2"
Jul 24 15:19:58.236114 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:58.236082    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad3fa403-b926-4005-bf62-95542d6c32c2-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2\" (UID: \"ad3fa403-b926-4005-bf62-95542d6c32c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2"
Jul 24 15:19:58.236210 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:58.236139    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad3fa403-b926-4005-bf62-95542d6c32c2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2\" (UID: \"ad3fa403-b926-4005-bf62-95542d6c32c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2"
Jul 24 15:19:58.237872 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:58.237854    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad3fa403-b926-4005-bf62-95542d6c32c2-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2\" (UID: \"ad3fa403-b926-4005-bf62-95542d6c32c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2"
Jul 24 15:19:58.238204 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:58.238188    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad3fa403-b926-4005-bf62-95542d6c32c2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2\" (UID: \"ad3fa403-b926-4005-bf62-95542d6c32c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2"
Jul 24 15:19:58.244023 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:58.244003    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljn6q\" (UniqueName: \"kubernetes.io/projected/ad3fa403-b926-4005-bf62-95542d6c32c2-kube-api-access-ljn6q\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2\" (UID: \"ad3fa403-b926-4005-bf62-95542d6c32c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2"
Jul 24 15:19:58.348709 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:58.348623    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2"
Jul 24 15:19:58.488658 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:58.488619    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2"]
Jul 24 15:19:58.490440 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:19:58.490408    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad3fa403_b926_4005_bf62_95542d6c32c2.slice/crio-df4525268e205084d16c0e05b74ef08700c19714b2c42cb7ff53f3d6cbc7a948 WatchSource:0}: Error finding container df4525268e205084d16c0e05b74ef08700c19714b2c42cb7ff53f3d6cbc7a948: Status 404 returned error can't find the container with id df4525268e205084d16c0e05b74ef08700c19714b2c42cb7ff53f3d6cbc7a948
Jul 24 15:19:58.492117 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:58.492095    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 15:19:58.803236 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:58.803197    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2" event={"ID":"ad3fa403-b926-4005-bf62-95542d6c32c2","Type":"ContainerStarted","Data":"c147f1fc8d5ae747973c4cc494818b91e99b891fbe2907fd57359efe532d8780"}
Jul 24 15:19:58.803432 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:58.803248    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2" event={"ID":"ad3fa403-b926-4005-bf62-95542d6c32c2","Type":"ContainerStarted","Data":"df4525268e205084d16c0e05b74ef08700c19714b2c42cb7ff53f3d6cbc7a948"}
Jul 24 15:19:58.825920 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:19:58.825862    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2" podStartSLOduration=0.59430316 podStartE2EDuration="825.842617ms" totalImagesPullingTime="231.539457ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:19:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:19:58.824584624 +0000 UTC m=+691.636936794" watchObservedRunningTime="2026-07-24 15:19:58.825842617 +0000 UTC m=+691.638194796"
Jul 24 15:20:01.430600 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:20:01.430556    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:20:08.349007 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:20:08.348966    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2"
Jul 24 15:20:08.349485 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:20:08.349025    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2"
Jul 24 15:20:08.360990 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:20:08.360963    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2"
Jul 24 15:20:08.861061 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:20:08.861032    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2"
Jul 24 15:20:59.095481 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:20:59.095447    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-55bbff9d46-fjk7k"]
Jul 24 15:20:59.098786 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:20:59.098765    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55bbff9d46-fjk7k"
Jul 24 15:20:59.108293 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:20:59.108270    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-55bbff9d46-fjk7k"]
Jul 24 15:20:59.196212 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:20:59.196153    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzvjp\" (UniqueName: \"kubernetes.io/projected/1f7f8458-c4d3-4f25-aa7c-b134af1c64cd-kube-api-access-vzvjp\") pod \"authorino-55bbff9d46-fjk7k\" (UID: \"1f7f8458-c4d3-4f25-aa7c-b134af1c64cd\") " pod="kuadrant-system/authorino-55bbff9d46-fjk7k"
Jul 24 15:20:59.196420 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:20:59.196306    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1f7f8458-c4d3-4f25-aa7c-b134af1c64cd-tls-cert\") pod \"authorino-55bbff9d46-fjk7k\" (UID: \"1f7f8458-c4d3-4f25-aa7c-b134af1c64cd\") " pod="kuadrant-system/authorino-55bbff9d46-fjk7k"
Jul 24 15:20:59.296801 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:20:59.296762    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1f7f8458-c4d3-4f25-aa7c-b134af1c64cd-tls-cert\") pod \"authorino-55bbff9d46-fjk7k\" (UID: \"1f7f8458-c4d3-4f25-aa7c-b134af1c64cd\") " pod="kuadrant-system/authorino-55bbff9d46-fjk7k"
Jul 24 15:20:59.296987 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:20:59.296876    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzvjp\" (UniqueName: \"kubernetes.io/projected/1f7f8458-c4d3-4f25-aa7c-b134af1c64cd-kube-api-access-vzvjp\") pod \"authorino-55bbff9d46-fjk7k\" (UID: \"1f7f8458-c4d3-4f25-aa7c-b134af1c64cd\") " pod="kuadrant-system/authorino-55bbff9d46-fjk7k"
Jul 24 15:20:59.299339 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:20:59.299313    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1f7f8458-c4d3-4f25-aa7c-b134af1c64cd-tls-cert\") pod \"authorino-55bbff9d46-fjk7k\" (UID: \"1f7f8458-c4d3-4f25-aa7c-b134af1c64cd\") " pod="kuadrant-system/authorino-55bbff9d46-fjk7k"
Jul 24 15:20:59.304741 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:20:59.304716    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzvjp\" (UniqueName: \"kubernetes.io/projected/1f7f8458-c4d3-4f25-aa7c-b134af1c64cd-kube-api-access-vzvjp\") pod \"authorino-55bbff9d46-fjk7k\" (UID: \"1f7f8458-c4d3-4f25-aa7c-b134af1c64cd\") " pod="kuadrant-system/authorino-55bbff9d46-fjk7k"
Jul 24 15:20:59.407534 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:20:59.407448    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55bbff9d46-fjk7k"
Jul 24 15:20:59.553078 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:20:59.553047    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-55bbff9d46-fjk7k"]
Jul 24 15:20:59.555743 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:20:59.555716    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f7f8458_c4d3_4f25_aa7c_b134af1c64cd.slice/crio-1fdd8f113bfeb0a6585ef58e8821a3313b049e2a0e582ef511c1303ad4b9bca7 WatchSource:0}: Error finding container 1fdd8f113bfeb0a6585ef58e8821a3313b049e2a0e582ef511c1303ad4b9bca7: Status 404 returned error can't find the container with id 1fdd8f113bfeb0a6585ef58e8821a3313b049e2a0e582ef511c1303ad4b9bca7
Jul 24 15:21:00.085644 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:21:00.085602    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55bbff9d46-fjk7k" event={"ID":"1f7f8458-c4d3-4f25-aa7c-b134af1c64cd","Type":"ContainerStarted","Data":"c04a495023b8a690dae8f4244a83d37702c774b1965232f7b819b5161c8207fd"}
Jul 24 15:21:00.085793 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:21:00.085658    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55bbff9d46-fjk7k" event={"ID":"1f7f8458-c4d3-4f25-aa7c-b134af1c64cd","Type":"ContainerStarted","Data":"1fdd8f113bfeb0a6585ef58e8821a3313b049e2a0e582ef511c1303ad4b9bca7"}
Jul 24 15:21:00.109034 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:21:00.108980    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-55bbff9d46-fjk7k" podStartSLOduration=0.687798697 podStartE2EDuration="1.108963789s" totalImagesPullingTime="421.165092ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:20:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:21:00.107442503 +0000 UTC m=+752.919794682" watchObservedRunningTime="2026-07-24 15:21:00.108963789 +0000 UTC m=+752.921316005"
Jul 24 15:21:00.168132 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:21:00.168093    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c48c595c9-sjvg5"]
Jul 24 15:21:00.168352 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:21:00.168327    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c48c595c9-sjvg5" podUID="5aa92f5f-3d33-4c43-9457-4a31917a9691" containerName="authorino" containerID="cri-o://5a51ed6a8016439cb1b67d8efd7e5c01b01237f8a58bc121b9ddeb2207d304ce" gracePeriod=30
Jul 24 15:21:00.425273 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:21:00.425241    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c48c595c9-sjvg5"
Jul 24 15:21:00.609453 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:21:00.609374    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5aa92f5f-3d33-4c43-9457-4a31917a9691-tls-cert\") pod \"5aa92f5f-3d33-4c43-9457-4a31917a9691\" (UID: \"5aa92f5f-3d33-4c43-9457-4a31917a9691\") "
Jul 24 15:21:00.609453 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:21:00.609447    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ms5m2\" (UniqueName: \"kubernetes.io/projected/5aa92f5f-3d33-4c43-9457-4a31917a9691-kube-api-access-ms5m2\") pod \"5aa92f5f-3d33-4c43-9457-4a31917a9691\" (UID: \"5aa92f5f-3d33-4c43-9457-4a31917a9691\") "
Jul 24 15:21:00.611480 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:21:00.611445    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aa92f5f-3d33-4c43-9457-4a31917a9691-kube-api-access-ms5m2" (OuterVolumeSpecName: "kube-api-access-ms5m2") pod "5aa92f5f-3d33-4c43-9457-4a31917a9691" (UID: "5aa92f5f-3d33-4c43-9457-4a31917a9691"). InnerVolumeSpecName "kube-api-access-ms5m2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:21:00.620306 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:21:00.620275    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5aa92f5f-3d33-4c43-9457-4a31917a9691-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "5aa92f5f-3d33-4c43-9457-4a31917a9691" (UID: "5aa92f5f-3d33-4c43-9457-4a31917a9691"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 15:21:00.710832 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:21:00.710785    2579 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5aa92f5f-3d33-4c43-9457-4a31917a9691-tls-cert\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:21:00.710832 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:21:00.710835    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms5m2\" (UniqueName: \"kubernetes.io/projected/5aa92f5f-3d33-4c43-9457-4a31917a9691-kube-api-access-ms5m2\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:21:01.092516 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:21:01.092477    2579 generic.go:356] "Generic (PLEG): container finished" podID="5aa92f5f-3d33-4c43-9457-4a31917a9691" containerID="5a51ed6a8016439cb1b67d8efd7e5c01b01237f8a58bc121b9ddeb2207d304ce" exitCode=0
Jul 24 15:21:01.092516 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:21:01.092517    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c48c595c9-sjvg5"
Jul 24 15:21:01.092777 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:21:01.092559    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c48c595c9-sjvg5" event={"ID":"5aa92f5f-3d33-4c43-9457-4a31917a9691","Type":"ContainerDied","Data":"5a51ed6a8016439cb1b67d8efd7e5c01b01237f8a58bc121b9ddeb2207d304ce"}
Jul 24 15:21:01.092777 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:21:01.092608    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c48c595c9-sjvg5" event={"ID":"5aa92f5f-3d33-4c43-9457-4a31917a9691","Type":"ContainerDied","Data":"c89ff6993585f08473c040e9a05d162a8b86add17160f21f1a1a29b303542b1c"}
Jul 24 15:21:01.092777 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:21:01.092631    2579 scope.go:117] "RemoveContainer" containerID="5a51ed6a8016439cb1b67d8efd7e5c01b01237f8a58bc121b9ddeb2207d304ce"
Jul 24 15:21:01.102998 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:21:01.102980    2579 scope.go:117] "RemoveContainer" containerID="5a51ed6a8016439cb1b67d8efd7e5c01b01237f8a58bc121b9ddeb2207d304ce"
Jul 24 15:21:01.103293 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:21:01.103272    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a51ed6a8016439cb1b67d8efd7e5c01b01237f8a58bc121b9ddeb2207d304ce\": container with ID starting with 5a51ed6a8016439cb1b67d8efd7e5c01b01237f8a58bc121b9ddeb2207d304ce not found: ID does not exist" containerID="5a51ed6a8016439cb1b67d8efd7e5c01b01237f8a58bc121b9ddeb2207d304ce"
Jul 24 15:21:01.103379 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:21:01.103300    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a51ed6a8016439cb1b67d8efd7e5c01b01237f8a58bc121b9ddeb2207d304ce"} err="failed to get container status \"5a51ed6a8016439cb1b67d8efd7e5c01b01237f8a58bc121b9ddeb2207d304ce\": rpc error: code = NotFound desc = could not find container \"5a51ed6a8016439cb1b67d8efd7e5c01b01237f8a58bc121b9ddeb2207d304ce\": container with ID starting with 5a51ed6a8016439cb1b67d8efd7e5c01b01237f8a58bc121b9ddeb2207d304ce not found: ID does not exist"
Jul 24 15:21:01.121127 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:21:01.121086    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c48c595c9-sjvg5"]
Jul 24 15:21:01.124897 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:21:01.124873    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c48c595c9-sjvg5"]
Jul 24 15:21:01.717835 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:21:01.717779    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5aa92f5f-3d33-4c43-9457-4a31917a9691" path="/var/lib/kubelet/pods/5aa92f5f-3d33-4c43-9457-4a31917a9691/volumes"
Jul 24 15:21:32.625036 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:21:32.624998    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:21:42.325720 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:21:42.325682    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:21:54.237484 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:21:54.237399    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:23:34.129540 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:23:34.129498    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:24:13.224170 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:24:13.224134    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:24:23.548897 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:24:23.548848    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:24:32.730896 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:24:32.730859    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:24:43.934357 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:24:43.934324    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:24:55.829506 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:24:55.829409    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:25:06.024330 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:25:06.024290    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:25:18.231057 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:25:18.231013    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:25:28.721480 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:25:28.721441    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:25:50.828022 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:25:50.827988    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:25:59.433408 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:25:59.433368    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:26:25.638714 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:26:25.638626    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:26:37.464619 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:26:37.464578    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:26:53.832407 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:26:53.832360    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:27:03.796654 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:27:03.796619    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:27:13.534548 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:27:13.534503    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:27:23.041468 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:27:23.041428    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:27:32.926112 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:27:32.926076    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:29:23.617195 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:29:23.617070    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:29:32.842788 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:29:32.842749    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:29:45.634255 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:29:45.634214    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:29:58.235831 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:29:58.235789    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:30:00.201333 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:30:00.201298    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748450-h2lwk"]
Jul 24 15:30:00.201897 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:30:00.201876    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5aa92f5f-3d33-4c43-9457-4a31917a9691" containerName="authorino"
Jul 24 15:30:00.201897 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:30:00.201898    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5aa92f5f-3d33-4c43-9457-4a31917a9691" containerName="authorino"
Jul 24 15:30:00.202044 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:30:00.202030    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5aa92f5f-3d33-4c43-9457-4a31917a9691" containerName="authorino"
Jul 24 15:30:00.205478 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:30:00.205455    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748450-h2lwk"
Jul 24 15:30:00.209328 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:30:00.209303    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r6lw7\""
Jul 24 15:30:00.216117 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:30:00.216092    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748450-h2lwk"
Jul 24 15:30:00.216597 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:30:00.216576    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748450-h2lwk"]
Jul 24 15:30:00.368558 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:30:00.368346    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748450-h2lwk"]
Jul 24 15:30:00.370955 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:30:00.370918    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 15:30:00.604503 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:30:00.604462    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748450-h2lwk" event={"ID":"288486bd-44c1-46ec-8b2f-de18bb743daa","Type":"ContainerStarted","Data":"5737c6334b21a719687e537a077a20b91fb6e3610590fc299cb769076158a38f"}
Jul 24 15:30:03.623648 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:30:03.623605    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748450-h2lwk" event={"ID":"288486bd-44c1-46ec-8b2f-de18bb743daa","Type":"ContainerStarted","Data":"2eee51b5dea35ff9841f21cd78fbd47cb432d7f6abd4ff76a1c243466f5f7ede"}
Jul 24 15:30:03.642045 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:30:03.641990    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748450-h2lwk" podStartSLOduration=0.726550646 podStartE2EDuration="3.641971127s" totalImagesPullingTime="2.915420481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:30:03.641279036 +0000 UTC m=+1296.453631215" watchObservedRunningTime="2026-07-24 15:30:03.641971127 +0000 UTC m=+1296.454323305"
Jul 24 15:30:04.630593 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:30:04.630556    2579 generic.go:356] "Generic (PLEG): container finished" podID="288486bd-44c1-46ec-8b2f-de18bb743daa" containerID="2eee51b5dea35ff9841f21cd78fbd47cb432d7f6abd4ff76a1c243466f5f7ede" exitCode=0
Jul 24 15:30:04.631027 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:30:04.630640    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748450-h2lwk" event={"ID":"288486bd-44c1-46ec-8b2f-de18bb743daa","Type":"ContainerDied","Data":"2eee51b5dea35ff9841f21cd78fbd47cb432d7f6abd4ff76a1c243466f5f7ede"}
Jul 24 15:30:05.769021 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:30:05.768994    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748450-h2lwk"
Jul 24 15:30:06.641224 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:30:06.641185    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748450-h2lwk" event={"ID":"288486bd-44c1-46ec-8b2f-de18bb743daa","Type":"ContainerDied","Data":"5737c6334b21a719687e537a077a20b91fb6e3610590fc299cb769076158a38f"}
Jul 24 15:30:06.641224 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:30:06.641222    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748450-h2lwk"
Jul 24 15:30:06.641489 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:30:06.641237    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5737c6334b21a719687e537a077a20b91fb6e3610590fc299cb769076158a38f"
Jul 24 15:30:08.032260 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:30:08.032220    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:30:20.736509 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:30:20.736473    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:30:28.460487 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:30:28.460443    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:30:38.838600 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:30:38.838559    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:30:50.261831 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:30:50.261726    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:30:58.926146 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:30:58.926107    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:31:09.428874 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:31:09.428842    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:31:32.638263 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:31:32.638223    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:31:42.429576 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:31:42.429530    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:31:43.208933 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:31:43.208890    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz"]
Jul 24 15:31:43.209289 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:31:43.209275    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="288486bd-44c1-46ec-8b2f-de18bb743daa" containerName="cleanup"
Jul 24 15:31:43.209333 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:31:43.209291    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="288486bd-44c1-46ec-8b2f-de18bb743daa" containerName="cleanup"
Jul 24 15:31:43.209367 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:31:43.209343    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="288486bd-44c1-46ec-8b2f-de18bb743daa" containerName="cleanup"
Jul 24 15:31:43.212088 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:31:43.212071    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz"
Jul 24 15:31:43.217073 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:31:43.217049    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-51dfd496-kserve-self-signed-certs\""
Jul 24 15:31:43.332189 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:31:43.332151    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz"]
Jul 24 15:31:43.341155 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:31:43.341122    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e463464-d975-4355-869e-a03d406db0a0-tls-certs\") pod \"e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz\" (UID: \"5e463464-d975-4355-869e-a03d406db0a0\") " pod="llm/e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz"
Jul 24 15:31:43.341332 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:31:43.341178    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e463464-d975-4355-869e-a03d406db0a0-tmp-dir\") pod \"e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz\" (UID: \"5e463464-d975-4355-869e-a03d406db0a0\") " pod="llm/e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz"
Jul 24 15:31:43.341332 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:31:43.341228    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e463464-d975-4355-869e-a03d406db0a0-dshm\") pod \"e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz\" (UID: \"5e463464-d975-4355-869e-a03d406db0a0\") " pod="llm/e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz"
Jul 24 15:31:43.341332 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:31:43.341301    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e463464-d975-4355-869e-a03d406db0a0-model-cache\") pod \"e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz\" (UID: \"5e463464-d975-4355-869e-a03d406db0a0\") " pod="llm/e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz"
Jul 24 15:31:43.341497 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:31:43.341347    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e463464-d975-4355-869e-a03d406db0a0-home\") pod \"e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz\" (UID: \"5e463464-d975-4355-869e-a03d406db0a0\") " pod="llm/e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz"
Jul 24 15:31:43.341497 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:31:43.341405    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhx4z\" (UniqueName: \"kubernetes.io/projected/5e463464-d975-4355-869e-a03d406db0a0-kube-api-access-dhx4z\") pod \"e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz\" (UID: \"5e463464-d975-4355-869e-a03d406db0a0\") " pod="llm/e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz"
Jul 24 15:31:43.442578 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:31:43.442538    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e463464-d975-4355-869e-a03d406db0a0-tls-certs\") pod \"e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz\" (UID: \"5e463464-d975-4355-869e-a03d406db0a0\") " pod="llm/e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz"
Jul 24 15:31:43.442578 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:31:43.442579    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e463464-d975-4355-869e-a03d406db0a0-tmp-dir\") pod \"e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz\" (UID: \"5e463464-d975-4355-869e-a03d406db0a0\") " pod="llm/e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz"
Jul 24 15:31:43.443144 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:31:43.442606    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e463464-d975-4355-869e-a03d406db0a0-dshm\") pod \"e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz\" (UID: \"5e463464-d975-4355-869e-a03d406db0a0\") " pod="llm/e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz"
Jul 24 15:31:43.443144 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:31:43.442738    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e463464-d975-4355-869e-a03d406db0a0-model-cache\") pod \"e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz\" (UID: \"5e463464-d975-4355-869e-a03d406db0a0\") " pod="llm/e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz"
Jul 24 15:31:43.443144 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:31:43.442786    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e463464-d975-4355-869e-a03d406db0a0-home\") pod \"e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz\" (UID: \"5e463464-d975-4355-869e-a03d406db0a0\") " pod="llm/e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz"
Jul 24 15:31:43.443144 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:31:43.442873    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhx4z\" (UniqueName: \"kubernetes.io/projected/5e463464-d975-4355-869e-a03d406db0a0-kube-api-access-dhx4z\") pod \"e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz\" (UID: \"5e463464-d975-4355-869e-a03d406db0a0\") " pod="llm/e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz"
Jul 24 15:31:43.443144 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:31:43.443081    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e463464-d975-4355-869e-a03d406db0a0-tmp-dir\") pod \"e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz\" (UID: \"5e463464-d975-4355-869e-a03d406db0a0\") " pod="llm/e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz"
Jul 24 15:31:43.443375 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:31:43.443167    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e463464-d975-4355-869e-a03d406db0a0-model-cache\") pod \"e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz\" (UID: \"5e463464-d975-4355-869e-a03d406db0a0\") " pod="llm/e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz"
Jul 24 15:31:43.443375 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:31:43.443293    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e463464-d975-4355-869e-a03d406db0a0-home\") pod \"e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz\" (UID: \"5e463464-d975-4355-869e-a03d406db0a0\") " pod="llm/e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz"
Jul 24 15:31:43.445066 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:31:43.445037    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e463464-d975-4355-869e-a03d406db0a0-dshm\") pod \"e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz\" (UID: \"5e463464-d975-4355-869e-a03d406db0a0\") " pod="llm/e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz"
Jul 24 15:31:43.445218 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:31:43.445198    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e463464-d975-4355-869e-a03d406db0a0-tls-certs\") pod \"e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz\" (UID: \"5e463464-d975-4355-869e-a03d406db0a0\") " pod="llm/e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz"
Jul 24 15:31:43.456243 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:31:43.456213    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhx4z\" (UniqueName: \"kubernetes.io/projected/5e463464-d975-4355-869e-a03d406db0a0-kube-api-access-dhx4z\") pod \"e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz\" (UID: \"5e463464-d975-4355-869e-a03d406db0a0\") " pod="llm/e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz"
Jul 24 15:31:43.521166 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:31:43.521121    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz"
Jul 24 15:31:43.699936 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:31:43.699907    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz"]
Jul 24 15:31:43.702547 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:31:43.702518    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e463464_d975_4355_869e_a03d406db0a0.slice/crio-72f504deae816ebbf1e1c7367711249bc2ba6ca2b35d0c3c64ef55b34e815810 WatchSource:0}: Error finding container 72f504deae816ebbf1e1c7367711249bc2ba6ca2b35d0c3c64ef55b34e815810: Status 404 returned error can't find the container with id 72f504deae816ebbf1e1c7367711249bc2ba6ca2b35d0c3c64ef55b34e815810
Jul 24 15:31:44.106481 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:31:44.106446    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz" event={"ID":"5e463464-d975-4355-869e-a03d406db0a0","Type":"ContainerStarted","Data":"14ce6e4c67e70b4421e2fbaf4e8606beffe5f11ae1a8c4cdebd2bb918492e371"}
Jul 24 15:31:44.106481 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:31:44.106485    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz" event={"ID":"5e463464-d975-4355-869e-a03d406db0a0","Type":"ContainerStarted","Data":"72f504deae816ebbf1e1c7367711249bc2ba6ca2b35d0c3c64ef55b34e815810"}
Jul 24 15:31:44.127861 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:31:44.127780    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz" podStartSLOduration=1.127758997 podStartE2EDuration="1.127758997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:31:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:31:44.127565647 +0000 UTC m=+1396.939917837" watchObservedRunningTime="2026-07-24 15:31:44.127758997 +0000 UTC m=+1396.940111165"
Jul 24 15:31:53.521829 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:31:53.521790    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz"
Jul 24 15:31:53.522268 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:31:53.521869    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz"
Jul 24 15:31:53.534145 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:31:53.534115    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz"
Jul 24 15:31:54.164179 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:31:54.164150    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz"
Jul 24 15:31:56.628960 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:31:56.628926    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:32:15.643806 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:15.643772    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6649c54c98-ct7sx"]
Jul 24 15:32:15.647252 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:15.647233    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6649c54c98-ct7sx"
Jul 24 15:32:15.651489 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:15.651466    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 24 15:32:15.652201 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:15.652182    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 24 15:32:15.652201 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:15.652194    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 24 15:32:15.652486 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:15.652277    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 24 15:32:15.652540 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:15.652500    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6fm9c\""
Jul 24 15:32:15.652688 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:15.652673    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 24 15:32:15.653125 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:15.653107    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 24 15:32:15.653125 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:15.653121    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 24 15:32:15.660946 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:15.660926    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6649c54c98-ct7sx"]
Jul 24 15:32:15.661707 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:15.661689    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 24 15:32:15.748766 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:15.748725    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b8027fe-6846-486d-8288-e4842ba1b2d5-oauth-serving-cert\") pod \"console-6649c54c98-ct7sx\" (UID: \"9b8027fe-6846-486d-8288-e4842ba1b2d5\") " pod="openshift-console/console-6649c54c98-ct7sx"
Jul 24 15:32:15.748963 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:15.748786    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mdkx\" (UniqueName: \"kubernetes.io/projected/9b8027fe-6846-486d-8288-e4842ba1b2d5-kube-api-access-5mdkx\") pod \"console-6649c54c98-ct7sx\" (UID: \"9b8027fe-6846-486d-8288-e4842ba1b2d5\") " pod="openshift-console/console-6649c54c98-ct7sx"
Jul 24 15:32:15.748963 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:15.748840    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b8027fe-6846-486d-8288-e4842ba1b2d5-console-oauth-config\") pod \"console-6649c54c98-ct7sx\" (UID: \"9b8027fe-6846-486d-8288-e4842ba1b2d5\") " pod="openshift-console/console-6649c54c98-ct7sx"
Jul 24 15:32:15.748963 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:15.748857    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b8027fe-6846-486d-8288-e4842ba1b2d5-service-ca\") pod \"console-6649c54c98-ct7sx\" (UID: \"9b8027fe-6846-486d-8288-e4842ba1b2d5\") " pod="openshift-console/console-6649c54c98-ct7sx"
Jul 24 15:32:15.748963 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:15.748907    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b8027fe-6846-486d-8288-e4842ba1b2d5-console-config\") pod \"console-6649c54c98-ct7sx\" (UID: \"9b8027fe-6846-486d-8288-e4842ba1b2d5\") " pod="openshift-console/console-6649c54c98-ct7sx"
Jul 24 15:32:15.748963 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:15.748926    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b8027fe-6846-486d-8288-e4842ba1b2d5-trusted-ca-bundle\") pod \"console-6649c54c98-ct7sx\" (UID: \"9b8027fe-6846-486d-8288-e4842ba1b2d5\") " pod="openshift-console/console-6649c54c98-ct7sx"
Jul 24 15:32:15.749133 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:15.748997    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b8027fe-6846-486d-8288-e4842ba1b2d5-console-serving-cert\") pod \"console-6649c54c98-ct7sx\" (UID: \"9b8027fe-6846-486d-8288-e4842ba1b2d5\") " pod="openshift-console/console-6649c54c98-ct7sx"
Jul 24 15:32:15.850030 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:15.849989    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mdkx\" (UniqueName: \"kubernetes.io/projected/9b8027fe-6846-486d-8288-e4842ba1b2d5-kube-api-access-5mdkx\") pod \"console-6649c54c98-ct7sx\" (UID: \"9b8027fe-6846-486d-8288-e4842ba1b2d5\") " pod="openshift-console/console-6649c54c98-ct7sx"
Jul 24 15:32:15.850030 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:15.850030    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b8027fe-6846-486d-8288-e4842ba1b2d5-console-oauth-config\") pod \"console-6649c54c98-ct7sx\" (UID: \"9b8027fe-6846-486d-8288-e4842ba1b2d5\") " pod="openshift-console/console-6649c54c98-ct7sx"
Jul 24 15:32:15.850249 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:15.850054    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b8027fe-6846-486d-8288-e4842ba1b2d5-service-ca\") pod \"console-6649c54c98-ct7sx\" (UID: \"9b8027fe-6846-486d-8288-e4842ba1b2d5\") " pod="openshift-console/console-6649c54c98-ct7sx"
Jul 24 15:32:15.850249 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:15.850103    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b8027fe-6846-486d-8288-e4842ba1b2d5-console-config\") pod \"console-6649c54c98-ct7sx\" (UID: \"9b8027fe-6846-486d-8288-e4842ba1b2d5\") " pod="openshift-console/console-6649c54c98-ct7sx"
Jul 24 15:32:15.850249 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:15.850123    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b8027fe-6846-486d-8288-e4842ba1b2d5-trusted-ca-bundle\") pod \"console-6649c54c98-ct7sx\" (UID: \"9b8027fe-6846-486d-8288-e4842ba1b2d5\") " pod="openshift-console/console-6649c54c98-ct7sx"
Jul 24 15:32:15.850249 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:15.850166    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b8027fe-6846-486d-8288-e4842ba1b2d5-console-serving-cert\") pod \"console-6649c54c98-ct7sx\" (UID: \"9b8027fe-6846-486d-8288-e4842ba1b2d5\") " pod="openshift-console/console-6649c54c98-ct7sx"
Jul 24 15:32:15.850249 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:15.850199    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b8027fe-6846-486d-8288-e4842ba1b2d5-oauth-serving-cert\") pod \"console-6649c54c98-ct7sx\" (UID: \"9b8027fe-6846-486d-8288-e4842ba1b2d5\") " pod="openshift-console/console-6649c54c98-ct7sx"
Jul 24 15:32:15.851076 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:15.851042    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b8027fe-6846-486d-8288-e4842ba1b2d5-oauth-serving-cert\") pod \"console-6649c54c98-ct7sx\" (UID: \"9b8027fe-6846-486d-8288-e4842ba1b2d5\") " pod="openshift-console/console-6649c54c98-ct7sx"
Jul 24 15:32:15.851076 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:15.851057    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b8027fe-6846-486d-8288-e4842ba1b2d5-console-config\") pod \"console-6649c54c98-ct7sx\" (UID: \"9b8027fe-6846-486d-8288-e4842ba1b2d5\") " pod="openshift-console/console-6649c54c98-ct7sx"
Jul 24 15:32:15.851235 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:15.851091    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b8027fe-6846-486d-8288-e4842ba1b2d5-service-ca\") pod \"console-6649c54c98-ct7sx\" (UID: \"9b8027fe-6846-486d-8288-e4842ba1b2d5\") " pod="openshift-console/console-6649c54c98-ct7sx"
Jul 24 15:32:15.851235 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:15.851101    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b8027fe-6846-486d-8288-e4842ba1b2d5-trusted-ca-bundle\") pod \"console-6649c54c98-ct7sx\" (UID: \"9b8027fe-6846-486d-8288-e4842ba1b2d5\") " pod="openshift-console/console-6649c54c98-ct7sx"
Jul 24 15:32:15.852677 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:15.852652    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b8027fe-6846-486d-8288-e4842ba1b2d5-console-oauth-config\") pod \"console-6649c54c98-ct7sx\" (UID: \"9b8027fe-6846-486d-8288-e4842ba1b2d5\") " pod="openshift-console/console-6649c54c98-ct7sx"
Jul 24 15:32:15.852755 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:15.852725    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b8027fe-6846-486d-8288-e4842ba1b2d5-console-serving-cert\") pod \"console-6649c54c98-ct7sx\" (UID: \"9b8027fe-6846-486d-8288-e4842ba1b2d5\") " pod="openshift-console/console-6649c54c98-ct7sx"
Jul 24 15:32:15.859280 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:15.859254    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mdkx\" (UniqueName: \"kubernetes.io/projected/9b8027fe-6846-486d-8288-e4842ba1b2d5-kube-api-access-5mdkx\") pod \"console-6649c54c98-ct7sx\" (UID: \"9b8027fe-6846-486d-8288-e4842ba1b2d5\") " pod="openshift-console/console-6649c54c98-ct7sx"
Jul 24 15:32:15.957448 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:15.957367    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6649c54c98-ct7sx"
Jul 24 15:32:16.089968 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:16.089943    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6649c54c98-ct7sx"]
Jul 24 15:32:16.092142 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:32:16.092119    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b8027fe_6846_486d_8288_e4842ba1b2d5.slice/crio-3f6d8cc7ec42a96a205c0f5606c495c28a73ca1f850193caeb419a7b41316b78 WatchSource:0}: Error finding container 3f6d8cc7ec42a96a205c0f5606c495c28a73ca1f850193caeb419a7b41316b78: Status 404 returned error can't find the container with id 3f6d8cc7ec42a96a205c0f5606c495c28a73ca1f850193caeb419a7b41316b78
Jul 24 15:32:16.262538 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:16.262447    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6649c54c98-ct7sx" event={"ID":"9b8027fe-6846-486d-8288-e4842ba1b2d5","Type":"ContainerStarted","Data":"623142b94bbb3b208e404fec65e948663cf7a3dfe66f3040e7fef4ffc6970393"}
Jul 24 15:32:16.262538 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:16.262505    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6649c54c98-ct7sx" event={"ID":"9b8027fe-6846-486d-8288-e4842ba1b2d5","Type":"ContainerStarted","Data":"3f6d8cc7ec42a96a205c0f5606c495c28a73ca1f850193caeb419a7b41316b78"}
Jul 24 15:32:16.282876 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:16.282807    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6649c54c98-ct7sx" podStartSLOduration=1.282789432 podStartE2EDuration="1.282789432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:32:16.281043243 +0000 UTC m=+1429.093395424" watchObservedRunningTime="2026-07-24 15:32:16.282789432 +0000 UTC m=+1429.095141610"
Jul 24 15:32:25.958186 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:25.958111    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6649c54c98-ct7sx"
Jul 24 15:32:25.958186 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:25.958166    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6649c54c98-ct7sx"
Jul 24 15:32:25.962783 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:25.962760    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6649c54c98-ct7sx"
Jul 24 15:32:26.314519 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:26.314490    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6649c54c98-ct7sx"
Jul 24 15:32:27.968176 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:27.968143    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz"]
Jul 24 15:32:29.631411 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:29.631367    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:32:40.046725 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:40.046686    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:32:42.987176 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:42.987112    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz" podUID="5e463464-d975-4355-869e-a03d406db0a0" containerName="main" containerID="cri-o://14ce6e4c67e70b4421e2fbaf4e8606beffe5f11ae1a8c4cdebd2bb918492e371" gracePeriod=45
Jul 24 15:32:43.244518 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:43.244453    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz"
Jul 24 15:32:43.304158 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:43.304126    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e463464-d975-4355-869e-a03d406db0a0-tls-certs\") pod \"5e463464-d975-4355-869e-a03d406db0a0\" (UID: \"5e463464-d975-4355-869e-a03d406db0a0\") "
Jul 24 15:32:43.304158 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:43.304165    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e463464-d975-4355-869e-a03d406db0a0-tmp-dir\") pod \"5e463464-d975-4355-869e-a03d406db0a0\" (UID: \"5e463464-d975-4355-869e-a03d406db0a0\") "
Jul 24 15:32:43.304372 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:43.304190    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e463464-d975-4355-869e-a03d406db0a0-home\") pod \"5e463464-d975-4355-869e-a03d406db0a0\" (UID: \"5e463464-d975-4355-869e-a03d406db0a0\") "
Jul 24 15:32:43.304372 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:43.304282    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e463464-d975-4355-869e-a03d406db0a0-dshm\") pod \"5e463464-d975-4355-869e-a03d406db0a0\" (UID: \"5e463464-d975-4355-869e-a03d406db0a0\") "
Jul 24 15:32:43.304372 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:43.304313    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e463464-d975-4355-869e-a03d406db0a0-model-cache\") pod \"5e463464-d975-4355-869e-a03d406db0a0\" (UID: \"5e463464-d975-4355-869e-a03d406db0a0\") "
Jul 24 15:32:43.304372 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:43.304342    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhx4z\" (UniqueName: \"kubernetes.io/projected/5e463464-d975-4355-869e-a03d406db0a0-kube-api-access-dhx4z\") pod \"5e463464-d975-4355-869e-a03d406db0a0\" (UID: \"5e463464-d975-4355-869e-a03d406db0a0\") "
Jul 24 15:32:43.304564 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:43.304460    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e463464-d975-4355-869e-a03d406db0a0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5e463464-d975-4355-869e-a03d406db0a0" (UID: "5e463464-d975-4355-869e-a03d406db0a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:32:43.304564 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:43.304470    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e463464-d975-4355-869e-a03d406db0a0-home" (OuterVolumeSpecName: "home") pod "5e463464-d975-4355-869e-a03d406db0a0" (UID: "5e463464-d975-4355-869e-a03d406db0a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:32:43.304693 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:43.304667    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e463464-d975-4355-869e-a03d406db0a0-model-cache" (OuterVolumeSpecName: "model-cache") pod "5e463464-d975-4355-869e-a03d406db0a0" (UID: "5e463464-d975-4355-869e-a03d406db0a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:32:43.304740 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:43.304702    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e463464-d975-4355-869e-a03d406db0a0-tmp-dir\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:32:43.304740 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:43.304722    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e463464-d975-4355-869e-a03d406db0a0-home\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:32:43.306478 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:43.306445    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e463464-d975-4355-869e-a03d406db0a0-dshm" (OuterVolumeSpecName: "dshm") pod "5e463464-d975-4355-869e-a03d406db0a0" (UID: "5e463464-d975-4355-869e-a03d406db0a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:32:43.306478 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:43.306463    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e463464-d975-4355-869e-a03d406db0a0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5e463464-d975-4355-869e-a03d406db0a0" (UID: "5e463464-d975-4355-869e-a03d406db0a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 15:32:43.306478 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:43.306456    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e463464-d975-4355-869e-a03d406db0a0-kube-api-access-dhx4z" (OuterVolumeSpecName: "kube-api-access-dhx4z") pod "5e463464-d975-4355-869e-a03d406db0a0" (UID: "5e463464-d975-4355-869e-a03d406db0a0"). InnerVolumeSpecName "kube-api-access-dhx4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:32:43.399671 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:43.399629    2579 generic.go:356] "Generic (PLEG): container finished" podID="5e463464-d975-4355-869e-a03d406db0a0" containerID="14ce6e4c67e70b4421e2fbaf4e8606beffe5f11ae1a8c4cdebd2bb918492e371" exitCode=0
Jul 24 15:32:43.399866 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:43.399678    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz"
Jul 24 15:32:43.399866 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:43.399722    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz" event={"ID":"5e463464-d975-4355-869e-a03d406db0a0","Type":"ContainerDied","Data":"14ce6e4c67e70b4421e2fbaf4e8606beffe5f11ae1a8c4cdebd2bb918492e371"}
Jul 24 15:32:43.399866 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:43.399771    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz" event={"ID":"5e463464-d975-4355-869e-a03d406db0a0","Type":"ContainerDied","Data":"72f504deae816ebbf1e1c7367711249bc2ba6ca2b35d0c3c64ef55b34e815810"}
Jul 24 15:32:43.399866 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:43.399791    2579 scope.go:117] "RemoveContainer" containerID="14ce6e4c67e70b4421e2fbaf4e8606beffe5f11ae1a8c4cdebd2bb918492e371"
Jul 24 15:32:43.408032 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:43.408002    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e463464-d975-4355-869e-a03d406db0a0-tls-certs\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:32:43.408226 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:43.408209    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e463464-d975-4355-869e-a03d406db0a0-dshm\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:32:43.408315 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:43.408305    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e463464-d975-4355-869e-a03d406db0a0-model-cache\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:32:43.408398 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:43.408386    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhx4z\" (UniqueName: \"kubernetes.io/projected/5e463464-d975-4355-869e-a03d406db0a0-kube-api-access-dhx4z\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:32:43.416173 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:43.416154    2579 scope.go:117] "RemoveContainer" containerID="14ce6e4c67e70b4421e2fbaf4e8606beffe5f11ae1a8c4cdebd2bb918492e371"
Jul 24 15:32:43.417408 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:32:43.417377    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14ce6e4c67e70b4421e2fbaf4e8606beffe5f11ae1a8c4cdebd2bb918492e371\": container with ID starting with 14ce6e4c67e70b4421e2fbaf4e8606beffe5f11ae1a8c4cdebd2bb918492e371 not found: ID does not exist" containerID="14ce6e4c67e70b4421e2fbaf4e8606beffe5f11ae1a8c4cdebd2bb918492e371"
Jul 24 15:32:43.417492 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:43.417415    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14ce6e4c67e70b4421e2fbaf4e8606beffe5f11ae1a8c4cdebd2bb918492e371"} err="failed to get container status \"14ce6e4c67e70b4421e2fbaf4e8606beffe5f11ae1a8c4cdebd2bb918492e371\": rpc error: code = NotFound desc = could not find container \"14ce6e4c67e70b4421e2fbaf4e8606beffe5f11ae1a8c4cdebd2bb918492e371\": container with ID starting with 14ce6e4c67e70b4421e2fbaf4e8606beffe5f11ae1a8c4cdebd2bb918492e371 not found: ID does not exist"
Jul 24 15:32:43.426544 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:43.426521    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz"]
Jul 24 15:32:43.429935 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:43.429914    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-51dfd496-kserve-7866bbc6c5-525kz"]
Jul 24 15:32:43.717443 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:43.717405    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e463464-d975-4355-869e-a03d406db0a0" path="/var/lib/kubelet/pods/5e463464-d975-4355-869e-a03d406db0a0/volumes"
Jul 24 15:32:52.441419 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:32:52.441375    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:33:07.521178 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:33:07.521142    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:33:21.631624 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:33:21.631592    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:33:31.155150 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:33:31.155113    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:33:44.329899 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:33:44.329859    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:33:53.739959 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:33:53.739866    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:34:06.836832 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:34:06.836775    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:34:16.294296 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:34:16.294261    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:34:26.325077 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:34:26.325041    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:34:36.634781 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:34:36.634751    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:34:48.128345 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:34:48.128311    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:34:58.244801 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:34:58.244756    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:35:08.047256 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:35:08.047216    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:35:24.346037 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:35:24.345953    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:35:34.142990 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:35:34.142955    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:35:49.034515 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:35:49.034479    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:36:03.078791 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:03.078750    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:36:11.961631 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:11.961586    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6f8777f9-94d6545b-jqjlf"]
Jul 24 15:36:11.962232 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:11.962195    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e463464-d975-4355-869e-a03d406db0a0" containerName="main"
Jul 24 15:36:11.962232 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:11.962217    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e463464-d975-4355-869e-a03d406db0a0" containerName="main"
Jul 24 15:36:11.962354 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:11.962340    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e463464-d975-4355-869e-a03d406db0a0" containerName="main"
Jul 24 15:36:11.965496 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:11.965474    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f8777f9-94d6545b-jqjlf"
Jul 24 15:36:11.968082 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:11.968062    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-6f8777f9\""
Jul 24 15:36:11.968191 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:11.968093    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-6f8777f9-dockercfg-8mx57\""
Jul 24 15:36:11.981123 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:11.981013    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6f8777f9-94d6545b-jqjlf"]
Jul 24 15:36:12.033321 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:12.033290    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6f8777f9-6bb9f848b4-mjd8q"]
Jul 24 15:36:12.036784 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:12.036758    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6f8777f9-6bb9f848b4-mjd8q"
Jul 24 15:36:12.049938 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:12.049911    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6f8777f9-6bb9f848b4-mjd8q"]
Jul 24 15:36:12.084646 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:12.084618    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmfs9\" (UniqueName: \"kubernetes.io/projected/ca4a16a9-28ed-48d9-b2ee-7252432d9bbc-kube-api-access-dmfs9\") pod \"payload-pre-processing-e2e-ait-6f8777f9-94d6545b-jqjlf\" (UID: \"ca4a16a9-28ed-48d9-b2ee-7252432d9bbc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6f8777f9-94d6545b-jqjlf"
Jul 24 15:36:12.084806 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:12.084664    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca4a16a9-28ed-48d9-b2ee-7252432d9bbc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6f8777f9-94d6545b-jqjlf\" (UID: \"ca4a16a9-28ed-48d9-b2ee-7252432d9bbc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6f8777f9-94d6545b-jqjlf"
Jul 24 15:36:12.185837 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:12.185773    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfw6h\" (UniqueName: \"kubernetes.io/projected/4b70fd54-22cb-4180-b070-fa97ec41bdd9-kube-api-access-xfw6h\") pod \"payload-processing-e2e-ait-6f8777f9-6bb9f848b4-mjd8q\" (UID: \"4b70fd54-22cb-4180-b070-fa97ec41bdd9\") " pod="openshift-ingress/payload-processing-e2e-ait-6f8777f9-6bb9f848b4-mjd8q"
Jul 24 15:36:12.186011 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:12.185894    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmfs9\" (UniqueName: \"kubernetes.io/projected/ca4a16a9-28ed-48d9-b2ee-7252432d9bbc-kube-api-access-dmfs9\") pod \"payload-pre-processing-e2e-ait-6f8777f9-94d6545b-jqjlf\" (UID: \"ca4a16a9-28ed-48d9-b2ee-7252432d9bbc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6f8777f9-94d6545b-jqjlf"
Jul 24 15:36:12.186011 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:12.185953    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca4a16a9-28ed-48d9-b2ee-7252432d9bbc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6f8777f9-94d6545b-jqjlf\" (UID: \"ca4a16a9-28ed-48d9-b2ee-7252432d9bbc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6f8777f9-94d6545b-jqjlf"
Jul 24 15:36:12.186011 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:12.185973    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b70fd54-22cb-4180-b070-fa97ec41bdd9-plugins-config-volume\") pod \"payload-processing-e2e-ait-6f8777f9-6bb9f848b4-mjd8q\" (UID: \"4b70fd54-22cb-4180-b070-fa97ec41bdd9\") " pod="openshift-ingress/payload-processing-e2e-ait-6f8777f9-6bb9f848b4-mjd8q"
Jul 24 15:36:12.186611 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:12.186591    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca4a16a9-28ed-48d9-b2ee-7252432d9bbc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6f8777f9-94d6545b-jqjlf\" (UID: \"ca4a16a9-28ed-48d9-b2ee-7252432d9bbc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6f8777f9-94d6545b-jqjlf"
Jul 24 15:36:12.196115 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:12.196086    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmfs9\" (UniqueName: \"kubernetes.io/projected/ca4a16a9-28ed-48d9-b2ee-7252432d9bbc-kube-api-access-dmfs9\") pod \"payload-pre-processing-e2e-ait-6f8777f9-94d6545b-jqjlf\" (UID: \"ca4a16a9-28ed-48d9-b2ee-7252432d9bbc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6f8777f9-94d6545b-jqjlf"
Jul 24 15:36:12.276313 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:12.276282    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f8777f9-94d6545b-jqjlf"
Jul 24 15:36:12.286732 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:12.286707    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfw6h\" (UniqueName: \"kubernetes.io/projected/4b70fd54-22cb-4180-b070-fa97ec41bdd9-kube-api-access-xfw6h\") pod \"payload-processing-e2e-ait-6f8777f9-6bb9f848b4-mjd8q\" (UID: \"4b70fd54-22cb-4180-b070-fa97ec41bdd9\") " pod="openshift-ingress/payload-processing-e2e-ait-6f8777f9-6bb9f848b4-mjd8q"
Jul 24 15:36:12.286855 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:12.286767    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b70fd54-22cb-4180-b070-fa97ec41bdd9-plugins-config-volume\") pod \"payload-processing-e2e-ait-6f8777f9-6bb9f848b4-mjd8q\" (UID: \"4b70fd54-22cb-4180-b070-fa97ec41bdd9\") " pod="openshift-ingress/payload-processing-e2e-ait-6f8777f9-6bb9f848b4-mjd8q"
Jul 24 15:36:12.287443 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:12.287422    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b70fd54-22cb-4180-b070-fa97ec41bdd9-plugins-config-volume\") pod \"payload-processing-e2e-ait-6f8777f9-6bb9f848b4-mjd8q\" (UID: \"4b70fd54-22cb-4180-b070-fa97ec41bdd9\") " pod="openshift-ingress/payload-processing-e2e-ait-6f8777f9-6bb9f848b4-mjd8q"
Jul 24 15:36:12.298786 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:12.298764    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfw6h\" (UniqueName: \"kubernetes.io/projected/4b70fd54-22cb-4180-b070-fa97ec41bdd9-kube-api-access-xfw6h\") pod \"payload-processing-e2e-ait-6f8777f9-6bb9f848b4-mjd8q\" (UID: \"4b70fd54-22cb-4180-b070-fa97ec41bdd9\") " pod="openshift-ingress/payload-processing-e2e-ait-6f8777f9-6bb9f848b4-mjd8q"
Jul 24 15:36:12.346788 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:12.346716    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6f8777f9-6bb9f848b4-mjd8q"
Jul 24 15:36:12.425592 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:12.425567    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6f8777f9-94d6545b-jqjlf"]
Jul 24 15:36:12.428158 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:36:12.428130    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca4a16a9_28ed_48d9_b2ee_7252432d9bbc.slice/crio-6da191e41b00bc4075b1194b1fb97bb267cf8d584f18d9b8ab24b980601aca21 WatchSource:0}: Error finding container 6da191e41b00bc4075b1194b1fb97bb267cf8d584f18d9b8ab24b980601aca21: Status 404 returned error can't find the container with id 6da191e41b00bc4075b1194b1fb97bb267cf8d584f18d9b8ab24b980601aca21
Jul 24 15:36:12.429867 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:12.429848    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 15:36:12.488877 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:12.488852    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6f8777f9-6bb9f848b4-mjd8q"]
Jul 24 15:36:12.490855 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:36:12.490830    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b70fd54_22cb_4180_b070_fa97ec41bdd9.slice/crio-fd67e2cc6896392e4687bd62862ded0bc2a551c2da76182512a49ead726c1140 WatchSource:0}: Error finding container fd67e2cc6896392e4687bd62862ded0bc2a551c2da76182512a49ead726c1140: Status 404 returned error can't find the container with id fd67e2cc6896392e4687bd62862ded0bc2a551c2da76182512a49ead726c1140
Jul 24 15:36:13.407766 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:13.407726    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6f8777f9-6bb9f848b4-mjd8q" event={"ID":"4b70fd54-22cb-4180-b070-fa97ec41bdd9","Type":"ContainerStarted","Data":"fd67e2cc6896392e4687bd62862ded0bc2a551c2da76182512a49ead726c1140"}
Jul 24 15:36:13.409913 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:13.409886    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f8777f9-94d6545b-jqjlf" event={"ID":"ca4a16a9-28ed-48d9-b2ee-7252432d9bbc","Type":"ContainerStarted","Data":"6da191e41b00bc4075b1194b1fb97bb267cf8d584f18d9b8ab24b980601aca21"}
Jul 24 15:36:13.734855 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:13.734758    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:36:15.421388 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:15.421353    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6f8777f9-6bb9f848b4-mjd8q" event={"ID":"4b70fd54-22cb-4180-b070-fa97ec41bdd9","Type":"ContainerStarted","Data":"1c74680e23869f46490365c95f52d5191fd81b21d0fa8d4a0da665270b0f862b"}
Jul 24 15:36:15.421874 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:15.421466    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-6f8777f9-6bb9f848b4-mjd8q"
Jul 24 15:36:15.424051 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:15.423534    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f8777f9-94d6545b-jqjlf" event={"ID":"ca4a16a9-28ed-48d9-b2ee-7252432d9bbc","Type":"ContainerStarted","Data":"b13d84304c45e38f9e44d9122bdf0e03e16e5cf9dc084d6f79556612700723cf"}
Jul 24 15:36:15.424051 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:15.423833    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f8777f9-94d6545b-jqjlf"
Jul 24 15:36:15.442971 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:15.442791    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-6f8777f9-6bb9f848b4-mjd8q" podStartSLOduration=1.156496265 podStartE2EDuration="3.442772645s" totalImagesPullingTime="2.28627638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:36:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:36:15.442103296 +0000 UTC m=+1668.254455498" watchObservedRunningTime="2026-07-24 15:36:15.442772645 +0000 UTC m=+1668.255124827"
Jul 24 15:36:15.461445 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:15.461395    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f8777f9-94d6545b-jqjlf" podStartSLOduration=2.118799433 podStartE2EDuration="4.461380834s" totalImagesPullingTime="2.342581401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:36:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:36:15.460040557 +0000 UTC m=+1668.272392737" watchObservedRunningTime="2026-07-24 15:36:15.461380834 +0000 UTC m=+1668.273733016"
Jul 24 15:36:26.430291 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:26.430261    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-6f8777f9-6bb9f848b4-mjd8q"
Jul 24 15:36:26.430761 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:26.430337    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f8777f9-94d6545b-jqjlf"
Jul 24 15:36:36.590784 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:36.590742    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f8777f9-rz844"]
Jul 24 15:36:36.594172 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:36.594148    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f8777f9-rz844"
Jul 24 15:36:36.596970 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:36.596949    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r6lw7\""
Jul 24 15:36:36.597710 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:36.597693    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 15:36:36.604102 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:36.604049    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f8777f9-rz844"]
Jul 24 15:36:36.688577 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:36.688533    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a41a62b3-0498-43d5-aced-055b959e58d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f8777f9-rz844\" (UID: \"a41a62b3-0498-43d5-aced-055b959e58d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f8777f9-rz844"
Jul 24 15:36:36.789559 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:36.789507    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a41a62b3-0498-43d5-aced-055b959e58d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f8777f9-rz844\" (UID: \"a41a62b3-0498-43d5-aced-055b959e58d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f8777f9-rz844"
Jul 24 15:36:36.790184 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:36.790155    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a41a62b3-0498-43d5-aced-055b959e58d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f8777f9-rz844\" (UID: \"a41a62b3-0498-43d5-aced-055b959e58d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f8777f9-rz844"
Jul 24 15:36:36.905919 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:36.905837    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f8777f9-rz844"
Jul 24 15:36:37.038838 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:37.038791    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f8777f9-rz844"]
Jul 24 15:36:37.040887 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:36:37.040858    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda41a62b3_0498_43d5_aced_055b959e58d2.slice/crio-7d1ef5369d93b8587592062bf4a0ff055ea5422f6a39ffff8f72a52ddf4c41cd WatchSource:0}: Error finding container 7d1ef5369d93b8587592062bf4a0ff055ea5422f6a39ffff8f72a52ddf4c41cd: Status 404 returned error can't find the container with id 7d1ef5369d93b8587592062bf4a0ff055ea5422f6a39ffff8f72a52ddf4c41cd
Jul 24 15:36:37.534488 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:37.534453    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f8777f9-rz844" event={"ID":"a41a62b3-0498-43d5-aced-055b959e58d2","Type":"ContainerStarted","Data":"daa9307861a33b5ac4b07d568743e7bdecdcd8addca21396d1297f5774e578d7"}
Jul 24 15:36:37.534665 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:37.534502    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f8777f9-rz844" event={"ID":"a41a62b3-0498-43d5-aced-055b959e58d2","Type":"ContainerStarted","Data":"7d1ef5369d93b8587592062bf4a0ff055ea5422f6a39ffff8f72a52ddf4c41cd"}
Jul 24 15:36:37.553317 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:37.553268    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f8777f9-rz844" podStartSLOduration=1.553250316 podStartE2EDuration="1.553250316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:36:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:36:37.551187951 +0000 UTC m=+1690.363540131" watchObservedRunningTime="2026-07-24 15:36:37.553250316 +0000 UTC m=+1690.365602495"
Jul 24 15:36:38.540720 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:38.540680    2579 generic.go:356] "Generic (PLEG): container finished" podID="a41a62b3-0498-43d5-aced-055b959e58d2" containerID="daa9307861a33b5ac4b07d568743e7bdecdcd8addca21396d1297f5774e578d7" exitCode=0
Jul 24 15:36:38.541120 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:38.540733    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f8777f9-rz844" event={"ID":"a41a62b3-0498-43d5-aced-055b959e58d2","Type":"ContainerDied","Data":"daa9307861a33b5ac4b07d568743e7bdecdcd8addca21396d1297f5774e578d7"}
Jul 24 15:36:39.681488 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:39.681462    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f8777f9-rz844"
Jul 24 15:36:39.708897 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:39.708870    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a41a62b3-0498-43d5-aced-055b959e58d2-maas-api-service-ca\") pod \"a41a62b3-0498-43d5-aced-055b959e58d2\" (UID: \"a41a62b3-0498-43d5-aced-055b959e58d2\") "
Jul 24 15:36:39.709204 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:39.709183    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a41a62b3-0498-43d5-aced-055b959e58d2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a41a62b3-0498-43d5-aced-055b959e58d2" (UID: "a41a62b3-0498-43d5-aced-055b959e58d2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:36:39.809721 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:39.809614    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a41a62b3-0498-43d5-aced-055b959e58d2-maas-api-service-ca\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:36:40.558344 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:40.558303    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f8777f9-rz844" event={"ID":"a41a62b3-0498-43d5-aced-055b959e58d2","Type":"ContainerDied","Data":"7d1ef5369d93b8587592062bf4a0ff055ea5422f6a39ffff8f72a52ddf4c41cd"}
Jul 24 15:36:40.558344 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:40.558341    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f8777f9-rz844"
Jul 24 15:36:40.558557 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:40.558390    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d1ef5369d93b8587592062bf4a0ff055ea5422f6a39ffff8f72a52ddf4c41cd"
Jul 24 15:36:47.372352 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:47.372318    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6f8777f9-6bb9f848b4-mjd8q"]
Jul 24 15:36:47.372885 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:47.372634    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-6f8777f9-6bb9f848b4-mjd8q" podUID="4b70fd54-22cb-4180-b070-fa97ec41bdd9" containerName="payload-processing" containerID="cri-o://1c74680e23869f46490365c95f52d5191fd81b21d0fa8d4a0da665270b0f862b" gracePeriod=30
Jul 24 15:36:47.380117 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:47.379743    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6f8777f9-94d6545b-jqjlf"]
Jul 24 15:36:47.380117 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:47.380050    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f8777f9-94d6545b-jqjlf" podUID="ca4a16a9-28ed-48d9-b2ee-7252432d9bbc" containerName="payload-pre-processing" containerID="cri-o://b13d84304c45e38f9e44d9122bdf0e03e16e5cf9dc084d6f79556612700723cf" gracePeriod=30
Jul 24 15:36:47.597415 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:47.597377    2579 generic.go:356] "Generic (PLEG): container finished" podID="4b70fd54-22cb-4180-b070-fa97ec41bdd9" containerID="1c74680e23869f46490365c95f52d5191fd81b21d0fa8d4a0da665270b0f862b" exitCode=0
Jul 24 15:36:47.597576 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:47.597467    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6f8777f9-6bb9f848b4-mjd8q" event={"ID":"4b70fd54-22cb-4180-b070-fa97ec41bdd9","Type":"ContainerDied","Data":"1c74680e23869f46490365c95f52d5191fd81b21d0fa8d4a0da665270b0f862b"}
Jul 24 15:36:47.599277 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:47.599219    2579 generic.go:356] "Generic (PLEG): container finished" podID="ca4a16a9-28ed-48d9-b2ee-7252432d9bbc" containerID="b13d84304c45e38f9e44d9122bdf0e03e16e5cf9dc084d6f79556612700723cf" exitCode=0
Jul 24 15:36:47.599398 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:47.599378    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f8777f9-94d6545b-jqjlf" event={"ID":"ca4a16a9-28ed-48d9-b2ee-7252432d9bbc","Type":"ContainerDied","Data":"b13d84304c45e38f9e44d9122bdf0e03e16e5cf9dc084d6f79556612700723cf"}
Jul 24 15:36:47.667978 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:47.667956    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6f8777f9-6bb9f848b4-mjd8q"
Jul 24 15:36:47.670717 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:47.670700    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f8777f9-94d6545b-jqjlf"
Jul 24 15:36:47.771857 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:47.771801    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca4a16a9-28ed-48d9-b2ee-7252432d9bbc-plugins-config-volume\") pod \"ca4a16a9-28ed-48d9-b2ee-7252432d9bbc\" (UID: \"ca4a16a9-28ed-48d9-b2ee-7252432d9bbc\") "
Jul 24 15:36:47.772057 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:47.771874    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b70fd54-22cb-4180-b070-fa97ec41bdd9-plugins-config-volume\") pod \"4b70fd54-22cb-4180-b070-fa97ec41bdd9\" (UID: \"4b70fd54-22cb-4180-b070-fa97ec41bdd9\") "
Jul 24 15:36:47.772057 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:47.771949    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfw6h\" (UniqueName: \"kubernetes.io/projected/4b70fd54-22cb-4180-b070-fa97ec41bdd9-kube-api-access-xfw6h\") pod \"4b70fd54-22cb-4180-b070-fa97ec41bdd9\" (UID: \"4b70fd54-22cb-4180-b070-fa97ec41bdd9\") "
Jul 24 15:36:47.772057 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:47.771968    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmfs9\" (UniqueName: \"kubernetes.io/projected/ca4a16a9-28ed-48d9-b2ee-7252432d9bbc-kube-api-access-dmfs9\") pod \"ca4a16a9-28ed-48d9-b2ee-7252432d9bbc\" (UID: \"ca4a16a9-28ed-48d9-b2ee-7252432d9bbc\") "
Jul 24 15:36:47.772248 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:47.772220    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca4a16a9-28ed-48d9-b2ee-7252432d9bbc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ca4a16a9-28ed-48d9-b2ee-7252432d9bbc" (UID: "ca4a16a9-28ed-48d9-b2ee-7252432d9bbc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:36:47.772316 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:47.772288    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b70fd54-22cb-4180-b070-fa97ec41bdd9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4b70fd54-22cb-4180-b070-fa97ec41bdd9" (UID: "4b70fd54-22cb-4180-b070-fa97ec41bdd9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:36:47.774244 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:47.774214    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b70fd54-22cb-4180-b070-fa97ec41bdd9-kube-api-access-xfw6h" (OuterVolumeSpecName: "kube-api-access-xfw6h") pod "4b70fd54-22cb-4180-b070-fa97ec41bdd9" (UID: "4b70fd54-22cb-4180-b070-fa97ec41bdd9"). InnerVolumeSpecName "kube-api-access-xfw6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:36:47.774318 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:47.774258    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca4a16a9-28ed-48d9-b2ee-7252432d9bbc-kube-api-access-dmfs9" (OuterVolumeSpecName: "kube-api-access-dmfs9") pod "ca4a16a9-28ed-48d9-b2ee-7252432d9bbc" (UID: "ca4a16a9-28ed-48d9-b2ee-7252432d9bbc"). InnerVolumeSpecName "kube-api-access-dmfs9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:36:47.873555 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:47.873517    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfw6h\" (UniqueName: \"kubernetes.io/projected/4b70fd54-22cb-4180-b070-fa97ec41bdd9-kube-api-access-xfw6h\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:36:47.873555 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:47.873549    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmfs9\" (UniqueName: \"kubernetes.io/projected/ca4a16a9-28ed-48d9-b2ee-7252432d9bbc-kube-api-access-dmfs9\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:36:47.873555 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:47.873559    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca4a16a9-28ed-48d9-b2ee-7252432d9bbc-plugins-config-volume\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:36:47.873791 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:47.873567    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b70fd54-22cb-4180-b070-fa97ec41bdd9-plugins-config-volume\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:36:48.605492 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:48.605459    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6f8777f9-6bb9f848b4-mjd8q"
Jul 24 15:36:48.605983 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:48.605458    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6f8777f9-6bb9f848b4-mjd8q" event={"ID":"4b70fd54-22cb-4180-b070-fa97ec41bdd9","Type":"ContainerDied","Data":"fd67e2cc6896392e4687bd62862ded0bc2a551c2da76182512a49ead726c1140"}
Jul 24 15:36:48.605983 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:48.605606    2579 scope.go:117] "RemoveContainer" containerID="1c74680e23869f46490365c95f52d5191fd81b21d0fa8d4a0da665270b0f862b"
Jul 24 15:36:48.607028 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:48.606844    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f8777f9-94d6545b-jqjlf"
Jul 24 15:36:48.607028 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:48.606847    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f8777f9-94d6545b-jqjlf" event={"ID":"ca4a16a9-28ed-48d9-b2ee-7252432d9bbc","Type":"ContainerDied","Data":"6da191e41b00bc4075b1194b1fb97bb267cf8d584f18d9b8ab24b980601aca21"}
Jul 24 15:36:48.616297 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:48.616023    2579 scope.go:117] "RemoveContainer" containerID="b13d84304c45e38f9e44d9122bdf0e03e16e5cf9dc084d6f79556612700723cf"
Jul 24 15:36:48.640879 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:48.640848    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6f8777f9-94d6545b-jqjlf"]
Jul 24 15:36:48.650710 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:48.650684    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6f8777f9-94d6545b-jqjlf"]
Jul 24 15:36:48.689922 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:48.689894    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6f8777f9-6bb9f848b4-mjd8q"]
Jul 24 15:36:48.704295 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:48.704266    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6f8777f9-6bb9f848b4-mjd8q"]
Jul 24 15:36:49.717521 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:49.717436    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b70fd54-22cb-4180-b070-fa97ec41bdd9" path="/var/lib/kubelet/pods/4b70fd54-22cb-4180-b070-fa97ec41bdd9/volumes"
Jul 24 15:36:49.717906 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:49.717800    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca4a16a9-28ed-48d9-b2ee-7252432d9bbc" path="/var/lib/kubelet/pods/ca4a16a9-28ed-48d9-b2ee-7252432d9bbc/volumes"
Jul 24 15:36:50.172276 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:50.172241    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:36:51.669339 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:51.669290    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f8777f9-rz844"]
Jul 24 15:36:51.671370 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:51.671343    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f8777f9-rz844"]
Jul 24 15:36:51.716955 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:51.716918    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a41a62b3-0498-43d5-aced-055b959e58d2" path="/var/lib/kubelet/pods/a41a62b3-0498-43d5-aced-055b959e58d2/volumes"
Jul 24 15:36:55.410906 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:55.410872    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-70e683b2-575f5b4b5-xjscl"]
Jul 24 15:36:55.411384 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:55.411229    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a41a62b3-0498-43d5-aced-055b959e58d2" containerName="revoke-keys"
Jul 24 15:36:55.411384 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:55.411240    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a41a62b3-0498-43d5-aced-055b959e58d2" containerName="revoke-keys"
Jul 24 15:36:55.411384 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:55.411252    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca4a16a9-28ed-48d9-b2ee-7252432d9bbc" containerName="payload-pre-processing"
Jul 24 15:36:55.411384 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:55.411257    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca4a16a9-28ed-48d9-b2ee-7252432d9bbc" containerName="payload-pre-processing"
Jul 24 15:36:55.411384 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:55.411273    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b70fd54-22cb-4180-b070-fa97ec41bdd9" containerName="payload-processing"
Jul 24 15:36:55.411384 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:55.411278    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b70fd54-22cb-4180-b070-fa97ec41bdd9" containerName="payload-processing"
Jul 24 15:36:55.411384 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:55.411342    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a41a62b3-0498-43d5-aced-055b959e58d2" containerName="revoke-keys"
Jul 24 15:36:55.411384 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:55.411350    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca4a16a9-28ed-48d9-b2ee-7252432d9bbc" containerName="payload-pre-processing"
Jul 24 15:36:55.411384 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:55.411357    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b70fd54-22cb-4180-b070-fa97ec41bdd9" containerName="payload-processing"
Jul 24 15:36:55.415524 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:55.415506    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70e683b2-575f5b4b5-xjscl"
Jul 24 15:36:55.419346 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:55.419327    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 15:36:55.420026 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:55.420010    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 15:36:55.436723 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:55.436695    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-70e683b2-serving-cert\""
Jul 24 15:36:55.437064 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:55.437037    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tkn9s\""
Jul 24 15:36:55.439477 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:55.439445    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e06cce3-ea3b-4ae8-95b9-9b59223ab630-openshift-service-ca\") pod \"maas-api-e2e-migrate-70e683b2-575f5b4b5-xjscl\" (UID: \"9e06cce3-ea3b-4ae8-95b9-9b59223ab630\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70e683b2-575f5b4b5-xjscl"
Jul 24 15:36:55.439610 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:55.439494    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t76s9\" (UniqueName: \"kubernetes.io/projected/9e06cce3-ea3b-4ae8-95b9-9b59223ab630-kube-api-access-t76s9\") pod \"maas-api-e2e-migrate-70e683b2-575f5b4b5-xjscl\" (UID: \"9e06cce3-ea3b-4ae8-95b9-9b59223ab630\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70e683b2-575f5b4b5-xjscl"
Jul 24 15:36:55.439610 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:55.439550    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e06cce3-ea3b-4ae8-95b9-9b59223ab630-maas-api-tls\") pod \"maas-api-e2e-migrate-70e683b2-575f5b4b5-xjscl\" (UID: \"9e06cce3-ea3b-4ae8-95b9-9b59223ab630\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70e683b2-575f5b4b5-xjscl"
Jul 24 15:36:55.439886 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:55.439862    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-70e683b2-575f5b4b5-xjscl"]
Jul 24 15:36:55.540463 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:55.540423    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e06cce3-ea3b-4ae8-95b9-9b59223ab630-openshift-service-ca\") pod \"maas-api-e2e-migrate-70e683b2-575f5b4b5-xjscl\" (UID: \"9e06cce3-ea3b-4ae8-95b9-9b59223ab630\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70e683b2-575f5b4b5-xjscl"
Jul 24 15:36:55.540463 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:55.540468    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t76s9\" (UniqueName: \"kubernetes.io/projected/9e06cce3-ea3b-4ae8-95b9-9b59223ab630-kube-api-access-t76s9\") pod \"maas-api-e2e-migrate-70e683b2-575f5b4b5-xjscl\" (UID: \"9e06cce3-ea3b-4ae8-95b9-9b59223ab630\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70e683b2-575f5b4b5-xjscl"
Jul 24 15:36:55.540685 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:55.540494    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e06cce3-ea3b-4ae8-95b9-9b59223ab630-maas-api-tls\") pod \"maas-api-e2e-migrate-70e683b2-575f5b4b5-xjscl\" (UID: \"9e06cce3-ea3b-4ae8-95b9-9b59223ab630\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70e683b2-575f5b4b5-xjscl"
Jul 24 15:36:55.540685 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:36:55.540595    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-70e683b2-serving-cert: secret "maas-api-e2e-migrate-70e683b2-serving-cert" not found
Jul 24 15:36:55.540685 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:36:55.540658    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e06cce3-ea3b-4ae8-95b9-9b59223ab630-maas-api-tls podName:9e06cce3-ea3b-4ae8-95b9-9b59223ab630 nodeName:}" failed. No retries permitted until 2026-07-24 15:36:56.040642757 +0000 UTC m=+1708.852994915 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9e06cce3-ea3b-4ae8-95b9-9b59223ab630-maas-api-tls") pod "maas-api-e2e-migrate-70e683b2-575f5b4b5-xjscl" (UID: "9e06cce3-ea3b-4ae8-95b9-9b59223ab630") : secret "maas-api-e2e-migrate-70e683b2-serving-cert" not found
Jul 24 15:36:55.541161 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:55.541143    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e06cce3-ea3b-4ae8-95b9-9b59223ab630-openshift-service-ca\") pod \"maas-api-e2e-migrate-70e683b2-575f5b4b5-xjscl\" (UID: \"9e06cce3-ea3b-4ae8-95b9-9b59223ab630\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70e683b2-575f5b4b5-xjscl"
Jul 24 15:36:55.559183 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:55.559158    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t76s9\" (UniqueName: \"kubernetes.io/projected/9e06cce3-ea3b-4ae8-95b9-9b59223ab630-kube-api-access-t76s9\") pod \"maas-api-e2e-migrate-70e683b2-575f5b4b5-xjscl\" (UID: \"9e06cce3-ea3b-4ae8-95b9-9b59223ab630\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70e683b2-575f5b4b5-xjscl"
Jul 24 15:36:56.043716 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:56.043665    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e06cce3-ea3b-4ae8-95b9-9b59223ab630-maas-api-tls\") pod \"maas-api-e2e-migrate-70e683b2-575f5b4b5-xjscl\" (UID: \"9e06cce3-ea3b-4ae8-95b9-9b59223ab630\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70e683b2-575f5b4b5-xjscl"
Jul 24 15:36:56.043949 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:36:56.043839    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-70e683b2-serving-cert: secret "maas-api-e2e-migrate-70e683b2-serving-cert" not found
Jul 24 15:36:56.043949 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:36:56.043903    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e06cce3-ea3b-4ae8-95b9-9b59223ab630-maas-api-tls podName:9e06cce3-ea3b-4ae8-95b9-9b59223ab630 nodeName:}" failed. No retries permitted until 2026-07-24 15:36:57.043887769 +0000 UTC m=+1709.856239925 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9e06cce3-ea3b-4ae8-95b9-9b59223ab630-maas-api-tls") pod "maas-api-e2e-migrate-70e683b2-575f5b4b5-xjscl" (UID: "9e06cce3-ea3b-4ae8-95b9-9b59223ab630") : secret "maas-api-e2e-migrate-70e683b2-serving-cert" not found
Jul 24 15:36:56.294189 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:56.294106    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-70e683b2-578b8d4d8-md5dw"]
Jul 24 15:36:56.296804 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:56.296785    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-70e683b2-578b8d4d8-md5dw"
Jul 24 15:36:56.298968 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:56.298943    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-70e683b2\""
Jul 24 15:36:56.302940 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:56.302916    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-70e683b2-66676849bf-pncxs"]
Jul 24 15:36:56.305291 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:56.305274    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-70e683b2-66676849bf-pncxs"
Jul 24 15:36:56.307811 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:56.307788    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-70e683b2-dockercfg-gl49k\""
Jul 24 15:36:56.310732 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:56.310710    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-70e683b2-578b8d4d8-md5dw"]
Jul 24 15:36:56.324571 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:56.324540    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-70e683b2-66676849bf-pncxs"]
Jul 24 15:36:56.347989 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:56.347937    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60cbb234-68c6-4c72-896b-aac787f2f990-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-70e683b2-578b8d4d8-md5dw\" (UID: \"60cbb234-68c6-4c72-896b-aac787f2f990\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-70e683b2-578b8d4d8-md5dw"
Jul 24 15:36:56.348159 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:56.347998    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p8kh\" (UniqueName: \"kubernetes.io/projected/11768387-9fd5-439e-9842-00176bb740d3-kube-api-access-5p8kh\") pod \"payload-processing-e2e-migrate-70e683b2-66676849bf-pncxs\" (UID: \"11768387-9fd5-439e-9842-00176bb740d3\") " pod="openshift-ingress/payload-processing-e2e-migrate-70e683b2-66676849bf-pncxs"
Jul 24 15:36:56.348159 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:56.348074    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2gjz\" (UniqueName: \"kubernetes.io/projected/60cbb234-68c6-4c72-896b-aac787f2f990-kube-api-access-p2gjz\") pod \"payload-pre-processing-e2e-migrate-70e683b2-578b8d4d8-md5dw\" (UID: \"60cbb234-68c6-4c72-896b-aac787f2f990\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-70e683b2-578b8d4d8-md5dw"
Jul 24 15:36:56.348231 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:56.348215    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11768387-9fd5-439e-9842-00176bb740d3-plugins-config-volume\") pod \"payload-processing-e2e-migrate-70e683b2-66676849bf-pncxs\" (UID: \"11768387-9fd5-439e-9842-00176bb740d3\") " pod="openshift-ingress/payload-processing-e2e-migrate-70e683b2-66676849bf-pncxs"
Jul 24 15:36:56.449603 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:56.449568    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11768387-9fd5-439e-9842-00176bb740d3-plugins-config-volume\") pod \"payload-processing-e2e-migrate-70e683b2-66676849bf-pncxs\" (UID: \"11768387-9fd5-439e-9842-00176bb740d3\") " pod="openshift-ingress/payload-processing-e2e-migrate-70e683b2-66676849bf-pncxs"
Jul 24 15:36:56.450020 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:56.449700    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60cbb234-68c6-4c72-896b-aac787f2f990-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-70e683b2-578b8d4d8-md5dw\" (UID: \"60cbb234-68c6-4c72-896b-aac787f2f990\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-70e683b2-578b8d4d8-md5dw"
Jul 24 15:36:56.450020 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:56.449742    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p8kh\" (UniqueName: \"kubernetes.io/projected/11768387-9fd5-439e-9842-00176bb740d3-kube-api-access-5p8kh\") pod \"payload-processing-e2e-migrate-70e683b2-66676849bf-pncxs\" (UID: \"11768387-9fd5-439e-9842-00176bb740d3\") " pod="openshift-ingress/payload-processing-e2e-migrate-70e683b2-66676849bf-pncxs"
Jul 24 15:36:56.450020 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:56.449786    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2gjz\" (UniqueName: \"kubernetes.io/projected/60cbb234-68c6-4c72-896b-aac787f2f990-kube-api-access-p2gjz\") pod \"payload-pre-processing-e2e-migrate-70e683b2-578b8d4d8-md5dw\" (UID: \"60cbb234-68c6-4c72-896b-aac787f2f990\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-70e683b2-578b8d4d8-md5dw"
Jul 24 15:36:56.450401 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:56.450381    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11768387-9fd5-439e-9842-00176bb740d3-plugins-config-volume\") pod \"payload-processing-e2e-migrate-70e683b2-66676849bf-pncxs\" (UID: \"11768387-9fd5-439e-9842-00176bb740d3\") " pod="openshift-ingress/payload-processing-e2e-migrate-70e683b2-66676849bf-pncxs"
Jul 24 15:36:56.450443 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:56.450422    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60cbb234-68c6-4c72-896b-aac787f2f990-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-70e683b2-578b8d4d8-md5dw\" (UID: \"60cbb234-68c6-4c72-896b-aac787f2f990\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-70e683b2-578b8d4d8-md5dw"
Jul 24 15:36:56.458439 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:56.458398    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2gjz\" (UniqueName: \"kubernetes.io/projected/60cbb234-68c6-4c72-896b-aac787f2f990-kube-api-access-p2gjz\") pod \"payload-pre-processing-e2e-migrate-70e683b2-578b8d4d8-md5dw\" (UID: \"60cbb234-68c6-4c72-896b-aac787f2f990\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-70e683b2-578b8d4d8-md5dw"
Jul 24 15:36:56.458576 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:56.458561    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p8kh\" (UniqueName: \"kubernetes.io/projected/11768387-9fd5-439e-9842-00176bb740d3-kube-api-access-5p8kh\") pod \"payload-processing-e2e-migrate-70e683b2-66676849bf-pncxs\" (UID: \"11768387-9fd5-439e-9842-00176bb740d3\") " pod="openshift-ingress/payload-processing-e2e-migrate-70e683b2-66676849bf-pncxs"
Jul 24 15:36:56.608920 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:56.608802    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-70e683b2-578b8d4d8-md5dw"
Jul 24 15:36:56.619438 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:56.619411    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-70e683b2-66676849bf-pncxs"
Jul 24 15:36:56.772379 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:56.768783    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-70e683b2-578b8d4d8-md5dw"]
Jul 24 15:36:56.775592 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:36:56.775557    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60cbb234_68c6_4c72_896b_aac787f2f990.slice/crio-664931fdf23146598061cfce1b84ead0a0e01a146ed1aa0b96845c6d142d33f7 WatchSource:0}: Error finding container 664931fdf23146598061cfce1b84ead0a0e01a146ed1aa0b96845c6d142d33f7: Status 404 returned error can't find the container with id 664931fdf23146598061cfce1b84ead0a0e01a146ed1aa0b96845c6d142d33f7
Jul 24 15:36:56.797532 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:56.797498    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-70e683b2-66676849bf-pncxs"]
Jul 24 15:36:56.800882 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:36:56.800848    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11768387_9fd5_439e_9842_00176bb740d3.slice/crio-800a3585b6702e4a97f8c0cf975c251bed84c122044b474062adb99dd8cf2559 WatchSource:0}: Error finding container 800a3585b6702e4a97f8c0cf975c251bed84c122044b474062adb99dd8cf2559: Status 404 returned error can't find the container with id 800a3585b6702e4a97f8c0cf975c251bed84c122044b474062adb99dd8cf2559
Jul 24 15:36:57.056548 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:57.056508    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e06cce3-ea3b-4ae8-95b9-9b59223ab630-maas-api-tls\") pod \"maas-api-e2e-migrate-70e683b2-575f5b4b5-xjscl\" (UID: \"9e06cce3-ea3b-4ae8-95b9-9b59223ab630\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70e683b2-575f5b4b5-xjscl"
Jul 24 15:36:57.058888 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:57.058867    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e06cce3-ea3b-4ae8-95b9-9b59223ab630-maas-api-tls\") pod \"maas-api-e2e-migrate-70e683b2-575f5b4b5-xjscl\" (UID: \"9e06cce3-ea3b-4ae8-95b9-9b59223ab630\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70e683b2-575f5b4b5-xjscl"
Jul 24 15:36:57.226304 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:57.226265    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70e683b2-575f5b4b5-xjscl"
Jul 24 15:36:57.367219 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:57.367080    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-70e683b2-575f5b4b5-xjscl"]
Jul 24 15:36:57.370060 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:36:57.370026    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e06cce3_ea3b_4ae8_95b9_9b59223ab630.slice/crio-e0af5596a37e7fdb3b65882c7744f16cc9f11efd6f5d0b451636750f3fcf912d WatchSource:0}: Error finding container e0af5596a37e7fdb3b65882c7744f16cc9f11efd6f5d0b451636750f3fcf912d: Status 404 returned error can't find the container with id e0af5596a37e7fdb3b65882c7744f16cc9f11efd6f5d0b451636750f3fcf912d
Jul 24 15:36:57.653892 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:57.653784    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-70e683b2-66676849bf-pncxs" event={"ID":"11768387-9fd5-439e-9842-00176bb740d3","Type":"ContainerStarted","Data":"6e3ee47c408916fd7063a6a900f57cc0acf059f38d6f55bead847350fbe1cfb0"}
Jul 24 15:36:57.653892 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:57.653861    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-70e683b2-66676849bf-pncxs" event={"ID":"11768387-9fd5-439e-9842-00176bb740d3","Type":"ContainerStarted","Data":"800a3585b6702e4a97f8c0cf975c251bed84c122044b474062adb99dd8cf2559"}
Jul 24 15:36:57.654399 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:57.653983    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-70e683b2-66676849bf-pncxs"
Jul 24 15:36:57.655007 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:57.654984    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70e683b2-575f5b4b5-xjscl" event={"ID":"9e06cce3-ea3b-4ae8-95b9-9b59223ab630","Type":"ContainerStarted","Data":"e0af5596a37e7fdb3b65882c7744f16cc9f11efd6f5d0b451636750f3fcf912d"}
Jul 24 15:36:57.656390 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:57.656367    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-70e683b2-578b8d4d8-md5dw" event={"ID":"60cbb234-68c6-4c72-896b-aac787f2f990","Type":"ContainerStarted","Data":"12858b078d95eaf0d7640d5573dfdfc96fda806fed65a16fdb23a2c2c73a7672"}
Jul 24 15:36:57.656490 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:57.656407    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-70e683b2-578b8d4d8-md5dw" event={"ID":"60cbb234-68c6-4c72-896b-aac787f2f990","Type":"ContainerStarted","Data":"664931fdf23146598061cfce1b84ead0a0e01a146ed1aa0b96845c6d142d33f7"}
Jul 24 15:36:57.656490 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:57.656428    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-70e683b2-578b8d4d8-md5dw"
Jul 24 15:36:57.673076 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:57.673025    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-70e683b2-66676849bf-pncxs" podStartSLOduration=1.673008219 podStartE2EDuration="1.673008219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:36:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:36:57.672463573 +0000 UTC m=+1710.484815753" watchObservedRunningTime="2026-07-24 15:36:57.673008219 +0000 UTC m=+1710.485360414"
Jul 24 15:36:57.698118 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:36:57.698067    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-70e683b2-578b8d4d8-md5dw" podStartSLOduration=1.6980543510000001 podStartE2EDuration="1.698054351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:36:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:36:57.696912821 +0000 UTC m=+1710.509265004" watchObservedRunningTime="2026-07-24 15:36:57.698054351 +0000 UTC m=+1710.510406521"
Jul 24 15:37:00.674799 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:00.674764    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70e683b2-575f5b4b5-xjscl" event={"ID":"9e06cce3-ea3b-4ae8-95b9-9b59223ab630","Type":"ContainerStarted","Data":"15c2ab9fd733e73029d991bf85b6bad7d09d73063295ae487a72b7a534325ddb"}
Jul 24 15:37:00.675278 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:00.674847    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70e683b2-575f5b4b5-xjscl"
Jul 24 15:37:00.698225 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:00.698166    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70e683b2-575f5b4b5-xjscl" podStartSLOduration=3.218174135 podStartE2EDuration="5.698127983s" totalImagesPullingTime="2.479953848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:36:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:37:00.696782206 +0000 UTC m=+1713.509134382" watchObservedRunningTime="2026-07-24 15:37:00.698127983 +0000 UTC m=+1713.510480161"
Jul 24 15:37:01.969523 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:01.969486    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:37:06.686025 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:06.685994    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70e683b2-575f5b4b5-xjscl"
Jul 24 15:37:08.664421 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:08.664385    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-70e683b2-578b8d4d8-md5dw"
Jul 24 15:37:08.664802 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:08.664462    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-70e683b2-66676849bf-pncxs"
Jul 24 15:37:30.926783 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:30.926747    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-70e683b2-575f5b4b5-xjscl"]
Jul 24 15:37:30.927229 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:30.926986    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70e683b2-575f5b4b5-xjscl" podUID="9e06cce3-ea3b-4ae8-95b9-9b59223ab630" containerName="maas-api" containerID="cri-o://15c2ab9fd733e73029d991bf85b6bad7d09d73063295ae487a72b7a534325ddb" gracePeriod=30
Jul 24 15:37:30.993327 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:30.993293    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-70e683b2-66676849bf-pncxs"]
Jul 24 15:37:30.993542 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:30.993504    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-70e683b2-66676849bf-pncxs" podUID="11768387-9fd5-439e-9842-00176bb740d3" containerName="payload-processing" containerID="cri-o://6e3ee47c408916fd7063a6a900f57cc0acf059f38d6f55bead847350fbe1cfb0" gracePeriod=30
Jul 24 15:37:31.001887 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.001862    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-70e683b2-578b8d4d8-md5dw"]
Jul 24 15:37:31.002104 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.002060    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-70e683b2-578b8d4d8-md5dw" podUID="60cbb234-68c6-4c72-896b-aac787f2f990" containerName="payload-pre-processing" containerID="cri-o://12858b078d95eaf0d7640d5573dfdfc96fda806fed65a16fdb23a2c2c73a7672" gracePeriod=30
Jul 24 15:37:31.253605 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.253578    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70e683b2-575f5b4b5-xjscl"
Jul 24 15:37:31.279518 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.279404    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e06cce3-ea3b-4ae8-95b9-9b59223ab630-maas-api-tls\") pod \"9e06cce3-ea3b-4ae8-95b9-9b59223ab630\" (UID: \"9e06cce3-ea3b-4ae8-95b9-9b59223ab630\") "
Jul 24 15:37:31.279518 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.279478    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t76s9\" (UniqueName: \"kubernetes.io/projected/9e06cce3-ea3b-4ae8-95b9-9b59223ab630-kube-api-access-t76s9\") pod \"9e06cce3-ea3b-4ae8-95b9-9b59223ab630\" (UID: \"9e06cce3-ea3b-4ae8-95b9-9b59223ab630\") "
Jul 24 15:37:31.279803 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.279527    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e06cce3-ea3b-4ae8-95b9-9b59223ab630-openshift-service-ca\") pod \"9e06cce3-ea3b-4ae8-95b9-9b59223ab630\" (UID: \"9e06cce3-ea3b-4ae8-95b9-9b59223ab630\") "
Jul 24 15:37:31.280328 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.280298    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e06cce3-ea3b-4ae8-95b9-9b59223ab630-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9e06cce3-ea3b-4ae8-95b9-9b59223ab630" (UID: "9e06cce3-ea3b-4ae8-95b9-9b59223ab630"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:37:31.280484 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.280464    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-70e683b2-578b8d4d8-md5dw"
Jul 24 15:37:31.282301 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.282198    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e06cce3-ea3b-4ae8-95b9-9b59223ab630-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9e06cce3-ea3b-4ae8-95b9-9b59223ab630" (UID: "9e06cce3-ea3b-4ae8-95b9-9b59223ab630"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 15:37:31.282301 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.282271    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e06cce3-ea3b-4ae8-95b9-9b59223ab630-kube-api-access-t76s9" (OuterVolumeSpecName: "kube-api-access-t76s9") pod "9e06cce3-ea3b-4ae8-95b9-9b59223ab630" (UID: "9e06cce3-ea3b-4ae8-95b9-9b59223ab630"). InnerVolumeSpecName "kube-api-access-t76s9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:37:31.290191 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.290169    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-70e683b2-66676849bf-pncxs"
Jul 24 15:37:31.380580 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.380547    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60cbb234-68c6-4c72-896b-aac787f2f990-plugins-config-volume\") pod \"60cbb234-68c6-4c72-896b-aac787f2f990\" (UID: \"60cbb234-68c6-4c72-896b-aac787f2f990\") "
Jul 24 15:37:31.380756 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.380599    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2gjz\" (UniqueName: \"kubernetes.io/projected/60cbb234-68c6-4c72-896b-aac787f2f990-kube-api-access-p2gjz\") pod \"60cbb234-68c6-4c72-896b-aac787f2f990\" (UID: \"60cbb234-68c6-4c72-896b-aac787f2f990\") "
Jul 24 15:37:31.380756 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.380720    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5p8kh\" (UniqueName: \"kubernetes.io/projected/11768387-9fd5-439e-9842-00176bb740d3-kube-api-access-5p8kh\") pod \"11768387-9fd5-439e-9842-00176bb740d3\" (UID: \"11768387-9fd5-439e-9842-00176bb740d3\") "
Jul 24 15:37:31.380887 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.380871    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11768387-9fd5-439e-9842-00176bb740d3-plugins-config-volume\") pod \"11768387-9fd5-439e-9842-00176bb740d3\" (UID: \"11768387-9fd5-439e-9842-00176bb740d3\") "
Jul 24 15:37:31.381029 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.381004    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60cbb234-68c6-4c72-896b-aac787f2f990-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "60cbb234-68c6-4c72-896b-aac787f2f990" (UID: "60cbb234-68c6-4c72-896b-aac787f2f990"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:37:31.381245 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.381226    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e06cce3-ea3b-4ae8-95b9-9b59223ab630-maas-api-tls\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:37:31.381245 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.381246    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60cbb234-68c6-4c72-896b-aac787f2f990-plugins-config-volume\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:37:31.381369 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.381244    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11768387-9fd5-439e-9842-00176bb740d3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "11768387-9fd5-439e-9842-00176bb740d3" (UID: "11768387-9fd5-439e-9842-00176bb740d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:37:31.381369 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.381256    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t76s9\" (UniqueName: \"kubernetes.io/projected/9e06cce3-ea3b-4ae8-95b9-9b59223ab630-kube-api-access-t76s9\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:37:31.381369 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.381299    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e06cce3-ea3b-4ae8-95b9-9b59223ab630-openshift-service-ca\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:37:31.382707 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.382683    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60cbb234-68c6-4c72-896b-aac787f2f990-kube-api-access-p2gjz" (OuterVolumeSpecName: "kube-api-access-p2gjz") pod "60cbb234-68c6-4c72-896b-aac787f2f990" (UID: "60cbb234-68c6-4c72-896b-aac787f2f990"). InnerVolumeSpecName "kube-api-access-p2gjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:37:31.382759 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.382712    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11768387-9fd5-439e-9842-00176bb740d3-kube-api-access-5p8kh" (OuterVolumeSpecName: "kube-api-access-5p8kh") pod "11768387-9fd5-439e-9842-00176bb740d3" (UID: "11768387-9fd5-439e-9842-00176bb740d3"). InnerVolumeSpecName "kube-api-access-5p8kh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:37:31.482093 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.481997    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11768387-9fd5-439e-9842-00176bb740d3-plugins-config-volume\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:37:31.482093 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.482033    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2gjz\" (UniqueName: \"kubernetes.io/projected/60cbb234-68c6-4c72-896b-aac787f2f990-kube-api-access-p2gjz\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:37:31.482093 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.482047    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p8kh\" (UniqueName: \"kubernetes.io/projected/11768387-9fd5-439e-9842-00176bb740d3-kube-api-access-5p8kh\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:37:31.827058 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.827019    2579 generic.go:356] "Generic (PLEG): container finished" podID="11768387-9fd5-439e-9842-00176bb740d3" containerID="6e3ee47c408916fd7063a6a900f57cc0acf059f38d6f55bead847350fbe1cfb0" exitCode=0
Jul 24 15:37:31.827214 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.827066    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-70e683b2-66676849bf-pncxs"
Jul 24 15:37:31.827214 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.827104    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-70e683b2-66676849bf-pncxs" event={"ID":"11768387-9fd5-439e-9842-00176bb740d3","Type":"ContainerDied","Data":"6e3ee47c408916fd7063a6a900f57cc0acf059f38d6f55bead847350fbe1cfb0"}
Jul 24 15:37:31.827214 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.827159    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-70e683b2-66676849bf-pncxs" event={"ID":"11768387-9fd5-439e-9842-00176bb740d3","Type":"ContainerDied","Data":"800a3585b6702e4a97f8c0cf975c251bed84c122044b474062adb99dd8cf2559"}
Jul 24 15:37:31.827214 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.827189    2579 scope.go:117] "RemoveContainer" containerID="6e3ee47c408916fd7063a6a900f57cc0acf059f38d6f55bead847350fbe1cfb0"
Jul 24 15:37:31.828520 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.828499    2579 generic.go:356] "Generic (PLEG): container finished" podID="9e06cce3-ea3b-4ae8-95b9-9b59223ab630" containerID="15c2ab9fd733e73029d991bf85b6bad7d09d73063295ae487a72b7a534325ddb" exitCode=0
Jul 24 15:37:31.828633 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.828528    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70e683b2-575f5b4b5-xjscl" event={"ID":"9e06cce3-ea3b-4ae8-95b9-9b59223ab630","Type":"ContainerDied","Data":"15c2ab9fd733e73029d991bf85b6bad7d09d73063295ae487a72b7a534325ddb"}
Jul 24 15:37:31.828633 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.828566    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70e683b2-575f5b4b5-xjscl"
Jul 24 15:37:31.828736 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.828572    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-70e683b2-575f5b4b5-xjscl" event={"ID":"9e06cce3-ea3b-4ae8-95b9-9b59223ab630","Type":"ContainerDied","Data":"e0af5596a37e7fdb3b65882c7744f16cc9f11efd6f5d0b451636750f3fcf912d"}
Jul 24 15:37:31.830225 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.830198    2579 generic.go:356] "Generic (PLEG): container finished" podID="60cbb234-68c6-4c72-896b-aac787f2f990" containerID="12858b078d95eaf0d7640d5573dfdfc96fda806fed65a16fdb23a2c2c73a7672" exitCode=0
Jul 24 15:37:31.830315 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.830237    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-70e683b2-578b8d4d8-md5dw"
Jul 24 15:37:31.830315 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.830237    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-70e683b2-578b8d4d8-md5dw" event={"ID":"60cbb234-68c6-4c72-896b-aac787f2f990","Type":"ContainerDied","Data":"12858b078d95eaf0d7640d5573dfdfc96fda806fed65a16fdb23a2c2c73a7672"}
Jul 24 15:37:31.830395 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.830339    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-70e683b2-578b8d4d8-md5dw" event={"ID":"60cbb234-68c6-4c72-896b-aac787f2f990","Type":"ContainerDied","Data":"664931fdf23146598061cfce1b84ead0a0e01a146ed1aa0b96845c6d142d33f7"}
Jul 24 15:37:31.837036 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.837021    2579 scope.go:117] "RemoveContainer" containerID="6e3ee47c408916fd7063a6a900f57cc0acf059f38d6f55bead847350fbe1cfb0"
Jul 24 15:37:31.837282 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:37:31.837262    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e3ee47c408916fd7063a6a900f57cc0acf059f38d6f55bead847350fbe1cfb0\": container with ID starting with 6e3ee47c408916fd7063a6a900f57cc0acf059f38d6f55bead847350fbe1cfb0 not found: ID does not exist" containerID="6e3ee47c408916fd7063a6a900f57cc0acf059f38d6f55bead847350fbe1cfb0"
Jul 24 15:37:31.837323 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.837290    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e3ee47c408916fd7063a6a900f57cc0acf059f38d6f55bead847350fbe1cfb0"} err="failed to get container status \"6e3ee47c408916fd7063a6a900f57cc0acf059f38d6f55bead847350fbe1cfb0\": rpc error: code = NotFound desc = could not find container \"6e3ee47c408916fd7063a6a900f57cc0acf059f38d6f55bead847350fbe1cfb0\": container with ID starting with 6e3ee47c408916fd7063a6a900f57cc0acf059f38d6f55bead847350fbe1cfb0 not found: ID does not exist"
Jul 24 15:37:31.837323 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.837307    2579 scope.go:117] "RemoveContainer" containerID="15c2ab9fd733e73029d991bf85b6bad7d09d73063295ae487a72b7a534325ddb"
Jul 24 15:37:31.844752 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.844735    2579 scope.go:117] "RemoveContainer" containerID="15c2ab9fd733e73029d991bf85b6bad7d09d73063295ae487a72b7a534325ddb"
Jul 24 15:37:31.845002 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:37:31.844985    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15c2ab9fd733e73029d991bf85b6bad7d09d73063295ae487a72b7a534325ddb\": container with ID starting with 15c2ab9fd733e73029d991bf85b6bad7d09d73063295ae487a72b7a534325ddb not found: ID does not exist" containerID="15c2ab9fd733e73029d991bf85b6bad7d09d73063295ae487a72b7a534325ddb"
Jul 24 15:37:31.845070 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.845010    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15c2ab9fd733e73029d991bf85b6bad7d09d73063295ae487a72b7a534325ddb"} err="failed to get container status \"15c2ab9fd733e73029d991bf85b6bad7d09d73063295ae487a72b7a534325ddb\": rpc error: code = NotFound desc = could not find container \"15c2ab9fd733e73029d991bf85b6bad7d09d73063295ae487a72b7a534325ddb\": container with ID starting with 15c2ab9fd733e73029d991bf85b6bad7d09d73063295ae487a72b7a534325ddb not found: ID does not exist"
Jul 24 15:37:31.845070 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.845031    2579 scope.go:117] "RemoveContainer" containerID="12858b078d95eaf0d7640d5573dfdfc96fda806fed65a16fdb23a2c2c73a7672"
Jul 24 15:37:31.852842 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.852811    2579 scope.go:117] "RemoveContainer" containerID="12858b078d95eaf0d7640d5573dfdfc96fda806fed65a16fdb23a2c2c73a7672"
Jul 24 15:37:31.853071 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.853053    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-70e683b2-66676849bf-pncxs"]
Jul 24 15:37:31.853116 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:37:31.853076    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12858b078d95eaf0d7640d5573dfdfc96fda806fed65a16fdb23a2c2c73a7672\": container with ID starting with 12858b078d95eaf0d7640d5573dfdfc96fda806fed65a16fdb23a2c2c73a7672 not found: ID does not exist" containerID="12858b078d95eaf0d7640d5573dfdfc96fda806fed65a16fdb23a2c2c73a7672"
Jul 24 15:37:31.853116 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.853094    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12858b078d95eaf0d7640d5573dfdfc96fda806fed65a16fdb23a2c2c73a7672"} err="failed to get container status \"12858b078d95eaf0d7640d5573dfdfc96fda806fed65a16fdb23a2c2c73a7672\": rpc error: code = NotFound desc = could not find container \"12858b078d95eaf0d7640d5573dfdfc96fda806fed65a16fdb23a2c2c73a7672\": container with ID starting with 12858b078d95eaf0d7640d5573dfdfc96fda806fed65a16fdb23a2c2c73a7672 not found: ID does not exist"
Jul 24 15:37:31.859051 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.859029    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-70e683b2-66676849bf-pncxs"]
Jul 24 15:37:31.870420 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.870400    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-70e683b2-578b8d4d8-md5dw"]
Jul 24 15:37:31.878866 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.878844    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-70e683b2-578b8d4d8-md5dw"]
Jul 24 15:37:31.894010 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.893981    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-70e683b2-575f5b4b5-xjscl"]
Jul 24 15:37:31.901080 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:31.901058    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-70e683b2-575f5b4b5-xjscl"]
Jul 24 15:37:33.718026 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:33.717989    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11768387-9fd5-439e-9842-00176bb740d3" path="/var/lib/kubelet/pods/11768387-9fd5-439e-9842-00176bb740d3/volumes"
Jul 24 15:37:33.718405 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:33.718336    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60cbb234-68c6-4c72-896b-aac787f2f990" path="/var/lib/kubelet/pods/60cbb234-68c6-4c72-896b-aac787f2f990/volumes"
Jul 24 15:37:33.718665 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:33.718651    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e06cce3-ea3b-4ae8-95b9-9b59223ab630" path="/var/lib/kubelet/pods/9e06cce3-ea3b-4ae8-95b9-9b59223ab630/volumes"
Jul 24 15:37:34.031890 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:34.031854    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:37:39.541763 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:39.541728    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-26f33d00-7b8d6755cc-hk6fh"]
Jul 24 15:37:39.542383 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:39.542306    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11768387-9fd5-439e-9842-00176bb740d3" containerName="payload-processing"
Jul 24 15:37:39.542383 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:39.542328    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="11768387-9fd5-439e-9842-00176bb740d3" containerName="payload-processing"
Jul 24 15:37:39.542383 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:39.542355    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60cbb234-68c6-4c72-896b-aac787f2f990" containerName="payload-pre-processing"
Jul 24 15:37:39.542383 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:39.542363    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="60cbb234-68c6-4c72-896b-aac787f2f990" containerName="payload-pre-processing"
Jul 24 15:37:39.542593 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:39.542389    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e06cce3-ea3b-4ae8-95b9-9b59223ab630" containerName="maas-api"
Jul 24 15:37:39.542593 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:39.542399    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e06cce3-ea3b-4ae8-95b9-9b59223ab630" containerName="maas-api"
Jul 24 15:37:39.542593 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:39.542518    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="11768387-9fd5-439e-9842-00176bb740d3" containerName="payload-processing"
Jul 24 15:37:39.542593 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:39.542533    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e06cce3-ea3b-4ae8-95b9-9b59223ab630" containerName="maas-api"
Jul 24 15:37:39.542593 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:39.542543    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="60cbb234-68c6-4c72-896b-aac787f2f990" containerName="payload-pre-processing"
Jul 24 15:37:39.547326 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:39.547305    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-26f33d00-7b8d6755cc-hk6fh"
Jul 24 15:37:39.555538 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:39.555518    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-26f33d00-serving-cert\""
Jul 24 15:37:39.558070 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:39.558052    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 15:37:39.558154 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:39.558058    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tkn9s\""
Jul 24 15:37:39.558317 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:39.558302    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 15:37:39.582531 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:39.582498    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-26f33d00-7b8d6755cc-hk6fh"]
Jul 24 15:37:39.661256 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:39.661219    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c6df170-4234-4658-907c-3d4b7c4fcc7e-maas-api-tls\") pod \"maas-api-e2e-ait-26f33d00-7b8d6755cc-hk6fh\" (UID: \"7c6df170-4234-4658-907c-3d4b7c4fcc7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-26f33d00-7b8d6755cc-hk6fh"
Jul 24 15:37:39.661256 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:39.661258    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c6df170-4234-4658-907c-3d4b7c4fcc7e-openshift-service-ca\") pod \"maas-api-e2e-ait-26f33d00-7b8d6755cc-hk6fh\" (UID: \"7c6df170-4234-4658-907c-3d4b7c4fcc7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-26f33d00-7b8d6755cc-hk6fh"
Jul 24 15:37:39.661501 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:39.661437    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txx29\" (UniqueName: \"kubernetes.io/projected/7c6df170-4234-4658-907c-3d4b7c4fcc7e-kube-api-access-txx29\") pod \"maas-api-e2e-ait-26f33d00-7b8d6755cc-hk6fh\" (UID: \"7c6df170-4234-4658-907c-3d4b7c4fcc7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-26f33d00-7b8d6755cc-hk6fh"
Jul 24 15:37:39.762726 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:39.762687    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txx29\" (UniqueName: \"kubernetes.io/projected/7c6df170-4234-4658-907c-3d4b7c4fcc7e-kube-api-access-txx29\") pod \"maas-api-e2e-ait-26f33d00-7b8d6755cc-hk6fh\" (UID: \"7c6df170-4234-4658-907c-3d4b7c4fcc7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-26f33d00-7b8d6755cc-hk6fh"
Jul 24 15:37:39.762942 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:39.762761    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c6df170-4234-4658-907c-3d4b7c4fcc7e-maas-api-tls\") pod \"maas-api-e2e-ait-26f33d00-7b8d6755cc-hk6fh\" (UID: \"7c6df170-4234-4658-907c-3d4b7c4fcc7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-26f33d00-7b8d6755cc-hk6fh"
Jul 24 15:37:39.762942 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:39.762787    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c6df170-4234-4658-907c-3d4b7c4fcc7e-openshift-service-ca\") pod \"maas-api-e2e-ait-26f33d00-7b8d6755cc-hk6fh\" (UID: \"7c6df170-4234-4658-907c-3d4b7c4fcc7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-26f33d00-7b8d6755cc-hk6fh"
Jul 24 15:37:39.762942 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:37:39.762890    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-26f33d00-serving-cert: secret "maas-api-e2e-ait-26f33d00-serving-cert" not found
Jul 24 15:37:39.763127 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:37:39.762978    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c6df170-4234-4658-907c-3d4b7c4fcc7e-maas-api-tls podName:7c6df170-4234-4658-907c-3d4b7c4fcc7e nodeName:}" failed. No retries permitted until 2026-07-24 15:37:40.262955111 +0000 UTC m=+1753.075307270 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7c6df170-4234-4658-907c-3d4b7c4fcc7e-maas-api-tls") pod "maas-api-e2e-ait-26f33d00-7b8d6755cc-hk6fh" (UID: "7c6df170-4234-4658-907c-3d4b7c4fcc7e") : secret "maas-api-e2e-ait-26f33d00-serving-cert" not found
Jul 24 15:37:39.763486 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:39.763459    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c6df170-4234-4658-907c-3d4b7c4fcc7e-openshift-service-ca\") pod \"maas-api-e2e-ait-26f33d00-7b8d6755cc-hk6fh\" (UID: \"7c6df170-4234-4658-907c-3d4b7c4fcc7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-26f33d00-7b8d6755cc-hk6fh"
Jul 24 15:37:39.772026 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:39.772005    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txx29\" (UniqueName: \"kubernetes.io/projected/7c6df170-4234-4658-907c-3d4b7c4fcc7e-kube-api-access-txx29\") pod \"maas-api-e2e-ait-26f33d00-7b8d6755cc-hk6fh\" (UID: \"7c6df170-4234-4658-907c-3d4b7c4fcc7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-26f33d00-7b8d6755cc-hk6fh"
Jul 24 15:37:40.268726 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:40.268687    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c6df170-4234-4658-907c-3d4b7c4fcc7e-maas-api-tls\") pod \"maas-api-e2e-ait-26f33d00-7b8d6755cc-hk6fh\" (UID: \"7c6df170-4234-4658-907c-3d4b7c4fcc7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-26f33d00-7b8d6755cc-hk6fh"
Jul 24 15:37:40.268918 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:37:40.268844    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-26f33d00-serving-cert: secret "maas-api-e2e-ait-26f33d00-serving-cert" not found
Jul 24 15:37:40.268964 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:37:40.268917    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c6df170-4234-4658-907c-3d4b7c4fcc7e-maas-api-tls podName:7c6df170-4234-4658-907c-3d4b7c4fcc7e nodeName:}" failed. No retries permitted until 2026-07-24 15:37:41.268902256 +0000 UTC m=+1754.081254413 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7c6df170-4234-4658-907c-3d4b7c4fcc7e-maas-api-tls") pod "maas-api-e2e-ait-26f33d00-7b8d6755cc-hk6fh" (UID: "7c6df170-4234-4658-907c-3d4b7c4fcc7e") : secret "maas-api-e2e-ait-26f33d00-serving-cert" not found
Jul 24 15:37:40.297464 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:40.297428    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-26f33d00-6744f84ccb-65vx7"]
Jul 24 15:37:40.301058 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:40.301033    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-26f33d00-6744f84ccb-65vx7"
Jul 24 15:37:40.304010 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:40.303989    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-26f33d00\""
Jul 24 15:37:40.317991 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:40.317969    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-26f33d00-6744f84ccb-65vx7"]
Jul 24 15:37:40.340024 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:40.339994    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-26f33d00-64869db456-b665c"]
Jul 24 15:37:40.343346 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:40.343327    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-26f33d00-64869db456-b665c"
Jul 24 15:37:40.346805 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:40.346787    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-26f33d00-dockercfg-xg8l8\""
Jul 24 15:37:40.357118 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:40.357098    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-26f33d00-64869db456-b665c"]
Jul 24 15:37:40.369882 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:40.369853    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/531e2c06-9836-4875-884a-e4af5ab9064c-plugins-config-volume\") pod \"payload-processing-e2e-ait-26f33d00-6744f84ccb-65vx7\" (UID: \"531e2c06-9836-4875-884a-e4af5ab9064c\") " pod="openshift-ingress/payload-processing-e2e-ait-26f33d00-6744f84ccb-65vx7"
Jul 24 15:37:40.370056 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:40.370006    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsdsr\" (UniqueName: \"kubernetes.io/projected/531e2c06-9836-4875-884a-e4af5ab9064c-kube-api-access-jsdsr\") pod \"payload-processing-e2e-ait-26f33d00-6744f84ccb-65vx7\" (UID: \"531e2c06-9836-4875-884a-e4af5ab9064c\") " pod="openshift-ingress/payload-processing-e2e-ait-26f33d00-6744f84ccb-65vx7"
Jul 24 15:37:40.470573 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:40.470540    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdkwx\" (UniqueName: \"kubernetes.io/projected/4d46645d-3b03-444d-ac01-dd36992dece2-kube-api-access-cdkwx\") pod \"payload-pre-processing-e2e-ait-26f33d00-64869db456-b665c\" (UID: \"4d46645d-3b03-444d-ac01-dd36992dece2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-26f33d00-64869db456-b665c"
Jul 24 15:37:40.470740 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:40.470642    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsdsr\" (UniqueName: \"kubernetes.io/projected/531e2c06-9836-4875-884a-e4af5ab9064c-kube-api-access-jsdsr\") pod \"payload-processing-e2e-ait-26f33d00-6744f84ccb-65vx7\" (UID: \"531e2c06-9836-4875-884a-e4af5ab9064c\") " pod="openshift-ingress/payload-processing-e2e-ait-26f33d00-6744f84ccb-65vx7"
Jul 24 15:37:40.470740 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:40.470687    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d46645d-3b03-444d-ac01-dd36992dece2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-26f33d00-64869db456-b665c\" (UID: \"4d46645d-3b03-444d-ac01-dd36992dece2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-26f33d00-64869db456-b665c"
Jul 24 15:37:40.470852 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:40.470736    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/531e2c06-9836-4875-884a-e4af5ab9064c-plugins-config-volume\") pod \"payload-processing-e2e-ait-26f33d00-6744f84ccb-65vx7\" (UID: \"531e2c06-9836-4875-884a-e4af5ab9064c\") " pod="openshift-ingress/payload-processing-e2e-ait-26f33d00-6744f84ccb-65vx7"
Jul 24 15:37:40.471466 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:40.471446    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/531e2c06-9836-4875-884a-e4af5ab9064c-plugins-config-volume\") pod \"payload-processing-e2e-ait-26f33d00-6744f84ccb-65vx7\" (UID: \"531e2c06-9836-4875-884a-e4af5ab9064c\") " pod="openshift-ingress/payload-processing-e2e-ait-26f33d00-6744f84ccb-65vx7"
Jul 24 15:37:40.480016 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:40.479991    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsdsr\" (UniqueName: \"kubernetes.io/projected/531e2c06-9836-4875-884a-e4af5ab9064c-kube-api-access-jsdsr\") pod \"payload-processing-e2e-ait-26f33d00-6744f84ccb-65vx7\" (UID: \"531e2c06-9836-4875-884a-e4af5ab9064c\") " pod="openshift-ingress/payload-processing-e2e-ait-26f33d00-6744f84ccb-65vx7"
Jul 24 15:37:40.572246 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:40.572168    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdkwx\" (UniqueName: \"kubernetes.io/projected/4d46645d-3b03-444d-ac01-dd36992dece2-kube-api-access-cdkwx\") pod \"payload-pre-processing-e2e-ait-26f33d00-64869db456-b665c\" (UID: \"4d46645d-3b03-444d-ac01-dd36992dece2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-26f33d00-64869db456-b665c"
Jul 24 15:37:40.572246 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:40.572247    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d46645d-3b03-444d-ac01-dd36992dece2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-26f33d00-64869db456-b665c\" (UID: \"4d46645d-3b03-444d-ac01-dd36992dece2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-26f33d00-64869db456-b665c"
Jul 24 15:37:40.572924 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:40.572906    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d46645d-3b03-444d-ac01-dd36992dece2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-26f33d00-64869db456-b665c\" (UID: \"4d46645d-3b03-444d-ac01-dd36992dece2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-26f33d00-64869db456-b665c"
Jul 24 15:37:40.581228 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:40.581202    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdkwx\" (UniqueName: \"kubernetes.io/projected/4d46645d-3b03-444d-ac01-dd36992dece2-kube-api-access-cdkwx\") pod \"payload-pre-processing-e2e-ait-26f33d00-64869db456-b665c\" (UID: \"4d46645d-3b03-444d-ac01-dd36992dece2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-26f33d00-64869db456-b665c"
Jul 24 15:37:40.612166 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:40.612141    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-26f33d00-6744f84ccb-65vx7"
Jul 24 15:37:40.653314 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:40.653279    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-26f33d00-64869db456-b665c"
Jul 24 15:37:40.789474 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:40.789442    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-26f33d00-6744f84ccb-65vx7"]
Jul 24 15:37:40.790615 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:37:40.790585    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod531e2c06_9836_4875_884a_e4af5ab9064c.slice/crio-3c106b7a42c6022dc18077ebbb14216a23230195a110fa1dee27318ce34dbe64 WatchSource:0}: Error finding container 3c106b7a42c6022dc18077ebbb14216a23230195a110fa1dee27318ce34dbe64: Status 404 returned error can't find the container with id 3c106b7a42c6022dc18077ebbb14216a23230195a110fa1dee27318ce34dbe64
Jul 24 15:37:40.822188 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:40.822161    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-26f33d00-64869db456-b665c"]
Jul 24 15:37:40.824971 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:37:40.824942    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d46645d_3b03_444d_ac01_dd36992dece2.slice/crio-858525d79b81194fe19aecdb6e72626b5fba5ae4b45f22d7b0553ae05b2d5691 WatchSource:0}: Error finding container 858525d79b81194fe19aecdb6e72626b5fba5ae4b45f22d7b0553ae05b2d5691: Status 404 returned error can't find the container with id 858525d79b81194fe19aecdb6e72626b5fba5ae4b45f22d7b0553ae05b2d5691
Jul 24 15:37:40.883632 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:40.883557    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-26f33d00-6744f84ccb-65vx7" event={"ID":"531e2c06-9836-4875-884a-e4af5ab9064c","Type":"ContainerStarted","Data":"a1e946e4f5ea947d980040b6defe9ec1197caaeaa14498ddcf89ea4f9212c9ba"}
Jul 24 15:37:40.883853 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:40.883672    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-26f33d00-6744f84ccb-65vx7" event={"ID":"531e2c06-9836-4875-884a-e4af5ab9064c","Type":"ContainerStarted","Data":"3c106b7a42c6022dc18077ebbb14216a23230195a110fa1dee27318ce34dbe64"}
Jul 24 15:37:40.883853 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:40.883720    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-26f33d00-6744f84ccb-65vx7"
Jul 24 15:37:40.885323 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:40.885293    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-26f33d00-64869db456-b665c" event={"ID":"4d46645d-3b03-444d-ac01-dd36992dece2","Type":"ContainerStarted","Data":"aacc84261cafd7d875a203bc603212b2a6d7f42a720ac0e792c91318afb1479a"}
Jul 24 15:37:40.885461 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:40.885344    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-26f33d00-64869db456-b665c" event={"ID":"4d46645d-3b03-444d-ac01-dd36992dece2","Type":"ContainerStarted","Data":"858525d79b81194fe19aecdb6e72626b5fba5ae4b45f22d7b0553ae05b2d5691"}
Jul 24 15:37:40.885461 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:40.885376    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-26f33d00-64869db456-b665c"
Jul 24 15:37:40.906157 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:40.906107    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-26f33d00-6744f84ccb-65vx7" podStartSLOduration=0.906090343 podStartE2EDuration="906.090343ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:37:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:37:40.904403605 +0000 UTC m=+1753.716755775" watchObservedRunningTime="2026-07-24 15:37:40.906090343 +0000 UTC m=+1753.718442523"
Jul 24 15:37:40.926018 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:40.925954    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-26f33d00-64869db456-b665c" podStartSLOduration=0.925938211 podStartE2EDuration="925.938211ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:37:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:37:40.923990523 +0000 UTC m=+1753.736342704" watchObservedRunningTime="2026-07-24 15:37:40.925938211 +0000 UTC m=+1753.738290390"
Jul 24 15:37:41.280266 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:41.280221    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c6df170-4234-4658-907c-3d4b7c4fcc7e-maas-api-tls\") pod \"maas-api-e2e-ait-26f33d00-7b8d6755cc-hk6fh\" (UID: \"7c6df170-4234-4658-907c-3d4b7c4fcc7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-26f33d00-7b8d6755cc-hk6fh"
Jul 24 15:37:41.282658 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:41.282632    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c6df170-4234-4658-907c-3d4b7c4fcc7e-maas-api-tls\") pod \"maas-api-e2e-ait-26f33d00-7b8d6755cc-hk6fh\" (UID: \"7c6df170-4234-4658-907c-3d4b7c4fcc7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-26f33d00-7b8d6755cc-hk6fh"
Jul 24 15:37:41.357353 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:41.357308    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-26f33d00-7b8d6755cc-hk6fh"
Jul 24 15:37:41.497573 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:41.497550    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-26f33d00-7b8d6755cc-hk6fh"]
Jul 24 15:37:41.499945 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:37:41.499922    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c6df170_4234_4658_907c_3d4b7c4fcc7e.slice/crio-2f4a9f470628d2f8aeaa066049317bb02e67babc58888360e14dfca1fe9da074 WatchSource:0}: Error finding container 2f4a9f470628d2f8aeaa066049317bb02e67babc58888360e14dfca1fe9da074: Status 404 returned error can't find the container with id 2f4a9f470628d2f8aeaa066049317bb02e67babc58888360e14dfca1fe9da074
Jul 24 15:37:41.892078 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:41.892036    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-26f33d00-7b8d6755cc-hk6fh" event={"ID":"7c6df170-4234-4658-907c-3d4b7c4fcc7e","Type":"ContainerStarted","Data":"2f4a9f470628d2f8aeaa066049317bb02e67babc58888360e14dfca1fe9da074"}
Jul 24 15:37:42.898167 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:42.898128    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-26f33d00-7b8d6755cc-hk6fh" event={"ID":"7c6df170-4234-4658-907c-3d4b7c4fcc7e","Type":"ContainerStarted","Data":"534d43bfbb8c61c0a36a791c7f46d546cbedd7c04b16a3fe21daae548c36c44c"}
Jul 24 15:37:42.898620 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:42.898258    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-26f33d00-7b8d6755cc-hk6fh"
Jul 24 15:37:42.921384 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:42.921331    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-26f33d00-7b8d6755cc-hk6fh" podStartSLOduration=3.433507369 podStartE2EDuration="3.921315457s" totalImagesPullingTime="487.808088ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:37:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:37:42.918725543 +0000 UTC m=+1755.731077722" watchObservedRunningTime="2026-07-24 15:37:42.921315457 +0000 UTC m=+1755.733667635"
Jul 24 15:37:45.131701 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:45.131664    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:37:48.909608 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:48.909579    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-26f33d00-7b8d6755cc-hk6fh"
Jul 24 15:37:51.894323 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:51.894289    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-26f33d00-6744f84ccb-65vx7"
Jul 24 15:37:51.894717 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:37:51.894467    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-26f33d00-64869db456-b665c"
Jul 24 15:38:15.917988 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:15.917948    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-26f33d00-7b8d6755cc-hk6fh"]
Jul 24 15:38:15.918597 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:15.918195    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-26f33d00-7b8d6755cc-hk6fh" podUID="7c6df170-4234-4658-907c-3d4b7c4fcc7e" containerName="maas-api" containerID="cri-o://534d43bfbb8c61c0a36a791c7f46d546cbedd7c04b16a3fe21daae548c36c44c" gracePeriod=30
Jul 24 15:38:15.998795 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:15.998761    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-26f33d00-6744f84ccb-65vx7"]
Jul 24 15:38:15.999082 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:15.999024    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-26f33d00-6744f84ccb-65vx7" podUID="531e2c06-9836-4875-884a-e4af5ab9064c" containerName="payload-processing" containerID="cri-o://a1e946e4f5ea947d980040b6defe9ec1197caaeaa14498ddcf89ea4f9212c9ba" gracePeriod=30
Jul 24 15:38:16.006416 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:16.006295    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-26f33d00-64869db456-b665c"]
Jul 24 15:38:16.007088 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:16.006531    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-26f33d00-64869db456-b665c" podUID="4d46645d-3b03-444d-ac01-dd36992dece2" containerName="payload-pre-processing" containerID="cri-o://aacc84261cafd7d875a203bc603212b2a6d7f42a720ac0e792c91318afb1479a" gracePeriod=30
Jul 24 15:38:16.057775 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:16.057741    2579 generic.go:356] "Generic (PLEG): container finished" podID="7c6df170-4234-4658-907c-3d4b7c4fcc7e" containerID="534d43bfbb8c61c0a36a791c7f46d546cbedd7c04b16a3fe21daae548c36c44c" exitCode=0
Jul 24 15:38:16.057952 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:16.057811    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-26f33d00-7b8d6755cc-hk6fh" event={"ID":"7c6df170-4234-4658-907c-3d4b7c4fcc7e","Type":"ContainerDied","Data":"534d43bfbb8c61c0a36a791c7f46d546cbedd7c04b16a3fe21daae548c36c44c"}
Jul 24 15:38:16.254293 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:16.254264    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-26f33d00-7b8d6755cc-hk6fh"
Jul 24 15:38:16.286907 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:16.286881    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-26f33d00-6744f84ccb-65vx7"
Jul 24 15:38:16.293248 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:16.293223    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-26f33d00-64869db456-b665c"
Jul 24 15:38:16.307616 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:16.307591    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c6df170-4234-4658-907c-3d4b7c4fcc7e-openshift-service-ca\") pod \"7c6df170-4234-4658-907c-3d4b7c4fcc7e\" (UID: \"7c6df170-4234-4658-907c-3d4b7c4fcc7e\") "
Jul 24 15:38:16.307740 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:16.307631    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txx29\" (UniqueName: \"kubernetes.io/projected/7c6df170-4234-4658-907c-3d4b7c4fcc7e-kube-api-access-txx29\") pod \"7c6df170-4234-4658-907c-3d4b7c4fcc7e\" (UID: \"7c6df170-4234-4658-907c-3d4b7c4fcc7e\") "
Jul 24 15:38:16.307835 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:16.307744    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c6df170-4234-4658-907c-3d4b7c4fcc7e-maas-api-tls\") pod \"7c6df170-4234-4658-907c-3d4b7c4fcc7e\" (UID: \"7c6df170-4234-4658-907c-3d4b7c4fcc7e\") "
Jul 24 15:38:16.307934 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:16.307911    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c6df170-4234-4658-907c-3d4b7c4fcc7e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7c6df170-4234-4658-907c-3d4b7c4fcc7e" (UID: "7c6df170-4234-4658-907c-3d4b7c4fcc7e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:38:16.309906 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:16.309880    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c6df170-4234-4658-907c-3d4b7c4fcc7e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7c6df170-4234-4658-907c-3d4b7c4fcc7e" (UID: "7c6df170-4234-4658-907c-3d4b7c4fcc7e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 15:38:16.310410 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:16.310383    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c6df170-4234-4658-907c-3d4b7c4fcc7e-kube-api-access-txx29" (OuterVolumeSpecName: "kube-api-access-txx29") pod "7c6df170-4234-4658-907c-3d4b7c4fcc7e" (UID: "7c6df170-4234-4658-907c-3d4b7c4fcc7e"). InnerVolumeSpecName "kube-api-access-txx29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:38:16.408982 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:16.408942    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdkwx\" (UniqueName: \"kubernetes.io/projected/4d46645d-3b03-444d-ac01-dd36992dece2-kube-api-access-cdkwx\") pod \"4d46645d-3b03-444d-ac01-dd36992dece2\" (UID: \"4d46645d-3b03-444d-ac01-dd36992dece2\") "
Jul 24 15:38:16.409185 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:16.408993    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/531e2c06-9836-4875-884a-e4af5ab9064c-plugins-config-volume\") pod \"531e2c06-9836-4875-884a-e4af5ab9064c\" (UID: \"531e2c06-9836-4875-884a-e4af5ab9064c\") "
Jul 24 15:38:16.409185 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:16.409034    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d46645d-3b03-444d-ac01-dd36992dece2-plugins-config-volume\") pod \"4d46645d-3b03-444d-ac01-dd36992dece2\" (UID: \"4d46645d-3b03-444d-ac01-dd36992dece2\") "
Jul 24 15:38:16.409185 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:16.409154    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jsdsr\" (UniqueName: \"kubernetes.io/projected/531e2c06-9836-4875-884a-e4af5ab9064c-kube-api-access-jsdsr\") pod \"531e2c06-9836-4875-884a-e4af5ab9064c\" (UID: \"531e2c06-9836-4875-884a-e4af5ab9064c\") "
Jul 24 15:38:16.409417 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:16.409400    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c6df170-4234-4658-907c-3d4b7c4fcc7e-openshift-service-ca\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:38:16.409474 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:16.409424    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txx29\" (UniqueName: \"kubernetes.io/projected/7c6df170-4234-4658-907c-3d4b7c4fcc7e-kube-api-access-txx29\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:38:16.409474 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:16.409439    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c6df170-4234-4658-907c-3d4b7c4fcc7e-maas-api-tls\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:38:16.409545 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:16.409463    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d46645d-3b03-444d-ac01-dd36992dece2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4d46645d-3b03-444d-ac01-dd36992dece2" (UID: "4d46645d-3b03-444d-ac01-dd36992dece2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:38:16.409545 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:16.409467    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/531e2c06-9836-4875-884a-e4af5ab9064c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "531e2c06-9836-4875-884a-e4af5ab9064c" (UID: "531e2c06-9836-4875-884a-e4af5ab9064c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:38:16.411049 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:16.411031    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d46645d-3b03-444d-ac01-dd36992dece2-kube-api-access-cdkwx" (OuterVolumeSpecName: "kube-api-access-cdkwx") pod "4d46645d-3b03-444d-ac01-dd36992dece2" (UID: "4d46645d-3b03-444d-ac01-dd36992dece2"). InnerVolumeSpecName "kube-api-access-cdkwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:38:16.411106 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:16.411085    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/531e2c06-9836-4875-884a-e4af5ab9064c-kube-api-access-jsdsr" (OuterVolumeSpecName: "kube-api-access-jsdsr") pod "531e2c06-9836-4875-884a-e4af5ab9064c" (UID: "531e2c06-9836-4875-884a-e4af5ab9064c"). InnerVolumeSpecName "kube-api-access-jsdsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:38:16.510100 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:16.510024    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsdsr\" (UniqueName: \"kubernetes.io/projected/531e2c06-9836-4875-884a-e4af5ab9064c-kube-api-access-jsdsr\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:38:16.510100 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:16.510053    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdkwx\" (UniqueName: \"kubernetes.io/projected/4d46645d-3b03-444d-ac01-dd36992dece2-kube-api-access-cdkwx\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:38:16.510100 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:16.510063    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/531e2c06-9836-4875-884a-e4af5ab9064c-plugins-config-volume\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:38:16.510100 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:16.510073    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d46645d-3b03-444d-ac01-dd36992dece2-plugins-config-volume\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:38:17.064304 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:17.064276    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-26f33d00-7b8d6755cc-hk6fh"
Jul 24 15:38:17.064304 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:17.064292    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-26f33d00-7b8d6755cc-hk6fh" event={"ID":"7c6df170-4234-4658-907c-3d4b7c4fcc7e","Type":"ContainerDied","Data":"2f4a9f470628d2f8aeaa066049317bb02e67babc58888360e14dfca1fe9da074"}
Jul 24 15:38:17.064886 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:17.064356    2579 scope.go:117] "RemoveContainer" containerID="534d43bfbb8c61c0a36a791c7f46d546cbedd7c04b16a3fe21daae548c36c44c"
Jul 24 15:38:17.065686 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:17.065541    2579 generic.go:356] "Generic (PLEG): container finished" podID="4d46645d-3b03-444d-ac01-dd36992dece2" containerID="aacc84261cafd7d875a203bc603212b2a6d7f42a720ac0e792c91318afb1479a" exitCode=0
Jul 24 15:38:17.065686 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:17.065590    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-26f33d00-64869db456-b665c"
Jul 24 15:38:17.065686 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:17.065629    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-26f33d00-64869db456-b665c" event={"ID":"4d46645d-3b03-444d-ac01-dd36992dece2","Type":"ContainerDied","Data":"aacc84261cafd7d875a203bc603212b2a6d7f42a720ac0e792c91318afb1479a"}
Jul 24 15:38:17.065880 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:17.065682    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-26f33d00-64869db456-b665c" event={"ID":"4d46645d-3b03-444d-ac01-dd36992dece2","Type":"ContainerDied","Data":"858525d79b81194fe19aecdb6e72626b5fba5ae4b45f22d7b0553ae05b2d5691"}
Jul 24 15:38:17.067135 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:17.067110    2579 generic.go:356] "Generic (PLEG): container finished" podID="531e2c06-9836-4875-884a-e4af5ab9064c" containerID="a1e946e4f5ea947d980040b6defe9ec1197caaeaa14498ddcf89ea4f9212c9ba" exitCode=0
Jul 24 15:38:17.067245 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:17.067145    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-26f33d00-6744f84ccb-65vx7"
Jul 24 15:38:17.067245 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:17.067153    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-26f33d00-6744f84ccb-65vx7" event={"ID":"531e2c06-9836-4875-884a-e4af5ab9064c","Type":"ContainerDied","Data":"a1e946e4f5ea947d980040b6defe9ec1197caaeaa14498ddcf89ea4f9212c9ba"}
Jul 24 15:38:17.067245 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:17.067192    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-26f33d00-6744f84ccb-65vx7" event={"ID":"531e2c06-9836-4875-884a-e4af5ab9064c","Type":"ContainerDied","Data":"3c106b7a42c6022dc18077ebbb14216a23230195a110fa1dee27318ce34dbe64"}
Jul 24 15:38:17.074800 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:17.074777    2579 scope.go:117] "RemoveContainer" containerID="aacc84261cafd7d875a203bc603212b2a6d7f42a720ac0e792c91318afb1479a"
Jul 24 15:38:17.082661 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:17.082645    2579 scope.go:117] "RemoveContainer" containerID="aacc84261cafd7d875a203bc603212b2a6d7f42a720ac0e792c91318afb1479a"
Jul 24 15:38:17.082915 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:38:17.082893    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aacc84261cafd7d875a203bc603212b2a6d7f42a720ac0e792c91318afb1479a\": container with ID starting with aacc84261cafd7d875a203bc603212b2a6d7f42a720ac0e792c91318afb1479a not found: ID does not exist" containerID="aacc84261cafd7d875a203bc603212b2a6d7f42a720ac0e792c91318afb1479a"
Jul 24 15:38:17.082964 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:17.082926    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aacc84261cafd7d875a203bc603212b2a6d7f42a720ac0e792c91318afb1479a"} err="failed to get container status \"aacc84261cafd7d875a203bc603212b2a6d7f42a720ac0e792c91318afb1479a\": rpc error: code = NotFound desc = could not find container \"aacc84261cafd7d875a203bc603212b2a6d7f42a720ac0e792c91318afb1479a\": container with ID starting with aacc84261cafd7d875a203bc603212b2a6d7f42a720ac0e792c91318afb1479a not found: ID does not exist"
Jul 24 15:38:17.082964 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:17.082947    2579 scope.go:117] "RemoveContainer" containerID="a1e946e4f5ea947d980040b6defe9ec1197caaeaa14498ddcf89ea4f9212c9ba"
Jul 24 15:38:17.090091 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:17.090074    2579 scope.go:117] "RemoveContainer" containerID="a1e946e4f5ea947d980040b6defe9ec1197caaeaa14498ddcf89ea4f9212c9ba"
Jul 24 15:38:17.090360 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:38:17.090340    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1e946e4f5ea947d980040b6defe9ec1197caaeaa14498ddcf89ea4f9212c9ba\": container with ID starting with a1e946e4f5ea947d980040b6defe9ec1197caaeaa14498ddcf89ea4f9212c9ba not found: ID does not exist" containerID="a1e946e4f5ea947d980040b6defe9ec1197caaeaa14498ddcf89ea4f9212c9ba"
Jul 24 15:38:17.090414 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:17.090371    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1e946e4f5ea947d980040b6defe9ec1197caaeaa14498ddcf89ea4f9212c9ba"} err="failed to get container status \"a1e946e4f5ea947d980040b6defe9ec1197caaeaa14498ddcf89ea4f9212c9ba\": rpc error: code = NotFound desc = could not find container \"a1e946e4f5ea947d980040b6defe9ec1197caaeaa14498ddcf89ea4f9212c9ba\": container with ID starting with a1e946e4f5ea947d980040b6defe9ec1197caaeaa14498ddcf89ea4f9212c9ba not found: ID does not exist"
Jul 24 15:38:17.093894 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:17.093874    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-26f33d00-7b8d6755cc-hk6fh"]
Jul 24 15:38:17.101124 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:17.101103    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-26f33d00-7b8d6755cc-hk6fh"]
Jul 24 15:38:17.116643 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:17.116617    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-26f33d00-64869db456-b665c"]
Jul 24 15:38:17.129381 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:17.129347    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-26f33d00-64869db456-b665c"]
Jul 24 15:38:17.154577 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:17.154546    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-26f33d00-6744f84ccb-65vx7"]
Jul 24 15:38:17.158572 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:17.158545    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-26f33d00-6744f84ccb-65vx7"]
Jul 24 15:38:17.718557 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:17.718524    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d46645d-3b03-444d-ac01-dd36992dece2" path="/var/lib/kubelet/pods/4d46645d-3b03-444d-ac01-dd36992dece2/volumes"
Jul 24 15:38:17.718908 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:17.718895    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="531e2c06-9836-4875-884a-e4af5ab9064c" path="/var/lib/kubelet/pods/531e2c06-9836-4875-884a-e4af5ab9064c/volumes"
Jul 24 15:38:17.719256 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:17.719244    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c6df170-4234-4658-907c-3d4b7c4fcc7e" path="/var/lib/kubelet/pods/7c6df170-4234-4658-907c-3d4b7c4fcc7e/volumes"
Jul 24 15:38:19.236362 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:19.236322    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:38:24.654305 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:24.654272    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-65c49f07-b9997f454-v8cvv"]
Jul 24 15:38:24.654864 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:24.654839    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d46645d-3b03-444d-ac01-dd36992dece2" containerName="payload-pre-processing"
Jul 24 15:38:24.654864 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:24.654864    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d46645d-3b03-444d-ac01-dd36992dece2" containerName="payload-pre-processing"
Jul 24 15:38:24.655042 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:24.654894    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c6df170-4234-4658-907c-3d4b7c4fcc7e" containerName="maas-api"
Jul 24 15:38:24.655042 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:24.654905    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c6df170-4234-4658-907c-3d4b7c4fcc7e" containerName="maas-api"
Jul 24 15:38:24.655042 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:24.654927    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="531e2c06-9836-4875-884a-e4af5ab9064c" containerName="payload-processing"
Jul 24 15:38:24.655042 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:24.654935    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="531e2c06-9836-4875-884a-e4af5ab9064c" containerName="payload-processing"
Jul 24 15:38:24.655042 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:24.655021    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c6df170-4234-4658-907c-3d4b7c4fcc7e" containerName="maas-api"
Jul 24 15:38:24.655042 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:24.655035    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="531e2c06-9836-4875-884a-e4af5ab9064c" containerName="payload-processing"
Jul 24 15:38:24.655245 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:24.655045    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d46645d-3b03-444d-ac01-dd36992dece2" containerName="payload-pre-processing"
Jul 24 15:38:24.660047 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:24.660025    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-65c49f07-b9997f454-v8cvv"
Jul 24 15:38:24.662868 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:24.662849    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-65c49f07-serving-cert\""
Jul 24 15:38:24.663181 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:24.663165    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 15:38:24.663272 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:24.663202    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tkn9s\""
Jul 24 15:38:24.663895 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:24.663878    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 15:38:24.672033 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:24.672011    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-65c49f07-b9997f454-v8cvv"]
Jul 24 15:38:24.784156 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:24.784117    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8f92261-b174-43ef-b06c-7d4d978e407e-maas-api-tls\") pod \"maas-api-e2e-derive-65c49f07-b9997f454-v8cvv\" (UID: \"e8f92261-b174-43ef-b06c-7d4d978e407e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-65c49f07-b9997f454-v8cvv"
Jul 24 15:38:24.784156 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:24.784158    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hphf8\" (UniqueName: \"kubernetes.io/projected/e8f92261-b174-43ef-b06c-7d4d978e407e-kube-api-access-hphf8\") pod \"maas-api-e2e-derive-65c49f07-b9997f454-v8cvv\" (UID: \"e8f92261-b174-43ef-b06c-7d4d978e407e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-65c49f07-b9997f454-v8cvv"
Jul 24 15:38:24.784372 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:24.784227    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8f92261-b174-43ef-b06c-7d4d978e407e-openshift-service-ca\") pod \"maas-api-e2e-derive-65c49f07-b9997f454-v8cvv\" (UID: \"e8f92261-b174-43ef-b06c-7d4d978e407e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-65c49f07-b9997f454-v8cvv"
Jul 24 15:38:24.885438 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:24.885402    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8f92261-b174-43ef-b06c-7d4d978e407e-maas-api-tls\") pod \"maas-api-e2e-derive-65c49f07-b9997f454-v8cvv\" (UID: \"e8f92261-b174-43ef-b06c-7d4d978e407e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-65c49f07-b9997f454-v8cvv"
Jul 24 15:38:24.885438 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:24.885441    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hphf8\" (UniqueName: \"kubernetes.io/projected/e8f92261-b174-43ef-b06c-7d4d978e407e-kube-api-access-hphf8\") pod \"maas-api-e2e-derive-65c49f07-b9997f454-v8cvv\" (UID: \"e8f92261-b174-43ef-b06c-7d4d978e407e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-65c49f07-b9997f454-v8cvv"
Jul 24 15:38:24.885649 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:24.885472    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8f92261-b174-43ef-b06c-7d4d978e407e-openshift-service-ca\") pod \"maas-api-e2e-derive-65c49f07-b9997f454-v8cvv\" (UID: \"e8f92261-b174-43ef-b06c-7d4d978e407e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-65c49f07-b9997f454-v8cvv"
Jul 24 15:38:24.886054 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:24.886026    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8f92261-b174-43ef-b06c-7d4d978e407e-openshift-service-ca\") pod \"maas-api-e2e-derive-65c49f07-b9997f454-v8cvv\" (UID: \"e8f92261-b174-43ef-b06c-7d4d978e407e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-65c49f07-b9997f454-v8cvv"
Jul 24 15:38:24.887866 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:24.887840    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8f92261-b174-43ef-b06c-7d4d978e407e-maas-api-tls\") pod \"maas-api-e2e-derive-65c49f07-b9997f454-v8cvv\" (UID: \"e8f92261-b174-43ef-b06c-7d4d978e407e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-65c49f07-b9997f454-v8cvv"
Jul 24 15:38:24.923464 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:24.923401    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hphf8\" (UniqueName: \"kubernetes.io/projected/e8f92261-b174-43ef-b06c-7d4d978e407e-kube-api-access-hphf8\") pod \"maas-api-e2e-derive-65c49f07-b9997f454-v8cvv\" (UID: \"e8f92261-b174-43ef-b06c-7d4d978e407e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-65c49f07-b9997f454-v8cvv"
Jul 24 15:38:24.972984 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:24.972957    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-65c49f07-b9997f454-v8cvv"
Jul 24 15:38:25.111107 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:25.111084    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-65c49f07-b9997f454-v8cvv"]
Jul 24 15:38:25.113529 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:38:25.113504    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8f92261_b174_43ef_b06c_7d4d978e407e.slice/crio-3719e15b7b664089c90ccd85217edf9018185b0ae039f1b8abc161e735876c7d WatchSource:0}: Error finding container 3719e15b7b664089c90ccd85217edf9018185b0ae039f1b8abc161e735876c7d: Status 404 returned error can't find the container with id 3719e15b7b664089c90ccd85217edf9018185b0ae039f1b8abc161e735876c7d
Jul 24 15:38:25.538301 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:25.538275    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-65c49f07-6f6bd96956-x5dsn"]
Jul 24 15:38:25.541765 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:25.541747    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-65c49f07-6f6bd96956-x5dsn"
Jul 24 15:38:25.545161 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:25.545142    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-65c49f07\""
Jul 24 15:38:25.568766 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:25.568738    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-65c49f07-7554d55fb6-gvhpv"]
Jul 24 15:38:25.576890 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:25.576503    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-65c49f07-6f6bd96956-x5dsn"]
Jul 24 15:38:25.576890 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:25.576667    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-65c49f07-7554d55fb6-gvhpv"
Jul 24 15:38:25.580437 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:25.580415    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-65c49f07-dockercfg-9ggms\""
Jul 24 15:38:25.589752 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:25.589728    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-65c49f07-7554d55fb6-gvhpv"]
Jul 24 15:38:25.692040 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:25.691952    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba328fe3-e40e-481b-a9c9-c5e7325cbe1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-65c49f07-6f6bd96956-x5dsn\" (UID: \"ba328fe3-e40e-481b-a9c9-c5e7325cbe1f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-65c49f07-6f6bd96956-x5dsn"
Jul 24 15:38:25.692442 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:25.692081    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb4hx\" (UniqueName: \"kubernetes.io/projected/ba328fe3-e40e-481b-a9c9-c5e7325cbe1f-kube-api-access-wb4hx\") pod \"payload-pre-processing-e2e-derive-65c49f07-6f6bd96956-x5dsn\" (UID: \"ba328fe3-e40e-481b-a9c9-c5e7325cbe1f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-65c49f07-6f6bd96956-x5dsn"
Jul 24 15:38:25.692442 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:25.692121    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fb55\" (UniqueName: \"kubernetes.io/projected/1d2ac9f2-86d3-41e7-b1fb-bbeb10624f28-kube-api-access-8fb55\") pod \"payload-processing-e2e-derive-65c49f07-7554d55fb6-gvhpv\" (UID: \"1d2ac9f2-86d3-41e7-b1fb-bbeb10624f28\") " pod="openshift-ingress/payload-processing-e2e-derive-65c49f07-7554d55fb6-gvhpv"
Jul 24 15:38:25.692442 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:25.692183    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d2ac9f2-86d3-41e7-b1fb-bbeb10624f28-plugins-config-volume\") pod \"payload-processing-e2e-derive-65c49f07-7554d55fb6-gvhpv\" (UID: \"1d2ac9f2-86d3-41e7-b1fb-bbeb10624f28\") " pod="openshift-ingress/payload-processing-e2e-derive-65c49f07-7554d55fb6-gvhpv"
Jul 24 15:38:25.793568 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:25.793528    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba328fe3-e40e-481b-a9c9-c5e7325cbe1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-65c49f07-6f6bd96956-x5dsn\" (UID: \"ba328fe3-e40e-481b-a9c9-c5e7325cbe1f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-65c49f07-6f6bd96956-x5dsn"
Jul 24 15:38:25.793731 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:25.793698    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb4hx\" (UniqueName: \"kubernetes.io/projected/ba328fe3-e40e-481b-a9c9-c5e7325cbe1f-kube-api-access-wb4hx\") pod \"payload-pre-processing-e2e-derive-65c49f07-6f6bd96956-x5dsn\" (UID: \"ba328fe3-e40e-481b-a9c9-c5e7325cbe1f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-65c49f07-6f6bd96956-x5dsn"
Jul 24 15:38:25.793795 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:25.793730    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fb55\" (UniqueName: \"kubernetes.io/projected/1d2ac9f2-86d3-41e7-b1fb-bbeb10624f28-kube-api-access-8fb55\") pod \"payload-processing-e2e-derive-65c49f07-7554d55fb6-gvhpv\" (UID: \"1d2ac9f2-86d3-41e7-b1fb-bbeb10624f28\") " pod="openshift-ingress/payload-processing-e2e-derive-65c49f07-7554d55fb6-gvhpv"
Jul 24 15:38:25.793795 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:25.793765    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d2ac9f2-86d3-41e7-b1fb-bbeb10624f28-plugins-config-volume\") pod \"payload-processing-e2e-derive-65c49f07-7554d55fb6-gvhpv\" (UID: \"1d2ac9f2-86d3-41e7-b1fb-bbeb10624f28\") " pod="openshift-ingress/payload-processing-e2e-derive-65c49f07-7554d55fb6-gvhpv"
Jul 24 15:38:25.794313 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:25.794294    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba328fe3-e40e-481b-a9c9-c5e7325cbe1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-65c49f07-6f6bd96956-x5dsn\" (UID: \"ba328fe3-e40e-481b-a9c9-c5e7325cbe1f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-65c49f07-6f6bd96956-x5dsn"
Jul 24 15:38:25.794467 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:25.794450    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d2ac9f2-86d3-41e7-b1fb-bbeb10624f28-plugins-config-volume\") pod \"payload-processing-e2e-derive-65c49f07-7554d55fb6-gvhpv\" (UID: \"1d2ac9f2-86d3-41e7-b1fb-bbeb10624f28\") " pod="openshift-ingress/payload-processing-e2e-derive-65c49f07-7554d55fb6-gvhpv"
Jul 24 15:38:25.804435 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:25.804403    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fb55\" (UniqueName: \"kubernetes.io/projected/1d2ac9f2-86d3-41e7-b1fb-bbeb10624f28-kube-api-access-8fb55\") pod \"payload-processing-e2e-derive-65c49f07-7554d55fb6-gvhpv\" (UID: \"1d2ac9f2-86d3-41e7-b1fb-bbeb10624f28\") " pod="openshift-ingress/payload-processing-e2e-derive-65c49f07-7554d55fb6-gvhpv"
Jul 24 15:38:25.805202 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:25.805186    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb4hx\" (UniqueName: \"kubernetes.io/projected/ba328fe3-e40e-481b-a9c9-c5e7325cbe1f-kube-api-access-wb4hx\") pod \"payload-pre-processing-e2e-derive-65c49f07-6f6bd96956-x5dsn\" (UID: \"ba328fe3-e40e-481b-a9c9-c5e7325cbe1f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-65c49f07-6f6bd96956-x5dsn"
Jul 24 15:38:25.852654 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:25.852617    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-65c49f07-6f6bd96956-x5dsn"
Jul 24 15:38:25.892620 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:25.891895    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-65c49f07-7554d55fb6-gvhpv"
Jul 24 15:38:26.008145 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:26.008113    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-65c49f07-6f6bd96956-x5dsn"]
Jul 24 15:38:26.009999 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:38:26.009966    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba328fe3_e40e_481b_a9c9_c5e7325cbe1f.slice/crio-2500fe6783981f8d5c4c2d8440464302dab395dff51507a656588a13419b4b72 WatchSource:0}: Error finding container 2500fe6783981f8d5c4c2d8440464302dab395dff51507a656588a13419b4b72: Status 404 returned error can't find the container with id 2500fe6783981f8d5c4c2d8440464302dab395dff51507a656588a13419b4b72
Jul 24 15:38:26.052249 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:26.052219    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-65c49f07-7554d55fb6-gvhpv"]
Jul 24 15:38:26.055848 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:38:26.055802    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d2ac9f2_86d3_41e7_b1fb_bbeb10624f28.slice/crio-d2c477a382713476808fa44c03db8577dae60a5baf56037dbed47f111b7f4361 WatchSource:0}: Error finding container d2c477a382713476808fa44c03db8577dae60a5baf56037dbed47f111b7f4361: Status 404 returned error can't find the container with id d2c477a382713476808fa44c03db8577dae60a5baf56037dbed47f111b7f4361
Jul 24 15:38:26.117534 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:26.117498    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-65c49f07-b9997f454-v8cvv" event={"ID":"e8f92261-b174-43ef-b06c-7d4d978e407e","Type":"ContainerStarted","Data":"d051edffaa82a3a28152cb131505bd0b9f396323520fd9ccf2fa7b4c618645af"}
Jul 24 15:38:26.117727 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:26.117559    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-65c49f07-b9997f454-v8cvv" event={"ID":"e8f92261-b174-43ef-b06c-7d4d978e407e","Type":"ContainerStarted","Data":"3719e15b7b664089c90ccd85217edf9018185b0ae039f1b8abc161e735876c7d"}
Jul 24 15:38:26.117727 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:26.117633    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-65c49f07-b9997f454-v8cvv"
Jul 24 15:38:26.119211 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:26.119184    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-65c49f07-7554d55fb6-gvhpv" event={"ID":"1d2ac9f2-86d3-41e7-b1fb-bbeb10624f28","Type":"ContainerStarted","Data":"164f79ffb5f5db86ecef4385bc479e9ec7e1b462687e7281be7b64ea3784eda9"}
Jul 24 15:38:26.119312 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:26.119227    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-65c49f07-7554d55fb6-gvhpv" event={"ID":"1d2ac9f2-86d3-41e7-b1fb-bbeb10624f28","Type":"ContainerStarted","Data":"d2c477a382713476808fa44c03db8577dae60a5baf56037dbed47f111b7f4361"}
Jul 24 15:38:26.119312 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:26.119253    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-65c49f07-7554d55fb6-gvhpv"
Jul 24 15:38:26.120871 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:26.120848    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-65c49f07-6f6bd96956-x5dsn" event={"ID":"ba328fe3-e40e-481b-a9c9-c5e7325cbe1f","Type":"ContainerStarted","Data":"9788133eb4de02cc801e82450dfc2f87aeb47d243c4ddc2403234a896d408694"}
Jul 24 15:38:26.120959 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:26.120899    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-65c49f07-6f6bd96956-x5dsn"
Jul 24 15:38:26.120959 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:26.120919    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-65c49f07-6f6bd96956-x5dsn" event={"ID":"ba328fe3-e40e-481b-a9c9-c5e7325cbe1f","Type":"ContainerStarted","Data":"2500fe6783981f8d5c4c2d8440464302dab395dff51507a656588a13419b4b72"}
Jul 24 15:38:26.139955 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:26.139898    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-65c49f07-b9997f454-v8cvv" podStartSLOduration=1.774401492 podStartE2EDuration="2.139880178s" totalImagesPullingTime="365.478686ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:38:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:38:26.138212568 +0000 UTC m=+1798.950564749" watchObservedRunningTime="2026-07-24 15:38:26.139880178 +0000 UTC m=+1798.952232357"
Jul 24 15:38:26.158809 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:26.158751    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-65c49f07-6f6bd96956-x5dsn" podStartSLOduration=1.15873497 podStartE2EDuration="1.15873497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:38:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:38:26.156472381 +0000 UTC m=+1798.968824553" watchObservedRunningTime="2026-07-24 15:38:26.15873497 +0000 UTC m=+1798.971087149"
Jul 24 15:38:26.179577 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:26.179512    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-65c49f07-7554d55fb6-gvhpv" podStartSLOduration=1.179488056 podStartE2EDuration="1.179488056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:38:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:38:26.175679574 +0000 UTC m=+1798.988031752" watchObservedRunningTime="2026-07-24 15:38:26.179488056 +0000 UTC m=+1798.991840237"
Jul 24 15:38:27.943310 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:27.943274    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:38:32.130998 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:32.130968    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-65c49f07-b9997f454-v8cvv"
Jul 24 15:38:37.127999 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:37.127963    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-65c49f07-7554d55fb6-gvhpv"
Jul 24 15:38:37.128443 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:38:37.128057    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-65c49f07-6f6bd96956-x5dsn"
Jul 24 15:39:00.233486 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:00.233445    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-65c49f07-b9997f454-v8cvv"]
Jul 24 15:39:00.234070 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:00.233643    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-65c49f07-b9997f454-v8cvv" podUID="e8f92261-b174-43ef-b06c-7d4d978e407e" containerName="maas-api" containerID="cri-o://d051edffaa82a3a28152cb131505bd0b9f396323520fd9ccf2fa7b4c618645af" gracePeriod=30
Jul 24 15:39:00.324688 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:00.324649    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-65c49f07-7554d55fb6-gvhpv"]
Jul 24 15:39:00.324945 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:00.324904    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-65c49f07-7554d55fb6-gvhpv" podUID="1d2ac9f2-86d3-41e7-b1fb-bbeb10624f28" containerName="payload-processing" containerID="cri-o://164f79ffb5f5db86ecef4385bc479e9ec7e1b462687e7281be7b64ea3784eda9" gracePeriod=30
Jul 24 15:39:00.342798 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:00.342757    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-65c49f07-6f6bd96956-x5dsn"]
Jul 24 15:39:00.343418 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:00.343001    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-65c49f07-6f6bd96956-x5dsn" podUID="ba328fe3-e40e-481b-a9c9-c5e7325cbe1f" containerName="payload-pre-processing" containerID="cri-o://9788133eb4de02cc801e82450dfc2f87aeb47d243c4ddc2403234a896d408694" gracePeriod=30
Jul 24 15:39:00.602560 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:00.602529    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-65c49f07-b9997f454-v8cvv"
Jul 24 15:39:00.631698 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:00.631669    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-65c49f07-7554d55fb6-gvhpv"
Jul 24 15:39:00.661268 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:00.661243    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-65c49f07-6f6bd96956-x5dsn"
Jul 24 15:39:00.721764 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:00.721730    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8f92261-b174-43ef-b06c-7d4d978e407e-maas-api-tls\") pod \"e8f92261-b174-43ef-b06c-7d4d978e407e\" (UID: \"e8f92261-b174-43ef-b06c-7d4d978e407e\") "
Jul 24 15:39:00.721974 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:00.721791    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hphf8\" (UniqueName: \"kubernetes.io/projected/e8f92261-b174-43ef-b06c-7d4d978e407e-kube-api-access-hphf8\") pod \"e8f92261-b174-43ef-b06c-7d4d978e407e\" (UID: \"e8f92261-b174-43ef-b06c-7d4d978e407e\") "
Jul 24 15:39:00.721974 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:00.721867    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fb55\" (UniqueName: \"kubernetes.io/projected/1d2ac9f2-86d3-41e7-b1fb-bbeb10624f28-kube-api-access-8fb55\") pod \"1d2ac9f2-86d3-41e7-b1fb-bbeb10624f28\" (UID: \"1d2ac9f2-86d3-41e7-b1fb-bbeb10624f28\") "
Jul 24 15:39:00.721974 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:00.721899    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba328fe3-e40e-481b-a9c9-c5e7325cbe1f-plugins-config-volume\") pod \"ba328fe3-e40e-481b-a9c9-c5e7325cbe1f\" (UID: \"ba328fe3-e40e-481b-a9c9-c5e7325cbe1f\") "
Jul 24 15:39:00.722116 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:00.721983    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d2ac9f2-86d3-41e7-b1fb-bbeb10624f28-plugins-config-volume\") pod \"1d2ac9f2-86d3-41e7-b1fb-bbeb10624f28\" (UID: \"1d2ac9f2-86d3-41e7-b1fb-bbeb10624f28\") "
Jul 24 15:39:00.722116 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:00.722030    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8f92261-b174-43ef-b06c-7d4d978e407e-openshift-service-ca\") pod \"e8f92261-b174-43ef-b06c-7d4d978e407e\" (UID: \"e8f92261-b174-43ef-b06c-7d4d978e407e\") "
Jul 24 15:39:00.722116 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:00.722061    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wb4hx\" (UniqueName: \"kubernetes.io/projected/ba328fe3-e40e-481b-a9c9-c5e7325cbe1f-kube-api-access-wb4hx\") pod \"ba328fe3-e40e-481b-a9c9-c5e7325cbe1f\" (UID: \"ba328fe3-e40e-481b-a9c9-c5e7325cbe1f\") "
Jul 24 15:39:00.722461 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:00.722410    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8f92261-b174-43ef-b06c-7d4d978e407e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e8f92261-b174-43ef-b06c-7d4d978e407e" (UID: "e8f92261-b174-43ef-b06c-7d4d978e407e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:39:00.722461 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:00.722431    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba328fe3-e40e-481b-a9c9-c5e7325cbe1f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ba328fe3-e40e-481b-a9c9-c5e7325cbe1f" (UID: "ba328fe3-e40e-481b-a9c9-c5e7325cbe1f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:39:00.722461 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:00.722439    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d2ac9f2-86d3-41e7-b1fb-bbeb10624f28-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1d2ac9f2-86d3-41e7-b1fb-bbeb10624f28" (UID: "1d2ac9f2-86d3-41e7-b1fb-bbeb10624f28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:39:00.724103 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:00.724073    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d2ac9f2-86d3-41e7-b1fb-bbeb10624f28-kube-api-access-8fb55" (OuterVolumeSpecName: "kube-api-access-8fb55") pod "1d2ac9f2-86d3-41e7-b1fb-bbeb10624f28" (UID: "1d2ac9f2-86d3-41e7-b1fb-bbeb10624f28"). InnerVolumeSpecName "kube-api-access-8fb55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:39:00.724265 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:00.724248    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba328fe3-e40e-481b-a9c9-c5e7325cbe1f-kube-api-access-wb4hx" (OuterVolumeSpecName: "kube-api-access-wb4hx") pod "ba328fe3-e40e-481b-a9c9-c5e7325cbe1f" (UID: "ba328fe3-e40e-481b-a9c9-c5e7325cbe1f"). InnerVolumeSpecName "kube-api-access-wb4hx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:39:00.724412 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:00.724396    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8f92261-b174-43ef-b06c-7d4d978e407e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e8f92261-b174-43ef-b06c-7d4d978e407e" (UID: "e8f92261-b174-43ef-b06c-7d4d978e407e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 15:39:00.724455 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:00.724407    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8f92261-b174-43ef-b06c-7d4d978e407e-kube-api-access-hphf8" (OuterVolumeSpecName: "kube-api-access-hphf8") pod "e8f92261-b174-43ef-b06c-7d4d978e407e" (UID: "e8f92261-b174-43ef-b06c-7d4d978e407e"). InnerVolumeSpecName "kube-api-access-hphf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:39:00.823147 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:00.823106    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fb55\" (UniqueName: \"kubernetes.io/projected/1d2ac9f2-86d3-41e7-b1fb-bbeb10624f28-kube-api-access-8fb55\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:39:00.823147 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:00.823142    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba328fe3-e40e-481b-a9c9-c5e7325cbe1f-plugins-config-volume\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:39:00.823147 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:00.823151    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d2ac9f2-86d3-41e7-b1fb-bbeb10624f28-plugins-config-volume\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:39:00.823391 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:00.823160    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8f92261-b174-43ef-b06c-7d4d978e407e-openshift-service-ca\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:39:00.823391 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:00.823169    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb4hx\" (UniqueName: \"kubernetes.io/projected/ba328fe3-e40e-481b-a9c9-c5e7325cbe1f-kube-api-access-wb4hx\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:39:00.823391 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:00.823178    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8f92261-b174-43ef-b06c-7d4d978e407e-maas-api-tls\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:39:00.823391 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:00.823186    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hphf8\" (UniqueName: \"kubernetes.io/projected/e8f92261-b174-43ef-b06c-7d4d978e407e-kube-api-access-hphf8\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:39:01.297859 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:01.297803    2579 generic.go:356] "Generic (PLEG): container finished" podID="e8f92261-b174-43ef-b06c-7d4d978e407e" containerID="d051edffaa82a3a28152cb131505bd0b9f396323520fd9ccf2fa7b4c618645af" exitCode=0
Jul 24 15:39:01.297859 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:01.297848    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-65c49f07-b9997f454-v8cvv" event={"ID":"e8f92261-b174-43ef-b06c-7d4d978e407e","Type":"ContainerDied","Data":"d051edffaa82a3a28152cb131505bd0b9f396323520fd9ccf2fa7b4c618645af"}
Jul 24 15:39:01.298399 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:01.297874    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-65c49f07-b9997f454-v8cvv"
Jul 24 15:39:01.298399 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:01.297910    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-65c49f07-b9997f454-v8cvv" event={"ID":"e8f92261-b174-43ef-b06c-7d4d978e407e","Type":"ContainerDied","Data":"3719e15b7b664089c90ccd85217edf9018185b0ae039f1b8abc161e735876c7d"}
Jul 24 15:39:01.298399 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:01.297933    2579 scope.go:117] "RemoveContainer" containerID="d051edffaa82a3a28152cb131505bd0b9f396323520fd9ccf2fa7b4c618645af"
Jul 24 15:39:01.299492 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:01.299240    2579 generic.go:356] "Generic (PLEG): container finished" podID="1d2ac9f2-86d3-41e7-b1fb-bbeb10624f28" containerID="164f79ffb5f5db86ecef4385bc479e9ec7e1b462687e7281be7b64ea3784eda9" exitCode=0
Jul 24 15:39:01.299492 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:01.299284    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-65c49f07-7554d55fb6-gvhpv"
Jul 24 15:39:01.299492 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:01.299306    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-65c49f07-7554d55fb6-gvhpv" event={"ID":"1d2ac9f2-86d3-41e7-b1fb-bbeb10624f28","Type":"ContainerDied","Data":"164f79ffb5f5db86ecef4385bc479e9ec7e1b462687e7281be7b64ea3784eda9"}
Jul 24 15:39:01.299492 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:01.299346    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-65c49f07-7554d55fb6-gvhpv" event={"ID":"1d2ac9f2-86d3-41e7-b1fb-bbeb10624f28","Type":"ContainerDied","Data":"d2c477a382713476808fa44c03db8577dae60a5baf56037dbed47f111b7f4361"}
Jul 24 15:39:01.300581 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:01.300561    2579 generic.go:356] "Generic (PLEG): container finished" podID="ba328fe3-e40e-481b-a9c9-c5e7325cbe1f" containerID="9788133eb4de02cc801e82450dfc2f87aeb47d243c4ddc2403234a896d408694" exitCode=0
Jul 24 15:39:01.300681 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:01.300607    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-65c49f07-6f6bd96956-x5dsn"
Jul 24 15:39:01.300681 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:01.300646    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-65c49f07-6f6bd96956-x5dsn" event={"ID":"ba328fe3-e40e-481b-a9c9-c5e7325cbe1f","Type":"ContainerDied","Data":"9788133eb4de02cc801e82450dfc2f87aeb47d243c4ddc2403234a896d408694"}
Jul 24 15:39:01.300761 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:01.300688    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-65c49f07-6f6bd96956-x5dsn" event={"ID":"ba328fe3-e40e-481b-a9c9-c5e7325cbe1f","Type":"ContainerDied","Data":"2500fe6783981f8d5c4c2d8440464302dab395dff51507a656588a13419b4b72"}
Jul 24 15:39:01.308090 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:01.308057    2579 scope.go:117] "RemoveContainer" containerID="d051edffaa82a3a28152cb131505bd0b9f396323520fd9ccf2fa7b4c618645af"
Jul 24 15:39:01.308349 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:39:01.308331    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d051edffaa82a3a28152cb131505bd0b9f396323520fd9ccf2fa7b4c618645af\": container with ID starting with d051edffaa82a3a28152cb131505bd0b9f396323520fd9ccf2fa7b4c618645af not found: ID does not exist" containerID="d051edffaa82a3a28152cb131505bd0b9f396323520fd9ccf2fa7b4c618645af"
Jul 24 15:39:01.308414 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:01.308356    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d051edffaa82a3a28152cb131505bd0b9f396323520fd9ccf2fa7b4c618645af"} err="failed to get container status \"d051edffaa82a3a28152cb131505bd0b9f396323520fd9ccf2fa7b4c618645af\": rpc error: code = NotFound desc = could not find container \"d051edffaa82a3a28152cb131505bd0b9f396323520fd9ccf2fa7b4c618645af\": container with ID starting with d051edffaa82a3a28152cb131505bd0b9f396323520fd9ccf2fa7b4c618645af not found: ID does not exist"
Jul 24 15:39:01.308414 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:01.308375    2579 scope.go:117] "RemoveContainer" containerID="164f79ffb5f5db86ecef4385bc479e9ec7e1b462687e7281be7b64ea3784eda9"
Jul 24 15:39:01.318002 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:01.317972    2579 scope.go:117] "RemoveContainer" containerID="164f79ffb5f5db86ecef4385bc479e9ec7e1b462687e7281be7b64ea3784eda9"
Jul 24 15:39:01.318225 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:39:01.318205    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"164f79ffb5f5db86ecef4385bc479e9ec7e1b462687e7281be7b64ea3784eda9\": container with ID starting with 164f79ffb5f5db86ecef4385bc479e9ec7e1b462687e7281be7b64ea3784eda9 not found: ID does not exist" containerID="164f79ffb5f5db86ecef4385bc479e9ec7e1b462687e7281be7b64ea3784eda9"
Jul 24 15:39:01.318273 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:01.318232    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"164f79ffb5f5db86ecef4385bc479e9ec7e1b462687e7281be7b64ea3784eda9"} err="failed to get container status \"164f79ffb5f5db86ecef4385bc479e9ec7e1b462687e7281be7b64ea3784eda9\": rpc error: code = NotFound desc = could not find container \"164f79ffb5f5db86ecef4385bc479e9ec7e1b462687e7281be7b64ea3784eda9\": container with ID starting with 164f79ffb5f5db86ecef4385bc479e9ec7e1b462687e7281be7b64ea3784eda9 not found: ID does not exist"
Jul 24 15:39:01.318273 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:01.318249    2579 scope.go:117] "RemoveContainer" containerID="9788133eb4de02cc801e82450dfc2f87aeb47d243c4ddc2403234a896d408694"
Jul 24 15:39:01.326097 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:01.326079    2579 scope.go:117] "RemoveContainer" containerID="9788133eb4de02cc801e82450dfc2f87aeb47d243c4ddc2403234a896d408694"
Jul 24 15:39:01.326379 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:39:01.326356    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9788133eb4de02cc801e82450dfc2f87aeb47d243c4ddc2403234a896d408694\": container with ID starting with 9788133eb4de02cc801e82450dfc2f87aeb47d243c4ddc2403234a896d408694 not found: ID does not exist" containerID="9788133eb4de02cc801e82450dfc2f87aeb47d243c4ddc2403234a896d408694"
Jul 24 15:39:01.326467 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:01.326395    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9788133eb4de02cc801e82450dfc2f87aeb47d243c4ddc2403234a896d408694"} err="failed to get container status \"9788133eb4de02cc801e82450dfc2f87aeb47d243c4ddc2403234a896d408694\": rpc error: code = NotFound desc = could not find container \"9788133eb4de02cc801e82450dfc2f87aeb47d243c4ddc2403234a896d408694\": container with ID starting with 9788133eb4de02cc801e82450dfc2f87aeb47d243c4ddc2403234a896d408694 not found: ID does not exist"
Jul 24 15:39:01.329562 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:01.329540    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-65c49f07-b9997f454-v8cvv"]
Jul 24 15:39:01.333038 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:01.333019    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-65c49f07-b9997f454-v8cvv"]
Jul 24 15:39:01.347607 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:01.347580    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-65c49f07-7554d55fb6-gvhpv"]
Jul 24 15:39:01.353828 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:01.353791    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-65c49f07-7554d55fb6-gvhpv"]
Jul 24 15:39:01.374278 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:01.374246    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-65c49f07-6f6bd96956-x5dsn"]
Jul 24 15:39:01.381719 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:01.381693    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-65c49f07-6f6bd96956-x5dsn"]
Jul 24 15:39:01.717913 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:01.717878    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d2ac9f2-86d3-41e7-b1fb-bbeb10624f28" path="/var/lib/kubelet/pods/1d2ac9f2-86d3-41e7-b1fb-bbeb10624f28/volumes"
Jul 24 15:39:01.718290 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:01.718276    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba328fe3-e40e-481b-a9c9-c5e7325cbe1f" path="/var/lib/kubelet/pods/ba328fe3-e40e-481b-a9c9-c5e7325cbe1f/volumes"
Jul 24 15:39:01.718641 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:01.718629    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8f92261-b174-43ef-b06c-7d4d978e407e" path="/var/lib/kubelet/pods/e8f92261-b174-43ef-b06c-7d4d978e407e/volumes"
Jul 24 15:39:02.971597 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:39:02.971555    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:40:34.441049 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:34.441011    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:40:47.134345 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:47.134302    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:40:51.090780 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:51.090740    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a586f80f-59c8f84cb-8gbkd"]
Jul 24 15:40:51.091321 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:51.091306    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba328fe3-e40e-481b-a9c9-c5e7325cbe1f" containerName="payload-pre-processing"
Jul 24 15:40:51.091368 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:51.091324    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba328fe3-e40e-481b-a9c9-c5e7325cbe1f" containerName="payload-pre-processing"
Jul 24 15:40:51.091368 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:51.091351    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8f92261-b174-43ef-b06c-7d4d978e407e" containerName="maas-api"
Jul 24 15:40:51.091368 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:51.091359    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8f92261-b174-43ef-b06c-7d4d978e407e" containerName="maas-api"
Jul 24 15:40:51.091467 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:51.091382    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d2ac9f2-86d3-41e7-b1fb-bbeb10624f28" containerName="payload-processing"
Jul 24 15:40:51.091467 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:51.091393    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d2ac9f2-86d3-41e7-b1fb-bbeb10624f28" containerName="payload-processing"
Jul 24 15:40:51.091530 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:51.091487    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba328fe3-e40e-481b-a9c9-c5e7325cbe1f" containerName="payload-pre-processing"
Jul 24 15:40:51.091530 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:51.091502    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8f92261-b174-43ef-b06c-7d4d978e407e" containerName="maas-api"
Jul 24 15:40:51.091530 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:51.091512    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d2ac9f2-86d3-41e7-b1fb-bbeb10624f28" containerName="payload-processing"
Jul 24 15:40:51.095102 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:51.095080    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a586f80f-59c8f84cb-8gbkd"
Jul 24 15:40:51.098447 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:51.098427    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 15:40:51.098772 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:51.098756    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tkn9s\""
Jul 24 15:40:51.099434 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:51.099413    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 15:40:51.099518 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:51.099413    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-a586f80f-serving-cert\""
Jul 24 15:40:51.105307 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:51.105283    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a586f80f-59c8f84cb-8gbkd"]
Jul 24 15:40:51.214258 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:51.214224    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5cc9c82-1536-4f34-a8e0-e92b4f487648-maas-api-tls\") pod \"maas-api-e2e-mt-a586f80f-59c8f84cb-8gbkd\" (UID: \"f5cc9c82-1536-4f34-a8e0-e92b4f487648\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a586f80f-59c8f84cb-8gbkd"
Jul 24 15:40:51.214484 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:51.214275    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5cc9c82-1536-4f34-a8e0-e92b4f487648-openshift-service-ca\") pod \"maas-api-e2e-mt-a586f80f-59c8f84cb-8gbkd\" (UID: \"f5cc9c82-1536-4f34-a8e0-e92b4f487648\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a586f80f-59c8f84cb-8gbkd"
Jul 24 15:40:51.214484 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:51.214405    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfbv8\" (UniqueName: \"kubernetes.io/projected/f5cc9c82-1536-4f34-a8e0-e92b4f487648-kube-api-access-sfbv8\") pod \"maas-api-e2e-mt-a586f80f-59c8f84cb-8gbkd\" (UID: \"f5cc9c82-1536-4f34-a8e0-e92b4f487648\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a586f80f-59c8f84cb-8gbkd"
Jul 24 15:40:51.315704 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:51.315651    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfbv8\" (UniqueName: \"kubernetes.io/projected/f5cc9c82-1536-4f34-a8e0-e92b4f487648-kube-api-access-sfbv8\") pod \"maas-api-e2e-mt-a586f80f-59c8f84cb-8gbkd\" (UID: \"f5cc9c82-1536-4f34-a8e0-e92b4f487648\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a586f80f-59c8f84cb-8gbkd"
Jul 24 15:40:51.315943 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:51.315760    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5cc9c82-1536-4f34-a8e0-e92b4f487648-maas-api-tls\") pod \"maas-api-e2e-mt-a586f80f-59c8f84cb-8gbkd\" (UID: \"f5cc9c82-1536-4f34-a8e0-e92b4f487648\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a586f80f-59c8f84cb-8gbkd"
Jul 24 15:40:51.315943 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:51.315799    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5cc9c82-1536-4f34-a8e0-e92b4f487648-openshift-service-ca\") pod \"maas-api-e2e-mt-a586f80f-59c8f84cb-8gbkd\" (UID: \"f5cc9c82-1536-4f34-a8e0-e92b4f487648\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a586f80f-59c8f84cb-8gbkd"
Jul 24 15:40:51.315943 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:40:51.315904    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-a586f80f-serving-cert: secret "maas-api-e2e-mt-a586f80f-serving-cert" not found
Jul 24 15:40:51.316109 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:40:51.315959    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5cc9c82-1536-4f34-a8e0-e92b4f487648-maas-api-tls podName:f5cc9c82-1536-4f34-a8e0-e92b4f487648 nodeName:}" failed. No retries permitted until 2026-07-24 15:40:51.815943725 +0000 UTC m=+1944.628295882 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f5cc9c82-1536-4f34-a8e0-e92b4f487648-maas-api-tls") pod "maas-api-e2e-mt-a586f80f-59c8f84cb-8gbkd" (UID: "f5cc9c82-1536-4f34-a8e0-e92b4f487648") : secret "maas-api-e2e-mt-a586f80f-serving-cert" not found
Jul 24 15:40:51.316439 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:51.316421    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5cc9c82-1536-4f34-a8e0-e92b4f487648-openshift-service-ca\") pod \"maas-api-e2e-mt-a586f80f-59c8f84cb-8gbkd\" (UID: \"f5cc9c82-1536-4f34-a8e0-e92b4f487648\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a586f80f-59c8f84cb-8gbkd"
Jul 24 15:40:51.328045 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:51.328018    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfbv8\" (UniqueName: \"kubernetes.io/projected/f5cc9c82-1536-4f34-a8e0-e92b4f487648-kube-api-access-sfbv8\") pod \"maas-api-e2e-mt-a586f80f-59c8f84cb-8gbkd\" (UID: \"f5cc9c82-1536-4f34-a8e0-e92b4f487648\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a586f80f-59c8f84cb-8gbkd"
Jul 24 15:40:51.822094 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:51.822053    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5cc9c82-1536-4f34-a8e0-e92b4f487648-maas-api-tls\") pod \"maas-api-e2e-mt-a586f80f-59c8f84cb-8gbkd\" (UID: \"f5cc9c82-1536-4f34-a8e0-e92b4f487648\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a586f80f-59c8f84cb-8gbkd"
Jul 24 15:40:51.824461 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:51.824434    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5cc9c82-1536-4f34-a8e0-e92b4f487648-maas-api-tls\") pod \"maas-api-e2e-mt-a586f80f-59c8f84cb-8gbkd\" (UID: \"f5cc9c82-1536-4f34-a8e0-e92b4f487648\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a586f80f-59c8f84cb-8gbkd"
Jul 24 15:40:51.893838 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:51.893787    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a586f80f-74bfc7579b-r7mk4"]
Jul 24 15:40:51.897628 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:51.897601    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a586f80f-59cf8fbc94-lscf2"]
Jul 24 15:40:51.897784 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:51.897710    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a586f80f-74bfc7579b-r7mk4"
Jul 24 15:40:51.901224 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:51.901202    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a586f80f-59cf8fbc94-lscf2"
Jul 24 15:40:51.901712 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:51.901696    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-a586f80f\""
Jul 24 15:40:51.909533 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:51.909505    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a586f80f-74bfc7579b-r7mk4"]
Jul 24 15:40:51.917147 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:51.917118    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a586f80f-59cf8fbc94-lscf2"]
Jul 24 15:40:52.006685 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:52.006639    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a586f80f-59c8f84cb-8gbkd"
Jul 24 15:40:52.024875 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:52.024844    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d31fe1d-72ba-473d-bf2e-f6c7f5577308-plugins-config-volume\") pod \"payload-processing-e2e-mt-a586f80f-59cf8fbc94-lscf2\" (UID: \"6d31fe1d-72ba-473d-bf2e-f6c7f5577308\") " pod="openshift-ingress/payload-processing-e2e-mt-a586f80f-59cf8fbc94-lscf2"
Jul 24 15:40:52.025029 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:52.024901    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/104dc7ec-b937-44c9-b5ea-eb6d4a45f183-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a586f80f-74bfc7579b-r7mk4\" (UID: \"104dc7ec-b937-44c9-b5ea-eb6d4a45f183\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a586f80f-74bfc7579b-r7mk4"
Jul 24 15:40:52.025029 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:52.024989    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcqzc\" (UniqueName: \"kubernetes.io/projected/6d31fe1d-72ba-473d-bf2e-f6c7f5577308-kube-api-access-tcqzc\") pod \"payload-processing-e2e-mt-a586f80f-59cf8fbc94-lscf2\" (UID: \"6d31fe1d-72ba-473d-bf2e-f6c7f5577308\") " pod="openshift-ingress/payload-processing-e2e-mt-a586f80f-59cf8fbc94-lscf2"
Jul 24 15:40:52.025152 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:52.025064    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldbr2\" (UniqueName: \"kubernetes.io/projected/104dc7ec-b937-44c9-b5ea-eb6d4a45f183-kube-api-access-ldbr2\") pod \"payload-pre-processing-e2e-mt-a586f80f-74bfc7579b-r7mk4\" (UID: \"104dc7ec-b937-44c9-b5ea-eb6d4a45f183\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a586f80f-74bfc7579b-r7mk4"
Jul 24 15:40:52.126294 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:52.126258    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldbr2\" (UniqueName: \"kubernetes.io/projected/104dc7ec-b937-44c9-b5ea-eb6d4a45f183-kube-api-access-ldbr2\") pod \"payload-pre-processing-e2e-mt-a586f80f-74bfc7579b-r7mk4\" (UID: \"104dc7ec-b937-44c9-b5ea-eb6d4a45f183\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a586f80f-74bfc7579b-r7mk4"
Jul 24 15:40:52.126677 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:52.126411    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d31fe1d-72ba-473d-bf2e-f6c7f5577308-plugins-config-volume\") pod \"payload-processing-e2e-mt-a586f80f-59cf8fbc94-lscf2\" (UID: \"6d31fe1d-72ba-473d-bf2e-f6c7f5577308\") " pod="openshift-ingress/payload-processing-e2e-mt-a586f80f-59cf8fbc94-lscf2"
Jul 24 15:40:52.126677 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:52.126452    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/104dc7ec-b937-44c9-b5ea-eb6d4a45f183-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a586f80f-74bfc7579b-r7mk4\" (UID: \"104dc7ec-b937-44c9-b5ea-eb6d4a45f183\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a586f80f-74bfc7579b-r7mk4"
Jul 24 15:40:52.126677 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:52.126470    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcqzc\" (UniqueName: \"kubernetes.io/projected/6d31fe1d-72ba-473d-bf2e-f6c7f5577308-kube-api-access-tcqzc\") pod \"payload-processing-e2e-mt-a586f80f-59cf8fbc94-lscf2\" (UID: \"6d31fe1d-72ba-473d-bf2e-f6c7f5577308\") " pod="openshift-ingress/payload-processing-e2e-mt-a586f80f-59cf8fbc94-lscf2"
Jul 24 15:40:52.127353 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:52.127329    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d31fe1d-72ba-473d-bf2e-f6c7f5577308-plugins-config-volume\") pod \"payload-processing-e2e-mt-a586f80f-59cf8fbc94-lscf2\" (UID: \"6d31fe1d-72ba-473d-bf2e-f6c7f5577308\") " pod="openshift-ingress/payload-processing-e2e-mt-a586f80f-59cf8fbc94-lscf2"
Jul 24 15:40:52.127453 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:52.127329    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/104dc7ec-b937-44c9-b5ea-eb6d4a45f183-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a586f80f-74bfc7579b-r7mk4\" (UID: \"104dc7ec-b937-44c9-b5ea-eb6d4a45f183\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a586f80f-74bfc7579b-r7mk4"
Jul 24 15:40:52.135186 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:52.135158    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldbr2\" (UniqueName: \"kubernetes.io/projected/104dc7ec-b937-44c9-b5ea-eb6d4a45f183-kube-api-access-ldbr2\") pod \"payload-pre-processing-e2e-mt-a586f80f-74bfc7579b-r7mk4\" (UID: \"104dc7ec-b937-44c9-b5ea-eb6d4a45f183\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a586f80f-74bfc7579b-r7mk4"
Jul 24 15:40:52.135769 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:52.135747    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcqzc\" (UniqueName: \"kubernetes.io/projected/6d31fe1d-72ba-473d-bf2e-f6c7f5577308-kube-api-access-tcqzc\") pod \"payload-processing-e2e-mt-a586f80f-59cf8fbc94-lscf2\" (UID: \"6d31fe1d-72ba-473d-bf2e-f6c7f5577308\") " pod="openshift-ingress/payload-processing-e2e-mt-a586f80f-59cf8fbc94-lscf2"
Jul 24 15:40:52.145530 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:52.145502    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a586f80f-59c8f84cb-8gbkd"]
Jul 24 15:40:52.147253 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:40:52.147229    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cc9c82_1536_4f34_a8e0_e92b4f487648.slice/crio-7837bf02e1ff9cac6ca245de055789385ddae3d162acbe0ea4865374e9ee524f WatchSource:0}: Error finding container 7837bf02e1ff9cac6ca245de055789385ddae3d162acbe0ea4865374e9ee524f: Status 404 returned error can't find the container with id 7837bf02e1ff9cac6ca245de055789385ddae3d162acbe0ea4865374e9ee524f
Jul 24 15:40:52.210265 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:52.210228    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a586f80f-74bfc7579b-r7mk4"
Jul 24 15:40:52.218736 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:52.218712    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a586f80f-59cf8fbc94-lscf2"
Jul 24 15:40:52.360181 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:52.360157    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a586f80f-74bfc7579b-r7mk4"]
Jul 24 15:40:52.362425 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:40:52.362393    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod104dc7ec_b937_44c9_b5ea_eb6d4a45f183.slice/crio-f2e0b2856c603ff01864ecd3188f379ee5c6d51bff5cbe9952eefda25396beb9 WatchSource:0}: Error finding container f2e0b2856c603ff01864ecd3188f379ee5c6d51bff5cbe9952eefda25396beb9: Status 404 returned error can't find the container with id f2e0b2856c603ff01864ecd3188f379ee5c6d51bff5cbe9952eefda25396beb9
Jul 24 15:40:52.377629 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:52.377607    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a586f80f-59cf8fbc94-lscf2"]
Jul 24 15:40:52.379446 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:40:52.379423    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d31fe1d_72ba_473d_bf2e_f6c7f5577308.slice/crio-3624d9c040ca24c2a5bdba1ced1911cc7521967d950283fe3ced0982c6511965 WatchSource:0}: Error finding container 3624d9c040ca24c2a5bdba1ced1911cc7521967d950283fe3ced0982c6511965: Status 404 returned error can't find the container with id 3624d9c040ca24c2a5bdba1ced1911cc7521967d950283fe3ced0982c6511965
Jul 24 15:40:52.842783 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:52.842743    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a586f80f-74bfc7579b-r7mk4" event={"ID":"104dc7ec-b937-44c9-b5ea-eb6d4a45f183","Type":"ContainerStarted","Data":"6e337dd2af519778435e0b9dfd26db5153b76ce024367fb3e2e89d5ca22d81c7"}
Jul 24 15:40:52.843044 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:52.842808    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a586f80f-74bfc7579b-r7mk4"
Jul 24 15:40:52.843044 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:52.842848    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a586f80f-74bfc7579b-r7mk4" event={"ID":"104dc7ec-b937-44c9-b5ea-eb6d4a45f183","Type":"ContainerStarted","Data":"f2e0b2856c603ff01864ecd3188f379ee5c6d51bff5cbe9952eefda25396beb9"}
Jul 24 15:40:52.844462 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:52.844426    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a586f80f-59c8f84cb-8gbkd" event={"ID":"f5cc9c82-1536-4f34-a8e0-e92b4f487648","Type":"ContainerStarted","Data":"5ee4ffb4d047eced03c1d4ac1be9fc27467a0a52b77c8e67eedf8638a7b313da"}
Jul 24 15:40:52.844581 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:52.844478    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a586f80f-59c8f84cb-8gbkd" event={"ID":"f5cc9c82-1536-4f34-a8e0-e92b4f487648","Type":"ContainerStarted","Data":"7837bf02e1ff9cac6ca245de055789385ddae3d162acbe0ea4865374e9ee524f"}
Jul 24 15:40:52.844581 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:52.844507    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a586f80f-59c8f84cb-8gbkd"
Jul 24 15:40:52.845995 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:52.845971    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a586f80f-59cf8fbc94-lscf2" event={"ID":"6d31fe1d-72ba-473d-bf2e-f6c7f5577308","Type":"ContainerStarted","Data":"bff8b9a8e45f1f48a74f4f711773d5c574be79a92f28884e220856a8d3bf2d51"}
Jul 24 15:40:52.846084 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:52.846016    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a586f80f-59cf8fbc94-lscf2" event={"ID":"6d31fe1d-72ba-473d-bf2e-f6c7f5577308","Type":"ContainerStarted","Data":"3624d9c040ca24c2a5bdba1ced1911cc7521967d950283fe3ced0982c6511965"}
Jul 24 15:40:52.846084 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:52.846058    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-a586f80f-59cf8fbc94-lscf2"
Jul 24 15:40:52.867240 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:52.867193    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-a586f80f-74bfc7579b-r7mk4" podStartSLOduration=1.867178196 podStartE2EDuration="1.867178196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:40:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:40:52.864104889 +0000 UTC m=+1945.676457069" watchObservedRunningTime="2026-07-24 15:40:52.867178196 +0000 UTC m=+1945.679530384"
Jul 24 15:40:52.883566 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:52.883515    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-a586f80f-59cf8fbc94-lscf2" podStartSLOduration=1.8835025399999998 podStartE2EDuration="1.88350254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:40:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:40:52.882099955 +0000 UTC m=+1945.694452134" watchObservedRunningTime="2026-07-24 15:40:52.88350254 +0000 UTC m=+1945.695854696"
Jul 24 15:40:52.908942 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:52.908793    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a586f80f-59c8f84cb-8gbkd" podStartSLOduration=1.385399374 podStartE2EDuration="1.908776997s" totalImagesPullingTime="523.377623ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:40:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:40:52.906891484 +0000 UTC m=+1945.719243664" watchObservedRunningTime="2026-07-24 15:40:52.908776997 +0000 UTC m=+1945.721129173"
Jul 24 15:40:58.856445 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:40:58.856414    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a586f80f-59c8f84cb-8gbkd"
Jul 24 15:41:01.734054 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:01.734019    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:41:03.853561 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:03.853530    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-a586f80f-59cf8fbc94-lscf2"
Jul 24 15:41:03.854064 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:03.853805    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a586f80f-74bfc7579b-r7mk4"
Jul 24 15:41:13.397622 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.397588    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a586f80f-59c8f84cb-8gbkd"]
Jul 24 15:41:13.398154 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.397807    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a586f80f-59c8f84cb-8gbkd" podUID="f5cc9c82-1536-4f34-a8e0-e92b4f487648" containerName="maas-api" containerID="cri-o://5ee4ffb4d047eced03c1d4ac1be9fc27467a0a52b77c8e67eedf8638a7b313da" gracePeriod=30
Jul 24 15:41:13.470729 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.470684    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a586f80f-59cf8fbc94-lscf2"]
Jul 24 15:41:13.470969 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.470938    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-a586f80f-59cf8fbc94-lscf2" podUID="6d31fe1d-72ba-473d-bf2e-f6c7f5577308" containerName="payload-processing" containerID="cri-o://bff8b9a8e45f1f48a74f4f711773d5c574be79a92f28884e220856a8d3bf2d51" gracePeriod=30
Jul 24 15:41:13.478913 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.478890    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a586f80f-74bfc7579b-r7mk4"]
Jul 24 15:41:13.479199 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.479143    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-a586f80f-74bfc7579b-r7mk4" podUID="104dc7ec-b937-44c9-b5ea-eb6d4a45f183" containerName="payload-pre-processing" containerID="cri-o://6e337dd2af519778435e0b9dfd26db5153b76ce024367fb3e2e89d5ca22d81c7" gracePeriod=30
Jul 24 15:41:13.754925 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.754896    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a586f80f-59cf8fbc94-lscf2"
Jul 24 15:41:13.762913 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.762885    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a586f80f-74bfc7579b-r7mk4"
Jul 24 15:41:13.771156 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.771134    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a586f80f-59c8f84cb-8gbkd"
Jul 24 15:41:13.833110 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.833078    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5cc9c82-1536-4f34-a8e0-e92b4f487648-maas-api-tls\") pod \"f5cc9c82-1536-4f34-a8e0-e92b4f487648\" (UID: \"f5cc9c82-1536-4f34-a8e0-e92b4f487648\") "
Jul 24 15:41:13.833269 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.833145    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/104dc7ec-b937-44c9-b5ea-eb6d4a45f183-plugins-config-volume\") pod \"104dc7ec-b937-44c9-b5ea-eb6d4a45f183\" (UID: \"104dc7ec-b937-44c9-b5ea-eb6d4a45f183\") "
Jul 24 15:41:13.833269 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.833238    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5cc9c82-1536-4f34-a8e0-e92b4f487648-openshift-service-ca\") pod \"f5cc9c82-1536-4f34-a8e0-e92b4f487648\" (UID: \"f5cc9c82-1536-4f34-a8e0-e92b4f487648\") "
Jul 24 15:41:13.833341 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.833290    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d31fe1d-72ba-473d-bf2e-f6c7f5577308-plugins-config-volume\") pod \"6d31fe1d-72ba-473d-bf2e-f6c7f5577308\" (UID: \"6d31fe1d-72ba-473d-bf2e-f6c7f5577308\") "
Jul 24 15:41:13.833341 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.833325    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tcqzc\" (UniqueName: \"kubernetes.io/projected/6d31fe1d-72ba-473d-bf2e-f6c7f5577308-kube-api-access-tcqzc\") pod \"6d31fe1d-72ba-473d-bf2e-f6c7f5577308\" (UID: \"6d31fe1d-72ba-473d-bf2e-f6c7f5577308\") "
Jul 24 15:41:13.833421 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.833352    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfbv8\" (UniqueName: \"kubernetes.io/projected/f5cc9c82-1536-4f34-a8e0-e92b4f487648-kube-api-access-sfbv8\") pod \"f5cc9c82-1536-4f34-a8e0-e92b4f487648\" (UID: \"f5cc9c82-1536-4f34-a8e0-e92b4f487648\") "
Jul 24 15:41:13.833421 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.833377    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldbr2\" (UniqueName: \"kubernetes.io/projected/104dc7ec-b937-44c9-b5ea-eb6d4a45f183-kube-api-access-ldbr2\") pod \"104dc7ec-b937-44c9-b5ea-eb6d4a45f183\" (UID: \"104dc7ec-b937-44c9-b5ea-eb6d4a45f183\") "
Jul 24 15:41:13.833525 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.833504    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/104dc7ec-b937-44c9-b5ea-eb6d4a45f183-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "104dc7ec-b937-44c9-b5ea-eb6d4a45f183" (UID: "104dc7ec-b937-44c9-b5ea-eb6d4a45f183"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:41:13.833618 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.833583    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5cc9c82-1536-4f34-a8e0-e92b4f487648-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f5cc9c82-1536-4f34-a8e0-e92b4f487648" (UID: "f5cc9c82-1536-4f34-a8e0-e92b4f487648"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:41:13.833839 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.833804    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/104dc7ec-b937-44c9-b5ea-eb6d4a45f183-plugins-config-volume\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:41:13.833939 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.833844    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5cc9c82-1536-4f34-a8e0-e92b4f487648-openshift-service-ca\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:41:13.833939 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.833848    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d31fe1d-72ba-473d-bf2e-f6c7f5577308-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6d31fe1d-72ba-473d-bf2e-f6c7f5577308" (UID: "6d31fe1d-72ba-473d-bf2e-f6c7f5577308"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:41:13.835300 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.835273    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5cc9c82-1536-4f34-a8e0-e92b4f487648-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f5cc9c82-1536-4f34-a8e0-e92b4f487648" (UID: "f5cc9c82-1536-4f34-a8e0-e92b4f487648"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 15:41:13.835430 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.835312    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d31fe1d-72ba-473d-bf2e-f6c7f5577308-kube-api-access-tcqzc" (OuterVolumeSpecName: "kube-api-access-tcqzc") pod "6d31fe1d-72ba-473d-bf2e-f6c7f5577308" (UID: "6d31fe1d-72ba-473d-bf2e-f6c7f5577308"). InnerVolumeSpecName "kube-api-access-tcqzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:41:13.835430 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.835414    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5cc9c82-1536-4f34-a8e0-e92b4f487648-kube-api-access-sfbv8" (OuterVolumeSpecName: "kube-api-access-sfbv8") pod "f5cc9c82-1536-4f34-a8e0-e92b4f487648" (UID: "f5cc9c82-1536-4f34-a8e0-e92b4f487648"). InnerVolumeSpecName "kube-api-access-sfbv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:41:13.835630 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.835607    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/104dc7ec-b937-44c9-b5ea-eb6d4a45f183-kube-api-access-ldbr2" (OuterVolumeSpecName: "kube-api-access-ldbr2") pod "104dc7ec-b937-44c9-b5ea-eb6d4a45f183" (UID: "104dc7ec-b937-44c9-b5ea-eb6d4a45f183"). InnerVolumeSpecName "kube-api-access-ldbr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:41:13.934385 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.934292    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d31fe1d-72ba-473d-bf2e-f6c7f5577308-plugins-config-volume\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:41:13.934385 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.934324    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcqzc\" (UniqueName: \"kubernetes.io/projected/6d31fe1d-72ba-473d-bf2e-f6c7f5577308-kube-api-access-tcqzc\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:41:13.934385 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.934334    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfbv8\" (UniqueName: \"kubernetes.io/projected/f5cc9c82-1536-4f34-a8e0-e92b4f487648-kube-api-access-sfbv8\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:41:13.934385 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.934343    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldbr2\" (UniqueName: \"kubernetes.io/projected/104dc7ec-b937-44c9-b5ea-eb6d4a45f183-kube-api-access-ldbr2\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:41:13.934385 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.934353    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5cc9c82-1536-4f34-a8e0-e92b4f487648-maas-api-tls\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:41:13.951701 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.951662    2579 generic.go:356] "Generic (PLEG): container finished" podID="6d31fe1d-72ba-473d-bf2e-f6c7f5577308" containerID="bff8b9a8e45f1f48a74f4f711773d5c574be79a92f28884e220856a8d3bf2d51" exitCode=0
Jul 24 15:41:13.951879 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.951716    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a586f80f-59cf8fbc94-lscf2"
Jul 24 15:41:13.951879 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.951754    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a586f80f-59cf8fbc94-lscf2" event={"ID":"6d31fe1d-72ba-473d-bf2e-f6c7f5577308","Type":"ContainerDied","Data":"bff8b9a8e45f1f48a74f4f711773d5c574be79a92f28884e220856a8d3bf2d51"}
Jul 24 15:41:13.951879 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.951810    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a586f80f-59cf8fbc94-lscf2" event={"ID":"6d31fe1d-72ba-473d-bf2e-f6c7f5577308","Type":"ContainerDied","Data":"3624d9c040ca24c2a5bdba1ced1911cc7521967d950283fe3ced0982c6511965"}
Jul 24 15:41:13.951879 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.951856    2579 scope.go:117] "RemoveContainer" containerID="bff8b9a8e45f1f48a74f4f711773d5c574be79a92f28884e220856a8d3bf2d51"
Jul 24 15:41:13.953037 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.953018    2579 generic.go:356] "Generic (PLEG): container finished" podID="104dc7ec-b937-44c9-b5ea-eb6d4a45f183" containerID="6e337dd2af519778435e0b9dfd26db5153b76ce024367fb3e2e89d5ca22d81c7" exitCode=0
Jul 24 15:41:13.953105 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.953059    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a586f80f-74bfc7579b-r7mk4"
Jul 24 15:41:13.953105 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.953069    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a586f80f-74bfc7579b-r7mk4" event={"ID":"104dc7ec-b937-44c9-b5ea-eb6d4a45f183","Type":"ContainerDied","Data":"6e337dd2af519778435e0b9dfd26db5153b76ce024367fb3e2e89d5ca22d81c7"}
Jul 24 15:41:13.953216 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.953117    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a586f80f-74bfc7579b-r7mk4" event={"ID":"104dc7ec-b937-44c9-b5ea-eb6d4a45f183","Type":"ContainerDied","Data":"f2e0b2856c603ff01864ecd3188f379ee5c6d51bff5cbe9952eefda25396beb9"}
Jul 24 15:41:13.954493 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.954468    2579 generic.go:356] "Generic (PLEG): container finished" podID="f5cc9c82-1536-4f34-a8e0-e92b4f487648" containerID="5ee4ffb4d047eced03c1d4ac1be9fc27467a0a52b77c8e67eedf8638a7b313da" exitCode=0
Jul 24 15:41:13.954599 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.954504    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a586f80f-59c8f84cb-8gbkd"
Jul 24 15:41:13.954599 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.954510    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a586f80f-59c8f84cb-8gbkd" event={"ID":"f5cc9c82-1536-4f34-a8e0-e92b4f487648","Type":"ContainerDied","Data":"5ee4ffb4d047eced03c1d4ac1be9fc27467a0a52b77c8e67eedf8638a7b313da"}
Jul 24 15:41:13.954599 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.954547    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a586f80f-59c8f84cb-8gbkd" event={"ID":"f5cc9c82-1536-4f34-a8e0-e92b4f487648","Type":"ContainerDied","Data":"7837bf02e1ff9cac6ca245de055789385ddae3d162acbe0ea4865374e9ee524f"}
Jul 24 15:41:13.960896 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.960875    2579 scope.go:117] "RemoveContainer" containerID="bff8b9a8e45f1f48a74f4f711773d5c574be79a92f28884e220856a8d3bf2d51"
Jul 24 15:41:13.961171 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:41:13.961149    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bff8b9a8e45f1f48a74f4f711773d5c574be79a92f28884e220856a8d3bf2d51\": container with ID starting with bff8b9a8e45f1f48a74f4f711773d5c574be79a92f28884e220856a8d3bf2d51 not found: ID does not exist" containerID="bff8b9a8e45f1f48a74f4f711773d5c574be79a92f28884e220856a8d3bf2d51"
Jul 24 15:41:13.961270 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.961179    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bff8b9a8e45f1f48a74f4f711773d5c574be79a92f28884e220856a8d3bf2d51"} err="failed to get container status \"bff8b9a8e45f1f48a74f4f711773d5c574be79a92f28884e220856a8d3bf2d51\": rpc error: code = NotFound desc = could not find container \"bff8b9a8e45f1f48a74f4f711773d5c574be79a92f28884e220856a8d3bf2d51\": container with ID starting with bff8b9a8e45f1f48a74f4f711773d5c574be79a92f28884e220856a8d3bf2d51 not found: ID does not exist"
Jul 24 15:41:13.961270 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.961196    2579 scope.go:117] "RemoveContainer" containerID="6e337dd2af519778435e0b9dfd26db5153b76ce024367fb3e2e89d5ca22d81c7"
Jul 24 15:41:13.970046 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.970025    2579 scope.go:117] "RemoveContainer" containerID="6e337dd2af519778435e0b9dfd26db5153b76ce024367fb3e2e89d5ca22d81c7"
Jul 24 15:41:13.970303 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:41:13.970284    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e337dd2af519778435e0b9dfd26db5153b76ce024367fb3e2e89d5ca22d81c7\": container with ID starting with 6e337dd2af519778435e0b9dfd26db5153b76ce024367fb3e2e89d5ca22d81c7 not found: ID does not exist" containerID="6e337dd2af519778435e0b9dfd26db5153b76ce024367fb3e2e89d5ca22d81c7"
Jul 24 15:41:13.970348 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.970312    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e337dd2af519778435e0b9dfd26db5153b76ce024367fb3e2e89d5ca22d81c7"} err="failed to get container status \"6e337dd2af519778435e0b9dfd26db5153b76ce024367fb3e2e89d5ca22d81c7\": rpc error: code = NotFound desc = could not find container \"6e337dd2af519778435e0b9dfd26db5153b76ce024367fb3e2e89d5ca22d81c7\": container with ID starting with 6e337dd2af519778435e0b9dfd26db5153b76ce024367fb3e2e89d5ca22d81c7 not found: ID does not exist"
Jul 24 15:41:13.970348 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.970333    2579 scope.go:117] "RemoveContainer" containerID="5ee4ffb4d047eced03c1d4ac1be9fc27467a0a52b77c8e67eedf8638a7b313da"
Jul 24 15:41:13.977961 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.977941    2579 scope.go:117] "RemoveContainer" containerID="5ee4ffb4d047eced03c1d4ac1be9fc27467a0a52b77c8e67eedf8638a7b313da"
Jul 24 15:41:13.978204 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:41:13.978186    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ee4ffb4d047eced03c1d4ac1be9fc27467a0a52b77c8e67eedf8638a7b313da\": container with ID starting with 5ee4ffb4d047eced03c1d4ac1be9fc27467a0a52b77c8e67eedf8638a7b313da not found: ID does not exist" containerID="5ee4ffb4d047eced03c1d4ac1be9fc27467a0a52b77c8e67eedf8638a7b313da"
Jul 24 15:41:13.978250 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.978212    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ee4ffb4d047eced03c1d4ac1be9fc27467a0a52b77c8e67eedf8638a7b313da"} err="failed to get container status \"5ee4ffb4d047eced03c1d4ac1be9fc27467a0a52b77c8e67eedf8638a7b313da\": rpc error: code = NotFound desc = could not find container \"5ee4ffb4d047eced03c1d4ac1be9fc27467a0a52b77c8e67eedf8638a7b313da\": container with ID starting with 5ee4ffb4d047eced03c1d4ac1be9fc27467a0a52b77c8e67eedf8638a7b313da not found: ID does not exist"
Jul 24 15:41:13.984075 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.984052    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a586f80f-59c8f84cb-8gbkd"]
Jul 24 15:41:13.987617 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.987599    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a586f80f-59c8f84cb-8gbkd"]
Jul 24 15:41:13.998694 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:13.998670    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a586f80f-59cf8fbc94-lscf2"]
Jul 24 15:41:14.003395 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:14.003376    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a586f80f-59cf8fbc94-lscf2"]
Jul 24 15:41:14.014432 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:14.014408    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a586f80f-74bfc7579b-r7mk4"]
Jul 24 15:41:14.021171 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:14.021139    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a586f80f-74bfc7579b-r7mk4"]
Jul 24 15:41:15.717914 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:15.717884    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="104dc7ec-b937-44c9-b5ea-eb6d4a45f183" path="/var/lib/kubelet/pods/104dc7ec-b937-44c9-b5ea-eb6d4a45f183/volumes"
Jul 24 15:41:15.718309 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:15.718256    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d31fe1d-72ba-473d-bf2e-f6c7f5577308" path="/var/lib/kubelet/pods/6d31fe1d-72ba-473d-bf2e-f6c7f5577308/volumes"
Jul 24 15:41:15.718594 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:15.718581    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5cc9c82-1536-4f34-a8e0-e92b4f487648" path="/var/lib/kubelet/pods/f5cc9c82-1536-4f34-a8e0-e92b4f487648/volumes"
Jul 24 15:41:16.430262 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:16.430223    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:41:48.162282 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:48.162248    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-30fd85f1-8659df6487-rbmks"]
Jul 24 15:41:48.162803 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:48.162787    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5cc9c82-1536-4f34-a8e0-e92b4f487648" containerName="maas-api"
Jul 24 15:41:48.162869 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:48.162806    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5cc9c82-1536-4f34-a8e0-e92b4f487648" containerName="maas-api"
Jul 24 15:41:48.162869 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:48.162862    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d31fe1d-72ba-473d-bf2e-f6c7f5577308" containerName="payload-processing"
Jul 24 15:41:48.162948 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:48.162872    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d31fe1d-72ba-473d-bf2e-f6c7f5577308" containerName="payload-processing"
Jul 24 15:41:48.162948 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:48.162903    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="104dc7ec-b937-44c9-b5ea-eb6d4a45f183" containerName="payload-pre-processing"
Jul 24 15:41:48.162948 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:48.162913    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="104dc7ec-b937-44c9-b5ea-eb6d4a45f183" containerName="payload-pre-processing"
Jul 24 15:41:48.163062 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:48.163002    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="104dc7ec-b937-44c9-b5ea-eb6d4a45f183" containerName="payload-pre-processing"
Jul 24 15:41:48.163062 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:48.163018    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d31fe1d-72ba-473d-bf2e-f6c7f5577308" containerName="payload-processing"
Jul 24 15:41:48.163062 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:48.163028    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5cc9c82-1536-4f34-a8e0-e92b4f487648" containerName="maas-api"
Jul 24 15:41:48.166759 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:48.166735    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-30fd85f1-8659df6487-rbmks"
Jul 24 15:41:48.169168 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:48.169147    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-30fd85f1-serving-cert\""
Jul 24 15:41:48.169738 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:48.169718    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 15:41:48.170172 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:48.170155    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 15:41:48.170286 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:48.170172    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tkn9s\""
Jul 24 15:41:48.176409 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:48.176386    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-30fd85f1-8659df6487-rbmks"]
Jul 24 15:41:48.249439 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:48.249398    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17c767d8-2f74-4545-aa37-92d071aa281a-openshift-service-ca\") pod \"maas-api-e2e-mt-30fd85f1-8659df6487-rbmks\" (UID: \"17c767d8-2f74-4545-aa37-92d071aa281a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-30fd85f1-8659df6487-rbmks"
Jul 24 15:41:48.249439 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:48.249440    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-796bh\" (UniqueName: \"kubernetes.io/projected/17c767d8-2f74-4545-aa37-92d071aa281a-kube-api-access-796bh\") pod \"maas-api-e2e-mt-30fd85f1-8659df6487-rbmks\" (UID: \"17c767d8-2f74-4545-aa37-92d071aa281a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-30fd85f1-8659df6487-rbmks"
Jul 24 15:41:48.249646 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:48.249507    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17c767d8-2f74-4545-aa37-92d071aa281a-maas-api-tls\") pod \"maas-api-e2e-mt-30fd85f1-8659df6487-rbmks\" (UID: \"17c767d8-2f74-4545-aa37-92d071aa281a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-30fd85f1-8659df6487-rbmks"
Jul 24 15:41:48.350184 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:48.350150    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17c767d8-2f74-4545-aa37-92d071aa281a-maas-api-tls\") pod \"maas-api-e2e-mt-30fd85f1-8659df6487-rbmks\" (UID: \"17c767d8-2f74-4545-aa37-92d071aa281a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-30fd85f1-8659df6487-rbmks"
Jul 24 15:41:48.350371 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:48.350252    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17c767d8-2f74-4545-aa37-92d071aa281a-openshift-service-ca\") pod \"maas-api-e2e-mt-30fd85f1-8659df6487-rbmks\" (UID: \"17c767d8-2f74-4545-aa37-92d071aa281a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-30fd85f1-8659df6487-rbmks"
Jul 24 15:41:48.350371 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:48.350282    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-796bh\" (UniqueName: \"kubernetes.io/projected/17c767d8-2f74-4545-aa37-92d071aa281a-kube-api-access-796bh\") pod \"maas-api-e2e-mt-30fd85f1-8659df6487-rbmks\" (UID: \"17c767d8-2f74-4545-aa37-92d071aa281a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-30fd85f1-8659df6487-rbmks"
Jul 24 15:41:48.350494 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:41:48.350377    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-30fd85f1-serving-cert: secret "maas-api-e2e-mt-30fd85f1-serving-cert" not found
Jul 24 15:41:48.350494 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:41:48.350475    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17c767d8-2f74-4545-aa37-92d071aa281a-maas-api-tls podName:17c767d8-2f74-4545-aa37-92d071aa281a nodeName:}" failed. No retries permitted until 2026-07-24 15:41:48.850452137 +0000 UTC m=+2001.662804300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/17c767d8-2f74-4545-aa37-92d071aa281a-maas-api-tls") pod "maas-api-e2e-mt-30fd85f1-8659df6487-rbmks" (UID: "17c767d8-2f74-4545-aa37-92d071aa281a") : secret "maas-api-e2e-mt-30fd85f1-serving-cert" not found
Jul 24 15:41:48.350948 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:48.350925    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17c767d8-2f74-4545-aa37-92d071aa281a-openshift-service-ca\") pod \"maas-api-e2e-mt-30fd85f1-8659df6487-rbmks\" (UID: \"17c767d8-2f74-4545-aa37-92d071aa281a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-30fd85f1-8659df6487-rbmks"
Jul 24 15:41:48.366136 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:48.366115    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-796bh\" (UniqueName: \"kubernetes.io/projected/17c767d8-2f74-4545-aa37-92d071aa281a-kube-api-access-796bh\") pod \"maas-api-e2e-mt-30fd85f1-8659df6487-rbmks\" (UID: \"17c767d8-2f74-4545-aa37-92d071aa281a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-30fd85f1-8659df6487-rbmks"
Jul 24 15:41:48.855507 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:48.855468    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17c767d8-2f74-4545-aa37-92d071aa281a-maas-api-tls\") pod \"maas-api-e2e-mt-30fd85f1-8659df6487-rbmks\" (UID: \"17c767d8-2f74-4545-aa37-92d071aa281a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-30fd85f1-8659df6487-rbmks"
Jul 24 15:41:48.857868 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:48.857845    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17c767d8-2f74-4545-aa37-92d071aa281a-maas-api-tls\") pod \"maas-api-e2e-mt-30fd85f1-8659df6487-rbmks\" (UID: \"17c767d8-2f74-4545-aa37-92d071aa281a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-30fd85f1-8659df6487-rbmks"
Jul 24 15:41:49.081137 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:49.081099    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-30fd85f1-8659df6487-rbmks"
Jul 24 15:41:49.081903 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:49.081883    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-30fd85f1-56c8d9fb4b-r98ds"]
Jul 24 15:41:49.085169 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:49.085146    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-30fd85f1-56c8d9fb4b-r98ds"
Jul 24 15:41:49.087670 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:49.087650    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-30fd85f1\""
Jul 24 15:41:49.097149 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:49.097114    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-30fd85f1-56c8d9fb4b-r98ds"]
Jul 24 15:41:49.157797 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:49.157765    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4tfz\" (UniqueName: \"kubernetes.io/projected/4499cb54-7b09-43bb-9bb0-a29633c9fafb-kube-api-access-j4tfz\") pod \"payload-processing-e2e-mt-30fd85f1-56c8d9fb4b-r98ds\" (UID: \"4499cb54-7b09-43bb-9bb0-a29633c9fafb\") " pod="openshift-ingress/payload-processing-e2e-mt-30fd85f1-56c8d9fb4b-r98ds"
Jul 24 15:41:49.157797 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:49.157800    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4499cb54-7b09-43bb-9bb0-a29633c9fafb-plugins-config-volume\") pod \"payload-processing-e2e-mt-30fd85f1-56c8d9fb4b-r98ds\" (UID: \"4499cb54-7b09-43bb-9bb0-a29633c9fafb\") " pod="openshift-ingress/payload-processing-e2e-mt-30fd85f1-56c8d9fb4b-r98ds"
Jul 24 15:41:49.163139 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:49.163111    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-30fd85f1-f55cf846d-9gxww"]
Jul 24 15:41:49.166933 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:49.166916    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-30fd85f1-f55cf846d-9gxww"
Jul 24 15:41:49.169441 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:49.169336    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-30fd85f1-dockercfg-4rl7v\""
Jul 24 15:41:49.182229 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:49.182197    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-30fd85f1-f55cf846d-9gxww"]
Jul 24 15:41:49.226375 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:49.226350    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-30fd85f1-8659df6487-rbmks"]
Jul 24 15:41:49.229134 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:41:49.229098    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17c767d8_2f74_4545_aa37_92d071aa281a.slice/crio-7ff9fb957d923cec98c68c8020acb5bb2a9c268fa6323baa13e717625693cef2 WatchSource:0}: Error finding container 7ff9fb957d923cec98c68c8020acb5bb2a9c268fa6323baa13e717625693cef2: Status 404 returned error can't find the container with id 7ff9fb957d923cec98c68c8020acb5bb2a9c268fa6323baa13e717625693cef2
Jul 24 15:41:49.230701 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:49.230679    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 15:41:49.259121 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:49.259093    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4tfz\" (UniqueName: \"kubernetes.io/projected/4499cb54-7b09-43bb-9bb0-a29633c9fafb-kube-api-access-j4tfz\") pod \"payload-processing-e2e-mt-30fd85f1-56c8d9fb4b-r98ds\" (UID: \"4499cb54-7b09-43bb-9bb0-a29633c9fafb\") " pod="openshift-ingress/payload-processing-e2e-mt-30fd85f1-56c8d9fb4b-r98ds"
Jul 24 15:41:49.259251 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:49.259135    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4499cb54-7b09-43bb-9bb0-a29633c9fafb-plugins-config-volume\") pod \"payload-processing-e2e-mt-30fd85f1-56c8d9fb4b-r98ds\" (UID: \"4499cb54-7b09-43bb-9bb0-a29633c9fafb\") " pod="openshift-ingress/payload-processing-e2e-mt-30fd85f1-56c8d9fb4b-r98ds"
Jul 24 15:41:49.259251 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:49.259174    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d4b20b4-b9f0-4fe6-8582-e4120cd260cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-30fd85f1-f55cf846d-9gxww\" (UID: \"2d4b20b4-b9f0-4fe6-8582-e4120cd260cf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-30fd85f1-f55cf846d-9gxww"
Jul 24 15:41:49.259251 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:49.259234    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmwjv\" (UniqueName: \"kubernetes.io/projected/2d4b20b4-b9f0-4fe6-8582-e4120cd260cf-kube-api-access-cmwjv\") pod \"payload-pre-processing-e2e-mt-30fd85f1-f55cf846d-9gxww\" (UID: \"2d4b20b4-b9f0-4fe6-8582-e4120cd260cf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-30fd85f1-f55cf846d-9gxww"
Jul 24 15:41:49.259858 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:49.259841    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4499cb54-7b09-43bb-9bb0-a29633c9fafb-plugins-config-volume\") pod \"payload-processing-e2e-mt-30fd85f1-56c8d9fb4b-r98ds\" (UID: \"4499cb54-7b09-43bb-9bb0-a29633c9fafb\") " pod="openshift-ingress/payload-processing-e2e-mt-30fd85f1-56c8d9fb4b-r98ds"
Jul 24 15:41:49.268923 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:49.268899    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4tfz\" (UniqueName: \"kubernetes.io/projected/4499cb54-7b09-43bb-9bb0-a29633c9fafb-kube-api-access-j4tfz\") pod \"payload-processing-e2e-mt-30fd85f1-56c8d9fb4b-r98ds\" (UID: \"4499cb54-7b09-43bb-9bb0-a29633c9fafb\") " pod="openshift-ingress/payload-processing-e2e-mt-30fd85f1-56c8d9fb4b-r98ds"
Jul 24 15:41:49.360491 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:49.360404    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d4b20b4-b9f0-4fe6-8582-e4120cd260cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-30fd85f1-f55cf846d-9gxww\" (UID: \"2d4b20b4-b9f0-4fe6-8582-e4120cd260cf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-30fd85f1-f55cf846d-9gxww"
Jul 24 15:41:49.360491 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:49.360461    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmwjv\" (UniqueName: \"kubernetes.io/projected/2d4b20b4-b9f0-4fe6-8582-e4120cd260cf-kube-api-access-cmwjv\") pod \"payload-pre-processing-e2e-mt-30fd85f1-f55cf846d-9gxww\" (UID: \"2d4b20b4-b9f0-4fe6-8582-e4120cd260cf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-30fd85f1-f55cf846d-9gxww"
Jul 24 15:41:49.361113 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:49.361093    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d4b20b4-b9f0-4fe6-8582-e4120cd260cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-30fd85f1-f55cf846d-9gxww\" (UID: \"2d4b20b4-b9f0-4fe6-8582-e4120cd260cf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-30fd85f1-f55cf846d-9gxww"
Jul 24 15:41:49.372787 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:49.372758    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmwjv\" (UniqueName: \"kubernetes.io/projected/2d4b20b4-b9f0-4fe6-8582-e4120cd260cf-kube-api-access-cmwjv\") pod \"payload-pre-processing-e2e-mt-30fd85f1-f55cf846d-9gxww\" (UID: \"2d4b20b4-b9f0-4fe6-8582-e4120cd260cf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-30fd85f1-f55cf846d-9gxww"
Jul 24 15:41:49.412603 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:49.412572    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-30fd85f1-56c8d9fb4b-r98ds"
Jul 24 15:41:49.481583 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:49.481550    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-30fd85f1-f55cf846d-9gxww"
Jul 24 15:41:49.560465 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:49.560435    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-30fd85f1-56c8d9fb4b-r98ds"]
Jul 24 15:41:49.561799 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:41:49.561769    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4499cb54_7b09_43bb_9bb0_a29633c9fafb.slice/crio-b5bd220e01d7551b89e8e2ccfbc258e28134565a93d5cc107dd93a57aaa72e80 WatchSource:0}: Error finding container b5bd220e01d7551b89e8e2ccfbc258e28134565a93d5cc107dd93a57aaa72e80: Status 404 returned error can't find the container with id b5bd220e01d7551b89e8e2ccfbc258e28134565a93d5cc107dd93a57aaa72e80
Jul 24 15:41:49.628283 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:49.628261    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-30fd85f1-f55cf846d-9gxww"]
Jul 24 15:41:49.629988 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:41:49.629961    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d4b20b4_b9f0_4fe6_8582_e4120cd260cf.slice/crio-3a0873dd8ca6925feaf1306a6a651d0f564879e08c1a8cad3d0eaef5858ccc88 WatchSource:0}: Error finding container 3a0873dd8ca6925feaf1306a6a651d0f564879e08c1a8cad3d0eaef5858ccc88: Status 404 returned error can't find the container with id 3a0873dd8ca6925feaf1306a6a651d0f564879e08c1a8cad3d0eaef5858ccc88
Jul 24 15:41:50.130594 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:50.130558    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-30fd85f1-8659df6487-rbmks" event={"ID":"17c767d8-2f74-4545-aa37-92d071aa281a","Type":"ContainerStarted","Data":"603870d1213f738d1d1f7bd33035a891c6d0c1957660aed1342cc49f60b942dd"}
Jul 24 15:41:50.130782 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:50.130617    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-30fd85f1-8659df6487-rbmks" event={"ID":"17c767d8-2f74-4545-aa37-92d071aa281a","Type":"ContainerStarted","Data":"7ff9fb957d923cec98c68c8020acb5bb2a9c268fa6323baa13e717625693cef2"}
Jul 24 15:41:50.130782 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:50.130674    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-30fd85f1-8659df6487-rbmks"
Jul 24 15:41:50.132045 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:50.132020    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-30fd85f1-f55cf846d-9gxww" event={"ID":"2d4b20b4-b9f0-4fe6-8582-e4120cd260cf","Type":"ContainerStarted","Data":"f281257ed0f2d9dce4bcc9fe4f50213a6ab9e4d97bedbef303195c385b1556c9"}
Jul 24 15:41:50.132166 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:50.132058    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-30fd85f1-f55cf846d-9gxww" event={"ID":"2d4b20b4-b9f0-4fe6-8582-e4120cd260cf","Type":"ContainerStarted","Data":"3a0873dd8ca6925feaf1306a6a651d0f564879e08c1a8cad3d0eaef5858ccc88"}
Jul 24 15:41:50.132166 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:50.132104    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-30fd85f1-f55cf846d-9gxww"
Jul 24 15:41:50.133487 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:50.133448    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-30fd85f1-56c8d9fb4b-r98ds" event={"ID":"4499cb54-7b09-43bb-9bb0-a29633c9fafb","Type":"ContainerStarted","Data":"20490e51a28c7c19dad96228a0a22222a10d5cd8fb5295af6d5d1559244939ec"}
Jul 24 15:41:50.133566 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:50.133515    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-30fd85f1-56c8d9fb4b-r98ds"
Jul 24 15:41:50.133566 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:50.133538    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-30fd85f1-56c8d9fb4b-r98ds" event={"ID":"4499cb54-7b09-43bb-9bb0-a29633c9fafb","Type":"ContainerStarted","Data":"b5bd220e01d7551b89e8e2ccfbc258e28134565a93d5cc107dd93a57aaa72e80"}
Jul 24 15:41:50.153517 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:50.153472    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-30fd85f1-8659df6487-rbmks" podStartSLOduration=1.712575248 podStartE2EDuration="2.153460968s" totalImagesPullingTime="440.88572ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:41:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:41:50.151612437 +0000 UTC m=+2002.963964617" watchObservedRunningTime="2026-07-24 15:41:50.153460968 +0000 UTC m=+2002.965813187"
Jul 24 15:41:50.171407 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:50.171355    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-30fd85f1-f55cf846d-9gxww" podStartSLOduration=1.171338187 podStartE2EDuration="1.171338187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:41:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:41:50.170907854 +0000 UTC m=+2002.983260045" watchObservedRunningTime="2026-07-24 15:41:50.171338187 +0000 UTC m=+2002.983690351"
Jul 24 15:41:50.189580 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:50.189535    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-30fd85f1-56c8d9fb4b-r98ds" podStartSLOduration=1.189520098 podStartE2EDuration="1.189520098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:41:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:41:50.188032004 +0000 UTC m=+2003.000384183" watchObservedRunningTime="2026-07-24 15:41:50.189520098 +0000 UTC m=+2003.001872276"
Jul 24 15:41:53.825755 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:53.825718    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:41:56.145084 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:41:56.145048    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-30fd85f1-8659df6487-rbmks"
Jul 24 15:42:01.141262 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:01.141232    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-30fd85f1-56c8d9fb4b-r98ds"
Jul 24 15:42:01.141729 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:01.141472    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-30fd85f1-f55cf846d-9gxww"
Jul 24 15:42:07.837152 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:07.837122    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-30fd85f1/e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg"]
Jul 24 15:42:07.840728 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:07.840711    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-30fd85f1/e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg"
Jul 24 15:42:07.843875 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:07.843848    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-30fd85f1\"/\"e2e-lifecycle-model-30fd85f1-kserve-self-signed-certs\""
Jul 24 15:42:07.843875 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:07.843870    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-30fd85f1\"/\"kube-root-ca.crt\""
Jul 24 15:42:07.844067 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:07.843870    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-30fd85f1\"/\"openshift-service-ca.crt\""
Jul 24 15:42:07.844067 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:07.843848    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-30fd85f1\"/\"default-dockercfg-8272l\""
Jul 24 15:42:07.851231 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:07.851207    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-30fd85f1/e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg"]
Jul 24 15:42:07.927124 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:07.927087    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqlj8\" (UniqueName: \"kubernetes.io/projected/a2047d5d-9f99-4ce1-b174-42e7fd51ec29-kube-api-access-bqlj8\") pod \"e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg\" (UID: \"a2047d5d-9f99-4ce1-b174-42e7fd51ec29\") " pod="ai-tenant-e2e-mt-30fd85f1/e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg"
Jul 24 15:42:07.927295 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:07.927140    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2047d5d-9f99-4ce1-b174-42e7fd51ec29-dshm\") pod \"e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg\" (UID: \"a2047d5d-9f99-4ce1-b174-42e7fd51ec29\") " pod="ai-tenant-e2e-mt-30fd85f1/e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg"
Jul 24 15:42:07.927295 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:07.927172    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2047d5d-9f99-4ce1-b174-42e7fd51ec29-tmp-dir\") pod \"e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg\" (UID: \"a2047d5d-9f99-4ce1-b174-42e7fd51ec29\") " pod="ai-tenant-e2e-mt-30fd85f1/e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg"
Jul 24 15:42:07.927295 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:07.927226    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2047d5d-9f99-4ce1-b174-42e7fd51ec29-model-cache\") pod \"e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg\" (UID: \"a2047d5d-9f99-4ce1-b174-42e7fd51ec29\") " pod="ai-tenant-e2e-mt-30fd85f1/e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg"
Jul 24 15:42:07.927461 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:07.927297    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2047d5d-9f99-4ce1-b174-42e7fd51ec29-home\") pod \"e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg\" (UID: \"a2047d5d-9f99-4ce1-b174-42e7fd51ec29\") " pod="ai-tenant-e2e-mt-30fd85f1/e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg"
Jul 24 15:42:07.927461 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:07.927361    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2047d5d-9f99-4ce1-b174-42e7fd51ec29-tls-certs\") pod \"e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg\" (UID: \"a2047d5d-9f99-4ce1-b174-42e7fd51ec29\") " pod="ai-tenant-e2e-mt-30fd85f1/e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg"
Jul 24 15:42:08.028142 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:08.028106    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqlj8\" (UniqueName: \"kubernetes.io/projected/a2047d5d-9f99-4ce1-b174-42e7fd51ec29-kube-api-access-bqlj8\") pod \"e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg\" (UID: \"a2047d5d-9f99-4ce1-b174-42e7fd51ec29\") " pod="ai-tenant-e2e-mt-30fd85f1/e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg"
Jul 24 15:42:08.028330 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:08.028164    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2047d5d-9f99-4ce1-b174-42e7fd51ec29-dshm\") pod \"e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg\" (UID: \"a2047d5d-9f99-4ce1-b174-42e7fd51ec29\") " pod="ai-tenant-e2e-mt-30fd85f1/e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg"
Jul 24 15:42:08.028330 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:08.028190    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2047d5d-9f99-4ce1-b174-42e7fd51ec29-tmp-dir\") pod \"e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg\" (UID: \"a2047d5d-9f99-4ce1-b174-42e7fd51ec29\") " pod="ai-tenant-e2e-mt-30fd85f1/e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg"
Jul 24 15:42:08.028462 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:08.028351    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2047d5d-9f99-4ce1-b174-42e7fd51ec29-model-cache\") pod \"e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg\" (UID: \"a2047d5d-9f99-4ce1-b174-42e7fd51ec29\") " pod="ai-tenant-e2e-mt-30fd85f1/e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg"
Jul 24 15:42:08.028462 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:08.028390    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2047d5d-9f99-4ce1-b174-42e7fd51ec29-home\") pod \"e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg\" (UID: \"a2047d5d-9f99-4ce1-b174-42e7fd51ec29\") " pod="ai-tenant-e2e-mt-30fd85f1/e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg"
Jul 24 15:42:08.028462 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:08.028442    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2047d5d-9f99-4ce1-b174-42e7fd51ec29-tls-certs\") pod \"e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg\" (UID: \"a2047d5d-9f99-4ce1-b174-42e7fd51ec29\") " pod="ai-tenant-e2e-mt-30fd85f1/e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg"
Jul 24 15:42:08.028735 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:08.028713    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2047d5d-9f99-4ce1-b174-42e7fd51ec29-tmp-dir\") pod \"e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg\" (UID: \"a2047d5d-9f99-4ce1-b174-42e7fd51ec29\") " pod="ai-tenant-e2e-mt-30fd85f1/e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg"
Jul 24 15:42:08.028796 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:08.028781    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2047d5d-9f99-4ce1-b174-42e7fd51ec29-model-cache\") pod \"e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg\" (UID: \"a2047d5d-9f99-4ce1-b174-42e7fd51ec29\") " pod="ai-tenant-e2e-mt-30fd85f1/e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg"
Jul 24 15:42:08.028857 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:08.028800    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2047d5d-9f99-4ce1-b174-42e7fd51ec29-home\") pod \"e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg\" (UID: \"a2047d5d-9f99-4ce1-b174-42e7fd51ec29\") " pod="ai-tenant-e2e-mt-30fd85f1/e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg"
Jul 24 15:42:08.030493 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:08.030458    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2047d5d-9f99-4ce1-b174-42e7fd51ec29-dshm\") pod \"e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg\" (UID: \"a2047d5d-9f99-4ce1-b174-42e7fd51ec29\") " pod="ai-tenant-e2e-mt-30fd85f1/e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg"
Jul 24 15:42:08.030869 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:08.030854    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2047d5d-9f99-4ce1-b174-42e7fd51ec29-tls-certs\") pod \"e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg\" (UID: \"a2047d5d-9f99-4ce1-b174-42e7fd51ec29\") " pod="ai-tenant-e2e-mt-30fd85f1/e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg"
Jul 24 15:42:08.037521 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:08.037496    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqlj8\" (UniqueName: \"kubernetes.io/projected/a2047d5d-9f99-4ce1-b174-42e7fd51ec29-kube-api-access-bqlj8\") pod \"e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg\" (UID: \"a2047d5d-9f99-4ce1-b174-42e7fd51ec29\") " pod="ai-tenant-e2e-mt-30fd85f1/e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg"
Jul 24 15:42:08.151661 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:08.151562    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-30fd85f1/e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg"
Jul 24 15:42:08.298525 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:08.298366    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-30fd85f1/e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg"]
Jul 24 15:42:08.301483 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:42:08.301454    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2047d5d_9f99_4ce1_b174_42e7fd51ec29.slice/crio-a04c352aed9e8434b9a76cad70c12fea435a06cb210429506b643c90217915ac WatchSource:0}: Error finding container a04c352aed9e8434b9a76cad70c12fea435a06cb210429506b643c90217915ac: Status 404 returned error can't find the container with id a04c352aed9e8434b9a76cad70c12fea435a06cb210429506b643c90217915ac
Jul 24 15:42:09.233008 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:09.232969    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-30fd85f1/e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg" event={"ID":"a2047d5d-9f99-4ce1-b174-42e7fd51ec29","Type":"ContainerStarted","Data":"13e9c5c8908c252ab92aeccfdc5e42c12a2181735a41283e2b0e54b56c6cf290"}
Jul 24 15:42:09.233397 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:09.233015    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-30fd85f1/e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg" event={"ID":"a2047d5d-9f99-4ce1-b174-42e7fd51ec29","Type":"ContainerStarted","Data":"a04c352aed9e8434b9a76cad70c12fea435a06cb210429506b643c90217915ac"}
Jul 24 15:42:09.258800 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:09.258743    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-30fd85f1/e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg" podStartSLOduration=2.258727662 podStartE2EDuration="2.258727662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:42:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:42:09.258616973 +0000 UTC m=+2022.070969151" watchObservedRunningTime="2026-07-24 15:42:09.258727662 +0000 UTC m=+2022.071079841"
Jul 24 15:42:11.146486 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:11.146450    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:42:18.152541 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:18.152502    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-30fd85f1/e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg"
Jul 24 15:42:18.153004 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:18.152557    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-30fd85f1/e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg"
Jul 24 15:42:18.165057 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:18.165032    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-30fd85f1/e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg"
Jul 24 15:42:18.285516 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:18.285488    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-30fd85f1/e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg"
Jul 24 15:42:27.736581 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:27.736550    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:42:46.228062 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:46.228021    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:42:47.595963 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:47.595923    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-30fd85f1-8659df6487-rbmks"]
Jul 24 15:42:47.596457 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:47.596217    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-30fd85f1-8659df6487-rbmks" podUID="17c767d8-2f74-4545-aa37-92d071aa281a" containerName="maas-api" containerID="cri-o://603870d1213f738d1d1f7bd33035a891c6d0c1957660aed1342cc49f60b942dd" gracePeriod=30
Jul 24 15:42:47.663337 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:47.663296    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-30fd85f1-56c8d9fb4b-r98ds"]
Jul 24 15:42:47.663562 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:47.663532    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-30fd85f1-56c8d9fb4b-r98ds" podUID="4499cb54-7b09-43bb-9bb0-a29633c9fafb" containerName="payload-processing" containerID="cri-o://20490e51a28c7c19dad96228a0a22222a10d5cd8fb5295af6d5d1559244939ec" gracePeriod=30
Jul 24 15:42:47.677109 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:47.677076    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-30fd85f1-f55cf846d-9gxww"]
Jul 24 15:42:47.677349 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:47.677319    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-30fd85f1-f55cf846d-9gxww" podUID="2d4b20b4-b9f0-4fe6-8582-e4120cd260cf" containerName="payload-pre-processing" containerID="cri-o://f281257ed0f2d9dce4bcc9fe4f50213a6ab9e4d97bedbef303195c385b1556c9" gracePeriod=30
Jul 24 15:42:47.956513 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:47.956482    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-30fd85f1-8659df6487-rbmks"
Jul 24 15:42:47.965725 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:47.965700    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-30fd85f1-56c8d9fb4b-r98ds"
Jul 24 15:42:47.975215 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:47.975188    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-30fd85f1-f55cf846d-9gxww"
Jul 24 15:42:48.027391 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.027360    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4499cb54-7b09-43bb-9bb0-a29633c9fafb-plugins-config-volume\") pod \"4499cb54-7b09-43bb-9bb0-a29633c9fafb\" (UID: \"4499cb54-7b09-43bb-9bb0-a29633c9fafb\") "
Jul 24 15:42:48.027391 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.027397    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17c767d8-2f74-4545-aa37-92d071aa281a-maas-api-tls\") pod \"17c767d8-2f74-4545-aa37-92d071aa281a\" (UID: \"17c767d8-2f74-4545-aa37-92d071aa281a\") "
Jul 24 15:42:48.027653 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.027452    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-796bh\" (UniqueName: \"kubernetes.io/projected/17c767d8-2f74-4545-aa37-92d071aa281a-kube-api-access-796bh\") pod \"17c767d8-2f74-4545-aa37-92d071aa281a\" (UID: \"17c767d8-2f74-4545-aa37-92d071aa281a\") "
Jul 24 15:42:48.027653 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.027545    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4tfz\" (UniqueName: \"kubernetes.io/projected/4499cb54-7b09-43bb-9bb0-a29633c9fafb-kube-api-access-j4tfz\") pod \"4499cb54-7b09-43bb-9bb0-a29633c9fafb\" (UID: \"4499cb54-7b09-43bb-9bb0-a29633c9fafb\") "
Jul 24 15:42:48.027653 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.027588    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17c767d8-2f74-4545-aa37-92d071aa281a-openshift-service-ca\") pod \"17c767d8-2f74-4545-aa37-92d071aa281a\" (UID: \"17c767d8-2f74-4545-aa37-92d071aa281a\") "
Jul 24 15:42:48.027954 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.027850    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4499cb54-7b09-43bb-9bb0-a29633c9fafb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4499cb54-7b09-43bb-9bb0-a29633c9fafb" (UID: "4499cb54-7b09-43bb-9bb0-a29633c9fafb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:42:48.028096 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.028038    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17c767d8-2f74-4545-aa37-92d071aa281a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "17c767d8-2f74-4545-aa37-92d071aa281a" (UID: "17c767d8-2f74-4545-aa37-92d071aa281a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:42:48.029713 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.029687    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17c767d8-2f74-4545-aa37-92d071aa281a-kube-api-access-796bh" (OuterVolumeSpecName: "kube-api-access-796bh") pod "17c767d8-2f74-4545-aa37-92d071aa281a" (UID: "17c767d8-2f74-4545-aa37-92d071aa281a"). InnerVolumeSpecName "kube-api-access-796bh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:42:48.029811 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.029728    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4499cb54-7b09-43bb-9bb0-a29633c9fafb-kube-api-access-j4tfz" (OuterVolumeSpecName: "kube-api-access-j4tfz") pod "4499cb54-7b09-43bb-9bb0-a29633c9fafb" (UID: "4499cb54-7b09-43bb-9bb0-a29633c9fafb"). InnerVolumeSpecName "kube-api-access-j4tfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:42:48.029811 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.029735    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17c767d8-2f74-4545-aa37-92d071aa281a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "17c767d8-2f74-4545-aa37-92d071aa281a" (UID: "17c767d8-2f74-4545-aa37-92d071aa281a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 15:42:48.128326 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.128242    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmwjv\" (UniqueName: \"kubernetes.io/projected/2d4b20b4-b9f0-4fe6-8582-e4120cd260cf-kube-api-access-cmwjv\") pod \"2d4b20b4-b9f0-4fe6-8582-e4120cd260cf\" (UID: \"2d4b20b4-b9f0-4fe6-8582-e4120cd260cf\") "
Jul 24 15:42:48.128482 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.128340    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d4b20b4-b9f0-4fe6-8582-e4120cd260cf-plugins-config-volume\") pod \"2d4b20b4-b9f0-4fe6-8582-e4120cd260cf\" (UID: \"2d4b20b4-b9f0-4fe6-8582-e4120cd260cf\") "
Jul 24 15:42:48.128533 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.128480    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4tfz\" (UniqueName: \"kubernetes.io/projected/4499cb54-7b09-43bb-9bb0-a29633c9fafb-kube-api-access-j4tfz\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:42:48.128533 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.128492    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17c767d8-2f74-4545-aa37-92d071aa281a-openshift-service-ca\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:42:48.128533 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.128502    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4499cb54-7b09-43bb-9bb0-a29633c9fafb-plugins-config-volume\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:42:48.128533 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.128511    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17c767d8-2f74-4545-aa37-92d071aa281a-maas-api-tls\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:42:48.128533 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.128519    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-796bh\" (UniqueName: \"kubernetes.io/projected/17c767d8-2f74-4545-aa37-92d071aa281a-kube-api-access-796bh\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:42:48.128710 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.128676    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d4b20b4-b9f0-4fe6-8582-e4120cd260cf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2d4b20b4-b9f0-4fe6-8582-e4120cd260cf" (UID: "2d4b20b4-b9f0-4fe6-8582-e4120cd260cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:42:48.130292 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.130272    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d4b20b4-b9f0-4fe6-8582-e4120cd260cf-kube-api-access-cmwjv" (OuterVolumeSpecName: "kube-api-access-cmwjv") pod "2d4b20b4-b9f0-4fe6-8582-e4120cd260cf" (UID: "2d4b20b4-b9f0-4fe6-8582-e4120cd260cf"). InnerVolumeSpecName "kube-api-access-cmwjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:42:48.229345 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.229311    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d4b20b4-b9f0-4fe6-8582-e4120cd260cf-plugins-config-volume\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:42:48.229345 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.229338    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmwjv\" (UniqueName: \"kubernetes.io/projected/2d4b20b4-b9f0-4fe6-8582-e4120cd260cf-kube-api-access-cmwjv\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:42:48.419944 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.419857    2579 generic.go:356] "Generic (PLEG): container finished" podID="17c767d8-2f74-4545-aa37-92d071aa281a" containerID="603870d1213f738d1d1f7bd33035a891c6d0c1957660aed1342cc49f60b942dd" exitCode=0
Jul 24 15:42:48.420094 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.419947    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-30fd85f1-8659df6487-rbmks"
Jul 24 15:42:48.420094 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.419942    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-30fd85f1-8659df6487-rbmks" event={"ID":"17c767d8-2f74-4545-aa37-92d071aa281a","Type":"ContainerDied","Data":"603870d1213f738d1d1f7bd33035a891c6d0c1957660aed1342cc49f60b942dd"}
Jul 24 15:42:48.420094 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.420057    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-30fd85f1-8659df6487-rbmks" event={"ID":"17c767d8-2f74-4545-aa37-92d071aa281a","Type":"ContainerDied","Data":"7ff9fb957d923cec98c68c8020acb5bb2a9c268fa6323baa13e717625693cef2"}
Jul 24 15:42:48.420245 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.420092    2579 scope.go:117] "RemoveContainer" containerID="603870d1213f738d1d1f7bd33035a891c6d0c1957660aed1342cc49f60b942dd"
Jul 24 15:42:48.421365 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.421299    2579 generic.go:356] "Generic (PLEG): container finished" podID="2d4b20b4-b9f0-4fe6-8582-e4120cd260cf" containerID="f281257ed0f2d9dce4bcc9fe4f50213a6ab9e4d97bedbef303195c385b1556c9" exitCode=0
Jul 24 15:42:48.421365 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.421334    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-30fd85f1-f55cf846d-9gxww"
Jul 24 15:42:48.421489 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.421395    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-30fd85f1-f55cf846d-9gxww" event={"ID":"2d4b20b4-b9f0-4fe6-8582-e4120cd260cf","Type":"ContainerDied","Data":"f281257ed0f2d9dce4bcc9fe4f50213a6ab9e4d97bedbef303195c385b1556c9"}
Jul 24 15:42:48.421489 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.421445    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-30fd85f1-f55cf846d-9gxww" event={"ID":"2d4b20b4-b9f0-4fe6-8582-e4120cd260cf","Type":"ContainerDied","Data":"3a0873dd8ca6925feaf1306a6a651d0f564879e08c1a8cad3d0eaef5858ccc88"}
Jul 24 15:42:48.422895 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.422870    2579 generic.go:356] "Generic (PLEG): container finished" podID="4499cb54-7b09-43bb-9bb0-a29633c9fafb" containerID="20490e51a28c7c19dad96228a0a22222a10d5cd8fb5295af6d5d1559244939ec" exitCode=0
Jul 24 15:42:48.422998 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.422912    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-30fd85f1-56c8d9fb4b-r98ds" event={"ID":"4499cb54-7b09-43bb-9bb0-a29633c9fafb","Type":"ContainerDied","Data":"20490e51a28c7c19dad96228a0a22222a10d5cd8fb5295af6d5d1559244939ec"}
Jul 24 15:42:48.422998 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.422956    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-30fd85f1-56c8d9fb4b-r98ds" event={"ID":"4499cb54-7b09-43bb-9bb0-a29633c9fafb","Type":"ContainerDied","Data":"b5bd220e01d7551b89e8e2ccfbc258e28134565a93d5cc107dd93a57aaa72e80"}
Jul 24 15:42:48.422998 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.422920    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-30fd85f1-56c8d9fb4b-r98ds"
Jul 24 15:42:48.429750 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.429735    2579 scope.go:117] "RemoveContainer" containerID="603870d1213f738d1d1f7bd33035a891c6d0c1957660aed1342cc49f60b942dd"
Jul 24 15:42:48.430155 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:42:48.430108    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"603870d1213f738d1d1f7bd33035a891c6d0c1957660aed1342cc49f60b942dd\": container with ID starting with 603870d1213f738d1d1f7bd33035a891c6d0c1957660aed1342cc49f60b942dd not found: ID does not exist" containerID="603870d1213f738d1d1f7bd33035a891c6d0c1957660aed1342cc49f60b942dd"
Jul 24 15:42:48.430155 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.430142    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"603870d1213f738d1d1f7bd33035a891c6d0c1957660aed1342cc49f60b942dd"} err="failed to get container status \"603870d1213f738d1d1f7bd33035a891c6d0c1957660aed1342cc49f60b942dd\": rpc error: code = NotFound desc = could not find container \"603870d1213f738d1d1f7bd33035a891c6d0c1957660aed1342cc49f60b942dd\": container with ID starting with 603870d1213f738d1d1f7bd33035a891c6d0c1957660aed1342cc49f60b942dd not found: ID does not exist"
Jul 24 15:42:48.430354 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.430166    2579 scope.go:117] "RemoveContainer" containerID="f281257ed0f2d9dce4bcc9fe4f50213a6ab9e4d97bedbef303195c385b1556c9"
Jul 24 15:42:48.442312 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.442257    2579 scope.go:117] "RemoveContainer" containerID="f281257ed0f2d9dce4bcc9fe4f50213a6ab9e4d97bedbef303195c385b1556c9"
Jul 24 15:42:48.443293 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:42:48.443265    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f281257ed0f2d9dce4bcc9fe4f50213a6ab9e4d97bedbef303195c385b1556c9\": container with ID starting with f281257ed0f2d9dce4bcc9fe4f50213a6ab9e4d97bedbef303195c385b1556c9 not found: ID does not exist" containerID="f281257ed0f2d9dce4bcc9fe4f50213a6ab9e4d97bedbef303195c385b1556c9"
Jul 24 15:42:48.443379 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.443303    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f281257ed0f2d9dce4bcc9fe4f50213a6ab9e4d97bedbef303195c385b1556c9"} err="failed to get container status \"f281257ed0f2d9dce4bcc9fe4f50213a6ab9e4d97bedbef303195c385b1556c9\": rpc error: code = NotFound desc = could not find container \"f281257ed0f2d9dce4bcc9fe4f50213a6ab9e4d97bedbef303195c385b1556c9\": container with ID starting with f281257ed0f2d9dce4bcc9fe4f50213a6ab9e4d97bedbef303195c385b1556c9 not found: ID does not exist"
Jul 24 15:42:48.443379 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.443326    2579 scope.go:117] "RemoveContainer" containerID="20490e51a28c7c19dad96228a0a22222a10d5cd8fb5295af6d5d1559244939ec"
Jul 24 15:42:48.444721 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.444702    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-30fd85f1-8659df6487-rbmks"]
Jul 24 15:42:48.448581 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.448559    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-30fd85f1-8659df6487-rbmks"]
Jul 24 15:42:48.451632 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.451619    2579 scope.go:117] "RemoveContainer" containerID="20490e51a28c7c19dad96228a0a22222a10d5cd8fb5295af6d5d1559244939ec"
Jul 24 15:42:48.451954 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:42:48.451937    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20490e51a28c7c19dad96228a0a22222a10d5cd8fb5295af6d5d1559244939ec\": container with ID starting with 20490e51a28c7c19dad96228a0a22222a10d5cd8fb5295af6d5d1559244939ec not found: ID does not exist" containerID="20490e51a28c7c19dad96228a0a22222a10d5cd8fb5295af6d5d1559244939ec"
Jul 24 15:42:48.452000 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.451961    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20490e51a28c7c19dad96228a0a22222a10d5cd8fb5295af6d5d1559244939ec"} err="failed to get container status \"20490e51a28c7c19dad96228a0a22222a10d5cd8fb5295af6d5d1559244939ec\": rpc error: code = NotFound desc = could not find container \"20490e51a28c7c19dad96228a0a22222a10d5cd8fb5295af6d5d1559244939ec\": container with ID starting with 20490e51a28c7c19dad96228a0a22222a10d5cd8fb5295af6d5d1559244939ec not found: ID does not exist"
Jul 24 15:42:48.459448 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.459430    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-30fd85f1-56c8d9fb4b-r98ds"]
Jul 24 15:42:48.466468 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.466448    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-30fd85f1-56c8d9fb4b-r98ds"]
Jul 24 15:42:48.479757 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.479731    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-30fd85f1-f55cf846d-9gxww"]
Jul 24 15:42:48.484936 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:48.484916    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-30fd85f1-f55cf846d-9gxww"]
Jul 24 15:42:49.717075 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:49.716998    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17c767d8-2f74-4545-aa37-92d071aa281a" path="/var/lib/kubelet/pods/17c767d8-2f74-4545-aa37-92d071aa281a/volumes"
Jul 24 15:42:49.717433 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:49.717423    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d4b20b4-b9f0-4fe6-8582-e4120cd260cf" path="/var/lib/kubelet/pods/2d4b20b4-b9f0-4fe6-8582-e4120cd260cf/volumes"
Jul 24 15:42:49.717736 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:49.717723    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4499cb54-7b09-43bb-9bb0-a29633c9fafb" path="/var/lib/kubelet/pods/4499cb54-7b09-43bb-9bb0-a29633c9fafb/volumes"
Jul 24 15:42:57.026593 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:42:57.026558    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:43:00.338131 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:43:00.338093    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-30fd85f1/e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg"]
Jul 24 15:43:15.357520 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:43:15.357462    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-30fd85f1/e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg" podUID="a2047d5d-9f99-4ce1-b174-42e7fd51ec29" containerName="main" containerID="cri-o://13e9c5c8908c252ab92aeccfdc5e42c12a2181735a41283e2b0e54b56c6cf290" gracePeriod=45
Jul 24 15:43:15.563459 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:43:15.563429    2579 generic.go:356] "Generic (PLEG): container finished" podID="a2047d5d-9f99-4ce1-b174-42e7fd51ec29" containerID="13e9c5c8908c252ab92aeccfdc5e42c12a2181735a41283e2b0e54b56c6cf290" exitCode=0
Jul 24 15:43:15.563611 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:43:15.563466    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-30fd85f1/e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg" event={"ID":"a2047d5d-9f99-4ce1-b174-42e7fd51ec29","Type":"ContainerDied","Data":"13e9c5c8908c252ab92aeccfdc5e42c12a2181735a41283e2b0e54b56c6cf290"}
Jul 24 15:43:15.605163 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:43:15.605139    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-30fd85f1/e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg"
Jul 24 15:43:15.695118 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:43:15.695033    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2047d5d-9f99-4ce1-b174-42e7fd51ec29-dshm\") pod \"a2047d5d-9f99-4ce1-b174-42e7fd51ec29\" (UID: \"a2047d5d-9f99-4ce1-b174-42e7fd51ec29\") "
Jul 24 15:43:15.695118 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:43:15.695078    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2047d5d-9f99-4ce1-b174-42e7fd51ec29-home\") pod \"a2047d5d-9f99-4ce1-b174-42e7fd51ec29\" (UID: \"a2047d5d-9f99-4ce1-b174-42e7fd51ec29\") "
Jul 24 15:43:15.695118 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:43:15.695102    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2047d5d-9f99-4ce1-b174-42e7fd51ec29-tls-certs\") pod \"a2047d5d-9f99-4ce1-b174-42e7fd51ec29\" (UID: \"a2047d5d-9f99-4ce1-b174-42e7fd51ec29\") "
Jul 24 15:43:15.695406 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:43:15.695157    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2047d5d-9f99-4ce1-b174-42e7fd51ec29-tmp-dir\") pod \"a2047d5d-9f99-4ce1-b174-42e7fd51ec29\" (UID: \"a2047d5d-9f99-4ce1-b174-42e7fd51ec29\") "
Jul 24 15:43:15.695406 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:43:15.695236    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqlj8\" (UniqueName: \"kubernetes.io/projected/a2047d5d-9f99-4ce1-b174-42e7fd51ec29-kube-api-access-bqlj8\") pod \"a2047d5d-9f99-4ce1-b174-42e7fd51ec29\" (UID: \"a2047d5d-9f99-4ce1-b174-42e7fd51ec29\") "
Jul 24 15:43:15.695406 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:43:15.695256    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2047d5d-9f99-4ce1-b174-42e7fd51ec29-model-cache\") pod \"a2047d5d-9f99-4ce1-b174-42e7fd51ec29\" (UID: \"a2047d5d-9f99-4ce1-b174-42e7fd51ec29\") "
Jul 24 15:43:15.695406 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:43:15.695337    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2047d5d-9f99-4ce1-b174-42e7fd51ec29-home" (OuterVolumeSpecName: "home") pod "a2047d5d-9f99-4ce1-b174-42e7fd51ec29" (UID: "a2047d5d-9f99-4ce1-b174-42e7fd51ec29"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:43:15.695608 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:43:15.695464    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2047d5d-9f99-4ce1-b174-42e7fd51ec29-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a2047d5d-9f99-4ce1-b174-42e7fd51ec29" (UID: "a2047d5d-9f99-4ce1-b174-42e7fd51ec29"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:43:15.695608 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:43:15.695545    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2047d5d-9f99-4ce1-b174-42e7fd51ec29-model-cache" (OuterVolumeSpecName: "model-cache") pod "a2047d5d-9f99-4ce1-b174-42e7fd51ec29" (UID: "a2047d5d-9f99-4ce1-b174-42e7fd51ec29"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:43:15.695608 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:43:15.695591    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2047d5d-9f99-4ce1-b174-42e7fd51ec29-home\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:43:15.695753 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:43:15.695610    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2047d5d-9f99-4ce1-b174-42e7fd51ec29-tmp-dir\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:43:15.697410 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:43:15.697379    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2047d5d-9f99-4ce1-b174-42e7fd51ec29-kube-api-access-bqlj8" (OuterVolumeSpecName: "kube-api-access-bqlj8") pod "a2047d5d-9f99-4ce1-b174-42e7fd51ec29" (UID: "a2047d5d-9f99-4ce1-b174-42e7fd51ec29"). InnerVolumeSpecName "kube-api-access-bqlj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:43:15.697523 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:43:15.697431    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2047d5d-9f99-4ce1-b174-42e7fd51ec29-dshm" (OuterVolumeSpecName: "dshm") pod "a2047d5d-9f99-4ce1-b174-42e7fd51ec29" (UID: "a2047d5d-9f99-4ce1-b174-42e7fd51ec29"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:43:15.697523 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:43:15.697497    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2047d5d-9f99-4ce1-b174-42e7fd51ec29-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a2047d5d-9f99-4ce1-b174-42e7fd51ec29" (UID: "a2047d5d-9f99-4ce1-b174-42e7fd51ec29"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 15:43:15.796753 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:43:15.796717    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqlj8\" (UniqueName: \"kubernetes.io/projected/a2047d5d-9f99-4ce1-b174-42e7fd51ec29-kube-api-access-bqlj8\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:43:15.796753 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:43:15.796752    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2047d5d-9f99-4ce1-b174-42e7fd51ec29-model-cache\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:43:15.796753 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:43:15.796765    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2047d5d-9f99-4ce1-b174-42e7fd51ec29-dshm\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:43:15.797040 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:43:15.796777    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2047d5d-9f99-4ce1-b174-42e7fd51ec29-tls-certs\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:43:16.570657 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:43:16.570631    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-30fd85f1/e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg"
Jul 24 15:43:16.571198 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:43:16.570630    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-30fd85f1/e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg" event={"ID":"a2047d5d-9f99-4ce1-b174-42e7fd51ec29","Type":"ContainerDied","Data":"a04c352aed9e8434b9a76cad70c12fea435a06cb210429506b643c90217915ac"}
Jul 24 15:43:16.571198 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:43:16.570771    2579 scope.go:117] "RemoveContainer" containerID="13e9c5c8908c252ab92aeccfdc5e42c12a2181735a41283e2b0e54b56c6cf290"
Jul 24 15:43:16.588176 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:43:16.588149    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-30fd85f1/e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg"]
Jul 24 15:43:16.592452 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:43:16.592430    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-30fd85f1/e2e-lifecycle-model-30fd85f1-kserve-7597df59cf-8vlkg"]
Jul 24 15:43:17.717528 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:43:17.717491    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2047d5d-9f99-4ce1-b174-42e7fd51ec29" path="/var/lib/kubelet/pods/a2047d5d-9f99-4ce1-b174-42e7fd51ec29/volumes"
Jul 24 15:43:27.853501 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:43:27.853467    2579 scope.go:117] "RemoveContainer" containerID="daa9307861a33b5ac4b07d568743e7bdecdcd8addca21396d1297f5774e578d7"
Jul 24 15:43:47.937739 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:43:47.937701    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:44:00.327788 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:00.327751    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:44:43.977952 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:43.977913    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2a1c00-79857b87b9-64h74"]
Jul 24 15:44:43.980651 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:43.978279    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4499cb54-7b09-43bb-9bb0-a29633c9fafb" containerName="payload-processing"
Jul 24 15:44:43.980651 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:43.978291    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4499cb54-7b09-43bb-9bb0-a29633c9fafb" containerName="payload-processing"
Jul 24 15:44:43.980651 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:43.978301    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2047d5d-9f99-4ce1-b174-42e7fd51ec29" containerName="main"
Jul 24 15:44:43.980651 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:43.978309    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2047d5d-9f99-4ce1-b174-42e7fd51ec29" containerName="main"
Jul 24 15:44:43.980651 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:43.978324    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17c767d8-2f74-4545-aa37-92d071aa281a" containerName="maas-api"
Jul 24 15:44:43.980651 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:43.978329    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="17c767d8-2f74-4545-aa37-92d071aa281a" containerName="maas-api"
Jul 24 15:44:43.980651 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:43.978338    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d4b20b4-b9f0-4fe6-8582-e4120cd260cf" containerName="payload-pre-processing"
Jul 24 15:44:43.980651 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:43.978343    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d4b20b4-b9f0-4fe6-8582-e4120cd260cf" containerName="payload-pre-processing"
Jul 24 15:44:43.980651 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:43.978402    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="17c767d8-2f74-4545-aa37-92d071aa281a" containerName="maas-api"
Jul 24 15:44:43.980651 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:43.978413    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2047d5d-9f99-4ce1-b174-42e7fd51ec29" containerName="main"
Jul 24 15:44:43.980651 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:43.978422    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4499cb54-7b09-43bb-9bb0-a29633c9fafb" containerName="payload-processing"
Jul 24 15:44:43.980651 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:43.978428    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d4b20b4-b9f0-4fe6-8582-e4120cd260cf" containerName="payload-pre-processing"
Jul 24 15:44:43.981554 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:43.981538    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a1c00-79857b87b9-64h74"
Jul 24 15:44:43.984429 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:43.984405    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-2a1c00-serving-cert\""
Jul 24 15:44:43.984563 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:43.984463    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 15:44:43.984741 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:43.984727    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 15:44:43.985278 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:43.985262    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tkn9s\""
Jul 24 15:44:43.993177 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:43.993154    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2a1c00-79857b87b9-64h74"]
Jul 24 15:44:44.091397 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:44.091361    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2a1c00-79857b87b9-64h74\" (UID: \"c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a1c00-79857b87b9-64h74"
Jul 24 15:44:44.091397 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:44.091398    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcxs7\" (UniqueName: \"kubernetes.io/projected/c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20-kube-api-access-kcxs7\") pod \"maas-api-e2e-inf-a-2a1c00-79857b87b9-64h74\" (UID: \"c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a1c00-79857b87b9-64h74"
Jul 24 15:44:44.091607 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:44.091493    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20-maas-api-tls\") pod \"maas-api-e2e-inf-a-2a1c00-79857b87b9-64h74\" (UID: \"c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a1c00-79857b87b9-64h74"
Jul 24 15:44:44.192767 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:44.192732    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2a1c00-79857b87b9-64h74\" (UID: \"c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a1c00-79857b87b9-64h74"
Jul 24 15:44:44.192767 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:44.192769    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcxs7\" (UniqueName: \"kubernetes.io/projected/c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20-kube-api-access-kcxs7\") pod \"maas-api-e2e-inf-a-2a1c00-79857b87b9-64h74\" (UID: \"c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a1c00-79857b87b9-64h74"
Jul 24 15:44:44.192996 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:44.192849    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20-maas-api-tls\") pod \"maas-api-e2e-inf-a-2a1c00-79857b87b9-64h74\" (UID: \"c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a1c00-79857b87b9-64h74"
Jul 24 15:44:44.193501 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:44.193478    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2a1c00-79857b87b9-64h74\" (UID: \"c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a1c00-79857b87b9-64h74"
Jul 24 15:44:44.195337 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:44.195311    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20-maas-api-tls\") pod \"maas-api-e2e-inf-a-2a1c00-79857b87b9-64h74\" (UID: \"c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a1c00-79857b87b9-64h74"
Jul 24 15:44:44.201682 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:44.201659    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcxs7\" (UniqueName: \"kubernetes.io/projected/c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20-kube-api-access-kcxs7\") pod \"maas-api-e2e-inf-a-2a1c00-79857b87b9-64h74\" (UID: \"c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a1c00-79857b87b9-64h74"
Jul 24 15:44:44.293427 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:44.293387    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a1c00-79857b87b9-64h74"
Jul 24 15:44:44.429584 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:44.429556    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2a1c00-79857b87b9-64h74"]
Jul 24 15:44:44.431754 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:44:44.431727    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5f0be5f_57eb_4a48_b3f5_f7cb5337ef20.slice/crio-814d93314f2be52afa8deb5a2fe85f2d5d77b6dd31cf5d935747cd3f4ed7e2c8 WatchSource:0}: Error finding container 814d93314f2be52afa8deb5a2fe85f2d5d77b6dd31cf5d935747cd3f4ed7e2c8: Status 404 returned error can't find the container with id 814d93314f2be52afa8deb5a2fe85f2d5d77b6dd31cf5d935747cd3f4ed7e2c8
Jul 24 15:44:44.982709 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:44.982672    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2a1c00-bf847bc88-dhqqm"]
Jul 24 15:44:44.987640 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:44.987614    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2a1c00-bf847bc88-dhqqm"
Jul 24 15:44:44.988125 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:44.988101    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a1c00-79857b87b9-64h74" event={"ID":"c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20","Type":"ContainerStarted","Data":"814d93314f2be52afa8deb5a2fe85f2d5d77b6dd31cf5d935747cd3f4ed7e2c8"}
Jul 24 15:44:44.990068 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:44.990050    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-2a1c00\""
Jul 24 15:44:45.000382 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:45.000355    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2a1c00-bf847bc88-dhqqm"]
Jul 24 15:44:45.058020 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:45.057985    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2a1c00-5d4d865ffb-dq97p"]
Jul 24 15:44:45.061341 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:45.061322    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2a1c00-5d4d865ffb-dq97p"
Jul 24 15:44:45.064271 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:45.064250    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-2a1c00-dockercfg-dtg8v\""
Jul 24 15:44:45.075848 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:45.075809    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2a1c00-5d4d865ffb-dq97p"]
Jul 24 15:44:45.101758 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:45.101724    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4dpk\" (UniqueName: \"kubernetes.io/projected/795d6360-c767-408a-be9d-4d06b532bfd6-kube-api-access-g4dpk\") pod \"payload-processing-e2e-inf-a-2a1c00-bf847bc88-dhqqm\" (UID: \"795d6360-c767-408a-be9d-4d06b532bfd6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2a1c00-bf847bc88-dhqqm"
Jul 24 15:44:45.101925 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:45.101850    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcvvb\" (UniqueName: \"kubernetes.io/projected/a38e1ff2-ae98-46f0-a0da-2eaf2a04fbaf-kube-api-access-pcvvb\") pod \"payload-pre-processing-e2e-inf-a-2a1c00-5d4d865ffb-dq97p\" (UID: \"a38e1ff2-ae98-46f0-a0da-2eaf2a04fbaf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2a1c00-5d4d865ffb-dq97p"
Jul 24 15:44:45.101925 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:45.101884    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/795d6360-c767-408a-be9d-4d06b532bfd6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2a1c00-bf847bc88-dhqqm\" (UID: \"795d6360-c767-408a-be9d-4d06b532bfd6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2a1c00-bf847bc88-dhqqm"
Jul 24 15:44:45.101925 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:45.101906    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a38e1ff2-ae98-46f0-a0da-2eaf2a04fbaf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2a1c00-5d4d865ffb-dq97p\" (UID: \"a38e1ff2-ae98-46f0-a0da-2eaf2a04fbaf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2a1c00-5d4d865ffb-dq97p"
Jul 24 15:44:45.203266 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:45.203242    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4dpk\" (UniqueName: \"kubernetes.io/projected/795d6360-c767-408a-be9d-4d06b532bfd6-kube-api-access-g4dpk\") pod \"payload-processing-e2e-inf-a-2a1c00-bf847bc88-dhqqm\" (UID: \"795d6360-c767-408a-be9d-4d06b532bfd6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2a1c00-bf847bc88-dhqqm"
Jul 24 15:44:45.203394 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:45.203290    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcvvb\" (UniqueName: \"kubernetes.io/projected/a38e1ff2-ae98-46f0-a0da-2eaf2a04fbaf-kube-api-access-pcvvb\") pod \"payload-pre-processing-e2e-inf-a-2a1c00-5d4d865ffb-dq97p\" (UID: \"a38e1ff2-ae98-46f0-a0da-2eaf2a04fbaf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2a1c00-5d4d865ffb-dq97p"
Jul 24 15:44:45.203394 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:45.203323    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/795d6360-c767-408a-be9d-4d06b532bfd6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2a1c00-bf847bc88-dhqqm\" (UID: \"795d6360-c767-408a-be9d-4d06b532bfd6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2a1c00-bf847bc88-dhqqm"
Jul 24 15:44:45.203394 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:45.203349    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a38e1ff2-ae98-46f0-a0da-2eaf2a04fbaf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2a1c00-5d4d865ffb-dq97p\" (UID: \"a38e1ff2-ae98-46f0-a0da-2eaf2a04fbaf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2a1c00-5d4d865ffb-dq97p"
Jul 24 15:44:45.204085 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:45.204061    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/795d6360-c767-408a-be9d-4d06b532bfd6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2a1c00-bf847bc88-dhqqm\" (UID: \"795d6360-c767-408a-be9d-4d06b532bfd6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2a1c00-bf847bc88-dhqqm"
Jul 24 15:44:45.204177 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:45.204106    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a38e1ff2-ae98-46f0-a0da-2eaf2a04fbaf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2a1c00-5d4d865ffb-dq97p\" (UID: \"a38e1ff2-ae98-46f0-a0da-2eaf2a04fbaf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2a1c00-5d4d865ffb-dq97p"
Jul 24 15:44:45.212167 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:45.212137    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcvvb\" (UniqueName: \"kubernetes.io/projected/a38e1ff2-ae98-46f0-a0da-2eaf2a04fbaf-kube-api-access-pcvvb\") pod \"payload-pre-processing-e2e-inf-a-2a1c00-5d4d865ffb-dq97p\" (UID: \"a38e1ff2-ae98-46f0-a0da-2eaf2a04fbaf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2a1c00-5d4d865ffb-dq97p"
Jul 24 15:44:45.212404 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:45.212388    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4dpk\" (UniqueName: \"kubernetes.io/projected/795d6360-c767-408a-be9d-4d06b532bfd6-kube-api-access-g4dpk\") pod \"payload-processing-e2e-inf-a-2a1c00-bf847bc88-dhqqm\" (UID: \"795d6360-c767-408a-be9d-4d06b532bfd6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2a1c00-bf847bc88-dhqqm"
Jul 24 15:44:45.299081 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:45.299049    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2a1c00-bf847bc88-dhqqm"
Jul 24 15:44:45.371655 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:45.371619    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2a1c00-5d4d865ffb-dq97p"
Jul 24 15:44:45.437415 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:45.437382    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2a1c00-bf847bc88-dhqqm"]
Jul 24 15:44:45.439693 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:44:45.439665    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod795d6360_c767_408a_be9d_4d06b532bfd6.slice/crio-130b3279c6c14f4d66497298dc8f91a51b4a61530ed6d31db19d394cc3e479fd WatchSource:0}: Error finding container 130b3279c6c14f4d66497298dc8f91a51b4a61530ed6d31db19d394cc3e479fd: Status 404 returned error can't find the container with id 130b3279c6c14f4d66497298dc8f91a51b4a61530ed6d31db19d394cc3e479fd
Jul 24 15:44:45.512152 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:45.512125    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2a1c00-5d4d865ffb-dq97p"]
Jul 24 15:44:45.513870 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:44:45.513841    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda38e1ff2_ae98_46f0_a0da_2eaf2a04fbaf.slice/crio-cc097f696386c3689ce3349b35a016fd4a77e470f60728404be2ae87b0df1b75 WatchSource:0}: Error finding container cc097f696386c3689ce3349b35a016fd4a77e470f60728404be2ae87b0df1b75: Status 404 returned error can't find the container with id cc097f696386c3689ce3349b35a016fd4a77e470f60728404be2ae87b0df1b75
Jul 24 15:44:45.994402 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:45.994297    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2a1c00-bf847bc88-dhqqm" event={"ID":"795d6360-c767-408a-be9d-4d06b532bfd6","Type":"ContainerStarted","Data":"2bba3257adb9db746a19e3577f12c196f931a911fda366b427e77c611071770e"}
Jul 24 15:44:45.994402 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:45.994374    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2a1c00-bf847bc88-dhqqm"
Jul 24 15:44:45.994402 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:45.994397    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2a1c00-bf847bc88-dhqqm" event={"ID":"795d6360-c767-408a-be9d-4d06b532bfd6","Type":"ContainerStarted","Data":"130b3279c6c14f4d66497298dc8f91a51b4a61530ed6d31db19d394cc3e479fd"}
Jul 24 15:44:45.995996 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:45.995948    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a1c00-79857b87b9-64h74" event={"ID":"c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20","Type":"ContainerStarted","Data":"4c70d316bc945c0fc9a668e791923c0b5c6293476d8352d12eb0a7e30487cdab"}
Jul 24 15:44:45.996143 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:45.996068    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a1c00-79857b87b9-64h74"
Jul 24 15:44:45.997420 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:45.997397    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2a1c00-5d4d865ffb-dq97p" event={"ID":"a38e1ff2-ae98-46f0-a0da-2eaf2a04fbaf","Type":"ContainerStarted","Data":"dab18a7e96faac93211887f25dfcfb623e6b830c11850eb955a76fc79563d08e"}
Jul 24 15:44:45.997508 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:45.997443    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2a1c00-5d4d865ffb-dq97p" event={"ID":"a38e1ff2-ae98-46f0-a0da-2eaf2a04fbaf","Type":"ContainerStarted","Data":"cc097f696386c3689ce3349b35a016fd4a77e470f60728404be2ae87b0df1b75"}
Jul 24 15:44:45.997508 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:45.997482    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2a1c00-5d4d865ffb-dq97p"
Jul 24 15:44:46.017281 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:46.017238    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-2a1c00-bf847bc88-dhqqm" podStartSLOduration=2.017225822 podStartE2EDuration="2.017225822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:44:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:44:46.014897104 +0000 UTC m=+2178.827249284" watchObservedRunningTime="2026-07-24 15:44:46.017225822 +0000 UTC m=+2178.829578039"
Jul 24 15:44:46.038859 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:46.038802    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2a1c00-5d4d865ffb-dq97p" podStartSLOduration=1.038790028 podStartE2EDuration="1.038790028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:44:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:44:46.036527168 +0000 UTC m=+2178.848879344" watchObservedRunningTime="2026-07-24 15:44:46.038790028 +0000 UTC m=+2178.851142243"
Jul 24 15:44:46.059829 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:46.059777    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a1c00-79857b87b9-64h74" podStartSLOduration=2.380324633 podStartE2EDuration="3.059763347s" totalImagesPullingTime="679.438714ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:44:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:44:46.057351052 +0000 UTC m=+2178.869703231" watchObservedRunningTime="2026-07-24 15:44:46.059763347 +0000 UTC m=+2178.872115525"
Jul 24 15:44:47.134441 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:47.134405    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:44:52.007705 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:52.007676    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a1c00-79857b87b9-64h74"
Jul 24 15:44:52.062831 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:52.062789    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b90afd-6c68469c95-pz7n5"]
Jul 24 15:44:52.067173 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:52.067146    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b90afd-6c68469c95-pz7n5"
Jul 24 15:44:52.073937 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:52.073913    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-b90afd-serving-cert\""
Jul 24 15:44:52.091429 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:52.091394    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b90afd-6c68469c95-pz7n5"]
Jul 24 15:44:52.175681 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:52.175647    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24bc147b-4fe5-480a-b7b5-99c9f859d5e8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b90afd-6c68469c95-pz7n5\" (UID: \"24bc147b-4fe5-480a-b7b5-99c9f859d5e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b90afd-6c68469c95-pz7n5"
Jul 24 15:44:52.175892 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:52.175735    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24bc147b-4fe5-480a-b7b5-99c9f859d5e8-maas-api-tls\") pod \"maas-api-e2e-inf-b-b90afd-6c68469c95-pz7n5\" (UID: \"24bc147b-4fe5-480a-b7b5-99c9f859d5e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b90afd-6c68469c95-pz7n5"
Jul 24 15:44:52.175892 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:52.175755    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnx7d\" (UniqueName: \"kubernetes.io/projected/24bc147b-4fe5-480a-b7b5-99c9f859d5e8-kube-api-access-nnx7d\") pod \"maas-api-e2e-inf-b-b90afd-6c68469c95-pz7n5\" (UID: \"24bc147b-4fe5-480a-b7b5-99c9f859d5e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b90afd-6c68469c95-pz7n5"
Jul 24 15:44:52.277210 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:52.277176    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24bc147b-4fe5-480a-b7b5-99c9f859d5e8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b90afd-6c68469c95-pz7n5\" (UID: \"24bc147b-4fe5-480a-b7b5-99c9f859d5e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b90afd-6c68469c95-pz7n5"
Jul 24 15:44:52.277379 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:52.277246    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24bc147b-4fe5-480a-b7b5-99c9f859d5e8-maas-api-tls\") pod \"maas-api-e2e-inf-b-b90afd-6c68469c95-pz7n5\" (UID: \"24bc147b-4fe5-480a-b7b5-99c9f859d5e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b90afd-6c68469c95-pz7n5"
Jul 24 15:44:52.277379 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:52.277264    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnx7d\" (UniqueName: \"kubernetes.io/projected/24bc147b-4fe5-480a-b7b5-99c9f859d5e8-kube-api-access-nnx7d\") pod \"maas-api-e2e-inf-b-b90afd-6c68469c95-pz7n5\" (UID: \"24bc147b-4fe5-480a-b7b5-99c9f859d5e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b90afd-6c68469c95-pz7n5"
Jul 24 15:44:52.277514 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:44:52.277415    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-b90afd-serving-cert: secret "maas-api-e2e-inf-b-b90afd-serving-cert" not found
Jul 24 15:44:52.277514 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:44:52.277492    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24bc147b-4fe5-480a-b7b5-99c9f859d5e8-maas-api-tls podName:24bc147b-4fe5-480a-b7b5-99c9f859d5e8 nodeName:}" failed. No retries permitted until 2026-07-24 15:44:52.777472679 +0000 UTC m=+2185.589824836 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/24bc147b-4fe5-480a-b7b5-99c9f859d5e8-maas-api-tls") pod "maas-api-e2e-inf-b-b90afd-6c68469c95-pz7n5" (UID: "24bc147b-4fe5-480a-b7b5-99c9f859d5e8") : secret "maas-api-e2e-inf-b-b90afd-serving-cert" not found
Jul 24 15:44:52.277790 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:52.277770    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24bc147b-4fe5-480a-b7b5-99c9f859d5e8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b90afd-6c68469c95-pz7n5\" (UID: \"24bc147b-4fe5-480a-b7b5-99c9f859d5e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b90afd-6c68469c95-pz7n5"
Jul 24 15:44:52.286916 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:52.286896    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnx7d\" (UniqueName: \"kubernetes.io/projected/24bc147b-4fe5-480a-b7b5-99c9f859d5e8-kube-api-access-nnx7d\") pod \"maas-api-e2e-inf-b-b90afd-6c68469c95-pz7n5\" (UID: \"24bc147b-4fe5-480a-b7b5-99c9f859d5e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b90afd-6c68469c95-pz7n5"
Jul 24 15:44:52.781945 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:52.781908    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24bc147b-4fe5-480a-b7b5-99c9f859d5e8-maas-api-tls\") pod \"maas-api-e2e-inf-b-b90afd-6c68469c95-pz7n5\" (UID: \"24bc147b-4fe5-480a-b7b5-99c9f859d5e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b90afd-6c68469c95-pz7n5"
Jul 24 15:44:52.784534 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:52.784476    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24bc147b-4fe5-480a-b7b5-99c9f859d5e8-maas-api-tls\") pod \"maas-api-e2e-inf-b-b90afd-6c68469c95-pz7n5\" (UID: \"24bc147b-4fe5-480a-b7b5-99c9f859d5e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b90afd-6c68469c95-pz7n5"
Jul 24 15:44:52.863958 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:52.863928    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b90afd-7898c794dd-hjjvr"]
Jul 24 15:44:52.867702 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:52.867683    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b90afd-7898c794dd-hjjvr"
Jul 24 15:44:52.869796 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:52.869777    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b90afd\""
Jul 24 15:44:52.881865 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:52.881840    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b90afd-7898c794dd-hjjvr"]
Jul 24 15:44:52.893070 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:52.893050    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b90afd-bf596f69b-fhcnn"]
Jul 24 15:44:52.896396 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:52.896361    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b90afd-bf596f69b-fhcnn"
Jul 24 15:44:52.899591 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:52.899571    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b90afd-dockercfg-628gx\""
Jul 24 15:44:52.908846 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:52.908811    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b90afd-bf596f69b-fhcnn"]
Jul 24 15:44:52.980560 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:52.980534    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b90afd-6c68469c95-pz7n5"
Jul 24 15:44:52.984592 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:52.984565    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg9lc\" (UniqueName: \"kubernetes.io/projected/1dbe50be-c5e4-475e-b9d4-f43266a8beb0-kube-api-access-mg9lc\") pod \"payload-pre-processing-e2e-inf-b-b90afd-7898c794dd-hjjvr\" (UID: \"1dbe50be-c5e4-475e-b9d4-f43266a8beb0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b90afd-7898c794dd-hjjvr"
Jul 24 15:44:52.984698 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:52.984627    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blmsw\" (UniqueName: \"kubernetes.io/projected/58f479fc-2f04-4b8d-9e08-85a6eedaa736-kube-api-access-blmsw\") pod \"payload-processing-e2e-inf-b-b90afd-bf596f69b-fhcnn\" (UID: \"58f479fc-2f04-4b8d-9e08-85a6eedaa736\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b90afd-bf596f69b-fhcnn"
Jul 24 15:44:52.984698 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:52.984682    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dbe50be-c5e4-475e-b9d4-f43266a8beb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b90afd-7898c794dd-hjjvr\" (UID: \"1dbe50be-c5e4-475e-b9d4-f43266a8beb0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b90afd-7898c794dd-hjjvr"
Jul 24 15:44:52.984791 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:52.984713    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f479fc-2f04-4b8d-9e08-85a6eedaa736-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b90afd-bf596f69b-fhcnn\" (UID: \"58f479fc-2f04-4b8d-9e08-85a6eedaa736\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b90afd-bf596f69b-fhcnn"
Jul 24 15:44:53.086202 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:53.086164    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f479fc-2f04-4b8d-9e08-85a6eedaa736-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b90afd-bf596f69b-fhcnn\" (UID: \"58f479fc-2f04-4b8d-9e08-85a6eedaa736\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b90afd-bf596f69b-fhcnn"
Jul 24 15:44:53.086640 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:53.086276    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg9lc\" (UniqueName: \"kubernetes.io/projected/1dbe50be-c5e4-475e-b9d4-f43266a8beb0-kube-api-access-mg9lc\") pod \"payload-pre-processing-e2e-inf-b-b90afd-7898c794dd-hjjvr\" (UID: \"1dbe50be-c5e4-475e-b9d4-f43266a8beb0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b90afd-7898c794dd-hjjvr"
Jul 24 15:44:53.086640 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:53.086361    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blmsw\" (UniqueName: \"kubernetes.io/projected/58f479fc-2f04-4b8d-9e08-85a6eedaa736-kube-api-access-blmsw\") pod \"payload-processing-e2e-inf-b-b90afd-bf596f69b-fhcnn\" (UID: \"58f479fc-2f04-4b8d-9e08-85a6eedaa736\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b90afd-bf596f69b-fhcnn"
Jul 24 15:44:53.086640 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:53.086421    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dbe50be-c5e4-475e-b9d4-f43266a8beb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b90afd-7898c794dd-hjjvr\" (UID: \"1dbe50be-c5e4-475e-b9d4-f43266a8beb0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b90afd-7898c794dd-hjjvr"
Jul 24 15:44:53.087082 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:53.087055    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f479fc-2f04-4b8d-9e08-85a6eedaa736-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b90afd-bf596f69b-fhcnn\" (UID: \"58f479fc-2f04-4b8d-9e08-85a6eedaa736\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b90afd-bf596f69b-fhcnn"
Jul 24 15:44:53.087340 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:53.087301    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dbe50be-c5e4-475e-b9d4-f43266a8beb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b90afd-7898c794dd-hjjvr\" (UID: \"1dbe50be-c5e4-475e-b9d4-f43266a8beb0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b90afd-7898c794dd-hjjvr"
Jul 24 15:44:53.096250 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:53.096223    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg9lc\" (UniqueName: \"kubernetes.io/projected/1dbe50be-c5e4-475e-b9d4-f43266a8beb0-kube-api-access-mg9lc\") pod \"payload-pre-processing-e2e-inf-b-b90afd-7898c794dd-hjjvr\" (UID: \"1dbe50be-c5e4-475e-b9d4-f43266a8beb0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b90afd-7898c794dd-hjjvr"
Jul 24 15:44:53.096379 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:53.096311    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blmsw\" (UniqueName: \"kubernetes.io/projected/58f479fc-2f04-4b8d-9e08-85a6eedaa736-kube-api-access-blmsw\") pod \"payload-processing-e2e-inf-b-b90afd-bf596f69b-fhcnn\" (UID: \"58f479fc-2f04-4b8d-9e08-85a6eedaa736\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b90afd-bf596f69b-fhcnn"
Jul 24 15:44:53.118841 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:53.118790    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b90afd-6c68469c95-pz7n5"]
Jul 24 15:44:53.120845 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:44:53.120808    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24bc147b_4fe5_480a_b7b5_99c9f859d5e8.slice/crio-a303436dff7814f470cbac7830df7d958bdd2f9d96149d2e20df9e34be700c11 WatchSource:0}: Error finding container a303436dff7814f470cbac7830df7d958bdd2f9d96149d2e20df9e34be700c11: Status 404 returned error can't find the container with id a303436dff7814f470cbac7830df7d958bdd2f9d96149d2e20df9e34be700c11
Jul 24 15:44:53.178671 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:53.178640    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b90afd-7898c794dd-hjjvr"
Jul 24 15:44:53.206123 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:53.206100    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b90afd-bf596f69b-fhcnn"
Jul 24 15:44:53.326486 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:53.326455    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b90afd-7898c794dd-hjjvr"]
Jul 24 15:44:53.328986 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:44:53.328927    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dbe50be_c5e4_475e_b9d4_f43266a8beb0.slice/crio-b577686bf98b0605bfa20322862eb1c562e976a9678ae2152215ea54990a879a WatchSource:0}: Error finding container b577686bf98b0605bfa20322862eb1c562e976a9678ae2152215ea54990a879a: Status 404 returned error can't find the container with id b577686bf98b0605bfa20322862eb1c562e976a9678ae2152215ea54990a879a
Jul 24 15:44:53.362777 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:53.362710    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b90afd-bf596f69b-fhcnn"]
Jul 24 15:44:53.373181 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:44:53.373154    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58f479fc_2f04_4b8d_9e08_85a6eedaa736.slice/crio-3d927dc5458f6d4799473312a57e84f556da41845695f12c7d6b82b289e21e48 WatchSource:0}: Error finding container 3d927dc5458f6d4799473312a57e84f556da41845695f12c7d6b82b289e21e48: Status 404 returned error can't find the container with id 3d927dc5458f6d4799473312a57e84f556da41845695f12c7d6b82b289e21e48
Jul 24 15:44:54.038503 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:54.038462    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b90afd-6c68469c95-pz7n5" event={"ID":"24bc147b-4fe5-480a-b7b5-99c9f859d5e8","Type":"ContainerStarted","Data":"d45bae9d16ccfa1c31e8710790c12bf750f259b1668ce0f2bb6d4cb3b2ccfc78"}
Jul 24 15:44:54.038704 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:54.038552    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b90afd-6c68469c95-pz7n5"
Jul 24 15:44:54.038704 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:54.038576    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b90afd-6c68469c95-pz7n5" event={"ID":"24bc147b-4fe5-480a-b7b5-99c9f859d5e8","Type":"ContainerStarted","Data":"a303436dff7814f470cbac7830df7d958bdd2f9d96149d2e20df9e34be700c11"}
Jul 24 15:44:54.040187 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:54.040156    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b90afd-7898c794dd-hjjvr" event={"ID":"1dbe50be-c5e4-475e-b9d4-f43266a8beb0","Type":"ContainerStarted","Data":"da4193ceaf7120c3e05f8dec591001ba402f93f62d2bcf7cd3a4374657e02ae5"}
Jul 24 15:44:54.040322 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:54.040214    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b90afd-7898c794dd-hjjvr" event={"ID":"1dbe50be-c5e4-475e-b9d4-f43266a8beb0","Type":"ContainerStarted","Data":"b577686bf98b0605bfa20322862eb1c562e976a9678ae2152215ea54990a879a"}
Jul 24 15:44:54.040322 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:54.040243    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b90afd-7898c794dd-hjjvr"
Jul 24 15:44:54.041793 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:54.041770    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b90afd-bf596f69b-fhcnn" event={"ID":"58f479fc-2f04-4b8d-9e08-85a6eedaa736","Type":"ContainerStarted","Data":"f120163d8bb0bb7eddec53da50ee2884ef7554e73f216bdc87840ce557b4854e"}
Jul 24 15:44:54.041915 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:54.041809    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b90afd-bf596f69b-fhcnn" event={"ID":"58f479fc-2f04-4b8d-9e08-85a6eedaa736","Type":"ContainerStarted","Data":"3d927dc5458f6d4799473312a57e84f556da41845695f12c7d6b82b289e21e48"}
Jul 24 15:44:54.041915 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:54.041901    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b90afd-bf596f69b-fhcnn"
Jul 24 15:44:54.072238 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:54.072188    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b90afd-6c68469c95-pz7n5" podStartSLOduration=1.621471296 podStartE2EDuration="2.072173061s" totalImagesPullingTime="450.701765ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:44:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:44:54.070744324 +0000 UTC m=+2186.883096505" watchObservedRunningTime="2026-07-24 15:44:54.072173061 +0000 UTC m=+2186.884525240"
Jul 24 15:44:54.105926 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:54.105887    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b90afd-7898c794dd-hjjvr" podStartSLOduration=2.105873614 podStartE2EDuration="2.105873614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:44:54.104932879 +0000 UTC m=+2186.917285071" watchObservedRunningTime="2026-07-24 15:44:54.105873614 +0000 UTC m=+2186.918225792"
Jul 24 15:44:54.131422 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:54.131377    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-b90afd-bf596f69b-fhcnn" podStartSLOduration=2.131361867 podStartE2EDuration="2.131361867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:44:54.129990288 +0000 UTC m=+2186.942342466" watchObservedRunningTime="2026-07-24 15:44:54.131361867 +0000 UTC m=+2186.943714047"
Jul 24 15:44:57.004761 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:57.004729    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2a1c00-5d4d865ffb-dq97p"
Jul 24 15:44:57.005180 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:44:57.004801    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2a1c00-bf847bc88-dhqqm"
Jul 24 15:45:00.054230 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:45:00.054201    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b90afd-6c68469c95-pz7n5"
Jul 24 15:45:04.454331 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:45:04.454299    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-2a1c00/test-model-2a1c00-kserve-74c9fb89f5-p246k"]
Jul 24 15:45:04.458006 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:45:04.457989    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2a1c00/test-model-2a1c00-kserve-74c9fb89f5-p246k"
Jul 24 15:45:04.460626 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:45:04.460603    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2a1c00\"/\"default-dockercfg-nrjf6\""
Jul 24 15:45:04.460745 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:45:04.460624    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2a1c00\"/\"openshift-service-ca.crt\""
Jul 24 15:45:04.461531 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:45:04.461511    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2a1c00\"/\"kube-root-ca.crt\""
Jul 24 15:45:04.461641 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:45:04.461517    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2a1c00\"/\"test-model-2a1c00-kserve-self-signed-certs\""
Jul 24 15:45:04.470666 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:45:04.470643    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2a1c00/test-model-2a1c00-kserve-74c9fb89f5-p246k"]
Jul 24 15:45:04.595090 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:45:04.595054    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/910c46fe-5bd2-458d-8fcc-9c4737f9a22a-dshm\") pod \"test-model-2a1c00-kserve-74c9fb89f5-p246k\" (UID: \"910c46fe-5bd2-458d-8fcc-9c4737f9a22a\") " pod="ai-tenant-e2e-inf-a-2a1c00/test-model-2a1c00-kserve-74c9fb89f5-p246k"
Jul 24 15:45:04.595309 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:45:04.595102    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/910c46fe-5bd2-458d-8fcc-9c4737f9a22a-tls-certs\") pod \"test-model-2a1c00-kserve-74c9fb89f5-p246k\" (UID: \"910c46fe-5bd2-458d-8fcc-9c4737f9a22a\") " pod="ai-tenant-e2e-inf-a-2a1c00/test-model-2a1c00-kserve-74c9fb89f5-p246k"
Jul 24 15:45:04.595309 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:45:04.595206    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv7xb\" (UniqueName: \"kubernetes.io/projected/910c46fe-5bd2-458d-8fcc-9c4737f9a22a-kube-api-access-cv7xb\") pod \"test-model-2a1c00-kserve-74c9fb89f5-p246k\" (UID: \"910c46fe-5bd2-458d-8fcc-9c4737f9a22a\") " pod="ai-tenant-e2e-inf-a-2a1c00/test-model-2a1c00-kserve-74c9fb89f5-p246k"
Jul 24 15:45:04.595309 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:45:04.595252    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/910c46fe-5bd2-458d-8fcc-9c4737f9a22a-home\") pod \"test-model-2a1c00-kserve-74c9fb89f5-p246k\" (UID: \"910c46fe-5bd2-458d-8fcc-9c4737f9a22a\") " pod="ai-tenant-e2e-inf-a-2a1c00/test-model-2a1c00-kserve-74c9fb89f5-p246k"
Jul 24 15:45:04.595309 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:45:04.595305    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/910c46fe-5bd2-458d-8fcc-9c4737f9a22a-model-cache\") pod \"test-model-2a1c00-kserve-74c9fb89f5-p246k\" (UID: \"910c46fe-5bd2-458d-8fcc-9c4737f9a22a\") " pod="ai-tenant-e2e-inf-a-2a1c00/test-model-2a1c00-kserve-74c9fb89f5-p246k"
Jul 24 15:45:04.595492 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:45:04.595346    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/910c46fe-5bd2-458d-8fcc-9c4737f9a22a-tmp-dir\") pod \"test-model-2a1c00-kserve-74c9fb89f5-p246k\" (UID: \"910c46fe-5bd2-458d-8fcc-9c4737f9a22a\") " pod="ai-tenant-e2e-inf-a-2a1c00/test-model-2a1c00-kserve-74c9fb89f5-p246k"
Jul 24 15:45:04.696391 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:45:04.696349    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv7xb\" (UniqueName: \"kubernetes.io/projected/910c46fe-5bd2-458d-8fcc-9c4737f9a22a-kube-api-access-cv7xb\") pod \"test-model-2a1c00-kserve-74c9fb89f5-p246k\" (UID: \"910c46fe-5bd2-458d-8fcc-9c4737f9a22a\") " pod="ai-tenant-e2e-inf-a-2a1c00/test-model-2a1c00-kserve-74c9fb89f5-p246k"
Jul 24 15:45:04.696558 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:45:04.696404    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/910c46fe-5bd2-458d-8fcc-9c4737f9a22a-home\") pod \"test-model-2a1c00-kserve-74c9fb89f5-p246k\" (UID: \"910c46fe-5bd2-458d-8fcc-9c4737f9a22a\") " pod="ai-tenant-e2e-inf-a-2a1c00/test-model-2a1c00-kserve-74c9fb89f5-p246k"
Jul 24 15:45:04.696558 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:45:04.696435    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/910c46fe-5bd2-458d-8fcc-9c4737f9a22a-model-cache\") pod \"test-model-2a1c00-kserve-74c9fb89f5-p246k\" (UID: \"910c46fe-5bd2-458d-8fcc-9c4737f9a22a\") " pod="ai-tenant-e2e-inf-a-2a1c00/test-model-2a1c00-kserve-74c9fb89f5-p246k"
Jul 24 15:45:04.696558 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:45:04.696467    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/910c46fe-5bd2-458d-8fcc-9c4737f9a22a-tmp-dir\") pod \"test-model-2a1c00-kserve-74c9fb89f5-p246k\" (UID: \"910c46fe-5bd2-458d-8fcc-9c4737f9a22a\") " pod="ai-tenant-e2e-inf-a-2a1c00/test-model-2a1c00-kserve-74c9fb89f5-p246k"
Jul 24 15:45:04.696558 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:45:04.696493    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/910c46fe-5bd2-458d-8fcc-9c4737f9a22a-dshm\") pod \"test-model-2a1c00-kserve-74c9fb89f5-p246k\" (UID: \"910c46fe-5bd2-458d-8fcc-9c4737f9a22a\") " pod="ai-tenant-e2e-inf-a-2a1c00/test-model-2a1c00-kserve-74c9fb89f5-p246k"
Jul 24 15:45:04.696558 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:45:04.696517    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/910c46fe-5bd2-458d-8fcc-9c4737f9a22a-tls-certs\") pod \"test-model-2a1c00-kserve-74c9fb89f5-p246k\" (UID: \"910c46fe-5bd2-458d-8fcc-9c4737f9a22a\") " pod="ai-tenant-e2e-inf-a-2a1c00/test-model-2a1c00-kserve-74c9fb89f5-p246k"
Jul 24 15:45:04.696936 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:45:04.696906    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/910c46fe-5bd2-458d-8fcc-9c4737f9a22a-model-cache\") pod \"test-model-2a1c00-kserve-74c9fb89f5-p246k\" (UID: \"910c46fe-5bd2-458d-8fcc-9c4737f9a22a\") " pod="ai-tenant-e2e-inf-a-2a1c00/test-model-2a1c00-kserve-74c9fb89f5-p246k"
Jul 24 15:45:04.697063 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:45:04.696949    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/910c46fe-5bd2-458d-8fcc-9c4737f9a22a-tmp-dir\") pod \"test-model-2a1c00-kserve-74c9fb89f5-p246k\" (UID: \"910c46fe-5bd2-458d-8fcc-9c4737f9a22a\") " pod="ai-tenant-e2e-inf-a-2a1c00/test-model-2a1c00-kserve-74c9fb89f5-p246k"
Jul 24 15:45:04.697063 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:45:04.697016    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/910c46fe-5bd2-458d-8fcc-9c4737f9a22a-home\") pod \"test-model-2a1c00-kserve-74c9fb89f5-p246k\" (UID: \"910c46fe-5bd2-458d-8fcc-9c4737f9a22a\") " pod="ai-tenant-e2e-inf-a-2a1c00/test-model-2a1c00-kserve-74c9fb89f5-p246k"
Jul 24 15:45:04.698797 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:45:04.698774    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/910c46fe-5bd2-458d-8fcc-9c4737f9a22a-dshm\") pod \"test-model-2a1c00-kserve-74c9fb89f5-p246k\" (UID: \"910c46fe-5bd2-458d-8fcc-9c4737f9a22a\") " pod="ai-tenant-e2e-inf-a-2a1c00/test-model-2a1c00-kserve-74c9fb89f5-p246k"
Jul 24 15:45:04.699005 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:45:04.698987    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/910c46fe-5bd2-458d-8fcc-9c4737f9a22a-tls-certs\") pod \"test-model-2a1c00-kserve-74c9fb89f5-p246k\" (UID: \"910c46fe-5bd2-458d-8fcc-9c4737f9a22a\") " pod="ai-tenant-e2e-inf-a-2a1c00/test-model-2a1c00-kserve-74c9fb89f5-p246k"
Jul 24 15:45:04.706989 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:45:04.706921    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv7xb\" (UniqueName: \"kubernetes.io/projected/910c46fe-5bd2-458d-8fcc-9c4737f9a22a-kube-api-access-cv7xb\") pod \"test-model-2a1c00-kserve-74c9fb89f5-p246k\" (UID: \"910c46fe-5bd2-458d-8fcc-9c4737f9a22a\") " pod="ai-tenant-e2e-inf-a-2a1c00/test-model-2a1c00-kserve-74c9fb89f5-p246k"
Jul 24 15:45:04.768774 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:45:04.768736    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2a1c00/test-model-2a1c00-kserve-74c9fb89f5-p246k"
Jul 24 15:45:04.909270 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:45:04.909245    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2a1c00/test-model-2a1c00-kserve-74c9fb89f5-p246k"]
Jul 24 15:45:04.911274 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:45:04.911241    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod910c46fe_5bd2_458d_8fcc_9c4737f9a22a.slice/crio-4e177bae19c0fb07665bd0d5b4b570fd73af8264f9b83cfe7e290e0481674ca7 WatchSource:0}: Error finding container 4e177bae19c0fb07665bd0d5b4b570fd73af8264f9b83cfe7e290e0481674ca7: Status 404 returned error can't find the container with id 4e177bae19c0fb07665bd0d5b4b570fd73af8264f9b83cfe7e290e0481674ca7
Jul 24 15:45:05.050160 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:45:05.050130    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b90afd-bf596f69b-fhcnn"
Jul 24 15:45:05.050316 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:45:05.050201    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b90afd-7898c794dd-hjjvr"
Jul 24 15:45:05.113436 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:45:05.113395    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2a1c00/test-model-2a1c00-kserve-74c9fb89f5-p246k" event={"ID":"910c46fe-5bd2-458d-8fcc-9c4737f9a22a","Type":"ContainerStarted","Data":"82dada6076f567f11aa8ebcd9c7dea53b15da825f691013967e02b2301150b42"}
Jul 24 15:45:05.113611 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:45:05.113467    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2a1c00/test-model-2a1c00-kserve-74c9fb89f5-p246k" event={"ID":"910c46fe-5bd2-458d-8fcc-9c4737f9a22a","Type":"ContainerStarted","Data":"4e177bae19c0fb07665bd0d5b4b570fd73af8264f9b83cfe7e290e0481674ca7"}
Jul 24 15:45:05.141675 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:45:05.141616    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-2a1c00/test-model-2a1c00-kserve-74c9fb89f5-p246k" podStartSLOduration=1.141595774 podStartE2EDuration="1.141595774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:45:05.136881044 +0000 UTC m=+2197.949233225" watchObservedRunningTime="2026-07-24 15:45:05.141595774 +0000 UTC m=+2197.953947955"
Jul 24 15:45:08.224492 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:45:08.224457    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:45:14.768889 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:45:14.768847    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-2a1c00/test-model-2a1c00-kserve-74c9fb89f5-p246k"
Jul 24 15:45:14.769341 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:45:14.768904    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-2a1c00/test-model-2a1c00-kserve-74c9fb89f5-p246k"
Jul 24 15:45:14.781337 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:45:14.781314    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-2a1c00/test-model-2a1c00-kserve-74c9fb89f5-p246k"
Jul 24 15:45:15.173465 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:45:15.173382    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-2a1c00/test-model-2a1c00-kserve-74c9fb89f5-p246k"
Jul 24 15:45:23.221472 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:45:23.221438    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:45:45.325627 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:45:45.325590    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:45:59.530656 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:45:59.530625    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:46:20.961717 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:20.961682    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2a1c00/test-model-2a1c00-kserve-74c9fb89f5-p246k"]
Jul 24 15:46:25.863017 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:25.862978    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:46:32.398086 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:32.398052    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2a1c00-79857b87b9-64h74"]
Jul 24 15:46:32.398590 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:32.398281    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a1c00-79857b87b9-64h74" podUID="c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20" containerName="maas-api" containerID="cri-o://4c70d316bc945c0fc9a668e791923c0b5c6293476d8352d12eb0a7e30487cdab" gracePeriod=30
Jul 24 15:46:32.452312 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:32.452274    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2a1c00-bf847bc88-dhqqm"]
Jul 24 15:46:32.452564 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:32.452531    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-2a1c00-bf847bc88-dhqqm" podUID="795d6360-c767-408a-be9d-4d06b532bfd6" containerName="payload-processing" containerID="cri-o://2bba3257adb9db746a19e3577f12c196f931a911fda366b427e77c611071770e" gracePeriod=30
Jul 24 15:46:32.459567 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:32.459542    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2a1c00-5d4d865ffb-dq97p"]
Jul 24 15:46:32.460019 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:32.459972    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2a1c00-5d4d865ffb-dq97p" podUID="a38e1ff2-ae98-46f0-a0da-2eaf2a04fbaf" containerName="payload-pre-processing" containerID="cri-o://dab18a7e96faac93211887f25dfcfb623e6b830c11850eb955a76fc79563d08e" gracePeriod=30
Jul 24 15:46:32.554384 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:32.554352    2579 generic.go:356] "Generic (PLEG): container finished" podID="c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20" containerID="4c70d316bc945c0fc9a668e791923c0b5c6293476d8352d12eb0a7e30487cdab" exitCode=0
Jul 24 15:46:32.554508 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:32.554427    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a1c00-79857b87b9-64h74" event={"ID":"c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20","Type":"ContainerDied","Data":"4c70d316bc945c0fc9a668e791923c0b5c6293476d8352d12eb0a7e30487cdab"}
Jul 24 15:46:32.764769 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:32.764743    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a1c00-79857b87b9-64h74"
Jul 24 15:46:32.768153 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:32.768129    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2a1c00-bf847bc88-dhqqm"
Jul 24 15:46:32.771763 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:32.771744    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2a1c00-5d4d865ffb-dq97p"
Jul 24 15:46:32.842422 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:32.842391    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4dpk\" (UniqueName: \"kubernetes.io/projected/795d6360-c767-408a-be9d-4d06b532bfd6-kube-api-access-g4dpk\") pod \"795d6360-c767-408a-be9d-4d06b532bfd6\" (UID: \"795d6360-c767-408a-be9d-4d06b532bfd6\") "
Jul 24 15:46:32.842594 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:32.842445    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20-maas-api-tls\") pod \"c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20\" (UID: \"c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20\") "
Jul 24 15:46:32.842594 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:32.842500    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a38e1ff2-ae98-46f0-a0da-2eaf2a04fbaf-plugins-config-volume\") pod \"a38e1ff2-ae98-46f0-a0da-2eaf2a04fbaf\" (UID: \"a38e1ff2-ae98-46f0-a0da-2eaf2a04fbaf\") "
Jul 24 15:46:32.842594 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:32.842547    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20-openshift-service-ca\") pod \"c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20\" (UID: \"c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20\") "
Jul 24 15:46:32.842753 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:32.842604    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcxs7\" (UniqueName: \"kubernetes.io/projected/c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20-kube-api-access-kcxs7\") pod \"c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20\" (UID: \"c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20\") "
Jul 24 15:46:32.842839 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:32.842742    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/795d6360-c767-408a-be9d-4d06b532bfd6-plugins-config-volume\") pod \"795d6360-c767-408a-be9d-4d06b532bfd6\" (UID: \"795d6360-c767-408a-be9d-4d06b532bfd6\") "
Jul 24 15:46:32.842839 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:32.842796    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pcvvb\" (UniqueName: \"kubernetes.io/projected/a38e1ff2-ae98-46f0-a0da-2eaf2a04fbaf-kube-api-access-pcvvb\") pod \"a38e1ff2-ae98-46f0-a0da-2eaf2a04fbaf\" (UID: \"a38e1ff2-ae98-46f0-a0da-2eaf2a04fbaf\") "
Jul 24 15:46:32.842952 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:32.842919    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a38e1ff2-ae98-46f0-a0da-2eaf2a04fbaf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a38e1ff2-ae98-46f0-a0da-2eaf2a04fbaf" (UID: "a38e1ff2-ae98-46f0-a0da-2eaf2a04fbaf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:46:32.843198 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:32.843167    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a38e1ff2-ae98-46f0-a0da-2eaf2a04fbaf-plugins-config-volume\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:46:32.843198 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:32.843174    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/795d6360-c767-408a-be9d-4d06b532bfd6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "795d6360-c767-408a-be9d-4d06b532bfd6" (UID: "795d6360-c767-408a-be9d-4d06b532bfd6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:46:32.843405 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:32.843359    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20" (UID: "c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:46:32.844605 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:32.844583    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20" (UID: "c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 15:46:32.844841 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:32.844809    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a38e1ff2-ae98-46f0-a0da-2eaf2a04fbaf-kube-api-access-pcvvb" (OuterVolumeSpecName: "kube-api-access-pcvvb") pod "a38e1ff2-ae98-46f0-a0da-2eaf2a04fbaf" (UID: "a38e1ff2-ae98-46f0-a0da-2eaf2a04fbaf"). InnerVolumeSpecName "kube-api-access-pcvvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:46:32.844990 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:32.844971    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20-kube-api-access-kcxs7" (OuterVolumeSpecName: "kube-api-access-kcxs7") pod "c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20" (UID: "c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20"). InnerVolumeSpecName "kube-api-access-kcxs7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:46:32.845053 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:32.844999    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/795d6360-c767-408a-be9d-4d06b532bfd6-kube-api-access-g4dpk" (OuterVolumeSpecName: "kube-api-access-g4dpk") pod "795d6360-c767-408a-be9d-4d06b532bfd6" (UID: "795d6360-c767-408a-be9d-4d06b532bfd6"). InnerVolumeSpecName "kube-api-access-g4dpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:46:32.944451 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:32.944350    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/795d6360-c767-408a-be9d-4d06b532bfd6-plugins-config-volume\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:46:32.944451 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:32.944381    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcvvb\" (UniqueName: \"kubernetes.io/projected/a38e1ff2-ae98-46f0-a0da-2eaf2a04fbaf-kube-api-access-pcvvb\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:46:32.944451 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:32.944396    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4dpk\" (UniqueName: \"kubernetes.io/projected/795d6360-c767-408a-be9d-4d06b532bfd6-kube-api-access-g4dpk\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:46:32.944451 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:32.944410    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20-maas-api-tls\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:46:32.944451 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:32.944421    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20-openshift-service-ca\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:46:32.944451 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:32.944435    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcxs7\" (UniqueName: \"kubernetes.io/projected/c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20-kube-api-access-kcxs7\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:46:33.560411 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:33.560380    2579 generic.go:356] "Generic (PLEG): container finished" podID="795d6360-c767-408a-be9d-4d06b532bfd6" containerID="2bba3257adb9db746a19e3577f12c196f931a911fda366b427e77c611071770e" exitCode=0
Jul 24 15:46:33.560872 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:33.560433    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2a1c00-bf847bc88-dhqqm"
Jul 24 15:46:33.560872 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:33.560467    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2a1c00-bf847bc88-dhqqm" event={"ID":"795d6360-c767-408a-be9d-4d06b532bfd6","Type":"ContainerDied","Data":"2bba3257adb9db746a19e3577f12c196f931a911fda366b427e77c611071770e"}
Jul 24 15:46:33.560872 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:33.560522    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2a1c00-bf847bc88-dhqqm" event={"ID":"795d6360-c767-408a-be9d-4d06b532bfd6","Type":"ContainerDied","Data":"130b3279c6c14f4d66497298dc8f91a51b4a61530ed6d31db19d394cc3e479fd"}
Jul 24 15:46:33.560872 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:33.560546    2579 scope.go:117] "RemoveContainer" containerID="2bba3257adb9db746a19e3577f12c196f931a911fda366b427e77c611071770e"
Jul 24 15:46:33.561908 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:33.561881    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a1c00-79857b87b9-64h74"
Jul 24 15:46:33.561908 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:33.561897    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a1c00-79857b87b9-64h74" event={"ID":"c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20","Type":"ContainerDied","Data":"814d93314f2be52afa8deb5a2fe85f2d5d77b6dd31cf5d935747cd3f4ed7e2c8"}
Jul 24 15:46:33.563307 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:33.563282    2579 generic.go:356] "Generic (PLEG): container finished" podID="a38e1ff2-ae98-46f0-a0da-2eaf2a04fbaf" containerID="dab18a7e96faac93211887f25dfcfb623e6b830c11850eb955a76fc79563d08e" exitCode=0
Jul 24 15:46:33.563410 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:33.563308    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2a1c00-5d4d865ffb-dq97p" event={"ID":"a38e1ff2-ae98-46f0-a0da-2eaf2a04fbaf","Type":"ContainerDied","Data":"dab18a7e96faac93211887f25dfcfb623e6b830c11850eb955a76fc79563d08e"}
Jul 24 15:46:33.563410 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:33.563336    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2a1c00-5d4d865ffb-dq97p"
Jul 24 15:46:33.563410 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:33.563357    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2a1c00-5d4d865ffb-dq97p" event={"ID":"a38e1ff2-ae98-46f0-a0da-2eaf2a04fbaf","Type":"ContainerDied","Data":"cc097f696386c3689ce3349b35a016fd4a77e470f60728404be2ae87b0df1b75"}
Jul 24 15:46:33.571638 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:33.571603    2579 scope.go:117] "RemoveContainer" containerID="2bba3257adb9db746a19e3577f12c196f931a911fda366b427e77c611071770e"
Jul 24 15:46:33.572119 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:46:33.572088    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bba3257adb9db746a19e3577f12c196f931a911fda366b427e77c611071770e\": container with ID starting with 2bba3257adb9db746a19e3577f12c196f931a911fda366b427e77c611071770e not found: ID does not exist" containerID="2bba3257adb9db746a19e3577f12c196f931a911fda366b427e77c611071770e"
Jul 24 15:46:33.572216 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:33.572141    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bba3257adb9db746a19e3577f12c196f931a911fda366b427e77c611071770e"} err="failed to get container status \"2bba3257adb9db746a19e3577f12c196f931a911fda366b427e77c611071770e\": rpc error: code = NotFound desc = could not find container \"2bba3257adb9db746a19e3577f12c196f931a911fda366b427e77c611071770e\": container with ID starting with 2bba3257adb9db746a19e3577f12c196f931a911fda366b427e77c611071770e not found: ID does not exist"
Jul 24 15:46:33.572216 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:33.572166    2579 scope.go:117] "RemoveContainer" containerID="4c70d316bc945c0fc9a668e791923c0b5c6293476d8352d12eb0a7e30487cdab"
Jul 24 15:46:33.584528 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:33.584507    2579 scope.go:117] "RemoveContainer" containerID="dab18a7e96faac93211887f25dfcfb623e6b830c11850eb955a76fc79563d08e"
Jul 24 15:46:33.592242 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:33.592224    2579 scope.go:117] "RemoveContainer" containerID="dab18a7e96faac93211887f25dfcfb623e6b830c11850eb955a76fc79563d08e"
Jul 24 15:46:33.592538 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:46:33.592515    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dab18a7e96faac93211887f25dfcfb623e6b830c11850eb955a76fc79563d08e\": container with ID starting with dab18a7e96faac93211887f25dfcfb623e6b830c11850eb955a76fc79563d08e not found: ID does not exist" containerID="dab18a7e96faac93211887f25dfcfb623e6b830c11850eb955a76fc79563d08e"
Jul 24 15:46:33.592598 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:33.592544    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dab18a7e96faac93211887f25dfcfb623e6b830c11850eb955a76fc79563d08e"} err="failed to get container status \"dab18a7e96faac93211887f25dfcfb623e6b830c11850eb955a76fc79563d08e\": rpc error: code = NotFound desc = could not find container \"dab18a7e96faac93211887f25dfcfb623e6b830c11850eb955a76fc79563d08e\": container with ID starting with dab18a7e96faac93211887f25dfcfb623e6b830c11850eb955a76fc79563d08e not found: ID does not exist"
Jul 24 15:46:33.592971 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:33.592954    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2a1c00-79857b87b9-64h74"]
Jul 24 15:46:33.599219 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:33.599200    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2a1c00-79857b87b9-64h74"]
Jul 24 15:46:33.607735 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:33.607714    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2a1c00-5d4d865ffb-dq97p"]
Jul 24 15:46:33.613486 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:33.613466    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2a1c00-5d4d865ffb-dq97p"]
Jul 24 15:46:33.622166 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:33.622145    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2a1c00-bf847bc88-dhqqm"]
Jul 24 15:46:33.624344 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:33.624325    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2a1c00-bf847bc88-dhqqm"]
Jul 24 15:46:33.718666 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:33.718579    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="795d6360-c767-408a-be9d-4d06b532bfd6" path="/var/lib/kubelet/pods/795d6360-c767-408a-be9d-4d06b532bfd6/volumes"
Jul 24 15:46:33.719081 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:33.719059    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a38e1ff2-ae98-46f0-a0da-2eaf2a04fbaf" path="/var/lib/kubelet/pods/a38e1ff2-ae98-46f0-a0da-2eaf2a04fbaf/volumes"
Jul 24 15:46:33.719500 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:33.719479    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20" path="/var/lib/kubelet/pods/c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20/volumes"
Jul 24 15:46:35.980739 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:35.980703    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-2a1c00/test-model-2a1c00-kserve-74c9fb89f5-p246k" podUID="910c46fe-5bd2-458d-8fcc-9c4737f9a22a" containerName="main" containerID="cri-o://82dada6076f567f11aa8ebcd9c7dea53b15da825f691013967e02b2301150b42" gracePeriod=45
Jul 24 15:46:36.242625 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:36.242567    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2a1c00/test-model-2a1c00-kserve-74c9fb89f5-p246k"
Jul 24 15:46:36.273036 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:36.273005    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/910c46fe-5bd2-458d-8fcc-9c4737f9a22a-tmp-dir\") pod \"910c46fe-5bd2-458d-8fcc-9c4737f9a22a\" (UID: \"910c46fe-5bd2-458d-8fcc-9c4737f9a22a\") "
Jul 24 15:46:36.273187 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:36.273097    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/910c46fe-5bd2-458d-8fcc-9c4737f9a22a-model-cache\") pod \"910c46fe-5bd2-458d-8fcc-9c4737f9a22a\" (UID: \"910c46fe-5bd2-458d-8fcc-9c4737f9a22a\") "
Jul 24 15:46:36.273187 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:36.273144    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/910c46fe-5bd2-458d-8fcc-9c4737f9a22a-home\") pod \"910c46fe-5bd2-458d-8fcc-9c4737f9a22a\" (UID: \"910c46fe-5bd2-458d-8fcc-9c4737f9a22a\") "
Jul 24 15:46:36.273315 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:36.273215    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cv7xb\" (UniqueName: \"kubernetes.io/projected/910c46fe-5bd2-458d-8fcc-9c4737f9a22a-kube-api-access-cv7xb\") pod \"910c46fe-5bd2-458d-8fcc-9c4737f9a22a\" (UID: \"910c46fe-5bd2-458d-8fcc-9c4737f9a22a\") "
Jul 24 15:46:36.273315 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:36.273274    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/910c46fe-5bd2-458d-8fcc-9c4737f9a22a-tls-certs\") pod \"910c46fe-5bd2-458d-8fcc-9c4737f9a22a\" (UID: \"910c46fe-5bd2-458d-8fcc-9c4737f9a22a\") "
Jul 24 15:46:36.273315 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:36.273312    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/910c46fe-5bd2-458d-8fcc-9c4737f9a22a-dshm\") pod \"910c46fe-5bd2-458d-8fcc-9c4737f9a22a\" (UID: \"910c46fe-5bd2-458d-8fcc-9c4737f9a22a\") "
Jul 24 15:46:36.273465 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:36.273317    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/910c46fe-5bd2-458d-8fcc-9c4737f9a22a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "910c46fe-5bd2-458d-8fcc-9c4737f9a22a" (UID: "910c46fe-5bd2-458d-8fcc-9c4737f9a22a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:46:36.273465 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:36.273394    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/910c46fe-5bd2-458d-8fcc-9c4737f9a22a-model-cache" (OuterVolumeSpecName: "model-cache") pod "910c46fe-5bd2-458d-8fcc-9c4737f9a22a" (UID: "910c46fe-5bd2-458d-8fcc-9c4737f9a22a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:46:36.273465 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:36.273421    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/910c46fe-5bd2-458d-8fcc-9c4737f9a22a-home" (OuterVolumeSpecName: "home") pod "910c46fe-5bd2-458d-8fcc-9c4737f9a22a" (UID: "910c46fe-5bd2-458d-8fcc-9c4737f9a22a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:46:36.273655 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:36.273635    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/910c46fe-5bd2-458d-8fcc-9c4737f9a22a-tmp-dir\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:46:36.273655 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:36.273654    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/910c46fe-5bd2-458d-8fcc-9c4737f9a22a-model-cache\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:46:36.273853 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:36.273663    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/910c46fe-5bd2-458d-8fcc-9c4737f9a22a-home\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:46:36.275393 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:36.275370    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/910c46fe-5bd2-458d-8fcc-9c4737f9a22a-dshm" (OuterVolumeSpecName: "dshm") pod "910c46fe-5bd2-458d-8fcc-9c4737f9a22a" (UID: "910c46fe-5bd2-458d-8fcc-9c4737f9a22a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:46:36.275670 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:36.275647    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/910c46fe-5bd2-458d-8fcc-9c4737f9a22a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "910c46fe-5bd2-458d-8fcc-9c4737f9a22a" (UID: "910c46fe-5bd2-458d-8fcc-9c4737f9a22a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 15:46:36.275756 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:36.275669    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/910c46fe-5bd2-458d-8fcc-9c4737f9a22a-kube-api-access-cv7xb" (OuterVolumeSpecName: "kube-api-access-cv7xb") pod "910c46fe-5bd2-458d-8fcc-9c4737f9a22a" (UID: "910c46fe-5bd2-458d-8fcc-9c4737f9a22a"). InnerVolumeSpecName "kube-api-access-cv7xb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:46:36.374548 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:36.374516    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv7xb\" (UniqueName: \"kubernetes.io/projected/910c46fe-5bd2-458d-8fcc-9c4737f9a22a-kube-api-access-cv7xb\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:46:36.374548 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:36.374543    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/910c46fe-5bd2-458d-8fcc-9c4737f9a22a-tls-certs\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:46:36.374548 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:36.374553    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/910c46fe-5bd2-458d-8fcc-9c4737f9a22a-dshm\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:46:36.581952 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:36.581916    2579 generic.go:356] "Generic (PLEG): container finished" podID="910c46fe-5bd2-458d-8fcc-9c4737f9a22a" containerID="82dada6076f567f11aa8ebcd9c7dea53b15da825f691013967e02b2301150b42" exitCode=0
Jul 24 15:46:36.582139 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:36.581965    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2a1c00/test-model-2a1c00-kserve-74c9fb89f5-p246k"
Jul 24 15:46:36.582139 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:36.581975    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2a1c00/test-model-2a1c00-kserve-74c9fb89f5-p246k" event={"ID":"910c46fe-5bd2-458d-8fcc-9c4737f9a22a","Type":"ContainerDied","Data":"82dada6076f567f11aa8ebcd9c7dea53b15da825f691013967e02b2301150b42"}
Jul 24 15:46:36.582139 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:36.582013    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2a1c00/test-model-2a1c00-kserve-74c9fb89f5-p246k" event={"ID":"910c46fe-5bd2-458d-8fcc-9c4737f9a22a","Type":"ContainerDied","Data":"4e177bae19c0fb07665bd0d5b4b570fd73af8264f9b83cfe7e290e0481674ca7"}
Jul 24 15:46:36.582139 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:36.582036    2579 scope.go:117] "RemoveContainer" containerID="82dada6076f567f11aa8ebcd9c7dea53b15da825f691013967e02b2301150b42"
Jul 24 15:46:36.591223 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:36.591194    2579 scope.go:117] "RemoveContainer" containerID="82dada6076f567f11aa8ebcd9c7dea53b15da825f691013967e02b2301150b42"
Jul 24 15:46:36.591472 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:46:36.591452    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82dada6076f567f11aa8ebcd9c7dea53b15da825f691013967e02b2301150b42\": container with ID starting with 82dada6076f567f11aa8ebcd9c7dea53b15da825f691013967e02b2301150b42 not found: ID does not exist" containerID="82dada6076f567f11aa8ebcd9c7dea53b15da825f691013967e02b2301150b42"
Jul 24 15:46:36.591532 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:36.591481    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82dada6076f567f11aa8ebcd9c7dea53b15da825f691013967e02b2301150b42"} err="failed to get container status \"82dada6076f567f11aa8ebcd9c7dea53b15da825f691013967e02b2301150b42\": rpc error: code = NotFound desc = could not find container \"82dada6076f567f11aa8ebcd9c7dea53b15da825f691013967e02b2301150b42\": container with ID starting with 82dada6076f567f11aa8ebcd9c7dea53b15da825f691013967e02b2301150b42 not found: ID does not exist"
Jul 24 15:46:36.603587 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:36.603557    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2a1c00/test-model-2a1c00-kserve-74c9fb89f5-p246k"]
Jul 24 15:46:36.605278 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:36.605260    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-2a1c00/test-model-2a1c00-kserve-74c9fb89f5-p246k"]
Jul 24 15:46:37.716889 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:37.716856    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="910c46fe-5bd2-458d-8fcc-9c4737f9a22a" path="/var/lib/kubelet/pods/910c46fe-5bd2-458d-8fcc-9c4737f9a22a/volumes"
Jul 24 15:46:41.516076 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:41.516038    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:46:54.411401 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:54.411367    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b90afd-6c68469c95-pz7n5"]
Jul 24 15:46:54.411953 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:54.411588    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b90afd-6c68469c95-pz7n5" podUID="24bc147b-4fe5-480a-b7b5-99c9f859d5e8" containerName="maas-api" containerID="cri-o://d45bae9d16ccfa1c31e8710790c12bf750f259b1668ce0f2bb6d4cb3b2ccfc78" gracePeriod=30
Jul 24 15:46:54.474473 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:54.474437    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b90afd-bf596f69b-fhcnn"]
Jul 24 15:46:54.474729 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:54.474697    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-b90afd-bf596f69b-fhcnn" podUID="58f479fc-2f04-4b8d-9e08-85a6eedaa736" containerName="payload-processing" containerID="cri-o://f120163d8bb0bb7eddec53da50ee2884ef7554e73f216bdc87840ce557b4854e" gracePeriod=30
Jul 24 15:46:54.483641 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:54.483617    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b90afd-7898c794dd-hjjvr"]
Jul 24 15:46:54.483859 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:54.483827    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b90afd-7898c794dd-hjjvr" podUID="1dbe50be-c5e4-475e-b9d4-f43266a8beb0" containerName="payload-pre-processing" containerID="cri-o://da4193ceaf7120c3e05f8dec591001ba402f93f62d2bcf7cd3a4374657e02ae5" gracePeriod=30
Jul 24 15:46:54.672588 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:54.672505    2579 generic.go:356] "Generic (PLEG): container finished" podID="1dbe50be-c5e4-475e-b9d4-f43266a8beb0" containerID="da4193ceaf7120c3e05f8dec591001ba402f93f62d2bcf7cd3a4374657e02ae5" exitCode=0
Jul 24 15:46:54.672727 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:54.672581    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b90afd-7898c794dd-hjjvr" event={"ID":"1dbe50be-c5e4-475e-b9d4-f43266a8beb0","Type":"ContainerDied","Data":"da4193ceaf7120c3e05f8dec591001ba402f93f62d2bcf7cd3a4374657e02ae5"}
Jul 24 15:46:54.674134 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:54.674108    2579 generic.go:356] "Generic (PLEG): container finished" podID="58f479fc-2f04-4b8d-9e08-85a6eedaa736" containerID="f120163d8bb0bb7eddec53da50ee2884ef7554e73f216bdc87840ce557b4854e" exitCode=0
Jul 24 15:46:54.674234 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:54.674165    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b90afd-bf596f69b-fhcnn" event={"ID":"58f479fc-2f04-4b8d-9e08-85a6eedaa736","Type":"ContainerDied","Data":"f120163d8bb0bb7eddec53da50ee2884ef7554e73f216bdc87840ce557b4854e"}
Jul 24 15:46:54.774882 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:54.774834    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b90afd-bf596f69b-fhcnn"
Jul 24 15:46:54.778087 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:54.778066    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b90afd-7898c794dd-hjjvr"
Jul 24 15:46:54.848872 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:54.848849    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blmsw\" (UniqueName: \"kubernetes.io/projected/58f479fc-2f04-4b8d-9e08-85a6eedaa736-kube-api-access-blmsw\") pod \"58f479fc-2f04-4b8d-9e08-85a6eedaa736\" (UID: \"58f479fc-2f04-4b8d-9e08-85a6eedaa736\") "
Jul 24 15:46:54.848996 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:54.848882    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f479fc-2f04-4b8d-9e08-85a6eedaa736-plugins-config-volume\") pod \"58f479fc-2f04-4b8d-9e08-85a6eedaa736\" (UID: \"58f479fc-2f04-4b8d-9e08-85a6eedaa736\") "
Jul 24 15:46:54.848996 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:54.848942    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mg9lc\" (UniqueName: \"kubernetes.io/projected/1dbe50be-c5e4-475e-b9d4-f43266a8beb0-kube-api-access-mg9lc\") pod \"1dbe50be-c5e4-475e-b9d4-f43266a8beb0\" (UID: \"1dbe50be-c5e4-475e-b9d4-f43266a8beb0\") "
Jul 24 15:46:54.849118 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:54.849010    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dbe50be-c5e4-475e-b9d4-f43266a8beb0-plugins-config-volume\") pod \"1dbe50be-c5e4-475e-b9d4-f43266a8beb0\" (UID: \"1dbe50be-c5e4-475e-b9d4-f43266a8beb0\") "
Jul 24 15:46:54.849361 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:54.849332    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58f479fc-2f04-4b8d-9e08-85a6eedaa736-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "58f479fc-2f04-4b8d-9e08-85a6eedaa736" (UID: "58f479fc-2f04-4b8d-9e08-85a6eedaa736"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:46:54.849456 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:54.849388    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dbe50be-c5e4-475e-b9d4-f43266a8beb0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1dbe50be-c5e4-475e-b9d4-f43266a8beb0" (UID: "1dbe50be-c5e4-475e-b9d4-f43266a8beb0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:46:54.851093 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:54.851070    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58f479fc-2f04-4b8d-9e08-85a6eedaa736-kube-api-access-blmsw" (OuterVolumeSpecName: "kube-api-access-blmsw") pod "58f479fc-2f04-4b8d-9e08-85a6eedaa736" (UID: "58f479fc-2f04-4b8d-9e08-85a6eedaa736"). InnerVolumeSpecName "kube-api-access-blmsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:46:54.851195 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:54.851173    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dbe50be-c5e4-475e-b9d4-f43266a8beb0-kube-api-access-mg9lc" (OuterVolumeSpecName: "kube-api-access-mg9lc") pod "1dbe50be-c5e4-475e-b9d4-f43266a8beb0" (UID: "1dbe50be-c5e4-475e-b9d4-f43266a8beb0"). InnerVolumeSpecName "kube-api-access-mg9lc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:46:54.858172 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:54.858153    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b90afd-6c68469c95-pz7n5"
Jul 24 15:46:54.950076 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:54.949994    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24bc147b-4fe5-480a-b7b5-99c9f859d5e8-maas-api-tls\") pod \"24bc147b-4fe5-480a-b7b5-99c9f859d5e8\" (UID: \"24bc147b-4fe5-480a-b7b5-99c9f859d5e8\") "
Jul 24 15:46:54.950076 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:54.950045    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnx7d\" (UniqueName: \"kubernetes.io/projected/24bc147b-4fe5-480a-b7b5-99c9f859d5e8-kube-api-access-nnx7d\") pod \"24bc147b-4fe5-480a-b7b5-99c9f859d5e8\" (UID: \"24bc147b-4fe5-480a-b7b5-99c9f859d5e8\") "
Jul 24 15:46:54.950286 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:54.950085    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24bc147b-4fe5-480a-b7b5-99c9f859d5e8-openshift-service-ca\") pod \"24bc147b-4fe5-480a-b7b5-99c9f859d5e8\" (UID: \"24bc147b-4fe5-480a-b7b5-99c9f859d5e8\") "
Jul 24 15:46:54.950349 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:54.950329    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg9lc\" (UniqueName: \"kubernetes.io/projected/1dbe50be-c5e4-475e-b9d4-f43266a8beb0-kube-api-access-mg9lc\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:46:54.950412 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:54.950350    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dbe50be-c5e4-475e-b9d4-f43266a8beb0-plugins-config-volume\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:46:54.950412 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:54.950366    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blmsw\" (UniqueName: \"kubernetes.io/projected/58f479fc-2f04-4b8d-9e08-85a6eedaa736-kube-api-access-blmsw\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:46:54.950412 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:54.950379    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f479fc-2f04-4b8d-9e08-85a6eedaa736-plugins-config-volume\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:46:54.950564 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:54.950525    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24bc147b-4fe5-480a-b7b5-99c9f859d5e8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "24bc147b-4fe5-480a-b7b5-99c9f859d5e8" (UID: "24bc147b-4fe5-480a-b7b5-99c9f859d5e8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:46:54.952025 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:54.952004    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24bc147b-4fe5-480a-b7b5-99c9f859d5e8-kube-api-access-nnx7d" (OuterVolumeSpecName: "kube-api-access-nnx7d") pod "24bc147b-4fe5-480a-b7b5-99c9f859d5e8" (UID: "24bc147b-4fe5-480a-b7b5-99c9f859d5e8"). InnerVolumeSpecName "kube-api-access-nnx7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:46:54.952121 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:54.952063    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24bc147b-4fe5-480a-b7b5-99c9f859d5e8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "24bc147b-4fe5-480a-b7b5-99c9f859d5e8" (UID: "24bc147b-4fe5-480a-b7b5-99c9f859d5e8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 15:46:55.050989 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:55.050948    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24bc147b-4fe5-480a-b7b5-99c9f859d5e8-openshift-service-ca\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:46:55.050989 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:55.050983    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24bc147b-4fe5-480a-b7b5-99c9f859d5e8-maas-api-tls\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:46:55.050989 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:55.050993    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnx7d\" (UniqueName: \"kubernetes.io/projected/24bc147b-4fe5-480a-b7b5-99c9f859d5e8-kube-api-access-nnx7d\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:46:55.680864 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:55.680810    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b90afd-bf596f69b-fhcnn"
Jul 24 15:46:55.680864 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:55.680848    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b90afd-bf596f69b-fhcnn" event={"ID":"58f479fc-2f04-4b8d-9e08-85a6eedaa736","Type":"ContainerDied","Data":"3d927dc5458f6d4799473312a57e84f556da41845695f12c7d6b82b289e21e48"}
Jul 24 15:46:55.681402 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:55.680920    2579 scope.go:117] "RemoveContainer" containerID="f120163d8bb0bb7eddec53da50ee2884ef7554e73f216bdc87840ce557b4854e"
Jul 24 15:46:55.682283 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:55.682212    2579 generic.go:356] "Generic (PLEG): container finished" podID="24bc147b-4fe5-480a-b7b5-99c9f859d5e8" containerID="d45bae9d16ccfa1c31e8710790c12bf750f259b1668ce0f2bb6d4cb3b2ccfc78" exitCode=0
Jul 24 15:46:55.682357 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:55.682268    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b90afd-6c68469c95-pz7n5" event={"ID":"24bc147b-4fe5-480a-b7b5-99c9f859d5e8","Type":"ContainerDied","Data":"d45bae9d16ccfa1c31e8710790c12bf750f259b1668ce0f2bb6d4cb3b2ccfc78"}
Jul 24 15:46:55.682357 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:55.682319    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b90afd-6c68469c95-pz7n5"
Jul 24 15:46:55.682357 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:55.682334    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b90afd-6c68469c95-pz7n5" event={"ID":"24bc147b-4fe5-480a-b7b5-99c9f859d5e8","Type":"ContainerDied","Data":"a303436dff7814f470cbac7830df7d958bdd2f9d96149d2e20df9e34be700c11"}
Jul 24 15:46:55.683804 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:55.683776    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b90afd-7898c794dd-hjjvr"
Jul 24 15:46:55.683931 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:55.683781    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b90afd-7898c794dd-hjjvr" event={"ID":"1dbe50be-c5e4-475e-b9d4-f43266a8beb0","Type":"ContainerDied","Data":"b577686bf98b0605bfa20322862eb1c562e976a9678ae2152215ea54990a879a"}
Jul 24 15:46:55.690142 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:55.690122    2579 scope.go:117] "RemoveContainer" containerID="d45bae9d16ccfa1c31e8710790c12bf750f259b1668ce0f2bb6d4cb3b2ccfc78"
Jul 24 15:46:55.700277 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:55.700258    2579 scope.go:117] "RemoveContainer" containerID="d45bae9d16ccfa1c31e8710790c12bf750f259b1668ce0f2bb6d4cb3b2ccfc78"
Jul 24 15:46:55.700569 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:46:55.700549    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d45bae9d16ccfa1c31e8710790c12bf750f259b1668ce0f2bb6d4cb3b2ccfc78\": container with ID starting with d45bae9d16ccfa1c31e8710790c12bf750f259b1668ce0f2bb6d4cb3b2ccfc78 not found: ID does not exist" containerID="d45bae9d16ccfa1c31e8710790c12bf750f259b1668ce0f2bb6d4cb3b2ccfc78"
Jul 24 15:46:55.700635 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:55.700580    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d45bae9d16ccfa1c31e8710790c12bf750f259b1668ce0f2bb6d4cb3b2ccfc78"} err="failed to get container status \"d45bae9d16ccfa1c31e8710790c12bf750f259b1668ce0f2bb6d4cb3b2ccfc78\": rpc error: code = NotFound desc = could not find container \"d45bae9d16ccfa1c31e8710790c12bf750f259b1668ce0f2bb6d4cb3b2ccfc78\": container with ID starting with d45bae9d16ccfa1c31e8710790c12bf750f259b1668ce0f2bb6d4cb3b2ccfc78 not found: ID does not exist"
Jul 24 15:46:55.700635 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:55.700596    2579 scope.go:117] "RemoveContainer" containerID="da4193ceaf7120c3e05f8dec591001ba402f93f62d2bcf7cd3a4374657e02ae5"
Jul 24 15:46:55.708671 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:55.708643    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b90afd-7898c794dd-hjjvr"]
Jul 24 15:46:55.716466 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:55.716444    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b90afd-7898c794dd-hjjvr"]
Jul 24 15:46:55.721317 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:55.721295    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b90afd-bf596f69b-fhcnn"]
Jul 24 15:46:55.727022 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:55.727001    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b90afd-bf596f69b-fhcnn"]
Jul 24 15:46:55.736455 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:55.736434    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b90afd-6c68469c95-pz7n5"]
Jul 24 15:46:55.739981 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:55.739959    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b90afd-6c68469c95-pz7n5"]
Jul 24 15:46:57.417054 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:57.417021    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:46:57.718357 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:57.718271    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dbe50be-c5e4-475e-b9d4-f43266a8beb0" path="/var/lib/kubelet/pods/1dbe50be-c5e4-475e-b9d4-f43266a8beb0/volumes"
Jul 24 15:46:57.718737 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:57.718722    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24bc147b-4fe5-480a-b7b5-99c9f859d5e8" path="/var/lib/kubelet/pods/24bc147b-4fe5-480a-b7b5-99c9f859d5e8/volumes"
Jul 24 15:46:57.719107 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:46:57.719093    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58f479fc-2f04-4b8d-9e08-85a6eedaa736" path="/var/lib/kubelet/pods/58f479fc-2f04-4b8d-9e08-85a6eedaa736/volumes"
Jul 24 15:47:09.651202 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:09.651167    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-aefb55-676f6b859c-6kfss"]
Jul 24 15:47:09.651664 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:09.651529    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="910c46fe-5bd2-458d-8fcc-9c4737f9a22a" containerName="main"
Jul 24 15:47:09.651664 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:09.651540    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="910c46fe-5bd2-458d-8fcc-9c4737f9a22a" containerName="main"
Jul 24 15:47:09.651664 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:09.651548    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20" containerName="maas-api"
Jul 24 15:47:09.651664 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:09.651553    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20" containerName="maas-api"
Jul 24 15:47:09.651664 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:09.651563    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dbe50be-c5e4-475e-b9d4-f43266a8beb0" containerName="payload-pre-processing"
Jul 24 15:47:09.651664 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:09.651569    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dbe50be-c5e4-475e-b9d4-f43266a8beb0" containerName="payload-pre-processing"
Jul 24 15:47:09.651664 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:09.651577    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="795d6360-c767-408a-be9d-4d06b532bfd6" containerName="payload-processing"
Jul 24 15:47:09.651664 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:09.651583    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="795d6360-c767-408a-be9d-4d06b532bfd6" containerName="payload-processing"
Jul 24 15:47:09.651664 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:09.651589    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24bc147b-4fe5-480a-b7b5-99c9f859d5e8" containerName="maas-api"
Jul 24 15:47:09.651664 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:09.651594    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="24bc147b-4fe5-480a-b7b5-99c9f859d5e8" containerName="maas-api"
Jul 24 15:47:09.651664 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:09.651609    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a38e1ff2-ae98-46f0-a0da-2eaf2a04fbaf" containerName="payload-pre-processing"
Jul 24 15:47:09.651664 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:09.651614    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a38e1ff2-ae98-46f0-a0da-2eaf2a04fbaf" containerName="payload-pre-processing"
Jul 24 15:47:09.651664 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:09.651619    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58f479fc-2f04-4b8d-9e08-85a6eedaa736" containerName="payload-processing"
Jul 24 15:47:09.651664 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:09.651625    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="58f479fc-2f04-4b8d-9e08-85a6eedaa736" containerName="payload-processing"
Jul 24 15:47:09.652192 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:09.651690    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a38e1ff2-ae98-46f0-a0da-2eaf2a04fbaf" containerName="payload-pre-processing"
Jul 24 15:47:09.652192 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:09.651700    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="58f479fc-2f04-4b8d-9e08-85a6eedaa736" containerName="payload-processing"
Jul 24 15:47:09.652192 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:09.651706    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="910c46fe-5bd2-458d-8fcc-9c4737f9a22a" containerName="main"
Jul 24 15:47:09.652192 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:09.651713    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dbe50be-c5e4-475e-b9d4-f43266a8beb0" containerName="payload-pre-processing"
Jul 24 15:47:09.652192 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:09.651721    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="795d6360-c767-408a-be9d-4d06b532bfd6" containerName="payload-processing"
Jul 24 15:47:09.652192 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:09.651727    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5f0be5f-57eb-4a48-b3f5-f7cb5337ef20" containerName="maas-api"
Jul 24 15:47:09.652192 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:09.651732    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="24bc147b-4fe5-480a-b7b5-99c9f859d5e8" containerName="maas-api"
Jul 24 15:47:09.654940 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:09.654924    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aefb55-676f6b859c-6kfss"
Jul 24 15:47:09.657275 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:09.657256    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-aefb55-serving-cert\""
Jul 24 15:47:09.657431 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:09.657411    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 15:47:09.658037 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:09.658022    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tkn9s\""
Jul 24 15:47:09.658121 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:09.658024    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 15:47:09.666064 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:09.666042    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-aefb55-676f6b859c-6kfss"]
Jul 24 15:47:09.789753 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:09.789716    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a1eddba-580d-473a-aff2-b9c52bd1d9b3-maas-api-tls\") pod \"maas-api-e2e-api-a-aefb55-676f6b859c-6kfss\" (UID: \"1a1eddba-580d-473a-aff2-b9c52bd1d9b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aefb55-676f6b859c-6kfss"
Jul 24 15:47:09.789944 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:09.789806    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2w4j\" (UniqueName: \"kubernetes.io/projected/1a1eddba-580d-473a-aff2-b9c52bd1d9b3-kube-api-access-c2w4j\") pod \"maas-api-e2e-api-a-aefb55-676f6b859c-6kfss\" (UID: \"1a1eddba-580d-473a-aff2-b9c52bd1d9b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aefb55-676f6b859c-6kfss"
Jul 24 15:47:09.789944 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:09.789864    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a1eddba-580d-473a-aff2-b9c52bd1d9b3-openshift-service-ca\") pod \"maas-api-e2e-api-a-aefb55-676f6b859c-6kfss\" (UID: \"1a1eddba-580d-473a-aff2-b9c52bd1d9b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aefb55-676f6b859c-6kfss"
Jul 24 15:47:09.891131 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:09.891027    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2w4j\" (UniqueName: \"kubernetes.io/projected/1a1eddba-580d-473a-aff2-b9c52bd1d9b3-kube-api-access-c2w4j\") pod \"maas-api-e2e-api-a-aefb55-676f6b859c-6kfss\" (UID: \"1a1eddba-580d-473a-aff2-b9c52bd1d9b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aefb55-676f6b859c-6kfss"
Jul 24 15:47:09.891321 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:09.891151    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a1eddba-580d-473a-aff2-b9c52bd1d9b3-openshift-service-ca\") pod \"maas-api-e2e-api-a-aefb55-676f6b859c-6kfss\" (UID: \"1a1eddba-580d-473a-aff2-b9c52bd1d9b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aefb55-676f6b859c-6kfss"
Jul 24 15:47:09.891321 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:09.891272    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a1eddba-580d-473a-aff2-b9c52bd1d9b3-maas-api-tls\") pod \"maas-api-e2e-api-a-aefb55-676f6b859c-6kfss\" (UID: \"1a1eddba-580d-473a-aff2-b9c52bd1d9b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aefb55-676f6b859c-6kfss"
Jul 24 15:47:09.891501 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:47:09.891422    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-aefb55-serving-cert: secret "maas-api-e2e-api-a-aefb55-serving-cert" not found
Jul 24 15:47:09.891637 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:47:09.891533    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a1eddba-580d-473a-aff2-b9c52bd1d9b3-maas-api-tls podName:1a1eddba-580d-473a-aff2-b9c52bd1d9b3 nodeName:}" failed. No retries permitted until 2026-07-24 15:47:10.391510586 +0000 UTC m=+2323.203862746 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1a1eddba-580d-473a-aff2-b9c52bd1d9b3-maas-api-tls") pod "maas-api-e2e-api-a-aefb55-676f6b859c-6kfss" (UID: "1a1eddba-580d-473a-aff2-b9c52bd1d9b3") : secret "maas-api-e2e-api-a-aefb55-serving-cert" not found
Jul 24 15:47:09.891877 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:09.891856    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a1eddba-580d-473a-aff2-b9c52bd1d9b3-openshift-service-ca\") pod \"maas-api-e2e-api-a-aefb55-676f6b859c-6kfss\" (UID: \"1a1eddba-580d-473a-aff2-b9c52bd1d9b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aefb55-676f6b859c-6kfss"
Jul 24 15:47:09.901687 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:09.901634    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2w4j\" (UniqueName: \"kubernetes.io/projected/1a1eddba-580d-473a-aff2-b9c52bd1d9b3-kube-api-access-c2w4j\") pod \"maas-api-e2e-api-a-aefb55-676f6b859c-6kfss\" (UID: \"1a1eddba-580d-473a-aff2-b9c52bd1d9b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aefb55-676f6b859c-6kfss"
Jul 24 15:47:10.396997 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:10.396961    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a1eddba-580d-473a-aff2-b9c52bd1d9b3-maas-api-tls\") pod \"maas-api-e2e-api-a-aefb55-676f6b859c-6kfss\" (UID: \"1a1eddba-580d-473a-aff2-b9c52bd1d9b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aefb55-676f6b859c-6kfss"
Jul 24 15:47:10.399960 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:10.399932    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a1eddba-580d-473a-aff2-b9c52bd1d9b3-maas-api-tls\") pod \"maas-api-e2e-api-a-aefb55-676f6b859c-6kfss\" (UID: \"1a1eddba-580d-473a-aff2-b9c52bd1d9b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aefb55-676f6b859c-6kfss"
Jul 24 15:47:10.507746 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:10.507709    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-aefb55-55c5776b88-rzbd9"]
Jul 24 15:47:10.511631 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:10.511606    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aefb55-55c5776b88-rzbd9"
Jul 24 15:47:10.513941 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:10.513922    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-aefb55\""
Jul 24 15:47:10.514081 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:10.514060    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-aefb55-dockercfg-kgs29\""
Jul 24 15:47:10.521777 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:10.521753    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-aefb55-55c5776b88-rzbd9"]
Jul 24 15:47:10.540702 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:10.540668    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-aefb55-5746d97b96-5zszh"]
Jul 24 15:47:10.544123 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:10.544105    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-aefb55-5746d97b96-5zszh"
Jul 24 15:47:10.560441 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:10.560413    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-aefb55-5746d97b96-5zszh"]
Jul 24 15:47:10.565648 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:10.565622    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aefb55-676f6b859c-6kfss"
Jul 24 15:47:10.599807 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:10.599763    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/504dc68a-e1f8-4ea5-a61e-002dfc5aa0b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-aefb55-55c5776b88-rzbd9\" (UID: \"504dc68a-e1f8-4ea5-a61e-002dfc5aa0b1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-aefb55-55c5776b88-rzbd9"
Jul 24 15:47:10.599962 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:10.599856    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r57tq\" (UniqueName: \"kubernetes.io/projected/504dc68a-e1f8-4ea5-a61e-002dfc5aa0b1-kube-api-access-r57tq\") pod \"payload-pre-processing-e2e-api-a-aefb55-55c5776b88-rzbd9\" (UID: \"504dc68a-e1f8-4ea5-a61e-002dfc5aa0b1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-aefb55-55c5776b88-rzbd9"
Jul 24 15:47:10.699281 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:10.699255    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-aefb55-676f6b859c-6kfss"]
Jul 24 15:47:10.700532 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:10.700507    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8c145a0-fa06-4388-a205-7232b366a7e0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-aefb55-5746d97b96-5zszh\" (UID: \"e8c145a0-fa06-4388-a205-7232b366a7e0\") " pod="openshift-ingress/payload-processing-e2e-api-a-aefb55-5746d97b96-5zszh"
Jul 24 15:47:10.700667 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:10.700558    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/504dc68a-e1f8-4ea5-a61e-002dfc5aa0b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-aefb55-55c5776b88-rzbd9\" (UID: \"504dc68a-e1f8-4ea5-a61e-002dfc5aa0b1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-aefb55-55c5776b88-rzbd9"
Jul 24 15:47:10.700667 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:10.700586    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r57tq\" (UniqueName: \"kubernetes.io/projected/504dc68a-e1f8-4ea5-a61e-002dfc5aa0b1-kube-api-access-r57tq\") pod \"payload-pre-processing-e2e-api-a-aefb55-55c5776b88-rzbd9\" (UID: \"504dc68a-e1f8-4ea5-a61e-002dfc5aa0b1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-aefb55-55c5776b88-rzbd9"
Jul 24 15:47:10.700784 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:10.700677    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9cdp\" (UniqueName: \"kubernetes.io/projected/e8c145a0-fa06-4388-a205-7232b366a7e0-kube-api-access-s9cdp\") pod \"payload-processing-e2e-api-a-aefb55-5746d97b96-5zszh\" (UID: \"e8c145a0-fa06-4388-a205-7232b366a7e0\") " pod="openshift-ingress/payload-processing-e2e-api-a-aefb55-5746d97b96-5zszh"
Jul 24 15:47:10.701392 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:10.701370    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/504dc68a-e1f8-4ea5-a61e-002dfc5aa0b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-aefb55-55c5776b88-rzbd9\" (UID: \"504dc68a-e1f8-4ea5-a61e-002dfc5aa0b1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-aefb55-55c5776b88-rzbd9"
Jul 24 15:47:10.701591 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:47:10.701567    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a1eddba_580d_473a_aff2_b9c52bd1d9b3.slice/crio-a81be51c3d9fc9cc138a16577755166608037844b517112928cd27b54b9843d9 WatchSource:0}: Error finding container a81be51c3d9fc9cc138a16577755166608037844b517112928cd27b54b9843d9: Status 404 returned error can't find the container with id a81be51c3d9fc9cc138a16577755166608037844b517112928cd27b54b9843d9
Jul 24 15:47:10.702906 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:10.702888    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 15:47:10.708282 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:10.708262    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r57tq\" (UniqueName: \"kubernetes.io/projected/504dc68a-e1f8-4ea5-a61e-002dfc5aa0b1-kube-api-access-r57tq\") pod \"payload-pre-processing-e2e-api-a-aefb55-55c5776b88-rzbd9\" (UID: \"504dc68a-e1f8-4ea5-a61e-002dfc5aa0b1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-aefb55-55c5776b88-rzbd9"
Jul 24 15:47:10.761594 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:10.761560    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aefb55-676f6b859c-6kfss" event={"ID":"1a1eddba-580d-473a-aff2-b9c52bd1d9b3","Type":"ContainerStarted","Data":"a81be51c3d9fc9cc138a16577755166608037844b517112928cd27b54b9843d9"}
Jul 24 15:47:10.802099 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:10.802056    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9cdp\" (UniqueName: \"kubernetes.io/projected/e8c145a0-fa06-4388-a205-7232b366a7e0-kube-api-access-s9cdp\") pod \"payload-processing-e2e-api-a-aefb55-5746d97b96-5zszh\" (UID: \"e8c145a0-fa06-4388-a205-7232b366a7e0\") " pod="openshift-ingress/payload-processing-e2e-api-a-aefb55-5746d97b96-5zszh"
Jul 24 15:47:10.802900 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:10.802182    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8c145a0-fa06-4388-a205-7232b366a7e0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-aefb55-5746d97b96-5zszh\" (UID: \"e8c145a0-fa06-4388-a205-7232b366a7e0\") " pod="openshift-ingress/payload-processing-e2e-api-a-aefb55-5746d97b96-5zszh"
Jul 24 15:47:10.803126 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:10.803107    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8c145a0-fa06-4388-a205-7232b366a7e0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-aefb55-5746d97b96-5zszh\" (UID: \"e8c145a0-fa06-4388-a205-7232b366a7e0\") " pod="openshift-ingress/payload-processing-e2e-api-a-aefb55-5746d97b96-5zszh"
Jul 24 15:47:10.810245 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:10.810220    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9cdp\" (UniqueName: \"kubernetes.io/projected/e8c145a0-fa06-4388-a205-7232b366a7e0-kube-api-access-s9cdp\") pod \"payload-processing-e2e-api-a-aefb55-5746d97b96-5zszh\" (UID: \"e8c145a0-fa06-4388-a205-7232b366a7e0\") " pod="openshift-ingress/payload-processing-e2e-api-a-aefb55-5746d97b96-5zszh"
Jul 24 15:47:10.823717 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:10.823688    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aefb55-55c5776b88-rzbd9"
Jul 24 15:47:10.855454 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:10.855431    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-aefb55-5746d97b96-5zszh"
Jul 24 15:47:10.971679 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:10.971647    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-aefb55-55c5776b88-rzbd9"]
Jul 24 15:47:10.975076 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:47:10.975047    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod504dc68a_e1f8_4ea5_a61e_002dfc5aa0b1.slice/crio-ed73b2bac9bd5f60870b5e78d977bfed16c08bff1e432439d5d18c43464be562 WatchSource:0}: Error finding container ed73b2bac9bd5f60870b5e78d977bfed16c08bff1e432439d5d18c43464be562: Status 404 returned error can't find the container with id ed73b2bac9bd5f60870b5e78d977bfed16c08bff1e432439d5d18c43464be562
Jul 24 15:47:11.003845 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:11.003806    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-aefb55-5746d97b96-5zszh"]
Jul 24 15:47:11.011954 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:47:11.011930    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8c145a0_fa06_4388_a205_7232b366a7e0.slice/crio-efee87eb80a872c51ec7616237b84d8d913630e749ac9c7f1b8fc6a9b72721f6 WatchSource:0}: Error finding container efee87eb80a872c51ec7616237b84d8d913630e749ac9c7f1b8fc6a9b72721f6: Status 404 returned error can't find the container with id efee87eb80a872c51ec7616237b84d8d913630e749ac9c7f1b8fc6a9b72721f6
Jul 24 15:47:11.771767 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:11.771734    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aefb55-676f6b859c-6kfss" event={"ID":"1a1eddba-580d-473a-aff2-b9c52bd1d9b3","Type":"ContainerStarted","Data":"986311fdb4a3100ba05ccd345af1745cdc02312e36387f8a7c51c1e5f3e3daf1"}
Jul 24 15:47:11.772263 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:11.771905    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aefb55-676f6b859c-6kfss"
Jul 24 15:47:11.773252 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:11.773226    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-aefb55-5746d97b96-5zszh" event={"ID":"e8c145a0-fa06-4388-a205-7232b366a7e0","Type":"ContainerStarted","Data":"297b03ad4bd1f0f7515cdb9effb99e31fd758d0c726c545caf5693f6432ebd83"}
Jul 24 15:47:11.773392 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:11.773272    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-aefb55-5746d97b96-5zszh"
Jul 24 15:47:11.773392 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:11.773296    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-aefb55-5746d97b96-5zszh" event={"ID":"e8c145a0-fa06-4388-a205-7232b366a7e0","Type":"ContainerStarted","Data":"efee87eb80a872c51ec7616237b84d8d913630e749ac9c7f1b8fc6a9b72721f6"}
Jul 24 15:47:11.774807 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:11.774784    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aefb55-55c5776b88-rzbd9" event={"ID":"504dc68a-e1f8-4ea5-a61e-002dfc5aa0b1","Type":"ContainerStarted","Data":"63dab29607cba6238c491df9cd89f7b2be3f9592b85abcb36e7ae07cbb822a5e"}
Jul 24 15:47:11.774950 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:11.774847    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aefb55-55c5776b88-rzbd9" event={"ID":"504dc68a-e1f8-4ea5-a61e-002dfc5aa0b1","Type":"ContainerStarted","Data":"ed73b2bac9bd5f60870b5e78d977bfed16c08bff1e432439d5d18c43464be562"}
Jul 24 15:47:11.774950 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:11.774874    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aefb55-55c5776b88-rzbd9"
Jul 24 15:47:11.791187 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:11.791140    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aefb55-676f6b859c-6kfss" podStartSLOduration=2.115653989 podStartE2EDuration="2.791126046s" totalImagesPullingTime="675.472057ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:47:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:47:11.789848901 +0000 UTC m=+2324.602201081" watchObservedRunningTime="2026-07-24 15:47:11.791126046 +0000 UTC m=+2324.603478225"
Jul 24 15:47:11.805057 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:11.805015    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aefb55-55c5776b88-rzbd9" podStartSLOduration=1.805002351 podStartE2EDuration="1.805002351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:47:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:47:11.804078664 +0000 UTC m=+2324.616430856" watchObservedRunningTime="2026-07-24 15:47:11.805002351 +0000 UTC m=+2324.617354520"
Jul 24 15:47:11.821129 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:11.821085    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-aefb55-5746d97b96-5zszh" podStartSLOduration=1.821073516 podStartE2EDuration="1.821073516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:47:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:47:11.819331269 +0000 UTC m=+2324.631683449" watchObservedRunningTime="2026-07-24 15:47:11.821073516 +0000 UTC m=+2324.633425694"
Jul 24 15:47:14.832588 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:14.832549    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:47:17.785943 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:17.785916    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aefb55-676f6b859c-6kfss"
Jul 24 15:47:17.893830 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:17.893786    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-735232-5f498d95db-m8b6r"]
Jul 24 15:47:17.899159 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:17.899137    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-735232-5f498d95db-m8b6r"
Jul 24 15:47:17.902048 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:17.902028    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-735232-serving-cert\""
Jul 24 15:47:17.916994 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:17.916970    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-735232-5f498d95db-m8b6r"]
Jul 24 15:47:17.968487 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:17.968451    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mg77\" (UniqueName: \"kubernetes.io/projected/5115bf21-3b75-4341-8204-da72fa6c1c7f-kube-api-access-2mg77\") pod \"maas-api-e2e-api-b-735232-5f498d95db-m8b6r\" (UID: \"5115bf21-3b75-4341-8204-da72fa6c1c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-735232-5f498d95db-m8b6r"
Jul 24 15:47:17.968639 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:17.968541    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5115bf21-3b75-4341-8204-da72fa6c1c7f-openshift-service-ca\") pod \"maas-api-e2e-api-b-735232-5f498d95db-m8b6r\" (UID: \"5115bf21-3b75-4341-8204-da72fa6c1c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-735232-5f498d95db-m8b6r"
Jul 24 15:47:17.968639 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:17.968565    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5115bf21-3b75-4341-8204-da72fa6c1c7f-maas-api-tls\") pod \"maas-api-e2e-api-b-735232-5f498d95db-m8b6r\" (UID: \"5115bf21-3b75-4341-8204-da72fa6c1c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-735232-5f498d95db-m8b6r"
Jul 24 15:47:18.069778 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:18.069698    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5115bf21-3b75-4341-8204-da72fa6c1c7f-openshift-service-ca\") pod \"maas-api-e2e-api-b-735232-5f498d95db-m8b6r\" (UID: \"5115bf21-3b75-4341-8204-da72fa6c1c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-735232-5f498d95db-m8b6r"
Jul 24 15:47:18.069778 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:18.069746    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5115bf21-3b75-4341-8204-da72fa6c1c7f-maas-api-tls\") pod \"maas-api-e2e-api-b-735232-5f498d95db-m8b6r\" (UID: \"5115bf21-3b75-4341-8204-da72fa6c1c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-735232-5f498d95db-m8b6r"
Jul 24 15:47:18.069961 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:18.069843    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mg77\" (UniqueName: \"kubernetes.io/projected/5115bf21-3b75-4341-8204-da72fa6c1c7f-kube-api-access-2mg77\") pod \"maas-api-e2e-api-b-735232-5f498d95db-m8b6r\" (UID: \"5115bf21-3b75-4341-8204-da72fa6c1c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-735232-5f498d95db-m8b6r"
Jul 24 15:47:18.069961 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:47:18.069916    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-735232-serving-cert: secret "maas-api-e2e-api-b-735232-serving-cert" not found
Jul 24 15:47:18.070045 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:47:18.069998    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5115bf21-3b75-4341-8204-da72fa6c1c7f-maas-api-tls podName:5115bf21-3b75-4341-8204-da72fa6c1c7f nodeName:}" failed. No retries permitted until 2026-07-24 15:47:18.569973732 +0000 UTC m=+2331.382325898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5115bf21-3b75-4341-8204-da72fa6c1c7f-maas-api-tls") pod "maas-api-e2e-api-b-735232-5f498d95db-m8b6r" (UID: "5115bf21-3b75-4341-8204-da72fa6c1c7f") : secret "maas-api-e2e-api-b-735232-serving-cert" not found
Jul 24 15:47:18.070390 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:18.070369    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5115bf21-3b75-4341-8204-da72fa6c1c7f-openshift-service-ca\") pod \"maas-api-e2e-api-b-735232-5f498d95db-m8b6r\" (UID: \"5115bf21-3b75-4341-8204-da72fa6c1c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-735232-5f498d95db-m8b6r"
Jul 24 15:47:18.078328 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:18.078296    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mg77\" (UniqueName: \"kubernetes.io/projected/5115bf21-3b75-4341-8204-da72fa6c1c7f-kube-api-access-2mg77\") pod \"maas-api-e2e-api-b-735232-5f498d95db-m8b6r\" (UID: \"5115bf21-3b75-4341-8204-da72fa6c1c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-735232-5f498d95db-m8b6r"
Jul 24 15:47:18.549373 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:18.549340    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-735232-f8d9c84b8-wtm5f"]
Jul 24 15:47:18.552975 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:18.552954    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-735232-f8d9c84b8-wtm5f"
Jul 24 15:47:18.555338 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:18.555320    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-735232\""
Jul 24 15:47:18.555428 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:18.555381    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-735232-dockercfg-nm49z\""
Jul 24 15:47:18.563462 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:18.563439    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-735232-f8d9c84b8-wtm5f"]
Jul 24 15:47:18.565746 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:18.565722    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-735232-df6995698-6qxxn"]
Jul 24 15:47:18.569413 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:18.569394    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-735232-df6995698-6qxxn"
Jul 24 15:47:18.573316 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:18.573255    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5flv\" (UniqueName: \"kubernetes.io/projected/16924896-d0f2-4958-8ff5-b372607d13af-kube-api-access-c5flv\") pod \"payload-pre-processing-e2e-api-b-735232-f8d9c84b8-wtm5f\" (UID: \"16924896-d0f2-4958-8ff5-b372607d13af\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-735232-f8d9c84b8-wtm5f"
Jul 24 15:47:18.573529 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:18.573503    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5115bf21-3b75-4341-8204-da72fa6c1c7f-maas-api-tls\") pod \"maas-api-e2e-api-b-735232-5f498d95db-m8b6r\" (UID: \"5115bf21-3b75-4341-8204-da72fa6c1c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-735232-5f498d95db-m8b6r"
Jul 24 15:47:18.573790 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:18.573766    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16924896-d0f2-4958-8ff5-b372607d13af-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-735232-f8d9c84b8-wtm5f\" (UID: \"16924896-d0f2-4958-8ff5-b372607d13af\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-735232-f8d9c84b8-wtm5f"
Jul 24 15:47:18.574011 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:47:18.573673    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-735232-serving-cert: secret "maas-api-e2e-api-b-735232-serving-cert" not found
Jul 24 15:47:18.574193 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:47:18.574179    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5115bf21-3b75-4341-8204-da72fa6c1c7f-maas-api-tls podName:5115bf21-3b75-4341-8204-da72fa6c1c7f nodeName:}" failed. No retries permitted until 2026-07-24 15:47:19.574158617 +0000 UTC m=+2332.386510776 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5115bf21-3b75-4341-8204-da72fa6c1c7f-maas-api-tls") pod "maas-api-e2e-api-b-735232-5f498d95db-m8b6r" (UID: "5115bf21-3b75-4341-8204-da72fa6c1c7f") : secret "maas-api-e2e-api-b-735232-serving-cert" not found
Jul 24 15:47:18.585074 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:18.585033    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-735232-df6995698-6qxxn"]
Jul 24 15:47:18.675051 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:18.675016    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df4a7af-435c-4aa3-88bb-0f971f45f8b0-plugins-config-volume\") pod \"payload-processing-e2e-api-b-735232-df6995698-6qxxn\" (UID: \"6df4a7af-435c-4aa3-88bb-0f971f45f8b0\") " pod="openshift-ingress/payload-processing-e2e-api-b-735232-df6995698-6qxxn"
Jul 24 15:47:18.675051 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:18.675054    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5flv\" (UniqueName: \"kubernetes.io/projected/16924896-d0f2-4958-8ff5-b372607d13af-kube-api-access-c5flv\") pod \"payload-pre-processing-e2e-api-b-735232-f8d9c84b8-wtm5f\" (UID: \"16924896-d0f2-4958-8ff5-b372607d13af\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-735232-f8d9c84b8-wtm5f"
Jul 24 15:47:18.675256 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:18.675183    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16924896-d0f2-4958-8ff5-b372607d13af-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-735232-f8d9c84b8-wtm5f\" (UID: \"16924896-d0f2-4958-8ff5-b372607d13af\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-735232-f8d9c84b8-wtm5f"
Jul 24 15:47:18.675256 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:18.675222    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qkj2\" (UniqueName: \"kubernetes.io/projected/6df4a7af-435c-4aa3-88bb-0f971f45f8b0-kube-api-access-2qkj2\") pod \"payload-processing-e2e-api-b-735232-df6995698-6qxxn\" (UID: \"6df4a7af-435c-4aa3-88bb-0f971f45f8b0\") " pod="openshift-ingress/payload-processing-e2e-api-b-735232-df6995698-6qxxn"
Jul 24 15:47:18.675878 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:18.675859    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16924896-d0f2-4958-8ff5-b372607d13af-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-735232-f8d9c84b8-wtm5f\" (UID: \"16924896-d0f2-4958-8ff5-b372607d13af\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-735232-f8d9c84b8-wtm5f"
Jul 24 15:47:18.684383 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:18.684361    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5flv\" (UniqueName: \"kubernetes.io/projected/16924896-d0f2-4958-8ff5-b372607d13af-kube-api-access-c5flv\") pod \"payload-pre-processing-e2e-api-b-735232-f8d9c84b8-wtm5f\" (UID: \"16924896-d0f2-4958-8ff5-b372607d13af\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-735232-f8d9c84b8-wtm5f"
Jul 24 15:47:18.776730 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:18.776609    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df4a7af-435c-4aa3-88bb-0f971f45f8b0-plugins-config-volume\") pod \"payload-processing-e2e-api-b-735232-df6995698-6qxxn\" (UID: \"6df4a7af-435c-4aa3-88bb-0f971f45f8b0\") " pod="openshift-ingress/payload-processing-e2e-api-b-735232-df6995698-6qxxn"
Jul 24 15:47:18.776928 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:18.776893    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qkj2\" (UniqueName: \"kubernetes.io/projected/6df4a7af-435c-4aa3-88bb-0f971f45f8b0-kube-api-access-2qkj2\") pod \"payload-processing-e2e-api-b-735232-df6995698-6qxxn\" (UID: \"6df4a7af-435c-4aa3-88bb-0f971f45f8b0\") " pod="openshift-ingress/payload-processing-e2e-api-b-735232-df6995698-6qxxn"
Jul 24 15:47:18.777383 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:18.777365    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df4a7af-435c-4aa3-88bb-0f971f45f8b0-plugins-config-volume\") pod \"payload-processing-e2e-api-b-735232-df6995698-6qxxn\" (UID: \"6df4a7af-435c-4aa3-88bb-0f971f45f8b0\") " pod="openshift-ingress/payload-processing-e2e-api-b-735232-df6995698-6qxxn"
Jul 24 15:47:18.784751 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:18.784727    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qkj2\" (UniqueName: \"kubernetes.io/projected/6df4a7af-435c-4aa3-88bb-0f971f45f8b0-kube-api-access-2qkj2\") pod \"payload-processing-e2e-api-b-735232-df6995698-6qxxn\" (UID: \"6df4a7af-435c-4aa3-88bb-0f971f45f8b0\") " pod="openshift-ingress/payload-processing-e2e-api-b-735232-df6995698-6qxxn"
Jul 24 15:47:18.865025 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:18.864931    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-735232-f8d9c84b8-wtm5f"
Jul 24 15:47:18.881476 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:18.881440    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-735232-df6995698-6qxxn"
Jul 24 15:47:19.012999 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:19.012798    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-735232-f8d9c84b8-wtm5f"]
Jul 24 15:47:19.014978 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:47:19.014945    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16924896_d0f2_4958_8ff5_b372607d13af.slice/crio-5e9b292f01db4606ac9e288e5baeb1d250524d3185e27fb1bdca701a93686ebf WatchSource:0}: Error finding container 5e9b292f01db4606ac9e288e5baeb1d250524d3185e27fb1bdca701a93686ebf: Status 404 returned error can't find the container with id 5e9b292f01db4606ac9e288e5baeb1d250524d3185e27fb1bdca701a93686ebf
Jul 24 15:47:19.248460 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:19.248417    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-735232-df6995698-6qxxn"]
Jul 24 15:47:19.250719 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:47:19.250684    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df4a7af_435c_4aa3_88bb_0f971f45f8b0.slice/crio-99bcf02b1339cb5c5ea69e918cbe211c910a3db2c18323e92fd4d098b9b1e468 WatchSource:0}: Error finding container 99bcf02b1339cb5c5ea69e918cbe211c910a3db2c18323e92fd4d098b9b1e468: Status 404 returned error can't find the container with id 99bcf02b1339cb5c5ea69e918cbe211c910a3db2c18323e92fd4d098b9b1e468
Jul 24 15:47:19.585930 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:19.585881    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5115bf21-3b75-4341-8204-da72fa6c1c7f-maas-api-tls\") pod \"maas-api-e2e-api-b-735232-5f498d95db-m8b6r\" (UID: \"5115bf21-3b75-4341-8204-da72fa6c1c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-735232-5f498d95db-m8b6r"
Jul 24 15:47:19.588336 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:19.588311    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5115bf21-3b75-4341-8204-da72fa6c1c7f-maas-api-tls\") pod \"maas-api-e2e-api-b-735232-5f498d95db-m8b6r\" (UID: \"5115bf21-3b75-4341-8204-da72fa6c1c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-735232-5f498d95db-m8b6r"
Jul 24 15:47:19.710923 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:19.710886    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-735232-5f498d95db-m8b6r"
Jul 24 15:47:19.820226 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:19.820187    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-735232-f8d9c84b8-wtm5f" event={"ID":"16924896-d0f2-4958-8ff5-b372607d13af","Type":"ContainerStarted","Data":"faa3b8b4dc5cfbb3f363b1ba172f8ccf1005ee12a6abb1c9fe56413c09a8d972"}
Jul 24 15:47:19.820405 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:19.820360    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-735232-f8d9c84b8-wtm5f"
Jul 24 15:47:19.820480 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:19.820416    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-735232-f8d9c84b8-wtm5f" event={"ID":"16924896-d0f2-4958-8ff5-b372607d13af","Type":"ContainerStarted","Data":"5e9b292f01db4606ac9e288e5baeb1d250524d3185e27fb1bdca701a93686ebf"}
Jul 24 15:47:19.822964 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:19.822935    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-735232-df6995698-6qxxn" event={"ID":"6df4a7af-435c-4aa3-88bb-0f971f45f8b0","Type":"ContainerStarted","Data":"45bd0ea3cd783e31d0857db4bb1f4ce0d328b5cc7213f8a2bddd3feafd225ca2"}
Jul 24 15:47:19.823071 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:19.822990    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-735232-df6995698-6qxxn" event={"ID":"6df4a7af-435c-4aa3-88bb-0f971f45f8b0","Type":"ContainerStarted","Data":"99bcf02b1339cb5c5ea69e918cbe211c910a3db2c18323e92fd4d098b9b1e468"}
Jul 24 15:47:19.823071 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:19.823020    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-735232-df6995698-6qxxn"
Jul 24 15:47:19.841091 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:19.840991    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-735232-f8d9c84b8-wtm5f" podStartSLOduration=1.8409706529999998 podStartE2EDuration="1.840970653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:47:19.837482863 +0000 UTC m=+2332.649835043" watchObservedRunningTime="2026-07-24 15:47:19.840970653 +0000 UTC m=+2332.653322838"
Jul 24 15:47:19.851553 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:19.851524    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-735232-5f498d95db-m8b6r"]
Jul 24 15:47:19.853160 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:47:19.853128    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5115bf21_3b75_4341_8204_da72fa6c1c7f.slice/crio-acead135a8050ca9cd6db023eae89de4c581a185ee32c15088dcff0357098cb1 WatchSource:0}: Error finding container acead135a8050ca9cd6db023eae89de4c581a185ee32c15088dcff0357098cb1: Status 404 returned error can't find the container with id acead135a8050ca9cd6db023eae89de4c581a185ee32c15088dcff0357098cb1
Jul 24 15:47:19.854880 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:19.854842    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-735232-df6995698-6qxxn" podStartSLOduration=1.854810827 podStartE2EDuration="1.854810827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:47:19.853320331 +0000 UTC m=+2332.665672524" watchObservedRunningTime="2026-07-24 15:47:19.854810827 +0000 UTC m=+2332.667163005"
Jul 24 15:47:20.829511 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:20.829475    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-735232-5f498d95db-m8b6r" event={"ID":"5115bf21-3b75-4341-8204-da72fa6c1c7f","Type":"ContainerStarted","Data":"46804e21d875a628441c5df47b50fc176ef6fbef069f96dfa8fc8c0d024f5c41"}
Jul 24 15:47:20.829968 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:20.829530    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-735232-5f498d95db-m8b6r" event={"ID":"5115bf21-3b75-4341-8204-da72fa6c1c7f","Type":"ContainerStarted","Data":"acead135a8050ca9cd6db023eae89de4c581a185ee32c15088dcff0357098cb1"}
Jul 24 15:47:20.829968 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:20.829712    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-735232-5f498d95db-m8b6r"
Jul 24 15:47:20.846796 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:20.846751    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-735232-5f498d95db-m8b6r" podStartSLOduration=3.485883232 podStartE2EDuration="3.846736434s" totalImagesPullingTime="360.853202ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:47:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:47:20.845598537 +0000 UTC m=+2333.657950716" watchObservedRunningTime="2026-07-24 15:47:20.846736434 +0000 UTC m=+2333.659088613"
Jul 24 15:47:22.783167 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:22.783136    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-aefb55-5746d97b96-5zszh"
Jul 24 15:47:22.783623 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:22.783218    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aefb55-55c5776b88-rzbd9"
Jul 24 15:47:26.841431 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:26.841404    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-735232-5f498d95db-m8b6r"
Jul 24 15:47:30.831428 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:30.831397    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-735232-df6995698-6qxxn"
Jul 24 15:47:30.831851 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:30.831471    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-735232-f8d9c84b8-wtm5f"
Jul 24 15:47:31.318784 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:31.318750    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:47:45.766711 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:45.766677    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-aefb55-676f6b859c-6kfss"]
Jul 24 15:47:45.767319 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:45.766941    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aefb55-676f6b859c-6kfss" podUID="1a1eddba-580d-473a-aff2-b9c52bd1d9b3" containerName="maas-api" containerID="cri-o://986311fdb4a3100ba05ccd345af1745cdc02312e36387f8a7c51c1e5f3e3daf1" gracePeriod=30
Jul 24 15:47:45.816640 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:45.816608    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-aefb55-5746d97b96-5zszh"]
Jul 24 15:47:45.816888 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:45.816863    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-aefb55-5746d97b96-5zszh" podUID="e8c145a0-fa06-4388-a205-7232b366a7e0" containerName="payload-processing" containerID="cri-o://297b03ad4bd1f0f7515cdb9effb99e31fd758d0c726c545caf5693f6432ebd83" gracePeriod=30
Jul 24 15:47:45.827223 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:45.827196    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-aefb55-55c5776b88-rzbd9"]
Jul 24 15:47:45.827472 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:45.827432    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aefb55-55c5776b88-rzbd9" podUID="504dc68a-e1f8-4ea5-a61e-002dfc5aa0b1" containerName="payload-pre-processing" containerID="cri-o://63dab29607cba6238c491df9cd89f7b2be3f9592b85abcb36e7ae07cbb822a5e" gracePeriod=30
Jul 24 15:47:45.968927 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:45.968886    2579 generic.go:356] "Generic (PLEG): container finished" podID="1a1eddba-580d-473a-aff2-b9c52bd1d9b3" containerID="986311fdb4a3100ba05ccd345af1745cdc02312e36387f8a7c51c1e5f3e3daf1" exitCode=0
Jul 24 15:47:45.969130 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:45.968990    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aefb55-676f6b859c-6kfss" event={"ID":"1a1eddba-580d-473a-aff2-b9c52bd1d9b3","Type":"ContainerDied","Data":"986311fdb4a3100ba05ccd345af1745cdc02312e36387f8a7c51c1e5f3e3daf1"}
Jul 24 15:47:45.971154 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:45.970999    2579 generic.go:356] "Generic (PLEG): container finished" podID="e8c145a0-fa06-4388-a205-7232b366a7e0" containerID="297b03ad4bd1f0f7515cdb9effb99e31fd758d0c726c545caf5693f6432ebd83" exitCode=0
Jul 24 15:47:45.971154 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:45.971083    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-aefb55-5746d97b96-5zszh" event={"ID":"e8c145a0-fa06-4388-a205-7232b366a7e0","Type":"ContainerDied","Data":"297b03ad4bd1f0f7515cdb9effb99e31fd758d0c726c545caf5693f6432ebd83"}
Jul 24 15:47:45.973627 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:45.973565    2579 generic.go:356] "Generic (PLEG): container finished" podID="504dc68a-e1f8-4ea5-a61e-002dfc5aa0b1" containerID="63dab29607cba6238c491df9cd89f7b2be3f9592b85abcb36e7ae07cbb822a5e" exitCode=0
Jul 24 15:47:45.973747 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:45.973710    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aefb55-55c5776b88-rzbd9" event={"ID":"504dc68a-e1f8-4ea5-a61e-002dfc5aa0b1","Type":"ContainerDied","Data":"63dab29607cba6238c491df9cd89f7b2be3f9592b85abcb36e7ae07cbb822a5e"}
Jul 24 15:47:46.073020 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:46.072994    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aefb55-676f6b859c-6kfss"
Jul 24 15:47:46.084288 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:46.084261    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-aefb55-5746d97b96-5zszh"
Jul 24 15:47:46.116713 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:46.116689    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aefb55-55c5776b88-rzbd9"
Jul 24 15:47:46.244277 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:46.244238    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2w4j\" (UniqueName: \"kubernetes.io/projected/1a1eddba-580d-473a-aff2-b9c52bd1d9b3-kube-api-access-c2w4j\") pod \"1a1eddba-580d-473a-aff2-b9c52bd1d9b3\" (UID: \"1a1eddba-580d-473a-aff2-b9c52bd1d9b3\") "
Jul 24 15:47:46.244277 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:46.244279    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a1eddba-580d-473a-aff2-b9c52bd1d9b3-openshift-service-ca\") pod \"1a1eddba-580d-473a-aff2-b9c52bd1d9b3\" (UID: \"1a1eddba-580d-473a-aff2-b9c52bd1d9b3\") "
Jul 24 15:47:46.244553 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:46.244351    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a1eddba-580d-473a-aff2-b9c52bd1d9b3-maas-api-tls\") pod \"1a1eddba-580d-473a-aff2-b9c52bd1d9b3\" (UID: \"1a1eddba-580d-473a-aff2-b9c52bd1d9b3\") "
Jul 24 15:47:46.244553 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:46.244370    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9cdp\" (UniqueName: \"kubernetes.io/projected/e8c145a0-fa06-4388-a205-7232b366a7e0-kube-api-access-s9cdp\") pod \"e8c145a0-fa06-4388-a205-7232b366a7e0\" (UID: \"e8c145a0-fa06-4388-a205-7232b366a7e0\") "
Jul 24 15:47:46.244553 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:46.244402    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/504dc68a-e1f8-4ea5-a61e-002dfc5aa0b1-plugins-config-volume\") pod \"504dc68a-e1f8-4ea5-a61e-002dfc5aa0b1\" (UID: \"504dc68a-e1f8-4ea5-a61e-002dfc5aa0b1\") "
Jul 24 15:47:46.244553 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:46.244439    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r57tq\" (UniqueName: \"kubernetes.io/projected/504dc68a-e1f8-4ea5-a61e-002dfc5aa0b1-kube-api-access-r57tq\") pod \"504dc68a-e1f8-4ea5-a61e-002dfc5aa0b1\" (UID: \"504dc68a-e1f8-4ea5-a61e-002dfc5aa0b1\") "
Jul 24 15:47:46.244795 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:46.244735    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8c145a0-fa06-4388-a205-7232b366a7e0-plugins-config-volume\") pod \"e8c145a0-fa06-4388-a205-7232b366a7e0\" (UID: \"e8c145a0-fa06-4388-a205-7232b366a7e0\") "
Jul 24 15:47:46.244937 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:46.244850    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a1eddba-580d-473a-aff2-b9c52bd1d9b3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1a1eddba-580d-473a-aff2-b9c52bd1d9b3" (UID: "1a1eddba-580d-473a-aff2-b9c52bd1d9b3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:47:46.245010 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:46.244954    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/504dc68a-e1f8-4ea5-a61e-002dfc5aa0b1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "504dc68a-e1f8-4ea5-a61e-002dfc5aa0b1" (UID: "504dc68a-e1f8-4ea5-a61e-002dfc5aa0b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:47:46.245216 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:46.245192    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a1eddba-580d-473a-aff2-b9c52bd1d9b3-openshift-service-ca\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:47:46.245293 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:46.245221    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/504dc68a-e1f8-4ea5-a61e-002dfc5aa0b1-plugins-config-volume\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:47:46.245353 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:46.245310    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8c145a0-fa06-4388-a205-7232b366a7e0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e8c145a0-fa06-4388-a205-7232b366a7e0" (UID: "e8c145a0-fa06-4388-a205-7232b366a7e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:47:46.246661 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:46.246637    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a1eddba-580d-473a-aff2-b9c52bd1d9b3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1a1eddba-580d-473a-aff2-b9c52bd1d9b3" (UID: "1a1eddba-580d-473a-aff2-b9c52bd1d9b3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 15:47:46.246661 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:46.246656    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/504dc68a-e1f8-4ea5-a61e-002dfc5aa0b1-kube-api-access-r57tq" (OuterVolumeSpecName: "kube-api-access-r57tq") pod "504dc68a-e1f8-4ea5-a61e-002dfc5aa0b1" (UID: "504dc68a-e1f8-4ea5-a61e-002dfc5aa0b1"). InnerVolumeSpecName "kube-api-access-r57tq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:47:46.246791 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:46.246640    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8c145a0-fa06-4388-a205-7232b366a7e0-kube-api-access-s9cdp" (OuterVolumeSpecName: "kube-api-access-s9cdp") pod "e8c145a0-fa06-4388-a205-7232b366a7e0" (UID: "e8c145a0-fa06-4388-a205-7232b366a7e0"). InnerVolumeSpecName "kube-api-access-s9cdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:47:46.246791 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:46.246644    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a1eddba-580d-473a-aff2-b9c52bd1d9b3-kube-api-access-c2w4j" (OuterVolumeSpecName: "kube-api-access-c2w4j") pod "1a1eddba-580d-473a-aff2-b9c52bd1d9b3" (UID: "1a1eddba-580d-473a-aff2-b9c52bd1d9b3"). InnerVolumeSpecName "kube-api-access-c2w4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:47:46.345791 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:46.345706    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a1eddba-580d-473a-aff2-b9c52bd1d9b3-maas-api-tls\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:47:46.345791 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:46.345736    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9cdp\" (UniqueName: \"kubernetes.io/projected/e8c145a0-fa06-4388-a205-7232b366a7e0-kube-api-access-s9cdp\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:47:46.345791 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:46.345745    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r57tq\" (UniqueName: \"kubernetes.io/projected/504dc68a-e1f8-4ea5-a61e-002dfc5aa0b1-kube-api-access-r57tq\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:47:46.345791 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:46.345753    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8c145a0-fa06-4388-a205-7232b366a7e0-plugins-config-volume\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:47:46.345791 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:46.345762    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2w4j\" (UniqueName: \"kubernetes.io/projected/1a1eddba-580d-473a-aff2-b9c52bd1d9b3-kube-api-access-c2w4j\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:47:46.980059 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:46.980026    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aefb55-676f6b859c-6kfss"
Jul 24 15:47:46.980059 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:46.980037    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aefb55-676f6b859c-6kfss" event={"ID":"1a1eddba-580d-473a-aff2-b9c52bd1d9b3","Type":"ContainerDied","Data":"a81be51c3d9fc9cc138a16577755166608037844b517112928cd27b54b9843d9"}
Jul 24 15:47:46.980588 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:46.980089    2579 scope.go:117] "RemoveContainer" containerID="986311fdb4a3100ba05ccd345af1745cdc02312e36387f8a7c51c1e5f3e3daf1"
Jul 24 15:47:46.981560 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:46.981419    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-aefb55-5746d97b96-5zszh" event={"ID":"e8c145a0-fa06-4388-a205-7232b366a7e0","Type":"ContainerDied","Data":"efee87eb80a872c51ec7616237b84d8d913630e749ac9c7f1b8fc6a9b72721f6"}
Jul 24 15:47:46.981560 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:46.981464    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-aefb55-5746d97b96-5zszh"
Jul 24 15:47:46.983016 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:46.982992    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aefb55-55c5776b88-rzbd9"
Jul 24 15:47:46.983016 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:46.982993    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aefb55-55c5776b88-rzbd9" event={"ID":"504dc68a-e1f8-4ea5-a61e-002dfc5aa0b1","Type":"ContainerDied","Data":"ed73b2bac9bd5f60870b5e78d977bfed16c08bff1e432439d5d18c43464be562"}
Jul 24 15:47:46.989936 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:46.989920    2579 scope.go:117] "RemoveContainer" containerID="297b03ad4bd1f0f7515cdb9effb99e31fd758d0c726c545caf5693f6432ebd83"
Jul 24 15:47:46.998570 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:46.998552    2579 scope.go:117] "RemoveContainer" containerID="63dab29607cba6238c491df9cd89f7b2be3f9592b85abcb36e7ae07cbb822a5e"
Jul 24 15:47:47.002852 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:47.002825    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-aefb55-676f6b859c-6kfss"]
Jul 24 15:47:47.009043 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:47.009023    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-aefb55-676f6b859c-6kfss"]
Jul 24 15:47:47.020546 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:47.020520    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-aefb55-55c5776b88-rzbd9"]
Jul 24 15:47:47.022136 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:47.022115    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-aefb55-55c5776b88-rzbd9"]
Jul 24 15:47:47.033952 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:47.033881    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-aefb55-5746d97b96-5zszh"]
Jul 24 15:47:47.039911 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:47.039885    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-aefb55-5746d97b96-5zszh"]
Jul 24 15:47:47.716991 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:47.716953    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a1eddba-580d-473a-aff2-b9c52bd1d9b3" path="/var/lib/kubelet/pods/1a1eddba-580d-473a-aff2-b9c52bd1d9b3/volumes"
Jul 24 15:47:47.717349 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:47.717336    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="504dc68a-e1f8-4ea5-a61e-002dfc5aa0b1" path="/var/lib/kubelet/pods/504dc68a-e1f8-4ea5-a61e-002dfc5aa0b1/volumes"
Jul 24 15:47:47.717661 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:47.717649    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8c145a0-fa06-4388-a205-7232b366a7e0" path="/var/lib/kubelet/pods/e8c145a0-fa06-4388-a205-7232b366a7e0/volumes"
Jul 24 15:47:51.029869 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:47:51.029833    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:48:12.460570 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:12.460533    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-735232-5f498d95db-m8b6r"]
Jul 24 15:48:12.461135 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:12.460798    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-735232-5f498d95db-m8b6r" podUID="5115bf21-3b75-4341-8204-da72fa6c1c7f" containerName="maas-api" containerID="cri-o://46804e21d875a628441c5df47b50fc176ef6fbef069f96dfa8fc8c0d024f5c41" gracePeriod=30
Jul 24 15:48:12.514915 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:12.514882    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-735232-df6995698-6qxxn"]
Jul 24 15:48:12.515168 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:12.515127    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-735232-df6995698-6qxxn" podUID="6df4a7af-435c-4aa3-88bb-0f971f45f8b0" containerName="payload-processing" containerID="cri-o://45bd0ea3cd783e31d0857db4bb1f4ce0d328b5cc7213f8a2bddd3feafd225ca2" gracePeriod=30
Jul 24 15:48:12.524622 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:12.524587    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-735232-f8d9c84b8-wtm5f"]
Jul 24 15:48:12.524901 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:12.524867    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-735232-f8d9c84b8-wtm5f" podUID="16924896-d0f2-4958-8ff5-b372607d13af" containerName="payload-pre-processing" containerID="cri-o://faa3b8b4dc5cfbb3f363b1ba172f8ccf1005ee12a6abb1c9fe56413c09a8d972" gracePeriod=30
Jul 24 15:48:12.790293 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:12.790266    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-735232-5f498d95db-m8b6r"
Jul 24 15:48:12.795133 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:12.795112    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-735232-df6995698-6qxxn"
Jul 24 15:48:12.804287 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:12.804268    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-735232-f8d9c84b8-wtm5f"
Jul 24 15:48:12.906385 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:12.906354    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df4a7af-435c-4aa3-88bb-0f971f45f8b0-plugins-config-volume\") pod \"6df4a7af-435c-4aa3-88bb-0f971f45f8b0\" (UID: \"6df4a7af-435c-4aa3-88bb-0f971f45f8b0\") "
Jul 24 15:48:12.906549 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:12.906402    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qkj2\" (UniqueName: \"kubernetes.io/projected/6df4a7af-435c-4aa3-88bb-0f971f45f8b0-kube-api-access-2qkj2\") pod \"6df4a7af-435c-4aa3-88bb-0f971f45f8b0\" (UID: \"6df4a7af-435c-4aa3-88bb-0f971f45f8b0\") "
Jul 24 15:48:12.906549 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:12.906496    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mg77\" (UniqueName: \"kubernetes.io/projected/5115bf21-3b75-4341-8204-da72fa6c1c7f-kube-api-access-2mg77\") pod \"5115bf21-3b75-4341-8204-da72fa6c1c7f\" (UID: \"5115bf21-3b75-4341-8204-da72fa6c1c7f\") "
Jul 24 15:48:12.906636 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:12.906561    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5115bf21-3b75-4341-8204-da72fa6c1c7f-maas-api-tls\") pod \"5115bf21-3b75-4341-8204-da72fa6c1c7f\" (UID: \"5115bf21-3b75-4341-8204-da72fa6c1c7f\") "
Jul 24 15:48:12.906636 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:12.906592    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5115bf21-3b75-4341-8204-da72fa6c1c7f-openshift-service-ca\") pod \"5115bf21-3b75-4341-8204-da72fa6c1c7f\" (UID: \"5115bf21-3b75-4341-8204-da72fa6c1c7f\") "
Jul 24 15:48:12.906748 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:12.906634    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5flv\" (UniqueName: \"kubernetes.io/projected/16924896-d0f2-4958-8ff5-b372607d13af-kube-api-access-c5flv\") pod \"16924896-d0f2-4958-8ff5-b372607d13af\" (UID: \"16924896-d0f2-4958-8ff5-b372607d13af\") "
Jul 24 15:48:12.906748 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:12.906665    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16924896-d0f2-4958-8ff5-b372607d13af-plugins-config-volume\") pod \"16924896-d0f2-4958-8ff5-b372607d13af\" (UID: \"16924896-d0f2-4958-8ff5-b372607d13af\") "
Jul 24 15:48:12.906877 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:12.906767    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6df4a7af-435c-4aa3-88bb-0f971f45f8b0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6df4a7af-435c-4aa3-88bb-0f971f45f8b0" (UID: "6df4a7af-435c-4aa3-88bb-0f971f45f8b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:48:12.906979 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:12.906962    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df4a7af-435c-4aa3-88bb-0f971f45f8b0-plugins-config-volume\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:48:12.907284 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:12.907176    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16924896-d0f2-4958-8ff5-b372607d13af-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "16924896-d0f2-4958-8ff5-b372607d13af" (UID: "16924896-d0f2-4958-8ff5-b372607d13af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:48:12.907380 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:12.907345    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5115bf21-3b75-4341-8204-da72fa6c1c7f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5115bf21-3b75-4341-8204-da72fa6c1c7f" (UID: "5115bf21-3b75-4341-8204-da72fa6c1c7f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:48:12.908688 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:12.908661    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5115bf21-3b75-4341-8204-da72fa6c1c7f-kube-api-access-2mg77" (OuterVolumeSpecName: "kube-api-access-2mg77") pod "5115bf21-3b75-4341-8204-da72fa6c1c7f" (UID: "5115bf21-3b75-4341-8204-da72fa6c1c7f"). InnerVolumeSpecName "kube-api-access-2mg77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:48:12.908780 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:12.908688    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df4a7af-435c-4aa3-88bb-0f971f45f8b0-kube-api-access-2qkj2" (OuterVolumeSpecName: "kube-api-access-2qkj2") pod "6df4a7af-435c-4aa3-88bb-0f971f45f8b0" (UID: "6df4a7af-435c-4aa3-88bb-0f971f45f8b0"). InnerVolumeSpecName "kube-api-access-2qkj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:48:12.908780 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:12.908690    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5115bf21-3b75-4341-8204-da72fa6c1c7f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5115bf21-3b75-4341-8204-da72fa6c1c7f" (UID: "5115bf21-3b75-4341-8204-da72fa6c1c7f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 15:48:12.908886 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:12.908785    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16924896-d0f2-4958-8ff5-b372607d13af-kube-api-access-c5flv" (OuterVolumeSpecName: "kube-api-access-c5flv") pod "16924896-d0f2-4958-8ff5-b372607d13af" (UID: "16924896-d0f2-4958-8ff5-b372607d13af"). InnerVolumeSpecName "kube-api-access-c5flv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:48:13.008361 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:13.008283    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5flv\" (UniqueName: \"kubernetes.io/projected/16924896-d0f2-4958-8ff5-b372607d13af-kube-api-access-c5flv\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:48:13.008361 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:13.008312    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16924896-d0f2-4958-8ff5-b372607d13af-plugins-config-volume\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:48:13.008361 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:13.008322    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qkj2\" (UniqueName: \"kubernetes.io/projected/6df4a7af-435c-4aa3-88bb-0f971f45f8b0-kube-api-access-2qkj2\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:48:13.008361 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:13.008331    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mg77\" (UniqueName: \"kubernetes.io/projected/5115bf21-3b75-4341-8204-da72fa6c1c7f-kube-api-access-2mg77\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:48:13.008361 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:13.008341    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5115bf21-3b75-4341-8204-da72fa6c1c7f-maas-api-tls\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:48:13.008361 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:13.008349    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5115bf21-3b75-4341-8204-da72fa6c1c7f-openshift-service-ca\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:48:13.122778 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:13.122741    2579 generic.go:356] "Generic (PLEG): container finished" podID="5115bf21-3b75-4341-8204-da72fa6c1c7f" containerID="46804e21d875a628441c5df47b50fc176ef6fbef069f96dfa8fc8c0d024f5c41" exitCode=0
Jul 24 15:48:13.122955 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:13.122845    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-735232-5f498d95db-m8b6r"
Jul 24 15:48:13.122955 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:13.122844    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-735232-5f498d95db-m8b6r" event={"ID":"5115bf21-3b75-4341-8204-da72fa6c1c7f","Type":"ContainerDied","Data":"46804e21d875a628441c5df47b50fc176ef6fbef069f96dfa8fc8c0d024f5c41"}
Jul 24 15:48:13.123099 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:13.122973    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-735232-5f498d95db-m8b6r" event={"ID":"5115bf21-3b75-4341-8204-da72fa6c1c7f","Type":"ContainerDied","Data":"acead135a8050ca9cd6db023eae89de4c581a185ee32c15088dcff0357098cb1"}
Jul 24 15:48:13.123099 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:13.122999    2579 scope.go:117] "RemoveContainer" containerID="46804e21d875a628441c5df47b50fc176ef6fbef069f96dfa8fc8c0d024f5c41"
Jul 24 15:48:13.124384 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:13.124364    2579 generic.go:356] "Generic (PLEG): container finished" podID="6df4a7af-435c-4aa3-88bb-0f971f45f8b0" containerID="45bd0ea3cd783e31d0857db4bb1f4ce0d328b5cc7213f8a2bddd3feafd225ca2" exitCode=0
Jul 24 15:48:13.124481 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:13.124418    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-735232-df6995698-6qxxn"
Jul 24 15:48:13.124802 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:13.124768    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-735232-df6995698-6qxxn" event={"ID":"6df4a7af-435c-4aa3-88bb-0f971f45f8b0","Type":"ContainerDied","Data":"45bd0ea3cd783e31d0857db4bb1f4ce0d328b5cc7213f8a2bddd3feafd225ca2"}
Jul 24 15:48:13.124946 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:13.124860    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-735232-df6995698-6qxxn" event={"ID":"6df4a7af-435c-4aa3-88bb-0f971f45f8b0","Type":"ContainerDied","Data":"99bcf02b1339cb5c5ea69e918cbe211c910a3db2c18323e92fd4d098b9b1e468"}
Jul 24 15:48:13.133643 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:13.133615    2579 generic.go:356] "Generic (PLEG): container finished" podID="16924896-d0f2-4958-8ff5-b372607d13af" containerID="faa3b8b4dc5cfbb3f363b1ba172f8ccf1005ee12a6abb1c9fe56413c09a8d972" exitCode=0
Jul 24 15:48:13.133754 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:13.133705    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-735232-f8d9c84b8-wtm5f" event={"ID":"16924896-d0f2-4958-8ff5-b372607d13af","Type":"ContainerDied","Data":"faa3b8b4dc5cfbb3f363b1ba172f8ccf1005ee12a6abb1c9fe56413c09a8d972"}
Jul 24 15:48:13.133828 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:13.133772    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-735232-f8d9c84b8-wtm5f" event={"ID":"16924896-d0f2-4958-8ff5-b372607d13af","Type":"ContainerDied","Data":"5e9b292f01db4606ac9e288e5baeb1d250524d3185e27fb1bdca701a93686ebf"}
Jul 24 15:48:13.133894 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:13.133878    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-735232-f8d9c84b8-wtm5f"
Jul 24 15:48:13.138056 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:13.137999    2579 scope.go:117] "RemoveContainer" containerID="46804e21d875a628441c5df47b50fc176ef6fbef069f96dfa8fc8c0d024f5c41"
Jul 24 15:48:13.138515 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:48:13.138477    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46804e21d875a628441c5df47b50fc176ef6fbef069f96dfa8fc8c0d024f5c41\": container with ID starting with 46804e21d875a628441c5df47b50fc176ef6fbef069f96dfa8fc8c0d024f5c41 not found: ID does not exist" containerID="46804e21d875a628441c5df47b50fc176ef6fbef069f96dfa8fc8c0d024f5c41"
Jul 24 15:48:13.138612 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:13.138524    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46804e21d875a628441c5df47b50fc176ef6fbef069f96dfa8fc8c0d024f5c41"} err="failed to get container status \"46804e21d875a628441c5df47b50fc176ef6fbef069f96dfa8fc8c0d024f5c41\": rpc error: code = NotFound desc = could not find container \"46804e21d875a628441c5df47b50fc176ef6fbef069f96dfa8fc8c0d024f5c41\": container with ID starting with 46804e21d875a628441c5df47b50fc176ef6fbef069f96dfa8fc8c0d024f5c41 not found: ID does not exist"
Jul 24 15:48:13.138612 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:13.138546    2579 scope.go:117] "RemoveContainer" containerID="45bd0ea3cd783e31d0857db4bb1f4ce0d328b5cc7213f8a2bddd3feafd225ca2"
Jul 24 15:48:13.146206 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:13.146190    2579 scope.go:117] "RemoveContainer" containerID="45bd0ea3cd783e31d0857db4bb1f4ce0d328b5cc7213f8a2bddd3feafd225ca2"
Jul 24 15:48:13.146431 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:48:13.146416    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45bd0ea3cd783e31d0857db4bb1f4ce0d328b5cc7213f8a2bddd3feafd225ca2\": container with ID starting with 45bd0ea3cd783e31d0857db4bb1f4ce0d328b5cc7213f8a2bddd3feafd225ca2 not found: ID does not exist" containerID="45bd0ea3cd783e31d0857db4bb1f4ce0d328b5cc7213f8a2bddd3feafd225ca2"
Jul 24 15:48:13.146471 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:13.146439    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45bd0ea3cd783e31d0857db4bb1f4ce0d328b5cc7213f8a2bddd3feafd225ca2"} err="failed to get container status \"45bd0ea3cd783e31d0857db4bb1f4ce0d328b5cc7213f8a2bddd3feafd225ca2\": rpc error: code = NotFound desc = could not find container \"45bd0ea3cd783e31d0857db4bb1f4ce0d328b5cc7213f8a2bddd3feafd225ca2\": container with ID starting with 45bd0ea3cd783e31d0857db4bb1f4ce0d328b5cc7213f8a2bddd3feafd225ca2 not found: ID does not exist"
Jul 24 15:48:13.146471 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:13.146453    2579 scope.go:117] "RemoveContainer" containerID="faa3b8b4dc5cfbb3f363b1ba172f8ccf1005ee12a6abb1c9fe56413c09a8d972"
Jul 24 15:48:13.151334 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:13.151311    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-735232-df6995698-6qxxn"]
Jul 24 15:48:13.155153 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:13.155136    2579 scope.go:117] "RemoveContainer" containerID="faa3b8b4dc5cfbb3f363b1ba172f8ccf1005ee12a6abb1c9fe56413c09a8d972"
Jul 24 15:48:13.155431 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:48:13.155406    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faa3b8b4dc5cfbb3f363b1ba172f8ccf1005ee12a6abb1c9fe56413c09a8d972\": container with ID starting with faa3b8b4dc5cfbb3f363b1ba172f8ccf1005ee12a6abb1c9fe56413c09a8d972 not found: ID does not exist" containerID="faa3b8b4dc5cfbb3f363b1ba172f8ccf1005ee12a6abb1c9fe56413c09a8d972"
Jul 24 15:48:13.155516 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:13.155437    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faa3b8b4dc5cfbb3f363b1ba172f8ccf1005ee12a6abb1c9fe56413c09a8d972"} err="failed to get container status \"faa3b8b4dc5cfbb3f363b1ba172f8ccf1005ee12a6abb1c9fe56413c09a8d972\": rpc error: code = NotFound desc = could not find container \"faa3b8b4dc5cfbb3f363b1ba172f8ccf1005ee12a6abb1c9fe56413c09a8d972\": container with ID starting with faa3b8b4dc5cfbb3f363b1ba172f8ccf1005ee12a6abb1c9fe56413c09a8d972 not found: ID does not exist"
Jul 24 15:48:13.155685 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:13.155668    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-735232-df6995698-6qxxn"]
Jul 24 15:48:13.164592 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:13.164570    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-735232-5f498d95db-m8b6r"]
Jul 24 15:48:13.171845 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:13.171804    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-735232-5f498d95db-m8b6r"]
Jul 24 15:48:13.182317 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:13.182295    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-735232-f8d9c84b8-wtm5f"]
Jul 24 15:48:13.186205 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:13.186186    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-735232-f8d9c84b8-wtm5f"]
Jul 24 15:48:13.716687 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:13.716657    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16924896-d0f2-4958-8ff5-b372607d13af" path="/var/lib/kubelet/pods/16924896-d0f2-4958-8ff5-b372607d13af/volumes"
Jul 24 15:48:13.717086 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:13.717070    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5115bf21-3b75-4341-8204-da72fa6c1c7f" path="/var/lib/kubelet/pods/5115bf21-3b75-4341-8204-da72fa6c1c7f/volumes"
Jul 24 15:48:13.717384 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:13.717373    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6df4a7af-435c-4aa3-88bb-0f971f45f8b0" path="/var/lib/kubelet/pods/6df4a7af-435c-4aa3-88bb-0f971f45f8b0/volumes"
Jul 24 15:48:17.225600 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:17.225566    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:48:27.153471 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.153437    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-2619ff-67c944cf6-56txk"]
Jul 24 15:48:27.153886 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.153837    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8c145a0-fa06-4388-a205-7232b366a7e0" containerName="payload-processing"
Jul 24 15:48:27.153886 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.153850    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8c145a0-fa06-4388-a205-7232b366a7e0" containerName="payload-processing"
Jul 24 15:48:27.153886 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.153863    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="504dc68a-e1f8-4ea5-a61e-002dfc5aa0b1" containerName="payload-pre-processing"
Jul 24 15:48:27.153886 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.153869    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="504dc68a-e1f8-4ea5-a61e-002dfc5aa0b1" containerName="payload-pre-processing"
Jul 24 15:48:27.153886 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.153882    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5115bf21-3b75-4341-8204-da72fa6c1c7f" containerName="maas-api"
Jul 24 15:48:27.153886 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.153887    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5115bf21-3b75-4341-8204-da72fa6c1c7f" containerName="maas-api"
Jul 24 15:48:27.154105 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.153898    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6df4a7af-435c-4aa3-88bb-0f971f45f8b0" containerName="payload-processing"
Jul 24 15:48:27.154105 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.153904    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6df4a7af-435c-4aa3-88bb-0f971f45f8b0" containerName="payload-processing"
Jul 24 15:48:27.154105 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.153912    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16924896-d0f2-4958-8ff5-b372607d13af" containerName="payload-pre-processing"
Jul 24 15:48:27.154105 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.153917    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="16924896-d0f2-4958-8ff5-b372607d13af" containerName="payload-pre-processing"
Jul 24 15:48:27.154105 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.153930    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a1eddba-580d-473a-aff2-b9c52bd1d9b3" containerName="maas-api"
Jul 24 15:48:27.154105 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.153935    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a1eddba-580d-473a-aff2-b9c52bd1d9b3" containerName="maas-api"
Jul 24 15:48:27.154105 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.153992    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8c145a0-fa06-4388-a205-7232b366a7e0" containerName="payload-processing"
Jul 24 15:48:27.154105 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.153998    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="16924896-d0f2-4958-8ff5-b372607d13af" containerName="payload-pre-processing"
Jul 24 15:48:27.154105 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.154004    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="504dc68a-e1f8-4ea5-a61e-002dfc5aa0b1" containerName="payload-pre-processing"
Jul 24 15:48:27.154105 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.154011    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5115bf21-3b75-4341-8204-da72fa6c1c7f" containerName="maas-api"
Jul 24 15:48:27.154105 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.154018    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a1eddba-580d-473a-aff2-b9c52bd1d9b3" containerName="maas-api"
Jul 24 15:48:27.154105 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.154024    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6df4a7af-435c-4aa3-88bb-0f971f45f8b0" containerName="payload-processing"
Jul 24 15:48:27.159088 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.159068    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2619ff-67c944cf6-56txk"
Jul 24 15:48:27.161751 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.161725    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-2619ff-serving-cert\""
Jul 24 15:48:27.161874 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.161747    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tkn9s\""
Jul 24 15:48:27.161874 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.161770    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 15:48:27.161874 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.161747    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 15:48:27.167611 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.167548    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-2619ff-67c944cf6-56txk"]
Jul 24 15:48:27.243651 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.243616    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kz95\" (UniqueName: \"kubernetes.io/projected/af885fa0-3295-4818-83b1-d38e540f292e-kube-api-access-4kz95\") pod \"maas-api-e2e-shared-a-2619ff-67c944cf6-56txk\" (UID: \"af885fa0-3295-4818-83b1-d38e540f292e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2619ff-67c944cf6-56txk"
Jul 24 15:48:27.243905 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.243658    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af885fa0-3295-4818-83b1-d38e540f292e-maas-api-tls\") pod \"maas-api-e2e-shared-a-2619ff-67c944cf6-56txk\" (UID: \"af885fa0-3295-4818-83b1-d38e540f292e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2619ff-67c944cf6-56txk"
Jul 24 15:48:27.243905 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.243687    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af885fa0-3295-4818-83b1-d38e540f292e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-2619ff-67c944cf6-56txk\" (UID: \"af885fa0-3295-4818-83b1-d38e540f292e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2619ff-67c944cf6-56txk"
Jul 24 15:48:27.344637 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.344599    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kz95\" (UniqueName: \"kubernetes.io/projected/af885fa0-3295-4818-83b1-d38e540f292e-kube-api-access-4kz95\") pod \"maas-api-e2e-shared-a-2619ff-67c944cf6-56txk\" (UID: \"af885fa0-3295-4818-83b1-d38e540f292e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2619ff-67c944cf6-56txk"
Jul 24 15:48:27.344637 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.344641    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af885fa0-3295-4818-83b1-d38e540f292e-maas-api-tls\") pod \"maas-api-e2e-shared-a-2619ff-67c944cf6-56txk\" (UID: \"af885fa0-3295-4818-83b1-d38e540f292e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2619ff-67c944cf6-56txk"
Jul 24 15:48:27.344885 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.344664    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af885fa0-3295-4818-83b1-d38e540f292e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-2619ff-67c944cf6-56txk\" (UID: \"af885fa0-3295-4818-83b1-d38e540f292e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2619ff-67c944cf6-56txk"
Jul 24 15:48:27.344885 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:48:27.344756    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-2619ff-serving-cert: secret "maas-api-e2e-shared-a-2619ff-serving-cert" not found
Jul 24 15:48:27.344885 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:48:27.344830    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af885fa0-3295-4818-83b1-d38e540f292e-maas-api-tls podName:af885fa0-3295-4818-83b1-d38e540f292e nodeName:}" failed. No retries permitted until 2026-07-24 15:48:27.844802883 +0000 UTC m=+2400.657155041 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/af885fa0-3295-4818-83b1-d38e540f292e-maas-api-tls") pod "maas-api-e2e-shared-a-2619ff-67c944cf6-56txk" (UID: "af885fa0-3295-4818-83b1-d38e540f292e") : secret "maas-api-e2e-shared-a-2619ff-serving-cert" not found
Jul 24 15:48:27.345277 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.345260    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af885fa0-3295-4818-83b1-d38e540f292e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-2619ff-67c944cf6-56txk\" (UID: \"af885fa0-3295-4818-83b1-d38e540f292e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2619ff-67c944cf6-56txk"
Jul 24 15:48:27.353516 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.353496    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kz95\" (UniqueName: \"kubernetes.io/projected/af885fa0-3295-4818-83b1-d38e540f292e-kube-api-access-4kz95\") pod \"maas-api-e2e-shared-a-2619ff-67c944cf6-56txk\" (UID: \"af885fa0-3295-4818-83b1-d38e540f292e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2619ff-67c944cf6-56txk"
Jul 24 15:48:27.848988 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.848955    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af885fa0-3295-4818-83b1-d38e540f292e-maas-api-tls\") pod \"maas-api-e2e-shared-a-2619ff-67c944cf6-56txk\" (UID: \"af885fa0-3295-4818-83b1-d38e540f292e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2619ff-67c944cf6-56txk"
Jul 24 15:48:27.851335 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.851303    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af885fa0-3295-4818-83b1-d38e540f292e-maas-api-tls\") pod \"maas-api-e2e-shared-a-2619ff-67c944cf6-56txk\" (UID: \"af885fa0-3295-4818-83b1-d38e540f292e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2619ff-67c944cf6-56txk"
Jul 24 15:48:27.898860 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.898830    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-2619ff-677c9878b4-fqr7x"]
Jul 24 15:48:27.902433 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.902414    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2619ff-677c9878b4-fqr7x"
Jul 24 15:48:27.904925 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.904890    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-2619ff-dockercfg-k6wn4\""
Jul 24 15:48:27.905270 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.905252    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-2619ff\""
Jul 24 15:48:27.911699 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.911676    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-2619ff-677c9878b4-fqr7x"]
Jul 24 15:48:27.927686 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.927658    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-2619ff-57fdbfb486-5gkbp"]
Jul 24 15:48:27.931086 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.931070    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-2619ff-57fdbfb486-5gkbp"
Jul 24 15:48:27.940605 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.940542    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-2619ff-57fdbfb486-5gkbp"]
Jul 24 15:48:27.949861 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.949835    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2b13488-f677-4987-bafe-df99527776e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-2619ff-677c9878b4-fqr7x\" (UID: \"f2b13488-f677-4987-bafe-df99527776e7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2619ff-677c9878b4-fqr7x"
Jul 24 15:48:27.950072 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.949896    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71111d85-3b11-48fe-a587-54c105cb74d7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-2619ff-57fdbfb486-5gkbp\" (UID: \"71111d85-3b11-48fe-a587-54c105cb74d7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-2619ff-57fdbfb486-5gkbp"
Jul 24 15:48:27.950072 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.950004    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8plp\" (UniqueName: \"kubernetes.io/projected/f2b13488-f677-4987-bafe-df99527776e7-kube-api-access-f8plp\") pod \"payload-pre-processing-e2e-shared-a-2619ff-677c9878b4-fqr7x\" (UID: \"f2b13488-f677-4987-bafe-df99527776e7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2619ff-677c9878b4-fqr7x"
Jul 24 15:48:27.950983 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:27.950109    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xhbr\" (UniqueName: \"kubernetes.io/projected/71111d85-3b11-48fe-a587-54c105cb74d7-kube-api-access-9xhbr\") pod \"payload-processing-e2e-shared-a-2619ff-57fdbfb486-5gkbp\" (UID: \"71111d85-3b11-48fe-a587-54c105cb74d7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-2619ff-57fdbfb486-5gkbp"
Jul 24 15:48:28.050882 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:28.050845    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8plp\" (UniqueName: \"kubernetes.io/projected/f2b13488-f677-4987-bafe-df99527776e7-kube-api-access-f8plp\") pod \"payload-pre-processing-e2e-shared-a-2619ff-677c9878b4-fqr7x\" (UID: \"f2b13488-f677-4987-bafe-df99527776e7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2619ff-677c9878b4-fqr7x"
Jul 24 15:48:28.051017 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:28.050897    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xhbr\" (UniqueName: \"kubernetes.io/projected/71111d85-3b11-48fe-a587-54c105cb74d7-kube-api-access-9xhbr\") pod \"payload-processing-e2e-shared-a-2619ff-57fdbfb486-5gkbp\" (UID: \"71111d85-3b11-48fe-a587-54c105cb74d7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-2619ff-57fdbfb486-5gkbp"
Jul 24 15:48:28.051068 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:28.051052    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2b13488-f677-4987-bafe-df99527776e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-2619ff-677c9878b4-fqr7x\" (UID: \"f2b13488-f677-4987-bafe-df99527776e7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2619ff-677c9878b4-fqr7x"
Jul 24 15:48:28.051137 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:28.051123    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71111d85-3b11-48fe-a587-54c105cb74d7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-2619ff-57fdbfb486-5gkbp\" (UID: \"71111d85-3b11-48fe-a587-54c105cb74d7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-2619ff-57fdbfb486-5gkbp"
Jul 24 15:48:28.051725 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:28.051707    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2b13488-f677-4987-bafe-df99527776e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-2619ff-677c9878b4-fqr7x\" (UID: \"f2b13488-f677-4987-bafe-df99527776e7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2619ff-677c9878b4-fqr7x"
Jul 24 15:48:28.051772 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:28.051731    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71111d85-3b11-48fe-a587-54c105cb74d7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-2619ff-57fdbfb486-5gkbp\" (UID: \"71111d85-3b11-48fe-a587-54c105cb74d7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-2619ff-57fdbfb486-5gkbp"
Jul 24 15:48:28.059633 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:28.059614    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8plp\" (UniqueName: \"kubernetes.io/projected/f2b13488-f677-4987-bafe-df99527776e7-kube-api-access-f8plp\") pod \"payload-pre-processing-e2e-shared-a-2619ff-677c9878b4-fqr7x\" (UID: \"f2b13488-f677-4987-bafe-df99527776e7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2619ff-677c9878b4-fqr7x"
Jul 24 15:48:28.059727 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:28.059706    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xhbr\" (UniqueName: \"kubernetes.io/projected/71111d85-3b11-48fe-a587-54c105cb74d7-kube-api-access-9xhbr\") pod \"payload-processing-e2e-shared-a-2619ff-57fdbfb486-5gkbp\" (UID: \"71111d85-3b11-48fe-a587-54c105cb74d7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-2619ff-57fdbfb486-5gkbp"
Jul 24 15:48:28.073516 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:28.073496    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tkn9s\""
Jul 24 15:48:28.082271 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:28.082250    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2619ff-67c944cf6-56txk"
Jul 24 15:48:28.214460 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:28.214435    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2619ff-677c9878b4-fqr7x"
Jul 24 15:48:28.215002 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:28.214980    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-2619ff-67c944cf6-56txk"]
Jul 24 15:48:28.216564 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:48:28.216537    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf885fa0_3295_4818_83b1_d38e540f292e.slice/crio-f7517882b4a4d427b221c3f4029acd03ba07a72266892a9a9c97106ab95e683e WatchSource:0}: Error finding container f7517882b4a4d427b221c3f4029acd03ba07a72266892a9a9c97106ab95e683e: Status 404 returned error can't find the container with id f7517882b4a4d427b221c3f4029acd03ba07a72266892a9a9c97106ab95e683e
Jul 24 15:48:28.243138 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:28.243111    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-2619ff-57fdbfb486-5gkbp"
Jul 24 15:48:28.390716 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:28.390688    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-2619ff-57fdbfb486-5gkbp"]
Jul 24 15:48:28.393062 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:48:28.393035    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71111d85_3b11_48fe_a587_54c105cb74d7.slice/crio-88f78970bd5e71efa6c30321ae8ca12cd1fd3effbcf06344b8988a5bbe2fae2d WatchSource:0}: Error finding container 88f78970bd5e71efa6c30321ae8ca12cd1fd3effbcf06344b8988a5bbe2fae2d: Status 404 returned error can't find the container with id 88f78970bd5e71efa6c30321ae8ca12cd1fd3effbcf06344b8988a5bbe2fae2d
Jul 24 15:48:28.568627 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:28.568601    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-2619ff-677c9878b4-fqr7x"]
Jul 24 15:48:28.570316 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:48:28.570282    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2b13488_f677_4987_bafe_df99527776e7.slice/crio-212965c41468658f00ab07bae64192b272d1ad5625e0fe3abfb423a4e2e103bc WatchSource:0}: Error finding container 212965c41468658f00ab07bae64192b272d1ad5625e0fe3abfb423a4e2e103bc: Status 404 returned error can't find the container with id 212965c41468658f00ab07bae64192b272d1ad5625e0fe3abfb423a4e2e103bc
Jul 24 15:48:29.213542 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:29.213502    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2619ff-677c9878b4-fqr7x" event={"ID":"f2b13488-f677-4987-bafe-df99527776e7","Type":"ContainerStarted","Data":"7523380cc5a739a76488709b5124045a9d96ce8f8e9a07ccce2f3e5dbbff6190"}
Jul 24 15:48:29.213729 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:29.213566    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2619ff-677c9878b4-fqr7x" event={"ID":"f2b13488-f677-4987-bafe-df99527776e7","Type":"ContainerStarted","Data":"212965c41468658f00ab07bae64192b272d1ad5625e0fe3abfb423a4e2e103bc"}
Jul 24 15:48:29.213729 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:29.213594    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2619ff-677c9878b4-fqr7x"
Jul 24 15:48:29.215059 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:29.215032    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-2619ff-57fdbfb486-5gkbp" event={"ID":"71111d85-3b11-48fe-a587-54c105cb74d7","Type":"ContainerStarted","Data":"28bc077ddc0c812f6a19e96473c54390618f06b7437d645aaf6227a303d4e01b"}
Jul 24 15:48:29.215453 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:29.215078    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-2619ff-57fdbfb486-5gkbp" event={"ID":"71111d85-3b11-48fe-a587-54c105cb74d7","Type":"ContainerStarted","Data":"88f78970bd5e71efa6c30321ae8ca12cd1fd3effbcf06344b8988a5bbe2fae2d"}
Jul 24 15:48:29.215453 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:29.215101    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-2619ff-57fdbfb486-5gkbp"
Jul 24 15:48:29.216575 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:29.216546    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2619ff-67c944cf6-56txk" event={"ID":"af885fa0-3295-4818-83b1-d38e540f292e","Type":"ContainerStarted","Data":"b81d9de77b4d57db37787762169b7f26e35023079a9a2a1dd7f6885b03d204cc"}
Jul 24 15:48:29.216694 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:29.216596    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2619ff-67c944cf6-56txk" event={"ID":"af885fa0-3295-4818-83b1-d38e540f292e","Type":"ContainerStarted","Data":"f7517882b4a4d427b221c3f4029acd03ba07a72266892a9a9c97106ab95e683e"}
Jul 24 15:48:29.216694 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:29.216646    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2619ff-67c944cf6-56txk"
Jul 24 15:48:29.228744 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:29.228697    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2619ff-677c9878b4-fqr7x" podStartSLOduration=2.2286810089999998 podStartE2EDuration="2.228681009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:48:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:48:29.228299738 +0000 UTC m=+2402.040651920" watchObservedRunningTime="2026-07-24 15:48:29.228681009 +0000 UTC m=+2402.041033192"
Jul 24 15:48:29.244306 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:29.244255    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2619ff-67c944cf6-56txk" podStartSLOduration=1.7769544179999999 podStartE2EDuration="2.244239246s" totalImagesPullingTime="467.284828ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:48:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:48:29.243008287 +0000 UTC m=+2402.055360468" watchObservedRunningTime="2026-07-24 15:48:29.244239246 +0000 UTC m=+2402.056591424"
Jul 24 15:48:29.262201 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:29.262147    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-2619ff-57fdbfb486-5gkbp" podStartSLOduration=2.262135508 podStartE2EDuration="2.262135508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:48:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:48:29.260572458 +0000 UTC m=+2402.072924639" watchObservedRunningTime="2026-07-24 15:48:29.262135508 +0000 UTC m=+2402.074487687"
Jul 24 15:48:32.519173 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:32.519141    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:48:35.228201 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:35.228168    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2619ff-67c944cf6-56txk"
Jul 24 15:48:35.318877 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:35.318843    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1578f9-c7bbcd9c-mhqkm"]
Jul 24 15:48:35.323288 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:35.323265    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1578f9-c7bbcd9c-mhqkm"
Jul 24 15:48:35.325738 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:35.325715    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-1578f9-serving-cert\""
Jul 24 15:48:35.331235 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:35.331207    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1578f9-c7bbcd9c-mhqkm"]
Jul 24 15:48:35.404321 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:35.404282    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bb038d2-9151-42db-b6a1-9e4b6286d220-openshift-service-ca\") pod \"maas-api-e2e-shared-b-1578f9-c7bbcd9c-mhqkm\" (UID: \"3bb038d2-9151-42db-b6a1-9e4b6286d220\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1578f9-c7bbcd9c-mhqkm"
Jul 24 15:48:35.404321 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:35.404321    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bb038d2-9151-42db-b6a1-9e4b6286d220-maas-api-tls\") pod \"maas-api-e2e-shared-b-1578f9-c7bbcd9c-mhqkm\" (UID: \"3bb038d2-9151-42db-b6a1-9e4b6286d220\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1578f9-c7bbcd9c-mhqkm"
Jul 24 15:48:35.404564 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:35.404340    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4g5b\" (UniqueName: \"kubernetes.io/projected/3bb038d2-9151-42db-b6a1-9e4b6286d220-kube-api-access-j4g5b\") pod \"maas-api-e2e-shared-b-1578f9-c7bbcd9c-mhqkm\" (UID: \"3bb038d2-9151-42db-b6a1-9e4b6286d220\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1578f9-c7bbcd9c-mhqkm"
Jul 24 15:48:35.505337 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:35.505246    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bb038d2-9151-42db-b6a1-9e4b6286d220-openshift-service-ca\") pod \"maas-api-e2e-shared-b-1578f9-c7bbcd9c-mhqkm\" (UID: \"3bb038d2-9151-42db-b6a1-9e4b6286d220\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1578f9-c7bbcd9c-mhqkm"
Jul 24 15:48:35.505337 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:35.505290    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bb038d2-9151-42db-b6a1-9e4b6286d220-maas-api-tls\") pod \"maas-api-e2e-shared-b-1578f9-c7bbcd9c-mhqkm\" (UID: \"3bb038d2-9151-42db-b6a1-9e4b6286d220\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1578f9-c7bbcd9c-mhqkm"
Jul 24 15:48:35.505337 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:35.505313    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4g5b\" (UniqueName: \"kubernetes.io/projected/3bb038d2-9151-42db-b6a1-9e4b6286d220-kube-api-access-j4g5b\") pod \"maas-api-e2e-shared-b-1578f9-c7bbcd9c-mhqkm\" (UID: \"3bb038d2-9151-42db-b6a1-9e4b6286d220\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1578f9-c7bbcd9c-mhqkm"
Jul 24 15:48:35.505968 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:35.505941    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bb038d2-9151-42db-b6a1-9e4b6286d220-openshift-service-ca\") pod \"maas-api-e2e-shared-b-1578f9-c7bbcd9c-mhqkm\" (UID: \"3bb038d2-9151-42db-b6a1-9e4b6286d220\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1578f9-c7bbcd9c-mhqkm"
Jul 24 15:48:35.507745 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:35.507727    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bb038d2-9151-42db-b6a1-9e4b6286d220-maas-api-tls\") pod \"maas-api-e2e-shared-b-1578f9-c7bbcd9c-mhqkm\" (UID: \"3bb038d2-9151-42db-b6a1-9e4b6286d220\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1578f9-c7bbcd9c-mhqkm"
Jul 24 15:48:35.516145 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:35.516123    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4g5b\" (UniqueName: \"kubernetes.io/projected/3bb038d2-9151-42db-b6a1-9e4b6286d220-kube-api-access-j4g5b\") pod \"maas-api-e2e-shared-b-1578f9-c7bbcd9c-mhqkm\" (UID: \"3bb038d2-9151-42db-b6a1-9e4b6286d220\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1578f9-c7bbcd9c-mhqkm"
Jul 24 15:48:35.636906 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:35.636872    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1578f9-c7bbcd9c-mhqkm"
Jul 24 15:48:35.776733 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:35.776708    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1578f9-c7bbcd9c-mhqkm"]
Jul 24 15:48:35.779382 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:48:35.779347    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bb038d2_9151_42db_b6a1_9e4b6286d220.slice/crio-0b0701295d5f375f2419075dec6e6c81303812b57ab4bd9d259fa9cc5ce86ab6 WatchSource:0}: Error finding container 0b0701295d5f375f2419075dec6e6c81303812b57ab4bd9d259fa9cc5ce86ab6: Status 404 returned error can't find the container with id 0b0701295d5f375f2419075dec6e6c81303812b57ab4bd9d259fa9cc5ce86ab6
Jul 24 15:48:35.963558 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:35.963527    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-1578f9-647bdc8c88-fqq67"]
Jul 24 15:48:35.968081 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:35.968064    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1578f9-647bdc8c88-fqq67"
Jul 24 15:48:35.970444 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:35.970420    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-1578f9-dockercfg-qlw8s\""
Jul 24 15:48:35.970536 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:35.970420    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-1578f9\""
Jul 24 15:48:35.977955 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:35.977925    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-1578f9-647bdc8c88-fqq67"]
Jul 24 15:48:35.999240 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:35.999208    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-1578f9-7fcfcd5566-hkxm4"]
Jul 24 15:48:36.002551 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:36.002535    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-1578f9-7fcfcd5566-hkxm4"
Jul 24 15:48:36.008316 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:36.008289    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtj27\" (UniqueName: \"kubernetes.io/projected/5f3c7c3e-9695-4364-8ede-5274813096a5-kube-api-access-dtj27\") pod \"payload-pre-processing-e2e-shared-b-1578f9-647bdc8c88-fqq67\" (UID: \"5f3c7c3e-9695-4364-8ede-5274813096a5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1578f9-647bdc8c88-fqq67"
Jul 24 15:48:36.008411 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:36.008383    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8r2g\" (UniqueName: \"kubernetes.io/projected/d447f9d4-1a69-41a2-a6f0-7d4d6b2b61c0-kube-api-access-f8r2g\") pod \"payload-processing-e2e-shared-b-1578f9-7fcfcd5566-hkxm4\" (UID: \"d447f9d4-1a69-41a2-a6f0-7d4d6b2b61c0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-1578f9-7fcfcd5566-hkxm4"
Jul 24 15:48:36.008473 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:36.008444    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f3c7c3e-9695-4364-8ede-5274813096a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-1578f9-647bdc8c88-fqq67\" (UID: \"5f3c7c3e-9695-4364-8ede-5274813096a5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1578f9-647bdc8c88-fqq67"
Jul 24 15:48:36.008557 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:36.008532    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d447f9d4-1a69-41a2-a6f0-7d4d6b2b61c0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-1578f9-7fcfcd5566-hkxm4\" (UID: \"d447f9d4-1a69-41a2-a6f0-7d4d6b2b61c0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-1578f9-7fcfcd5566-hkxm4"
Jul 24 15:48:36.022239 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:36.022098    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-1578f9-7fcfcd5566-hkxm4"]
Jul 24 15:48:36.109010 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:36.108984    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtj27\" (UniqueName: \"kubernetes.io/projected/5f3c7c3e-9695-4364-8ede-5274813096a5-kube-api-access-dtj27\") pod \"payload-pre-processing-e2e-shared-b-1578f9-647bdc8c88-fqq67\" (UID: \"5f3c7c3e-9695-4364-8ede-5274813096a5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1578f9-647bdc8c88-fqq67"
Jul 24 15:48:36.109108 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:36.109037    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8r2g\" (UniqueName: \"kubernetes.io/projected/d447f9d4-1a69-41a2-a6f0-7d4d6b2b61c0-kube-api-access-f8r2g\") pod \"payload-processing-e2e-shared-b-1578f9-7fcfcd5566-hkxm4\" (UID: \"d447f9d4-1a69-41a2-a6f0-7d4d6b2b61c0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-1578f9-7fcfcd5566-hkxm4"
Jul 24 15:48:36.109108 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:36.109076    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f3c7c3e-9695-4364-8ede-5274813096a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-1578f9-647bdc8c88-fqq67\" (UID: \"5f3c7c3e-9695-4364-8ede-5274813096a5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1578f9-647bdc8c88-fqq67"
Jul 24 15:48:36.109108 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:36.109106    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d447f9d4-1a69-41a2-a6f0-7d4d6b2b61c0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-1578f9-7fcfcd5566-hkxm4\" (UID: \"d447f9d4-1a69-41a2-a6f0-7d4d6b2b61c0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-1578f9-7fcfcd5566-hkxm4"
Jul 24 15:48:36.109765 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:36.109748    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d447f9d4-1a69-41a2-a6f0-7d4d6b2b61c0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-1578f9-7fcfcd5566-hkxm4\" (UID: \"d447f9d4-1a69-41a2-a6f0-7d4d6b2b61c0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-1578f9-7fcfcd5566-hkxm4"
Jul 24 15:48:36.109807 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:36.109785    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f3c7c3e-9695-4364-8ede-5274813096a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-1578f9-647bdc8c88-fqq67\" (UID: \"5f3c7c3e-9695-4364-8ede-5274813096a5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1578f9-647bdc8c88-fqq67"
Jul 24 15:48:36.116975 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:36.116957    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8r2g\" (UniqueName: \"kubernetes.io/projected/d447f9d4-1a69-41a2-a6f0-7d4d6b2b61c0-kube-api-access-f8r2g\") pod \"payload-processing-e2e-shared-b-1578f9-7fcfcd5566-hkxm4\" (UID: \"d447f9d4-1a69-41a2-a6f0-7d4d6b2b61c0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-1578f9-7fcfcd5566-hkxm4"
Jul 24 15:48:36.117060 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:36.116965    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtj27\" (UniqueName: \"kubernetes.io/projected/5f3c7c3e-9695-4364-8ede-5274813096a5-kube-api-access-dtj27\") pod \"payload-pre-processing-e2e-shared-b-1578f9-647bdc8c88-fqq67\" (UID: \"5f3c7c3e-9695-4364-8ede-5274813096a5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1578f9-647bdc8c88-fqq67"
Jul 24 15:48:36.249515 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:36.249480    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1578f9-c7bbcd9c-mhqkm" event={"ID":"3bb038d2-9151-42db-b6a1-9e4b6286d220","Type":"ContainerStarted","Data":"0dd6ee28feef6e4ea03ee08d9576a1f3059ce9dc2f9ec99cfc4b1d0a5b1cda21"}
Jul 24 15:48:36.249973 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:36.249530    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1578f9-c7bbcd9c-mhqkm" event={"ID":"3bb038d2-9151-42db-b6a1-9e4b6286d220","Type":"ContainerStarted","Data":"0b0701295d5f375f2419075dec6e6c81303812b57ab4bd9d259fa9cc5ce86ab6"}
Jul 24 15:48:36.249973 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:36.249633    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1578f9-c7bbcd9c-mhqkm"
Jul 24 15:48:36.280340 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:36.280309    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1578f9-647bdc8c88-fqq67"
Jul 24 15:48:36.312267 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:36.312236    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-1578f9-7fcfcd5566-hkxm4"
Jul 24 15:48:36.428242 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:36.428196    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1578f9-c7bbcd9c-mhqkm" podStartSLOduration=1.120429883 podStartE2EDuration="1.428179632s" totalImagesPullingTime="307.749749ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:48:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:48:36.269786507 +0000 UTC m=+2409.082138685" watchObservedRunningTime="2026-07-24 15:48:36.428179632 +0000 UTC m=+2409.240531858"
Jul 24 15:48:36.428467 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:36.428448    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-1578f9-647bdc8c88-fqq67"]
Jul 24 15:48:36.430539 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:48:36.430512    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f3c7c3e_9695_4364_8ede_5274813096a5.slice/crio-5ccf6813566e0a48c8726c7392af0046be3314ea55bb472a16d7f09826b29eca WatchSource:0}: Error finding container 5ccf6813566e0a48c8726c7392af0046be3314ea55bb472a16d7f09826b29eca: Status 404 returned error can't find the container with id 5ccf6813566e0a48c8726c7392af0046be3314ea55bb472a16d7f09826b29eca
Jul 24 15:48:36.671691 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:36.671661    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-1578f9-7fcfcd5566-hkxm4"]
Jul 24 15:48:36.673807 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:48:36.673779    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd447f9d4_1a69_41a2_a6f0_7d4d6b2b61c0.slice/crio-f88cd8d6f3807fba8265cd5cdb0901e01895a15191d46829d0569206a77d920c WatchSource:0}: Error finding container f88cd8d6f3807fba8265cd5cdb0901e01895a15191d46829d0569206a77d920c: Status 404 returned error can't find the container with id f88cd8d6f3807fba8265cd5cdb0901e01895a15191d46829d0569206a77d920c
Jul 24 15:48:37.257607 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:37.257569    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1578f9-647bdc8c88-fqq67" event={"ID":"5f3c7c3e-9695-4364-8ede-5274813096a5","Type":"ContainerStarted","Data":"431d64ee4f2d69a05bb6dd58fccedb2c86f56d0d9c1c7f5d48368db78f667981"}
Jul 24 15:48:37.258125 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:37.257639    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1578f9-647bdc8c88-fqq67" event={"ID":"5f3c7c3e-9695-4364-8ede-5274813096a5","Type":"ContainerStarted","Data":"5ccf6813566e0a48c8726c7392af0046be3314ea55bb472a16d7f09826b29eca"}
Jul 24 15:48:37.258125 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:37.257672    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1578f9-647bdc8c88-fqq67"
Jul 24 15:48:37.259153 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:37.259125    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-1578f9-7fcfcd5566-hkxm4" event={"ID":"d447f9d4-1a69-41a2-a6f0-7d4d6b2b61c0","Type":"ContainerStarted","Data":"acbffae8806269dd6ced1985d12344b6d4825639a8e8daeb732564057d6eba83"}
Jul 24 15:48:37.259247 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:37.259176    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-1578f9-7fcfcd5566-hkxm4" event={"ID":"d447f9d4-1a69-41a2-a6f0-7d4d6b2b61c0","Type":"ContainerStarted","Data":"f88cd8d6f3807fba8265cd5cdb0901e01895a15191d46829d0569206a77d920c"}
Jul 24 15:48:37.259327 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:37.259308    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-1578f9-7fcfcd5566-hkxm4"
Jul 24 15:48:37.279562 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:37.279513    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1578f9-647bdc8c88-fqq67" podStartSLOduration=2.279499306 podStartE2EDuration="2.279499306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:48:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:48:37.276973094 +0000 UTC m=+2410.089325274" watchObservedRunningTime="2026-07-24 15:48:37.279499306 +0000 UTC m=+2410.091851484"
Jul 24 15:48:37.293209 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:37.293170    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-1578f9-7fcfcd5566-hkxm4" podStartSLOduration=2.293156765 podStartE2EDuration="2.293156765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:48:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:48:37.29158049 +0000 UTC m=+2410.103932672" watchObservedRunningTime="2026-07-24 15:48:37.293156765 +0000 UTC m=+2410.105508943"
Jul 24 15:48:40.223688 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:40.223656    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2619ff-677c9878b4-fqr7x"
Jul 24 15:48:40.224122 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:40.223776    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-2619ff-57fdbfb486-5gkbp"
Jul 24 15:48:42.265995 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:42.265965    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1578f9-c7bbcd9c-mhqkm"
Jul 24 15:48:48.265748 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:48.265720    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-1578f9-7fcfcd5566-hkxm4"
Jul 24 15:48:48.266175 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:48.265787    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1578f9-647bdc8c88-fqq67"
Jul 24 15:48:49.116806 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:49.116769    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:48:53.113489 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:53.113454    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-2619ff/auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv"]
Jul 24 15:48:53.118154 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:53.118134    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2619ff/auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv"
Jul 24 15:48:53.121516 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:53.121492    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-2619ff\"/\"openshift-service-ca.crt\""
Jul 24 15:48:53.121644 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:53.121497    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2619ff\"/\"default-dockercfg-ltzlf\""
Jul 24 15:48:53.121644 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:53.121525    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-2619ff\"/\"kube-root-ca.crt\""
Jul 24 15:48:53.121783 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:53.121733    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2619ff\"/\"auth-test-model-2619ff-kserve-self-signed-certs\""
Jul 24 15:48:53.128903 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:53.128883    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2619ff/auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv"]
Jul 24 15:48:53.167522 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:53.167494    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/380684ce-7ab7-4f24-8d2c-9032d5a28efd-tmp-dir\") pod \"auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv\" (UID: \"380684ce-7ab7-4f24-8d2c-9032d5a28efd\") " pod="ai-tenant-e2e-shared-a-2619ff/auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv"
Jul 24 15:48:53.167678 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:53.167530    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/380684ce-7ab7-4f24-8d2c-9032d5a28efd-dshm\") pod \"auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv\" (UID: \"380684ce-7ab7-4f24-8d2c-9032d5a28efd\") " pod="ai-tenant-e2e-shared-a-2619ff/auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv"
Jul 24 15:48:53.167678 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:53.167554    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6986t\" (UniqueName: \"kubernetes.io/projected/380684ce-7ab7-4f24-8d2c-9032d5a28efd-kube-api-access-6986t\") pod \"auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv\" (UID: \"380684ce-7ab7-4f24-8d2c-9032d5a28efd\") " pod="ai-tenant-e2e-shared-a-2619ff/auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv"
Jul 24 15:48:53.167678 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:53.167605    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/380684ce-7ab7-4f24-8d2c-9032d5a28efd-model-cache\") pod \"auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv\" (UID: \"380684ce-7ab7-4f24-8d2c-9032d5a28efd\") " pod="ai-tenant-e2e-shared-a-2619ff/auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv"
Jul 24 15:48:53.167678 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:53.167625    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/380684ce-7ab7-4f24-8d2c-9032d5a28efd-home\") pod \"auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv\" (UID: \"380684ce-7ab7-4f24-8d2c-9032d5a28efd\") " pod="ai-tenant-e2e-shared-a-2619ff/auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv"
Jul 24 15:48:53.167856 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:53.167688    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/380684ce-7ab7-4f24-8d2c-9032d5a28efd-tls-certs\") pod \"auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv\" (UID: \"380684ce-7ab7-4f24-8d2c-9032d5a28efd\") " pod="ai-tenant-e2e-shared-a-2619ff/auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv"
Jul 24 15:48:53.268426 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:53.268390    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/380684ce-7ab7-4f24-8d2c-9032d5a28efd-tmp-dir\") pod \"auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv\" (UID: \"380684ce-7ab7-4f24-8d2c-9032d5a28efd\") " pod="ai-tenant-e2e-shared-a-2619ff/auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv"
Jul 24 15:48:53.268426 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:53.268429    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/380684ce-7ab7-4f24-8d2c-9032d5a28efd-dshm\") pod \"auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv\" (UID: \"380684ce-7ab7-4f24-8d2c-9032d5a28efd\") " pod="ai-tenant-e2e-shared-a-2619ff/auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv"
Jul 24 15:48:53.268669 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:53.268462    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6986t\" (UniqueName: \"kubernetes.io/projected/380684ce-7ab7-4f24-8d2c-9032d5a28efd-kube-api-access-6986t\") pod \"auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv\" (UID: \"380684ce-7ab7-4f24-8d2c-9032d5a28efd\") " pod="ai-tenant-e2e-shared-a-2619ff/auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv"
Jul 24 15:48:53.268669 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:53.268503    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/380684ce-7ab7-4f24-8d2c-9032d5a28efd-model-cache\") pod \"auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv\" (UID: \"380684ce-7ab7-4f24-8d2c-9032d5a28efd\") " pod="ai-tenant-e2e-shared-a-2619ff/auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv"
Jul 24 15:48:53.268669 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:53.268535    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/380684ce-7ab7-4f24-8d2c-9032d5a28efd-home\") pod \"auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv\" (UID: \"380684ce-7ab7-4f24-8d2c-9032d5a28efd\") " pod="ai-tenant-e2e-shared-a-2619ff/auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv"
Jul 24 15:48:53.268669 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:53.268576    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/380684ce-7ab7-4f24-8d2c-9032d5a28efd-tls-certs\") pod \"auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv\" (UID: \"380684ce-7ab7-4f24-8d2c-9032d5a28efd\") " pod="ai-tenant-e2e-shared-a-2619ff/auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv"
Jul 24 15:48:53.269142 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:53.268923    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/380684ce-7ab7-4f24-8d2c-9032d5a28efd-tmp-dir\") pod \"auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv\" (UID: \"380684ce-7ab7-4f24-8d2c-9032d5a28efd\") " pod="ai-tenant-e2e-shared-a-2619ff/auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv"
Jul 24 15:48:53.269142 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:53.269000    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/380684ce-7ab7-4f24-8d2c-9032d5a28efd-home\") pod \"auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv\" (UID: \"380684ce-7ab7-4f24-8d2c-9032d5a28efd\") " pod="ai-tenant-e2e-shared-a-2619ff/auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv"
Jul 24 15:48:53.269142 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:53.269048    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/380684ce-7ab7-4f24-8d2c-9032d5a28efd-model-cache\") pod \"auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv\" (UID: \"380684ce-7ab7-4f24-8d2c-9032d5a28efd\") " pod="ai-tenant-e2e-shared-a-2619ff/auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv"
Jul 24 15:48:53.270798 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:53.270775    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/380684ce-7ab7-4f24-8d2c-9032d5a28efd-dshm\") pod \"auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv\" (UID: \"380684ce-7ab7-4f24-8d2c-9032d5a28efd\") " pod="ai-tenant-e2e-shared-a-2619ff/auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv"
Jul 24 15:48:53.271007 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:53.270988    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/380684ce-7ab7-4f24-8d2c-9032d5a28efd-tls-certs\") pod \"auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv\" (UID: \"380684ce-7ab7-4f24-8d2c-9032d5a28efd\") " pod="ai-tenant-e2e-shared-a-2619ff/auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv"
Jul 24 15:48:53.276405 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:53.276385    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6986t\" (UniqueName: \"kubernetes.io/projected/380684ce-7ab7-4f24-8d2c-9032d5a28efd-kube-api-access-6986t\") pod \"auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv\" (UID: \"380684ce-7ab7-4f24-8d2c-9032d5a28efd\") " pod="ai-tenant-e2e-shared-a-2619ff/auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv"
Jul 24 15:48:53.429370 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:53.429291    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2619ff/auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv"
Jul 24 15:48:53.569444 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:53.569419    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2619ff/auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv"]
Jul 24 15:48:53.571338 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:48:53.571311    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod380684ce_7ab7_4f24_8d2c_9032d5a28efd.slice/crio-3002cd89ab60b75ebf3e6c43124ec4950f978a7e0cf5edb38fbe957e64291bd5 WatchSource:0}: Error finding container 3002cd89ab60b75ebf3e6c43124ec4950f978a7e0cf5edb38fbe957e64291bd5: Status 404 returned error can't find the container with id 3002cd89ab60b75ebf3e6c43124ec4950f978a7e0cf5edb38fbe957e64291bd5
Jul 24 15:48:54.349881 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:54.349837    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2619ff/auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv" event={"ID":"380684ce-7ab7-4f24-8d2c-9032d5a28efd","Type":"ContainerStarted","Data":"5cba8ddfb2007b19292aa63fc8fa658e6c3d14c03c6c4ff6b8b03139ef34365a"}
Jul 24 15:48:54.350280 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:54.349888    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2619ff/auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv" event={"ID":"380684ce-7ab7-4f24-8d2c-9032d5a28efd","Type":"ContainerStarted","Data":"3002cd89ab60b75ebf3e6c43124ec4950f978a7e0cf5edb38fbe957e64291bd5"}
Jul 24 15:48:54.369352 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:54.369305    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-2619ff/auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv" podStartSLOduration=1.369290905 podStartE2EDuration="1.369290905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:48:54.367722803 +0000 UTC m=+2427.180074982" watchObservedRunningTime="2026-07-24 15:48:54.369290905 +0000 UTC m=+2427.181643116"
Jul 24 15:48:58.520677 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:48:58.520639    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:49:03.430012 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:49:03.429966    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-2619ff/auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv"
Jul 24 15:49:03.430012 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:49:03.430018    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-2619ff/auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv"
Jul 24 15:49:03.442714 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:49:03.442684    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-2619ff/auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv"
Jul 24 15:49:04.413535 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:49:04.413505    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-2619ff/auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv"
Jul 24 15:49:24.329389 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:49:24.329346    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:49:45.926825 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:49:45.926773    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:50:05.116297 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:50:05.116262    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:50:18.120741 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:50:18.120706    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:50:37.322323 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:50:37.322225    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:50:53.019755 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:50:53.019722    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:51:12.326941 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:12.326905    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:51:28.163623 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:28.163588    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2619ff/auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv"]
Jul 24 15:51:30.827626 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:30.827586    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:51:43.182873 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:43.182788    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-2619ff/auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv" podUID="380684ce-7ab7-4f24-8d2c-9032d5a28efd" containerName="main" containerID="cri-o://5cba8ddfb2007b19292aa63fc8fa658e6c3d14c03c6c4ff6b8b03139ef34365a" gracePeriod=45
Jul 24 15:51:43.439218 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:43.439152    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2619ff/auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv"
Jul 24 15:51:43.615976 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:43.615936    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/380684ce-7ab7-4f24-8d2c-9032d5a28efd-dshm\") pod \"380684ce-7ab7-4f24-8d2c-9032d5a28efd\" (UID: \"380684ce-7ab7-4f24-8d2c-9032d5a28efd\") "
Jul 24 15:51:43.615976 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:43.615983    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/380684ce-7ab7-4f24-8d2c-9032d5a28efd-tmp-dir\") pod \"380684ce-7ab7-4f24-8d2c-9032d5a28efd\" (UID: \"380684ce-7ab7-4f24-8d2c-9032d5a28efd\") "
Jul 24 15:51:43.616242 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:43.616012    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6986t\" (UniqueName: \"kubernetes.io/projected/380684ce-7ab7-4f24-8d2c-9032d5a28efd-kube-api-access-6986t\") pod \"380684ce-7ab7-4f24-8d2c-9032d5a28efd\" (UID: \"380684ce-7ab7-4f24-8d2c-9032d5a28efd\") "
Jul 24 15:51:43.616242 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:43.616040    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/380684ce-7ab7-4f24-8d2c-9032d5a28efd-tls-certs\") pod \"380684ce-7ab7-4f24-8d2c-9032d5a28efd\" (UID: \"380684ce-7ab7-4f24-8d2c-9032d5a28efd\") "
Jul 24 15:51:43.616242 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:43.616063    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/380684ce-7ab7-4f24-8d2c-9032d5a28efd-model-cache\") pod \"380684ce-7ab7-4f24-8d2c-9032d5a28efd\" (UID: \"380684ce-7ab7-4f24-8d2c-9032d5a28efd\") "
Jul 24 15:51:43.616242 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:43.616101    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/380684ce-7ab7-4f24-8d2c-9032d5a28efd-home\") pod \"380684ce-7ab7-4f24-8d2c-9032d5a28efd\" (UID: \"380684ce-7ab7-4f24-8d2c-9032d5a28efd\") "
Jul 24 15:51:43.616433 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:43.616380    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/380684ce-7ab7-4f24-8d2c-9032d5a28efd-model-cache" (OuterVolumeSpecName: "model-cache") pod "380684ce-7ab7-4f24-8d2c-9032d5a28efd" (UID: "380684ce-7ab7-4f24-8d2c-9032d5a28efd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:51:43.616433 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:43.616381    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/380684ce-7ab7-4f24-8d2c-9032d5a28efd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "380684ce-7ab7-4f24-8d2c-9032d5a28efd" (UID: "380684ce-7ab7-4f24-8d2c-9032d5a28efd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:51:43.616532 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:43.616511    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/380684ce-7ab7-4f24-8d2c-9032d5a28efd-model-cache\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:51:43.616568 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:43.616547    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/380684ce-7ab7-4f24-8d2c-9032d5a28efd-tmp-dir\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:51:43.616568 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:43.616539    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/380684ce-7ab7-4f24-8d2c-9032d5a28efd-home" (OuterVolumeSpecName: "home") pod "380684ce-7ab7-4f24-8d2c-9032d5a28efd" (UID: "380684ce-7ab7-4f24-8d2c-9032d5a28efd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:51:43.618326 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:43.618306    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/380684ce-7ab7-4f24-8d2c-9032d5a28efd-kube-api-access-6986t" (OuterVolumeSpecName: "kube-api-access-6986t") pod "380684ce-7ab7-4f24-8d2c-9032d5a28efd" (UID: "380684ce-7ab7-4f24-8d2c-9032d5a28efd"). InnerVolumeSpecName "kube-api-access-6986t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:51:43.618596 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:43.618580    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/380684ce-7ab7-4f24-8d2c-9032d5a28efd-dshm" (OuterVolumeSpecName: "dshm") pod "380684ce-7ab7-4f24-8d2c-9032d5a28efd" (UID: "380684ce-7ab7-4f24-8d2c-9032d5a28efd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:51:43.618596 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:43.618584    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/380684ce-7ab7-4f24-8d2c-9032d5a28efd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "380684ce-7ab7-4f24-8d2c-9032d5a28efd" (UID: "380684ce-7ab7-4f24-8d2c-9032d5a28efd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 15:51:43.717315 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:43.717219    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/380684ce-7ab7-4f24-8d2c-9032d5a28efd-dshm\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:51:43.717315 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:43.717250    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6986t\" (UniqueName: \"kubernetes.io/projected/380684ce-7ab7-4f24-8d2c-9032d5a28efd-kube-api-access-6986t\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:51:43.717315 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:43.717262    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/380684ce-7ab7-4f24-8d2c-9032d5a28efd-tls-certs\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:51:43.717315 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:43.717270    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/380684ce-7ab7-4f24-8d2c-9032d5a28efd-home\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:51:44.212479 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:44.212442    2579 generic.go:356] "Generic (PLEG): container finished" podID="380684ce-7ab7-4f24-8d2c-9032d5a28efd" containerID="5cba8ddfb2007b19292aa63fc8fa658e6c3d14c03c6c4ff6b8b03139ef34365a" exitCode=0
Jul 24 15:51:44.212908 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:44.212484    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2619ff/auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv"
Jul 24 15:51:44.212908 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:44.212512    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2619ff/auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv" event={"ID":"380684ce-7ab7-4f24-8d2c-9032d5a28efd","Type":"ContainerDied","Data":"5cba8ddfb2007b19292aa63fc8fa658e6c3d14c03c6c4ff6b8b03139ef34365a"}
Jul 24 15:51:44.212908 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:44.212551    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2619ff/auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv" event={"ID":"380684ce-7ab7-4f24-8d2c-9032d5a28efd","Type":"ContainerDied","Data":"3002cd89ab60b75ebf3e6c43124ec4950f978a7e0cf5edb38fbe957e64291bd5"}
Jul 24 15:51:44.212908 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:44.212569    2579 scope.go:117] "RemoveContainer" containerID="5cba8ddfb2007b19292aa63fc8fa658e6c3d14c03c6c4ff6b8b03139ef34365a"
Jul 24 15:51:44.221468 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:44.221449    2579 scope.go:117] "RemoveContainer" containerID="5cba8ddfb2007b19292aa63fc8fa658e6c3d14c03c6c4ff6b8b03139ef34365a"
Jul 24 15:51:44.221721 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:51:44.221698    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cba8ddfb2007b19292aa63fc8fa658e6c3d14c03c6c4ff6b8b03139ef34365a\": container with ID starting with 5cba8ddfb2007b19292aa63fc8fa658e6c3d14c03c6c4ff6b8b03139ef34365a not found: ID does not exist" containerID="5cba8ddfb2007b19292aa63fc8fa658e6c3d14c03c6c4ff6b8b03139ef34365a"
Jul 24 15:51:44.221777 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:44.221730    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cba8ddfb2007b19292aa63fc8fa658e6c3d14c03c6c4ff6b8b03139ef34365a"} err="failed to get container status \"5cba8ddfb2007b19292aa63fc8fa658e6c3d14c03c6c4ff6b8b03139ef34365a\": rpc error: code = NotFound desc = could not find container \"5cba8ddfb2007b19292aa63fc8fa658e6c3d14c03c6c4ff6b8b03139ef34365a\": container with ID starting with 5cba8ddfb2007b19292aa63fc8fa658e6c3d14c03c6c4ff6b8b03139ef34365a not found: ID does not exist"
Jul 24 15:51:44.228784 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:44.228758    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2619ff/auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv"]
Jul 24 15:51:44.232621 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:44.232599    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-2619ff/auth-test-model-2619ff-kserve-5fc9776d8c-qz6tv"]
Jul 24 15:51:45.717586 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:45.717551    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="380684ce-7ab7-4f24-8d2c-9032d5a28efd" path="/var/lib/kubelet/pods/380684ce-7ab7-4f24-8d2c-9032d5a28efd/volumes"
Jul 24 15:51:49.630569 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:49.630493    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-2619ff/sub-test-model-2619ff-kserve-85b69684cf-p9wpd"]
Jul 24 15:51:49.630952 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:49.630939    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="380684ce-7ab7-4f24-8d2c-9032d5a28efd" containerName="main"
Jul 24 15:51:49.630952 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:49.630954    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="380684ce-7ab7-4f24-8d2c-9032d5a28efd" containerName="main"
Jul 24 15:51:49.631065 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:49.631052    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="380684ce-7ab7-4f24-8d2c-9032d5a28efd" containerName="main"
Jul 24 15:51:49.635343 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:49.635327    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2619ff/sub-test-model-2619ff-kserve-85b69684cf-p9wpd"
Jul 24 15:51:49.637830 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:49.637796    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-2619ff\"/\"kube-root-ca.crt\""
Jul 24 15:51:49.637978 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:49.637852    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-2619ff\"/\"openshift-service-ca.crt\""
Jul 24 15:51:49.637978 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:49.637921    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2619ff\"/\"default-dockercfg-ltzlf\""
Jul 24 15:51:49.638626 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:49.638601    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2619ff\"/\"sub-test-model-2619ff-kserve-self-signed-certs\""
Jul 24 15:51:49.645907 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:49.645885    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2619ff/sub-test-model-2619ff-kserve-85b69684cf-p9wpd"]
Jul 24 15:51:49.775627 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:49.775574    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a4da93a-0b63-45f8-9363-76ac097c1a6a-dshm\") pod \"sub-test-model-2619ff-kserve-85b69684cf-p9wpd\" (UID: \"5a4da93a-0b63-45f8-9363-76ac097c1a6a\") " pod="ai-tenant-e2e-shared-a-2619ff/sub-test-model-2619ff-kserve-85b69684cf-p9wpd"
Jul 24 15:51:49.775627 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:49.775636    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a4da93a-0b63-45f8-9363-76ac097c1a6a-model-cache\") pod \"sub-test-model-2619ff-kserve-85b69684cf-p9wpd\" (UID: \"5a4da93a-0b63-45f8-9363-76ac097c1a6a\") " pod="ai-tenant-e2e-shared-a-2619ff/sub-test-model-2619ff-kserve-85b69684cf-p9wpd"
Jul 24 15:51:49.775894 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:49.775696    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a4da93a-0b63-45f8-9363-76ac097c1a6a-tls-certs\") pod \"sub-test-model-2619ff-kserve-85b69684cf-p9wpd\" (UID: \"5a4da93a-0b63-45f8-9363-76ac097c1a6a\") " pod="ai-tenant-e2e-shared-a-2619ff/sub-test-model-2619ff-kserve-85b69684cf-p9wpd"
Jul 24 15:51:49.775894 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:49.775722    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a4da93a-0b63-45f8-9363-76ac097c1a6a-home\") pod \"sub-test-model-2619ff-kserve-85b69684cf-p9wpd\" (UID: \"5a4da93a-0b63-45f8-9363-76ac097c1a6a\") " pod="ai-tenant-e2e-shared-a-2619ff/sub-test-model-2619ff-kserve-85b69684cf-p9wpd"
Jul 24 15:51:49.775894 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:49.775763    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9b8j\" (UniqueName: \"kubernetes.io/projected/5a4da93a-0b63-45f8-9363-76ac097c1a6a-kube-api-access-h9b8j\") pod \"sub-test-model-2619ff-kserve-85b69684cf-p9wpd\" (UID: \"5a4da93a-0b63-45f8-9363-76ac097c1a6a\") " pod="ai-tenant-e2e-shared-a-2619ff/sub-test-model-2619ff-kserve-85b69684cf-p9wpd"
Jul 24 15:51:49.775894 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:49.775842    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a4da93a-0b63-45f8-9363-76ac097c1a6a-tmp-dir\") pod \"sub-test-model-2619ff-kserve-85b69684cf-p9wpd\" (UID: \"5a4da93a-0b63-45f8-9363-76ac097c1a6a\") " pod="ai-tenant-e2e-shared-a-2619ff/sub-test-model-2619ff-kserve-85b69684cf-p9wpd"
Jul 24 15:51:49.877043 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:49.876999    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a4da93a-0b63-45f8-9363-76ac097c1a6a-tls-certs\") pod \"sub-test-model-2619ff-kserve-85b69684cf-p9wpd\" (UID: \"5a4da93a-0b63-45f8-9363-76ac097c1a6a\") " pod="ai-tenant-e2e-shared-a-2619ff/sub-test-model-2619ff-kserve-85b69684cf-p9wpd"
Jul 24 15:51:49.877244 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:49.877065    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a4da93a-0b63-45f8-9363-76ac097c1a6a-home\") pod \"sub-test-model-2619ff-kserve-85b69684cf-p9wpd\" (UID: \"5a4da93a-0b63-45f8-9363-76ac097c1a6a\") " pod="ai-tenant-e2e-shared-a-2619ff/sub-test-model-2619ff-kserve-85b69684cf-p9wpd"
Jul 24 15:51:49.877244 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:49.877122    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9b8j\" (UniqueName: \"kubernetes.io/projected/5a4da93a-0b63-45f8-9363-76ac097c1a6a-kube-api-access-h9b8j\") pod \"sub-test-model-2619ff-kserve-85b69684cf-p9wpd\" (UID: \"5a4da93a-0b63-45f8-9363-76ac097c1a6a\") " pod="ai-tenant-e2e-shared-a-2619ff/sub-test-model-2619ff-kserve-85b69684cf-p9wpd"
Jul 24 15:51:49.877244 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:49.877150    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a4da93a-0b63-45f8-9363-76ac097c1a6a-tmp-dir\") pod \"sub-test-model-2619ff-kserve-85b69684cf-p9wpd\" (UID: \"5a4da93a-0b63-45f8-9363-76ac097c1a6a\") " pod="ai-tenant-e2e-shared-a-2619ff/sub-test-model-2619ff-kserve-85b69684cf-p9wpd"
Jul 24 15:51:49.877244 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:49.877209    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a4da93a-0b63-45f8-9363-76ac097c1a6a-dshm\") pod \"sub-test-model-2619ff-kserve-85b69684cf-p9wpd\" (UID: \"5a4da93a-0b63-45f8-9363-76ac097c1a6a\") " pod="ai-tenant-e2e-shared-a-2619ff/sub-test-model-2619ff-kserve-85b69684cf-p9wpd"
Jul 24 15:51:49.877244 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:49.877235    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a4da93a-0b63-45f8-9363-76ac097c1a6a-model-cache\") pod \"sub-test-model-2619ff-kserve-85b69684cf-p9wpd\" (UID: \"5a4da93a-0b63-45f8-9363-76ac097c1a6a\") " pod="ai-tenant-e2e-shared-a-2619ff/sub-test-model-2619ff-kserve-85b69684cf-p9wpd"
Jul 24 15:51:49.877520 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:49.877486    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a4da93a-0b63-45f8-9363-76ac097c1a6a-home\") pod \"sub-test-model-2619ff-kserve-85b69684cf-p9wpd\" (UID: \"5a4da93a-0b63-45f8-9363-76ac097c1a6a\") " pod="ai-tenant-e2e-shared-a-2619ff/sub-test-model-2619ff-kserve-85b69684cf-p9wpd"
Jul 24 15:51:49.877568 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:49.877536    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a4da93a-0b63-45f8-9363-76ac097c1a6a-tmp-dir\") pod \"sub-test-model-2619ff-kserve-85b69684cf-p9wpd\" (UID: \"5a4da93a-0b63-45f8-9363-76ac097c1a6a\") " pod="ai-tenant-e2e-shared-a-2619ff/sub-test-model-2619ff-kserve-85b69684cf-p9wpd"
Jul 24 15:51:49.877666 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:49.877648    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a4da93a-0b63-45f8-9363-76ac097c1a6a-model-cache\") pod \"sub-test-model-2619ff-kserve-85b69684cf-p9wpd\" (UID: \"5a4da93a-0b63-45f8-9363-76ac097c1a6a\") " pod="ai-tenant-e2e-shared-a-2619ff/sub-test-model-2619ff-kserve-85b69684cf-p9wpd"
Jul 24 15:51:49.879359 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:49.879340    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a4da93a-0b63-45f8-9363-76ac097c1a6a-dshm\") pod \"sub-test-model-2619ff-kserve-85b69684cf-p9wpd\" (UID: \"5a4da93a-0b63-45f8-9363-76ac097c1a6a\") " pod="ai-tenant-e2e-shared-a-2619ff/sub-test-model-2619ff-kserve-85b69684cf-p9wpd"
Jul 24 15:51:49.879525 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:49.879508    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a4da93a-0b63-45f8-9363-76ac097c1a6a-tls-certs\") pod \"sub-test-model-2619ff-kserve-85b69684cf-p9wpd\" (UID: \"5a4da93a-0b63-45f8-9363-76ac097c1a6a\") " pod="ai-tenant-e2e-shared-a-2619ff/sub-test-model-2619ff-kserve-85b69684cf-p9wpd"
Jul 24 15:51:49.885561 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:49.885504    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9b8j\" (UniqueName: \"kubernetes.io/projected/5a4da93a-0b63-45f8-9363-76ac097c1a6a-kube-api-access-h9b8j\") pod \"sub-test-model-2619ff-kserve-85b69684cf-p9wpd\" (UID: \"5a4da93a-0b63-45f8-9363-76ac097c1a6a\") " pod="ai-tenant-e2e-shared-a-2619ff/sub-test-model-2619ff-kserve-85b69684cf-p9wpd"
Jul 24 15:51:49.946054 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:49.946017    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2619ff/sub-test-model-2619ff-kserve-85b69684cf-p9wpd"
Jul 24 15:51:50.082386 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:50.082357    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2619ff/sub-test-model-2619ff-kserve-85b69684cf-p9wpd"]
Jul 24 15:51:50.084421 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:51:50.084394    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a4da93a_0b63_45f8_9363_76ac097c1a6a.slice/crio-3375712793ad4e3213618609c7666bcd3c6ca360f7d7389d11f3c0b7bedda74d WatchSource:0}: Error finding container 3375712793ad4e3213618609c7666bcd3c6ca360f7d7389d11f3c0b7bedda74d: Status 404 returned error can't find the container with id 3375712793ad4e3213618609c7666bcd3c6ca360f7d7389d11f3c0b7bedda74d
Jul 24 15:51:50.243553 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:50.243516    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2619ff/sub-test-model-2619ff-kserve-85b69684cf-p9wpd" event={"ID":"5a4da93a-0b63-45f8-9363-76ac097c1a6a","Type":"ContainerStarted","Data":"60521306567db0975d4b9e35849e07ba971b14eb6a0b2d89b1cb150e17d283c0"}
Jul 24 15:51:50.243722 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:50.243565    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2619ff/sub-test-model-2619ff-kserve-85b69684cf-p9wpd" event={"ID":"5a4da93a-0b63-45f8-9363-76ac097c1a6a","Type":"ContainerStarted","Data":"3375712793ad4e3213618609c7666bcd3c6ca360f7d7389d11f3c0b7bedda74d"}
Jul 24 15:51:50.262994 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:50.262933    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-2619ff/sub-test-model-2619ff-kserve-85b69684cf-p9wpd" podStartSLOduration=1.26291719 podStartE2EDuration="1.26291719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:51:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:51:50.261430321 +0000 UTC m=+2603.073782526" watchObservedRunningTime="2026-07-24 15:51:50.26291719 +0000 UTC m=+2603.075269347"
Jul 24 15:51:53.120445 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:53.120407    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:51:59.947017 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:59.946971    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-2619ff/sub-test-model-2619ff-kserve-85b69684cf-p9wpd"
Jul 24 15:51:59.947523 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:59.947036    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-2619ff/sub-test-model-2619ff-kserve-85b69684cf-p9wpd"
Jul 24 15:51:59.959907 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:51:59.959877    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-2619ff/sub-test-model-2619ff-kserve-85b69684cf-p9wpd"
Jul 24 15:52:00.317560 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:52:00.317522    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-2619ff/sub-test-model-2619ff-kserve-85b69684cf-p9wpd"
Jul 24 15:52:15.318375 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:52:15.318341    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:52:30.727246 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:52:30.727208    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:52:51.129650 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:52:51.129612    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:52:55.979148 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:52:55.979105    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2619ff/sub-test-model-2619ff-kserve-85b69684cf-p9wpd"]
Jul 24 15:53:08.018035 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:08.018002    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:53:10.997809 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:10.997767    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-2619ff/sub-test-model-2619ff-kserve-85b69684cf-p9wpd" podUID="5a4da93a-0b63-45f8-9363-76ac097c1a6a" containerName="main" containerID="cri-o://60521306567db0975d4b9e35849e07ba971b14eb6a0b2d89b1cb150e17d283c0" gracePeriod=45
Jul 24 15:53:11.256253 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:11.256190    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2619ff/sub-test-model-2619ff-kserve-85b69684cf-p9wpd"
Jul 24 15:53:11.352287 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:11.352245    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a4da93a-0b63-45f8-9363-76ac097c1a6a-dshm\") pod \"5a4da93a-0b63-45f8-9363-76ac097c1a6a\" (UID: \"5a4da93a-0b63-45f8-9363-76ac097c1a6a\") "
Jul 24 15:53:11.352287 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:11.352287    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a4da93a-0b63-45f8-9363-76ac097c1a6a-home\") pod \"5a4da93a-0b63-45f8-9363-76ac097c1a6a\" (UID: \"5a4da93a-0b63-45f8-9363-76ac097c1a6a\") "
Jul 24 15:53:11.352558 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:11.352326    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9b8j\" (UniqueName: \"kubernetes.io/projected/5a4da93a-0b63-45f8-9363-76ac097c1a6a-kube-api-access-h9b8j\") pod \"5a4da93a-0b63-45f8-9363-76ac097c1a6a\" (UID: \"5a4da93a-0b63-45f8-9363-76ac097c1a6a\") "
Jul 24 15:53:11.352558 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:11.352395    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a4da93a-0b63-45f8-9363-76ac097c1a6a-model-cache\") pod \"5a4da93a-0b63-45f8-9363-76ac097c1a6a\" (UID: \"5a4da93a-0b63-45f8-9363-76ac097c1a6a\") "
Jul 24 15:53:11.352558 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:11.352419    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a4da93a-0b63-45f8-9363-76ac097c1a6a-tmp-dir\") pod \"5a4da93a-0b63-45f8-9363-76ac097c1a6a\" (UID: \"5a4da93a-0b63-45f8-9363-76ac097c1a6a\") "
Jul 24 15:53:11.352558 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:11.352452    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a4da93a-0b63-45f8-9363-76ac097c1a6a-tls-certs\") pod \"5a4da93a-0b63-45f8-9363-76ac097c1a6a\" (UID: \"5a4da93a-0b63-45f8-9363-76ac097c1a6a\") "
Jul 24 15:53:11.352761 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:11.352672    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4da93a-0b63-45f8-9363-76ac097c1a6a-home" (OuterVolumeSpecName: "home") pod "5a4da93a-0b63-45f8-9363-76ac097c1a6a" (UID: "5a4da93a-0b63-45f8-9363-76ac097c1a6a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:53:11.352761 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:11.352677    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4da93a-0b63-45f8-9363-76ac097c1a6a-model-cache" (OuterVolumeSpecName: "model-cache") pod "5a4da93a-0b63-45f8-9363-76ac097c1a6a" (UID: "5a4da93a-0b63-45f8-9363-76ac097c1a6a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:53:11.352761 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:11.352716    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4da93a-0b63-45f8-9363-76ac097c1a6a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5a4da93a-0b63-45f8-9363-76ac097c1a6a" (UID: "5a4da93a-0b63-45f8-9363-76ac097c1a6a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:53:11.352932 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:11.352808    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a4da93a-0b63-45f8-9363-76ac097c1a6a-home\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:53:11.352932 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:11.352853    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a4da93a-0b63-45f8-9363-76ac097c1a6a-model-cache\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:53:11.352932 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:11.352888    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a4da93a-0b63-45f8-9363-76ac097c1a6a-tmp-dir\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:53:11.354562 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:11.354530    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a4da93a-0b63-45f8-9363-76ac097c1a6a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5a4da93a-0b63-45f8-9363-76ac097c1a6a" (UID: "5a4da93a-0b63-45f8-9363-76ac097c1a6a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 15:53:11.354923 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:11.354894    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4da93a-0b63-45f8-9363-76ac097c1a6a-dshm" (OuterVolumeSpecName: "dshm") pod "5a4da93a-0b63-45f8-9363-76ac097c1a6a" (UID: "5a4da93a-0b63-45f8-9363-76ac097c1a6a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:53:11.354923 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:11.354909    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a4da93a-0b63-45f8-9363-76ac097c1a6a-kube-api-access-h9b8j" (OuterVolumeSpecName: "kube-api-access-h9b8j") pod "5a4da93a-0b63-45f8-9363-76ac097c1a6a" (UID: "5a4da93a-0b63-45f8-9363-76ac097c1a6a"). InnerVolumeSpecName "kube-api-access-h9b8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:53:11.453930 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:11.453900    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a4da93a-0b63-45f8-9363-76ac097c1a6a-tls-certs\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:53:11.453930 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:11.453924    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a4da93a-0b63-45f8-9363-76ac097c1a6a-dshm\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:53:11.453930 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:11.453934    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9b8j\" (UniqueName: \"kubernetes.io/projected/5a4da93a-0b63-45f8-9363-76ac097c1a6a-kube-api-access-h9b8j\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:53:11.655412 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:11.655375    2579 generic.go:356] "Generic (PLEG): container finished" podID="5a4da93a-0b63-45f8-9363-76ac097c1a6a" containerID="60521306567db0975d4b9e35849e07ba971b14eb6a0b2d89b1cb150e17d283c0" exitCode=0
Jul 24 15:53:11.655596 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:11.655417    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2619ff/sub-test-model-2619ff-kserve-85b69684cf-p9wpd" event={"ID":"5a4da93a-0b63-45f8-9363-76ac097c1a6a","Type":"ContainerDied","Data":"60521306567db0975d4b9e35849e07ba971b14eb6a0b2d89b1cb150e17d283c0"}
Jul 24 15:53:11.655596 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:11.655450    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2619ff/sub-test-model-2619ff-kserve-85b69684cf-p9wpd" event={"ID":"5a4da93a-0b63-45f8-9363-76ac097c1a6a","Type":"ContainerDied","Data":"3375712793ad4e3213618609c7666bcd3c6ca360f7d7389d11f3c0b7bedda74d"}
Jul 24 15:53:11.655596 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:11.655468    2579 scope.go:117] "RemoveContainer" containerID="60521306567db0975d4b9e35849e07ba971b14eb6a0b2d89b1cb150e17d283c0"
Jul 24 15:53:11.655596 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:11.655419    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2619ff/sub-test-model-2619ff-kserve-85b69684cf-p9wpd"
Jul 24 15:53:11.664208 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:11.664191    2579 scope.go:117] "RemoveContainer" containerID="60521306567db0975d4b9e35849e07ba971b14eb6a0b2d89b1cb150e17d283c0"
Jul 24 15:53:11.664462 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:53:11.664446    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60521306567db0975d4b9e35849e07ba971b14eb6a0b2d89b1cb150e17d283c0\": container with ID starting with 60521306567db0975d4b9e35849e07ba971b14eb6a0b2d89b1cb150e17d283c0 not found: ID does not exist" containerID="60521306567db0975d4b9e35849e07ba971b14eb6a0b2d89b1cb150e17d283c0"
Jul 24 15:53:11.664504 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:11.664470    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60521306567db0975d4b9e35849e07ba971b14eb6a0b2d89b1cb150e17d283c0"} err="failed to get container status \"60521306567db0975d4b9e35849e07ba971b14eb6a0b2d89b1cb150e17d283c0\": rpc error: code = NotFound desc = could not find container \"60521306567db0975d4b9e35849e07ba971b14eb6a0b2d89b1cb150e17d283c0\": container with ID starting with 60521306567db0975d4b9e35849e07ba971b14eb6a0b2d89b1cb150e17d283c0 not found: ID does not exist"
Jul 24 15:53:11.681402 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:11.681376    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2619ff/sub-test-model-2619ff-kserve-85b69684cf-p9wpd"]
Jul 24 15:53:11.684622 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:11.684599    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-2619ff/sub-test-model-2619ff-kserve-85b69684cf-p9wpd"]
Jul 24 15:53:11.717906 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:11.717877    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a4da93a-0b63-45f8-9363-76ac097c1a6a" path="/var/lib/kubelet/pods/5a4da93a-0b63-45f8-9363-76ac097c1a6a/volumes"
Jul 24 15:53:17.625369 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:17.625325    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:53:43.823861 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:43.823794    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-1578f9/rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd"]
Jul 24 15:53:43.824381 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:43.824364    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a4da93a-0b63-45f8-9363-76ac097c1a6a" containerName="main"
Jul 24 15:53:43.824433 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:43.824383    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a4da93a-0b63-45f8-9363-76ac097c1a6a" containerName="main"
Jul 24 15:53:43.824471 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:43.824445    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a4da93a-0b63-45f8-9363-76ac097c1a6a" containerName="main"
Jul 24 15:53:43.827498 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:43.827480    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1578f9/rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd"
Jul 24 15:53:43.830575 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:43.830551    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1578f9\"/\"default-dockercfg-rb2tp\""
Jul 24 15:53:43.830711 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:43.830575    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1578f9\"/\"kube-root-ca.crt\""
Jul 24 15:53:43.830711 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:43.830575    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1578f9\"/\"openshift-service-ca.crt\""
Jul 24 15:53:43.830711 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:43.830576    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1578f9\"/\"rate-test-model-1578f9-kserve-self-signed-certs\""
Jul 24 15:53:43.836481 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:43.836461    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1578f9/rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd"]
Jul 24 15:53:43.959571 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:43.959535    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c-tls-certs\") pod \"rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd\" (UID: \"f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c\") " pod="ai-tenant-e2e-shared-b-1578f9/rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd"
Jul 24 15:53:43.959571 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:43.959574    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c-home\") pod \"rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd\" (UID: \"f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c\") " pod="ai-tenant-e2e-shared-b-1578f9/rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd"
Jul 24 15:53:43.959795 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:43.959646    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c-model-cache\") pod \"rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd\" (UID: \"f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c\") " pod="ai-tenant-e2e-shared-b-1578f9/rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd"
Jul 24 15:53:43.959795 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:43.959707    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j5wh\" (UniqueName: \"kubernetes.io/projected/f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c-kube-api-access-7j5wh\") pod \"rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd\" (UID: \"f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c\") " pod="ai-tenant-e2e-shared-b-1578f9/rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd"
Jul 24 15:53:43.959795 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:43.959736    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c-dshm\") pod \"rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd\" (UID: \"f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c\") " pod="ai-tenant-e2e-shared-b-1578f9/rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd"
Jul 24 15:53:43.959936 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:43.959892    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c-tmp-dir\") pod \"rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd\" (UID: \"f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c\") " pod="ai-tenant-e2e-shared-b-1578f9/rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd"
Jul 24 15:53:44.061246 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:44.061212    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c-model-cache\") pod \"rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd\" (UID: \"f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c\") " pod="ai-tenant-e2e-shared-b-1578f9/rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd"
Jul 24 15:53:44.061428 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:44.061261    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j5wh\" (UniqueName: \"kubernetes.io/projected/f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c-kube-api-access-7j5wh\") pod \"rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd\" (UID: \"f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c\") " pod="ai-tenant-e2e-shared-b-1578f9/rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd"
Jul 24 15:53:44.061428 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:44.061371    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c-dshm\") pod \"rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd\" (UID: \"f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c\") " pod="ai-tenant-e2e-shared-b-1578f9/rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd"
Jul 24 15:53:44.061502 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:44.061468    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c-tmp-dir\") pod \"rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd\" (UID: \"f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c\") " pod="ai-tenant-e2e-shared-b-1578f9/rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd"
Jul 24 15:53:44.061502 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:44.061499    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c-tls-certs\") pod \"rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd\" (UID: \"f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c\") " pod="ai-tenant-e2e-shared-b-1578f9/rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd"
Jul 24 15:53:44.061580 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:44.061520    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c-home\") pod \"rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd\" (UID: \"f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c\") " pod="ai-tenant-e2e-shared-b-1578f9/rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd"
Jul 24 15:53:44.061636 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:44.061614    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c-model-cache\") pod \"rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd\" (UID: \"f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c\") " pod="ai-tenant-e2e-shared-b-1578f9/rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd"
Jul 24 15:53:44.061804 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:44.061779    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c-tmp-dir\") pod \"rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd\" (UID: \"f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c\") " pod="ai-tenant-e2e-shared-b-1578f9/rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd"
Jul 24 15:53:44.061804 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:44.061794    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c-home\") pod \"rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd\" (UID: \"f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c\") " pod="ai-tenant-e2e-shared-b-1578f9/rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd"
Jul 24 15:53:44.063632 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:44.063614    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c-dshm\") pod \"rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd\" (UID: \"f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c\") " pod="ai-tenant-e2e-shared-b-1578f9/rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd"
Jul 24 15:53:44.063906 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:44.063890    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c-tls-certs\") pod \"rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd\" (UID: \"f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c\") " pod="ai-tenant-e2e-shared-b-1578f9/rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd"
Jul 24 15:53:44.069176 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:44.069152    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j5wh\" (UniqueName: \"kubernetes.io/projected/f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c-kube-api-access-7j5wh\") pod \"rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd\" (UID: \"f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c\") " pod="ai-tenant-e2e-shared-b-1578f9/rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd"
Jul 24 15:53:44.138031 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:44.137943    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1578f9/rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd"
Jul 24 15:53:44.269894 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:44.269867    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1578f9/rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd"]
Jul 24 15:53:44.271926 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:53:44.271898    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3079b47_d6cc_43ea_bcb0_6c89f4adaa0c.slice/crio-e333d53433ce7837dfa0b8aa52d1ab6b705d7cd853e5faddcd5a2499b67c1231 WatchSource:0}: Error finding container e333d53433ce7837dfa0b8aa52d1ab6b705d7cd853e5faddcd5a2499b67c1231: Status 404 returned error can't find the container with id e333d53433ce7837dfa0b8aa52d1ab6b705d7cd853e5faddcd5a2499b67c1231
Jul 24 15:53:44.824683 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:44.824648    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1578f9/rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd" event={"ID":"f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c","Type":"ContainerStarted","Data":"334bd6d28cc8101aebc8c7e0056c44241118197a78d8667d40081f741f9a761f"}
Jul 24 15:53:44.824683 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:44.824690    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1578f9/rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd" event={"ID":"f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c","Type":"ContainerStarted","Data":"e333d53433ce7837dfa0b8aa52d1ab6b705d7cd853e5faddcd5a2499b67c1231"}
Jul 24 15:53:44.845230 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:44.845163    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-1578f9/rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd" podStartSLOduration=1.8451449690000001 podStartE2EDuration="1.845144969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:53:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:53:44.843624405 +0000 UTC m=+2717.655976584" watchObservedRunningTime="2026-07-24 15:53:44.845144969 +0000 UTC m=+2717.657497140"
Jul 24 15:53:47.828399 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:47.828361    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:53:54.138301 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:54.138262    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-1578f9/rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd"
Jul 24 15:53:54.138725 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:54.138344    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-1578f9/rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd"
Jul 24 15:53:54.150898 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:54.150870    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-1578f9/rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd"
Jul 24 15:53:54.885980 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:53:54.885948    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-1578f9/rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd"
Jul 24 15:54:03.823941 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:03.823902    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:54:18.523244 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:18.523210    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:54:27.009765 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:27.009734    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1578f9/rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd"]
Jul 24 15:54:32.923438 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:32.923398    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:54:42.029430 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:42.029381    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-1578f9/rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd" podUID="f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c" containerName="main" containerID="cri-o://334bd6d28cc8101aebc8c7e0056c44241118197a78d8667d40081f741f9a761f" gracePeriod=45
Jul 24 15:54:42.289484 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:42.289414    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1578f9/rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd"
Jul 24 15:54:42.409848 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:42.409783    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7j5wh\" (UniqueName: \"kubernetes.io/projected/f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c-kube-api-access-7j5wh\") pod \"f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c\" (UID: \"f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c\") "
Jul 24 15:54:42.410028 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:42.409903    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c-home\") pod \"f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c\" (UID: \"f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c\") "
Jul 24 15:54:42.410028 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:42.409940    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c-tmp-dir\") pod \"f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c\" (UID: \"f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c\") "
Jul 24 15:54:42.410028 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:42.410013    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c-model-cache\") pod \"f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c\" (UID: \"f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c\") "
Jul 24 15:54:42.410196 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:42.410049    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c-tls-certs\") pod \"f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c\" (UID: \"f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c\") "
Jul 24 15:54:42.410196 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:42.410133    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c-dshm\") pod \"f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c\" (UID: \"f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c\") "
Jul 24 15:54:42.410310 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:42.410196    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c-home" (OuterVolumeSpecName: "home") pod "f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c" (UID: "f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:54:42.410310 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:42.410246    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c" (UID: "f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:54:42.410310 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:42.410288    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c-model-cache" (OuterVolumeSpecName: "model-cache") pod "f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c" (UID: "f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:54:42.410540 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:42.410522    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c-home\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:54:42.410603 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:42.410546    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c-tmp-dir\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:54:42.410603 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:42.410564    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c-model-cache\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:54:42.412184 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:42.412147    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c-dshm" (OuterVolumeSpecName: "dshm") pod "f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c" (UID: "f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:54:42.412302 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:42.412233    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c-kube-api-access-7j5wh" (OuterVolumeSpecName: "kube-api-access-7j5wh") pod "f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c" (UID: "f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c"). InnerVolumeSpecName "kube-api-access-7j5wh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:54:42.412302 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:42.412236    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c" (UID: "f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 15:54:42.511744 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:42.511702    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c-tls-certs\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:54:42.511744 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:42.511735    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c-dshm\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:54:42.511744 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:42.511743    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j5wh\" (UniqueName: \"kubernetes.io/projected/f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c-kube-api-access-7j5wh\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:54:43.113128 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:43.113093    2579 generic.go:356] "Generic (PLEG): container finished" podID="f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c" containerID="334bd6d28cc8101aebc8c7e0056c44241118197a78d8667d40081f741f9a761f" exitCode=0
Jul 24 15:54:43.113592 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:43.113139    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1578f9/rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd"
Jul 24 15:54:43.113592 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:43.113166    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1578f9/rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd" event={"ID":"f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c","Type":"ContainerDied","Data":"334bd6d28cc8101aebc8c7e0056c44241118197a78d8667d40081f741f9a761f"}
Jul 24 15:54:43.113592 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:43.113198    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1578f9/rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd" event={"ID":"f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c","Type":"ContainerDied","Data":"e333d53433ce7837dfa0b8aa52d1ab6b705d7cd853e5faddcd5a2499b67c1231"}
Jul 24 15:54:43.113592 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:43.113213    2579 scope.go:117] "RemoveContainer" containerID="334bd6d28cc8101aebc8c7e0056c44241118197a78d8667d40081f741f9a761f"
Jul 24 15:54:43.122587 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:43.122570    2579 scope.go:117] "RemoveContainer" containerID="334bd6d28cc8101aebc8c7e0056c44241118197a78d8667d40081f741f9a761f"
Jul 24 15:54:43.122871 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:54:43.122850    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"334bd6d28cc8101aebc8c7e0056c44241118197a78d8667d40081f741f9a761f\": container with ID starting with 334bd6d28cc8101aebc8c7e0056c44241118197a78d8667d40081f741f9a761f not found: ID does not exist" containerID="334bd6d28cc8101aebc8c7e0056c44241118197a78d8667d40081f741f9a761f"
Jul 24 15:54:43.122947 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:43.122877    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"334bd6d28cc8101aebc8c7e0056c44241118197a78d8667d40081f741f9a761f"} err="failed to get container status \"334bd6d28cc8101aebc8c7e0056c44241118197a78d8667d40081f741f9a761f\": rpc error: code = NotFound desc = could not find container \"334bd6d28cc8101aebc8c7e0056c44241118197a78d8667d40081f741f9a761f\": container with ID starting with 334bd6d28cc8101aebc8c7e0056c44241118197a78d8667d40081f741f9a761f not found: ID does not exist"
Jul 24 15:54:43.135592 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:43.135564    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1578f9/rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd"]
Jul 24 15:54:43.140022 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:43.139998    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-1578f9/rate-test-model-1578f9-kserve-7fcb9cf5c5-s8wmd"]
Jul 24 15:54:43.717795 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:43.717750    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c" path="/var/lib/kubelet/pods/f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c/volumes"
Jul 24 15:54:51.029670 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:51.029580    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a64ba4-6c48d9846-qgs67"]
Jul 24 15:54:51.032340 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:51.030233    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c" containerName="main"
Jul 24 15:54:51.032340 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:51.030254    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c" containerName="main"
Jul 24 15:54:51.032340 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:51.030342    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3079b47-d6cc-43ea-bcb0-6c89f4adaa0c" containerName="main"
Jul 24 15:54:51.033619 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:51.033600    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a64ba4-6c48d9846-qgs67"
Jul 24 15:54:51.035678 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:51.035657    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a64ba4\""
Jul 24 15:54:51.035874 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:51.035781    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a64ba4-dockercfg-ttnc5\""
Jul 24 15:54:51.048930 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:51.048903    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a64ba4-6c48d9846-qgs67"]
Jul 24 15:54:51.095424 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:51.095381    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19cffcc0-896f-436f-a962-b6291852f1b3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a64ba4-6c48d9846-qgs67\" (UID: \"19cffcc0-896f-436f-a962-b6291852f1b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a64ba4-6c48d9846-qgs67"
Jul 24 15:54:51.095606 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:51.095454    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzzws\" (UniqueName: \"kubernetes.io/projected/19cffcc0-896f-436f-a962-b6291852f1b3-kube-api-access-vzzws\") pod \"payload-processing-e2e-ipp-a-a64ba4-6c48d9846-qgs67\" (UID: \"19cffcc0-896f-436f-a962-b6291852f1b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a64ba4-6c48d9846-qgs67"
Jul 24 15:54:51.196200 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:51.196157    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19cffcc0-896f-436f-a962-b6291852f1b3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a64ba4-6c48d9846-qgs67\" (UID: \"19cffcc0-896f-436f-a962-b6291852f1b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a64ba4-6c48d9846-qgs67"
Jul 24 15:54:51.196406 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:51.196212    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzzws\" (UniqueName: \"kubernetes.io/projected/19cffcc0-896f-436f-a962-b6291852f1b3-kube-api-access-vzzws\") pod \"payload-processing-e2e-ipp-a-a64ba4-6c48d9846-qgs67\" (UID: \"19cffcc0-896f-436f-a962-b6291852f1b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a64ba4-6c48d9846-qgs67"
Jul 24 15:54:51.196915 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:51.196892    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19cffcc0-896f-436f-a962-b6291852f1b3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a64ba4-6c48d9846-qgs67\" (UID: \"19cffcc0-896f-436f-a962-b6291852f1b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a64ba4-6c48d9846-qgs67"
Jul 24 15:54:51.207734 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:51.207701    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzzws\" (UniqueName: \"kubernetes.io/projected/19cffcc0-896f-436f-a962-b6291852f1b3-kube-api-access-vzzws\") pod \"payload-processing-e2e-ipp-a-a64ba4-6c48d9846-qgs67\" (UID: \"19cffcc0-896f-436f-a962-b6291852f1b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a64ba4-6c48d9846-qgs67"
Jul 24 15:54:51.345683 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:51.345582    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a64ba4-6c48d9846-qgs67"
Jul 24 15:54:51.483336 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:51.483309    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a64ba4-6c48d9846-qgs67"]
Jul 24 15:54:51.485054 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:54:51.485030    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19cffcc0_896f_436f_a962_b6291852f1b3.slice/crio-6f6af35ca404a37380742f96993269c1bceecdba81fc60b8dd715b502ef397d0 WatchSource:0}: Error finding container 6f6af35ca404a37380742f96993269c1bceecdba81fc60b8dd715b502ef397d0: Status 404 returned error can't find the container with id 6f6af35ca404a37380742f96993269c1bceecdba81fc60b8dd715b502ef397d0
Jul 24 15:54:52.160977 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:52.160936    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a64ba4-6c48d9846-qgs67" event={"ID":"19cffcc0-896f-436f-a962-b6291852f1b3","Type":"ContainerStarted","Data":"fa8f10d965d137562c78a0cfc505b8ce668c8cccf080800e88fbb0d6de1a45c1"}
Jul 24 15:54:52.161490 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:52.160991    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a64ba4-6c48d9846-qgs67" event={"ID":"19cffcc0-896f-436f-a962-b6291852f1b3","Type":"ContainerStarted","Data":"6f6af35ca404a37380742f96993269c1bceecdba81fc60b8dd715b502ef397d0"}
Jul 24 15:54:52.161490 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:52.161031    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a64ba4-6c48d9846-qgs67"
Jul 24 15:54:52.178707 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:52.178650    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a64ba4-6c48d9846-qgs67" podStartSLOduration=1.178635331 podStartE2EDuration="1.178635331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:54:52.177701704 +0000 UTC m=+2784.990053892" watchObservedRunningTime="2026-07-24 15:54:52.178635331 +0000 UTC m=+2784.990987552"
Jul 24 15:54:56.631320 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:54:56.631284    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:55:03.167730 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:03.167696    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a64ba4-6c48d9846-qgs67"
Jul 24 15:55:09.962639 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:09.962602    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4dda13-5fff879bb4-ndshl"]
Jul 24 15:55:09.966631 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:09.966608    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4dda13-5fff879bb4-ndshl"
Jul 24 15:55:09.969069 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:09.969043    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-4dda13-dockercfg-tbj7z\""
Jul 24 15:55:09.969177 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:09.969048    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-4dda13\""
Jul 24 15:55:09.972715 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:09.972694    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4dda13-7b9796cb6b-w8fgx"]
Jul 24 15:55:09.978604 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:09.978582    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4dda13-5fff879bb4-ndshl"]
Jul 24 15:55:09.978711 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:09.978670    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4dda13-7b9796cb6b-w8fgx"
Jul 24 15:55:09.990522 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:09.990496    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4dda13-7b9796cb6b-w8fgx"]
Jul 24 15:55:10.082944 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:10.082911    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m47cv\" (UniqueName: \"kubernetes.io/projected/a3c304a1-65d0-4a80-8868-90d5471ecf4b-kube-api-access-m47cv\") pod \"payload-processing-e2e-ipp-b-4dda13-7b9796cb6b-w8fgx\" (UID: \"a3c304a1-65d0-4a80-8868-90d5471ecf4b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4dda13-7b9796cb6b-w8fgx"
Jul 24 15:55:10.083125 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:10.082973    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3c304a1-65d0-4a80-8868-90d5471ecf4b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4dda13-7b9796cb6b-w8fgx\" (UID: \"a3c304a1-65d0-4a80-8868-90d5471ecf4b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4dda13-7b9796cb6b-w8fgx"
Jul 24 15:55:10.083125 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:10.083009    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkvh7\" (UniqueName: \"kubernetes.io/projected/8cd34332-3dd4-4350-9fc0-26f1fee180f0-kube-api-access-jkvh7\") pod \"payload-pre-processing-e2e-ipp-b-4dda13-5fff879bb4-ndshl\" (UID: \"8cd34332-3dd4-4350-9fc0-26f1fee180f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4dda13-5fff879bb4-ndshl"
Jul 24 15:55:10.083240 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:10.083125    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cd34332-3dd4-4350-9fc0-26f1fee180f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4dda13-5fff879bb4-ndshl\" (UID: \"8cd34332-3dd4-4350-9fc0-26f1fee180f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4dda13-5fff879bb4-ndshl"
Jul 24 15:55:10.184026 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:10.183984    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m47cv\" (UniqueName: \"kubernetes.io/projected/a3c304a1-65d0-4a80-8868-90d5471ecf4b-kube-api-access-m47cv\") pod \"payload-processing-e2e-ipp-b-4dda13-7b9796cb6b-w8fgx\" (UID: \"a3c304a1-65d0-4a80-8868-90d5471ecf4b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4dda13-7b9796cb6b-w8fgx"
Jul 24 15:55:10.184211 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:10.184069    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3c304a1-65d0-4a80-8868-90d5471ecf4b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4dda13-7b9796cb6b-w8fgx\" (UID: \"a3c304a1-65d0-4a80-8868-90d5471ecf4b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4dda13-7b9796cb6b-w8fgx"
Jul 24 15:55:10.184211 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:10.184099    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkvh7\" (UniqueName: \"kubernetes.io/projected/8cd34332-3dd4-4350-9fc0-26f1fee180f0-kube-api-access-jkvh7\") pod \"payload-pre-processing-e2e-ipp-b-4dda13-5fff879bb4-ndshl\" (UID: \"8cd34332-3dd4-4350-9fc0-26f1fee180f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4dda13-5fff879bb4-ndshl"
Jul 24 15:55:10.184211 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:10.184189    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cd34332-3dd4-4350-9fc0-26f1fee180f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4dda13-5fff879bb4-ndshl\" (UID: \"8cd34332-3dd4-4350-9fc0-26f1fee180f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4dda13-5fff879bb4-ndshl"
Jul 24 15:55:10.184918 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:10.184893    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cd34332-3dd4-4350-9fc0-26f1fee180f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4dda13-5fff879bb4-ndshl\" (UID: \"8cd34332-3dd4-4350-9fc0-26f1fee180f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4dda13-5fff879bb4-ndshl"
Jul 24 15:55:10.185012 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:10.184897    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3c304a1-65d0-4a80-8868-90d5471ecf4b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4dda13-7b9796cb6b-w8fgx\" (UID: \"a3c304a1-65d0-4a80-8868-90d5471ecf4b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4dda13-7b9796cb6b-w8fgx"
Jul 24 15:55:10.192501 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:10.192478    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m47cv\" (UniqueName: \"kubernetes.io/projected/a3c304a1-65d0-4a80-8868-90d5471ecf4b-kube-api-access-m47cv\") pod \"payload-processing-e2e-ipp-b-4dda13-7b9796cb6b-w8fgx\" (UID: \"a3c304a1-65d0-4a80-8868-90d5471ecf4b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4dda13-7b9796cb6b-w8fgx"
Jul 24 15:55:10.192600 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:10.192509    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkvh7\" (UniqueName: \"kubernetes.io/projected/8cd34332-3dd4-4350-9fc0-26f1fee180f0-kube-api-access-jkvh7\") pod \"payload-pre-processing-e2e-ipp-b-4dda13-5fff879bb4-ndshl\" (UID: \"8cd34332-3dd4-4350-9fc0-26f1fee180f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4dda13-5fff879bb4-ndshl"
Jul 24 15:55:10.278048 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:10.278014    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4dda13-5fff879bb4-ndshl"
Jul 24 15:55:10.290350 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:10.290328    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4dda13-7b9796cb6b-w8fgx"
Jul 24 15:55:10.460974 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:10.460947    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4dda13-7b9796cb6b-w8fgx"]
Jul 24 15:55:10.463035 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:55:10.463008    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3c304a1_65d0_4a80_8868_90d5471ecf4b.slice/crio-b1536eb4d48ab9eb9cccf720b5762d10366ef4235e5f79b0c9558467b9bc3bb7 WatchSource:0}: Error finding container b1536eb4d48ab9eb9cccf720b5762d10366ef4235e5f79b0c9558467b9bc3bb7: Status 404 returned error can't find the container with id b1536eb4d48ab9eb9cccf720b5762d10366ef4235e5f79b0c9558467b9bc3bb7
Jul 24 15:55:10.628223 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:10.628195    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4dda13-5fff879bb4-ndshl"]
Jul 24 15:55:10.632140 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:55:10.632114    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cd34332_3dd4_4350_9fc0_26f1fee180f0.slice/crio-280e1fc5c4f02988655f1364b1302a6c91c9ef67144a5063b7d1953c32436c07 WatchSource:0}: Error finding container 280e1fc5c4f02988655f1364b1302a6c91c9ef67144a5063b7d1953c32436c07: Status 404 returned error can't find the container with id 280e1fc5c4f02988655f1364b1302a6c91c9ef67144a5063b7d1953c32436c07
Jul 24 15:55:11.260865 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:11.260824    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4dda13-5fff879bb4-ndshl" event={"ID":"8cd34332-3dd4-4350-9fc0-26f1fee180f0","Type":"ContainerStarted","Data":"2dc98b6a39483d6b28bbb6cd79baf4603cb8091135e6689e7d69e30911efb7ca"}
Jul 24 15:55:11.261363 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:11.260884    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4dda13-5fff879bb4-ndshl" event={"ID":"8cd34332-3dd4-4350-9fc0-26f1fee180f0","Type":"ContainerStarted","Data":"280e1fc5c4f02988655f1364b1302a6c91c9ef67144a5063b7d1953c32436c07"}
Jul 24 15:55:11.261363 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:11.260915    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4dda13-5fff879bb4-ndshl"
Jul 24 15:55:11.262365 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:11.262338    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4dda13-7b9796cb6b-w8fgx" event={"ID":"a3c304a1-65d0-4a80-8868-90d5471ecf4b","Type":"ContainerStarted","Data":"801b2da9fc66a38995cfedd10ab5ec03e51e454cdff17cf5b383de1c95d77e99"}
Jul 24 15:55:11.262482 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:11.262386    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4dda13-7b9796cb6b-w8fgx"
Jul 24 15:55:11.262482 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:11.262403    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4dda13-7b9796cb6b-w8fgx" event={"ID":"a3c304a1-65d0-4a80-8868-90d5471ecf4b","Type":"ContainerStarted","Data":"b1536eb4d48ab9eb9cccf720b5762d10366ef4235e5f79b0c9558467b9bc3bb7"}
Jul 24 15:55:11.278003 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:11.277952    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4dda13-5fff879bb4-ndshl" podStartSLOduration=2.277935156 podStartE2EDuration="2.277935156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:55:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:55:11.27693319 +0000 UTC m=+2804.089285369" watchObservedRunningTime="2026-07-24 15:55:11.277935156 +0000 UTC m=+2804.090287359"
Jul 24 15:55:11.293397 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:11.293356    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-4dda13-7b9796cb6b-w8fgx" podStartSLOduration=2.293344084 podStartE2EDuration="2.293344084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:55:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:55:11.291684639 +0000 UTC m=+2804.104036818" watchObservedRunningTime="2026-07-24 15:55:11.293344084 +0000 UTC m=+2804.105696240"
Jul 24 15:55:18.420989 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:18.420944    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:55:22.269380 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:22.269349    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4dda13-7b9796cb6b-w8fgx"
Jul 24 15:55:22.269786 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:22.269419    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4dda13-5fff879bb4-ndshl"
Jul 24 15:55:30.755356 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:30.755324    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-4dda13-7b9796cb6b-w8fgx_a3c304a1-65d0-4a80-8868-90d5471ecf4b/payload-processing/0.log"
Jul 24 15:55:53.025676 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:53.025641    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-a64ba4/ipp-route-a64ba4-kserve-5d56955d7-pfj8s"]
Jul 24 15:55:53.029373 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:53.029352    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a64ba4/ipp-route-a64ba4-kserve-5d56955d7-pfj8s"
Jul 24 15:55:53.031830 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:53.031786    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-a64ba4\"/\"openshift-service-ca.crt\""
Jul 24 15:55:53.031959 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:53.031788    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-a64ba4\"/\"kube-root-ca.crt\""
Jul 24 15:55:53.032653 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:53.032634    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-a64ba4\"/\"ipp-route-a64ba4-kserve-self-signed-certs\""
Jul 24 15:55:53.032752 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:53.032633    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-a64ba4\"/\"default-dockercfg-fjrkn\""
Jul 24 15:55:53.039452 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:53.039426    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-a64ba4/ipp-route-a64ba4-kserve-5d56955d7-pfj8s"]
Jul 24 15:55:53.099915 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:53.099877    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/426663e4-70cb-44b7-a7cb-444ecdbae049-tls-certs\") pod \"ipp-route-a64ba4-kserve-5d56955d7-pfj8s\" (UID: \"426663e4-70cb-44b7-a7cb-444ecdbae049\") " pod="ai-tenant-e2e-ipp-a-a64ba4/ipp-route-a64ba4-kserve-5d56955d7-pfj8s"
Jul 24 15:55:53.100096 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:53.099946    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/426663e4-70cb-44b7-a7cb-444ecdbae049-model-cache\") pod \"ipp-route-a64ba4-kserve-5d56955d7-pfj8s\" (UID: \"426663e4-70cb-44b7-a7cb-444ecdbae049\") " pod="ai-tenant-e2e-ipp-a-a64ba4/ipp-route-a64ba4-kserve-5d56955d7-pfj8s"
Jul 24 15:55:53.100096 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:53.099964    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/426663e4-70cb-44b7-a7cb-444ecdbae049-tmp-dir\") pod \"ipp-route-a64ba4-kserve-5d56955d7-pfj8s\" (UID: \"426663e4-70cb-44b7-a7cb-444ecdbae049\") " pod="ai-tenant-e2e-ipp-a-a64ba4/ipp-route-a64ba4-kserve-5d56955d7-pfj8s"
Jul 24 15:55:53.100096 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:53.099995    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/426663e4-70cb-44b7-a7cb-444ecdbae049-dshm\") pod \"ipp-route-a64ba4-kserve-5d56955d7-pfj8s\" (UID: \"426663e4-70cb-44b7-a7cb-444ecdbae049\") " pod="ai-tenant-e2e-ipp-a-a64ba4/ipp-route-a64ba4-kserve-5d56955d7-pfj8s"
Jul 24 15:55:53.100096 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:53.100013    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bckhp\" (UniqueName: \"kubernetes.io/projected/426663e4-70cb-44b7-a7cb-444ecdbae049-kube-api-access-bckhp\") pod \"ipp-route-a64ba4-kserve-5d56955d7-pfj8s\" (UID: \"426663e4-70cb-44b7-a7cb-444ecdbae049\") " pod="ai-tenant-e2e-ipp-a-a64ba4/ipp-route-a64ba4-kserve-5d56955d7-pfj8s"
Jul 24 15:55:53.100237 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:53.100085    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/426663e4-70cb-44b7-a7cb-444ecdbae049-home\") pod \"ipp-route-a64ba4-kserve-5d56955d7-pfj8s\" (UID: \"426663e4-70cb-44b7-a7cb-444ecdbae049\") " pod="ai-tenant-e2e-ipp-a-a64ba4/ipp-route-a64ba4-kserve-5d56955d7-pfj8s"
Jul 24 15:55:53.201025 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:53.200988    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/426663e4-70cb-44b7-a7cb-444ecdbae049-dshm\") pod \"ipp-route-a64ba4-kserve-5d56955d7-pfj8s\" (UID: \"426663e4-70cb-44b7-a7cb-444ecdbae049\") " pod="ai-tenant-e2e-ipp-a-a64ba4/ipp-route-a64ba4-kserve-5d56955d7-pfj8s"
Jul 24 15:55:53.201025 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:53.201028    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bckhp\" (UniqueName: \"kubernetes.io/projected/426663e4-70cb-44b7-a7cb-444ecdbae049-kube-api-access-bckhp\") pod \"ipp-route-a64ba4-kserve-5d56955d7-pfj8s\" (UID: \"426663e4-70cb-44b7-a7cb-444ecdbae049\") " pod="ai-tenant-e2e-ipp-a-a64ba4/ipp-route-a64ba4-kserve-5d56955d7-pfj8s"
Jul 24 15:55:53.201284 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:53.201058    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/426663e4-70cb-44b7-a7cb-444ecdbae049-home\") pod \"ipp-route-a64ba4-kserve-5d56955d7-pfj8s\" (UID: \"426663e4-70cb-44b7-a7cb-444ecdbae049\") " pod="ai-tenant-e2e-ipp-a-a64ba4/ipp-route-a64ba4-kserve-5d56955d7-pfj8s"
Jul 24 15:55:53.201284 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:53.201109    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/426663e4-70cb-44b7-a7cb-444ecdbae049-tls-certs\") pod \"ipp-route-a64ba4-kserve-5d56955d7-pfj8s\" (UID: \"426663e4-70cb-44b7-a7cb-444ecdbae049\") " pod="ai-tenant-e2e-ipp-a-a64ba4/ipp-route-a64ba4-kserve-5d56955d7-pfj8s"
Jul 24 15:55:53.201284 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:53.201157    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/426663e4-70cb-44b7-a7cb-444ecdbae049-model-cache\") pod \"ipp-route-a64ba4-kserve-5d56955d7-pfj8s\" (UID: \"426663e4-70cb-44b7-a7cb-444ecdbae049\") " pod="ai-tenant-e2e-ipp-a-a64ba4/ipp-route-a64ba4-kserve-5d56955d7-pfj8s"
Jul 24 15:55:53.201284 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:53.201172    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/426663e4-70cb-44b7-a7cb-444ecdbae049-tmp-dir\") pod \"ipp-route-a64ba4-kserve-5d56955d7-pfj8s\" (UID: \"426663e4-70cb-44b7-a7cb-444ecdbae049\") " pod="ai-tenant-e2e-ipp-a-a64ba4/ipp-route-a64ba4-kserve-5d56955d7-pfj8s"
Jul 24 15:55:53.201585 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:53.201558    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/426663e4-70cb-44b7-a7cb-444ecdbae049-home\") pod \"ipp-route-a64ba4-kserve-5d56955d7-pfj8s\" (UID: \"426663e4-70cb-44b7-a7cb-444ecdbae049\") " pod="ai-tenant-e2e-ipp-a-a64ba4/ipp-route-a64ba4-kserve-5d56955d7-pfj8s"
Jul 24 15:55:53.201585 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:53.201574    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/426663e4-70cb-44b7-a7cb-444ecdbae049-model-cache\") pod \"ipp-route-a64ba4-kserve-5d56955d7-pfj8s\" (UID: \"426663e4-70cb-44b7-a7cb-444ecdbae049\") " pod="ai-tenant-e2e-ipp-a-a64ba4/ipp-route-a64ba4-kserve-5d56955d7-pfj8s"
Jul 24 15:55:53.201774 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:53.201605    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/426663e4-70cb-44b7-a7cb-444ecdbae049-tmp-dir\") pod \"ipp-route-a64ba4-kserve-5d56955d7-pfj8s\" (UID: \"426663e4-70cb-44b7-a7cb-444ecdbae049\") " pod="ai-tenant-e2e-ipp-a-a64ba4/ipp-route-a64ba4-kserve-5d56955d7-pfj8s"
Jul 24 15:55:53.203414 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:53.203389    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/426663e4-70cb-44b7-a7cb-444ecdbae049-dshm\") pod \"ipp-route-a64ba4-kserve-5d56955d7-pfj8s\" (UID: \"426663e4-70cb-44b7-a7cb-444ecdbae049\") " pod="ai-tenant-e2e-ipp-a-a64ba4/ipp-route-a64ba4-kserve-5d56955d7-pfj8s"
Jul 24 15:55:53.203556 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:53.203540    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/426663e4-70cb-44b7-a7cb-444ecdbae049-tls-certs\") pod \"ipp-route-a64ba4-kserve-5d56955d7-pfj8s\" (UID: \"426663e4-70cb-44b7-a7cb-444ecdbae049\") " pod="ai-tenant-e2e-ipp-a-a64ba4/ipp-route-a64ba4-kserve-5d56955d7-pfj8s"
Jul 24 15:55:53.209127 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:53.209102    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bckhp\" (UniqueName: \"kubernetes.io/projected/426663e4-70cb-44b7-a7cb-444ecdbae049-kube-api-access-bckhp\") pod \"ipp-route-a64ba4-kserve-5d56955d7-pfj8s\" (UID: \"426663e4-70cb-44b7-a7cb-444ecdbae049\") " pod="ai-tenant-e2e-ipp-a-a64ba4/ipp-route-a64ba4-kserve-5d56955d7-pfj8s"
Jul 24 15:55:53.340202 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:53.340108    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a64ba4/ipp-route-a64ba4-kserve-5d56955d7-pfj8s"
Jul 24 15:55:53.487057 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:53.487033    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-a64ba4/ipp-route-a64ba4-kserve-5d56955d7-pfj8s"]
Jul 24 15:55:53.489414 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:55:53.489387    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod426663e4_70cb_44b7_a7cb_444ecdbae049.slice/crio-d9c399252540efe41920c7d27faf0f3efafc7e1b3bf770293d7815fc8c649e9f WatchSource:0}: Error finding container d9c399252540efe41920c7d27faf0f3efafc7e1b3bf770293d7815fc8c649e9f: Status 404 returned error can't find the container with id d9c399252540efe41920c7d27faf0f3efafc7e1b3bf770293d7815fc8c649e9f
Jul 24 15:55:54.482892 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:54.482856    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a64ba4/ipp-route-a64ba4-kserve-5d56955d7-pfj8s" event={"ID":"426663e4-70cb-44b7-a7cb-444ecdbae049","Type":"ContainerStarted","Data":"76b31db7e08e514fbf19bcbbc33d4dfd02da2bb10bf1ddd4d78b9efa25b436fe"}
Jul 24 15:55:54.483267 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:54.482900    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a64ba4/ipp-route-a64ba4-kserve-5d56955d7-pfj8s" event={"ID":"426663e4-70cb-44b7-a7cb-444ecdbae049","Type":"ContainerStarted","Data":"d9c399252540efe41920c7d27faf0f3efafc7e1b3bf770293d7815fc8c649e9f"}
Jul 24 15:55:54.501550 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:54.501496    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-a64ba4/ipp-route-a64ba4-kserve-5d56955d7-pfj8s" podStartSLOduration=1.501479453 podStartE2EDuration="1.501479453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:55:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:55:54.499750104 +0000 UTC m=+2847.312102282" watchObservedRunningTime="2026-07-24 15:55:54.501479453 +0000 UTC m=+2847.313831631"
Jul 24 15:55:57.529748 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:55:57.529710    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:56:03.340516 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:56:03.340473    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-a64ba4/ipp-route-a64ba4-kserve-5d56955d7-pfj8s"
Jul 24 15:56:03.340959 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:56:03.340534    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-a64ba4/ipp-route-a64ba4-kserve-5d56955d7-pfj8s"
Jul 24 15:56:03.353176 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:56:03.353147    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-a64ba4/ipp-route-a64ba4-kserve-5d56955d7-pfj8s"
Jul 24 15:56:03.539502 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:56:03.539470    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-a64ba4/ipp-route-a64ba4-kserve-5d56955d7-pfj8s"
Jul 24 15:56:21.329481 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:56:21.329397    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:56:32.818767 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:56:32.818729    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-a64ba4-6c48d9846-qgs67_19cffcc0-896f-436f-a962-b6291852f1b3/payload-processing/0.log"
Jul 24 15:56:33.057960 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:56:33.057926    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-4dda13-7b9796cb6b-w8fgx_a3c304a1-65d0-4a80-8868-90d5471ecf4b/payload-processing/0.log"
Jul 24 15:56:37.597231 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:56:37.597194    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-6b1d8d-96459949d-vtjh4"]
Jul 24 15:56:37.601017 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:56:37.600999    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-6b1d8d-96459949d-vtjh4"
Jul 24 15:56:37.607129 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:56:37.607108    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-6b1d8d-dockercfg-t69nd\""
Jul 24 15:56:37.607671 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:56:37.607484    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-6b1d8d\""
Jul 24 15:56:37.617868 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:56:37.617837    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-6b1d8d-96459949d-vtjh4"]
Jul 24 15:56:37.652191 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:56:37.652155    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbkwf\" (UniqueName: \"kubernetes.io/projected/63faa5c3-0db1-4f31-bd0a-b6f73e0f710c-kube-api-access-wbkwf\") pod \"payload-processing-e2e-ipp-cleanup-6b1d8d-96459949d-vtjh4\" (UID: \"63faa5c3-0db1-4f31-bd0a-b6f73e0f710c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-6b1d8d-96459949d-vtjh4"
Jul 24 15:56:37.652369 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:56:37.652225    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63faa5c3-0db1-4f31-bd0a-b6f73e0f710c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-6b1d8d-96459949d-vtjh4\" (UID: \"63faa5c3-0db1-4f31-bd0a-b6f73e0f710c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-6b1d8d-96459949d-vtjh4"
Jul 24 15:56:37.753621 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:56:37.753575    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbkwf\" (UniqueName: \"kubernetes.io/projected/63faa5c3-0db1-4f31-bd0a-b6f73e0f710c-kube-api-access-wbkwf\") pod \"payload-processing-e2e-ipp-cleanup-6b1d8d-96459949d-vtjh4\" (UID: \"63faa5c3-0db1-4f31-bd0a-b6f73e0f710c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-6b1d8d-96459949d-vtjh4"
Jul 24 15:56:37.753860 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:56:37.753666    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63faa5c3-0db1-4f31-bd0a-b6f73e0f710c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-6b1d8d-96459949d-vtjh4\" (UID: \"63faa5c3-0db1-4f31-bd0a-b6f73e0f710c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-6b1d8d-96459949d-vtjh4"
Jul 24 15:56:37.754500 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:56:37.754478    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63faa5c3-0db1-4f31-bd0a-b6f73e0f710c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-6b1d8d-96459949d-vtjh4\" (UID: \"63faa5c3-0db1-4f31-bd0a-b6f73e0f710c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-6b1d8d-96459949d-vtjh4"
Jul 24 15:56:37.762276 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:56:37.762254    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbkwf\" (UniqueName: \"kubernetes.io/projected/63faa5c3-0db1-4f31-bd0a-b6f73e0f710c-kube-api-access-wbkwf\") pod \"payload-processing-e2e-ipp-cleanup-6b1d8d-96459949d-vtjh4\" (UID: \"63faa5c3-0db1-4f31-bd0a-b6f73e0f710c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-6b1d8d-96459949d-vtjh4"
Jul 24 15:56:37.911581 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:56:37.911493    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-6b1d8d-96459949d-vtjh4"
Jul 24 15:56:38.050189 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:56:38.050163    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-6b1d8d-96459949d-vtjh4"]
Jul 24 15:56:38.053521 ip-10-0-140-160 kubenswrapper[2579]: W0724 15:56:38.053493    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63faa5c3_0db1_4f31_bd0a_b6f73e0f710c.slice/crio-378a1276901533bf608994528bde41b44fa57867277b49cbea0191a5adec52b3 WatchSource:0}: Error finding container 378a1276901533bf608994528bde41b44fa57867277b49cbea0191a5adec52b3: Status 404 returned error can't find the container with id 378a1276901533bf608994528bde41b44fa57867277b49cbea0191a5adec52b3
Jul 24 15:56:38.713248 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:56:38.713210    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-6b1d8d-96459949d-vtjh4" event={"ID":"63faa5c3-0db1-4f31-bd0a-b6f73e0f710c","Type":"ContainerStarted","Data":"1cecc7cbdef31da0874283cc6897e2c179105ae15567b12e41b118f22649d178"}
Jul 24 15:56:38.713688 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:56:38.713372    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-6b1d8d-96459949d-vtjh4"
Jul 24 15:56:38.713688 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:56:38.713408    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-6b1d8d-96459949d-vtjh4" event={"ID":"63faa5c3-0db1-4f31-bd0a-b6f73e0f710c","Type":"ContainerStarted","Data":"378a1276901533bf608994528bde41b44fa57867277b49cbea0191a5adec52b3"}
Jul 24 15:56:38.733156 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:56:38.733105    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-6b1d8d-96459949d-vtjh4" podStartSLOduration=1.733085618 podStartE2EDuration="1.733085618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:56:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:56:38.732673723 +0000 UTC m=+2891.545025903" watchObservedRunningTime="2026-07-24 15:56:38.733085618 +0000 UTC m=+2891.545437798"
Jul 24 15:56:41.726729 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:56:41.726691    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:56:49.720120 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:56:49.720091    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-6b1d8d-96459949d-vtjh4"
Jul 24 15:57:02.747014 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:02.746974    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-6b1d8d-96459949d-vtjh4"]
Jul 24 15:57:02.747552 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:02.747237    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-6b1d8d-96459949d-vtjh4" podUID="63faa5c3-0db1-4f31-bd0a-b6f73e0f710c" containerName="payload-processing" containerID="cri-o://1cecc7cbdef31da0874283cc6897e2c179105ae15567b12e41b118f22649d178" gracePeriod=30
Jul 24 15:57:03.001844 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:03.001752    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-6b1d8d-96459949d-vtjh4"
Jul 24 15:57:03.111320 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:03.111281    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbkwf\" (UniqueName: \"kubernetes.io/projected/63faa5c3-0db1-4f31-bd0a-b6f73e0f710c-kube-api-access-wbkwf\") pod \"63faa5c3-0db1-4f31-bd0a-b6f73e0f710c\" (UID: \"63faa5c3-0db1-4f31-bd0a-b6f73e0f710c\") "
Jul 24 15:57:03.111501 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:03.111398    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63faa5c3-0db1-4f31-bd0a-b6f73e0f710c-plugins-config-volume\") pod \"63faa5c3-0db1-4f31-bd0a-b6f73e0f710c\" (UID: \"63faa5c3-0db1-4f31-bd0a-b6f73e0f710c\") "
Jul 24 15:57:03.111796 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:03.111773    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63faa5c3-0db1-4f31-bd0a-b6f73e0f710c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "63faa5c3-0db1-4f31-bd0a-b6f73e0f710c" (UID: "63faa5c3-0db1-4f31-bd0a-b6f73e0f710c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:57:03.113445 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:03.113421    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63faa5c3-0db1-4f31-bd0a-b6f73e0f710c-kube-api-access-wbkwf" (OuterVolumeSpecName: "kube-api-access-wbkwf") pod "63faa5c3-0db1-4f31-bd0a-b6f73e0f710c" (UID: "63faa5c3-0db1-4f31-bd0a-b6f73e0f710c"). InnerVolumeSpecName "kube-api-access-wbkwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:57:03.213038 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:03.213003    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbkwf\" (UniqueName: \"kubernetes.io/projected/63faa5c3-0db1-4f31-bd0a-b6f73e0f710c-kube-api-access-wbkwf\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:57:03.213038 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:03.213032    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63faa5c3-0db1-4f31-bd0a-b6f73e0f710c-plugins-config-volume\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:57:03.842733 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:03.842692    2579 generic.go:356] "Generic (PLEG): container finished" podID="63faa5c3-0db1-4f31-bd0a-b6f73e0f710c" containerID="1cecc7cbdef31da0874283cc6897e2c179105ae15567b12e41b118f22649d178" exitCode=0
Jul 24 15:57:03.843181 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:03.842740    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-6b1d8d-96459949d-vtjh4"
Jul 24 15:57:03.843181 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:03.842774    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-6b1d8d-96459949d-vtjh4" event={"ID":"63faa5c3-0db1-4f31-bd0a-b6f73e0f710c","Type":"ContainerDied","Data":"1cecc7cbdef31da0874283cc6897e2c179105ae15567b12e41b118f22649d178"}
Jul 24 15:57:03.843181 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:03.842858    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-6b1d8d-96459949d-vtjh4" event={"ID":"63faa5c3-0db1-4f31-bd0a-b6f73e0f710c","Type":"ContainerDied","Data":"378a1276901533bf608994528bde41b44fa57867277b49cbea0191a5adec52b3"}
Jul 24 15:57:03.843181 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:03.842884    2579 scope.go:117] "RemoveContainer" containerID="1cecc7cbdef31da0874283cc6897e2c179105ae15567b12e41b118f22649d178"
Jul 24 15:57:03.851923 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:03.851903    2579 scope.go:117] "RemoveContainer" containerID="1cecc7cbdef31da0874283cc6897e2c179105ae15567b12e41b118f22649d178"
Jul 24 15:57:03.852191 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:57:03.852170    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cecc7cbdef31da0874283cc6897e2c179105ae15567b12e41b118f22649d178\": container with ID starting with 1cecc7cbdef31da0874283cc6897e2c179105ae15567b12e41b118f22649d178 not found: ID does not exist" containerID="1cecc7cbdef31da0874283cc6897e2c179105ae15567b12e41b118f22649d178"
Jul 24 15:57:03.852249 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:03.852197    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cecc7cbdef31da0874283cc6897e2c179105ae15567b12e41b118f22649d178"} err="failed to get container status \"1cecc7cbdef31da0874283cc6897e2c179105ae15567b12e41b118f22649d178\": rpc error: code = NotFound desc = could not find container \"1cecc7cbdef31da0874283cc6897e2c179105ae15567b12e41b118f22649d178\": container with ID starting with 1cecc7cbdef31da0874283cc6897e2c179105ae15567b12e41b118f22649d178 not found: ID does not exist"
Jul 24 15:57:03.859135 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:03.859111    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-6b1d8d-96459949d-vtjh4"]
Jul 24 15:57:03.862752 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:03.862724    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-6b1d8d-96459949d-vtjh4"]
Jul 24 15:57:05.718187 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:05.718153    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63faa5c3-0db1-4f31-bd0a-b6f73e0f710c" path="/var/lib/kubelet/pods/63faa5c3-0db1-4f31-bd0a-b6f73e0f710c/volumes"
Jul 24 15:57:07.525555 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:07.525517    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:57:32.121844 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:32.121785    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:57:33.115995 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:33.115958    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a64ba4-6c48d9846-qgs67"]
Jul 24 15:57:33.116229 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:33.116201    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a64ba4-6c48d9846-qgs67" podUID="19cffcc0-896f-436f-a962-b6291852f1b3" containerName="payload-processing" containerID="cri-o://fa8f10d965d137562c78a0cfc505b8ce668c8cccf080800e88fbb0d6de1a45c1" gracePeriod=30
Jul 24 15:57:33.166987 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:33.166933    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-a64ba4-6c48d9846-qgs67" podUID="19cffcc0-896f-436f-a962-b6291852f1b3" containerName="payload-processing" probeResult="failure" output="dial tcp 10.133.0.77:9004: connect: connection refused"
Jul 24 15:57:33.378212 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:33.378148    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a64ba4-6c48d9846-qgs67"
Jul 24 15:57:33.416879 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:33.416849    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19cffcc0-896f-436f-a962-b6291852f1b3-plugins-config-volume\") pod \"19cffcc0-896f-436f-a962-b6291852f1b3\" (UID: \"19cffcc0-896f-436f-a962-b6291852f1b3\") "
Jul 24 15:57:33.416879 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:33.416883    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzzws\" (UniqueName: \"kubernetes.io/projected/19cffcc0-896f-436f-a962-b6291852f1b3-kube-api-access-vzzws\") pod \"19cffcc0-896f-436f-a962-b6291852f1b3\" (UID: \"19cffcc0-896f-436f-a962-b6291852f1b3\") "
Jul 24 15:57:33.417184 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:33.417161    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19cffcc0-896f-436f-a962-b6291852f1b3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "19cffcc0-896f-436f-a962-b6291852f1b3" (UID: "19cffcc0-896f-436f-a962-b6291852f1b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:57:33.419021 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:33.418998    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19cffcc0-896f-436f-a962-b6291852f1b3-kube-api-access-vzzws" (OuterVolumeSpecName: "kube-api-access-vzzws") pod "19cffcc0-896f-436f-a962-b6291852f1b3" (UID: "19cffcc0-896f-436f-a962-b6291852f1b3"). InnerVolumeSpecName "kube-api-access-vzzws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:57:33.517913 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:33.517865    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19cffcc0-896f-436f-a962-b6291852f1b3-plugins-config-volume\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:57:33.517913 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:33.517906    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzzws\" (UniqueName: \"kubernetes.io/projected/19cffcc0-896f-436f-a962-b6291852f1b3-kube-api-access-vzzws\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:57:34.001457 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:34.001367    2579 generic.go:356] "Generic (PLEG): container finished" podID="19cffcc0-896f-436f-a962-b6291852f1b3" containerID="fa8f10d965d137562c78a0cfc505b8ce668c8cccf080800e88fbb0d6de1a45c1" exitCode=0
Jul 24 15:57:34.001457 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:34.001415    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a64ba4-6c48d9846-qgs67"
Jul 24 15:57:34.001638 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:34.001459    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a64ba4-6c48d9846-qgs67" event={"ID":"19cffcc0-896f-436f-a962-b6291852f1b3","Type":"ContainerDied","Data":"fa8f10d965d137562c78a0cfc505b8ce668c8cccf080800e88fbb0d6de1a45c1"}
Jul 24 15:57:34.001638 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:34.001512    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a64ba4-6c48d9846-qgs67" event={"ID":"19cffcc0-896f-436f-a962-b6291852f1b3","Type":"ContainerDied","Data":"6f6af35ca404a37380742f96993269c1bceecdba81fc60b8dd715b502ef397d0"}
Jul 24 15:57:34.001638 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:34.001533    2579 scope.go:117] "RemoveContainer" containerID="fa8f10d965d137562c78a0cfc505b8ce668c8cccf080800e88fbb0d6de1a45c1"
Jul 24 15:57:34.010312 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:34.010293    2579 scope.go:117] "RemoveContainer" containerID="fa8f10d965d137562c78a0cfc505b8ce668c8cccf080800e88fbb0d6de1a45c1"
Jul 24 15:57:34.010549 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:57:34.010531    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa8f10d965d137562c78a0cfc505b8ce668c8cccf080800e88fbb0d6de1a45c1\": container with ID starting with fa8f10d965d137562c78a0cfc505b8ce668c8cccf080800e88fbb0d6de1a45c1 not found: ID does not exist" containerID="fa8f10d965d137562c78a0cfc505b8ce668c8cccf080800e88fbb0d6de1a45c1"
Jul 24 15:57:34.010632 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:34.010555    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa8f10d965d137562c78a0cfc505b8ce668c8cccf080800e88fbb0d6de1a45c1"} err="failed to get container status \"fa8f10d965d137562c78a0cfc505b8ce668c8cccf080800e88fbb0d6de1a45c1\": rpc error: code = NotFound desc = could not find container \"fa8f10d965d137562c78a0cfc505b8ce668c8cccf080800e88fbb0d6de1a45c1\": container with ID starting with fa8f10d965d137562c78a0cfc505b8ce668c8cccf080800e88fbb0d6de1a45c1 not found: ID does not exist"
Jul 24 15:57:34.018419 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:34.018397    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a64ba4-6c48d9846-qgs67"]
Jul 24 15:57:34.022761 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:34.022739    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a64ba4-6c48d9846-qgs67"]
Jul 24 15:57:35.717917 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:35.717882    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19cffcc0-896f-436f-a962-b6291852f1b3" path="/var/lib/kubelet/pods/19cffcc0-896f-436f-a962-b6291852f1b3/volumes"
Jul 24 15:57:42.928266 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:42.928227    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:57:42.978740 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:42.978706    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-a64ba4/ipp-route-a64ba4-kserve-5d56955d7-pfj8s"]
Jul 24 15:57:58.000713 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:58.000656    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-a64ba4/ipp-route-a64ba4-kserve-5d56955d7-pfj8s" podUID="426663e4-70cb-44b7-a7cb-444ecdbae049" containerName="main" containerID="cri-o://76b31db7e08e514fbf19bcbbc33d4dfd02da2bb10bf1ddd4d78b9efa25b436fe" gracePeriod=45
Jul 24 15:57:58.126469 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:58.126438    2579 generic.go:356] "Generic (PLEG): container finished" podID="426663e4-70cb-44b7-a7cb-444ecdbae049" containerID="76b31db7e08e514fbf19bcbbc33d4dfd02da2bb10bf1ddd4d78b9efa25b436fe" exitCode=0
Jul 24 15:57:58.126602 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:58.126519    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a64ba4/ipp-route-a64ba4-kserve-5d56955d7-pfj8s" event={"ID":"426663e4-70cb-44b7-a7cb-444ecdbae049","Type":"ContainerDied","Data":"76b31db7e08e514fbf19bcbbc33d4dfd02da2bb10bf1ddd4d78b9efa25b436fe"}
Jul 24 15:57:58.258543 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:58.258482    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a64ba4/ipp-route-a64ba4-kserve-5d56955d7-pfj8s"
Jul 24 15:57:58.364740 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:58.364710    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/426663e4-70cb-44b7-a7cb-444ecdbae049-tmp-dir\") pod \"426663e4-70cb-44b7-a7cb-444ecdbae049\" (UID: \"426663e4-70cb-44b7-a7cb-444ecdbae049\") "
Jul 24 15:57:58.364956 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:58.364760    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/426663e4-70cb-44b7-a7cb-444ecdbae049-model-cache\") pod \"426663e4-70cb-44b7-a7cb-444ecdbae049\" (UID: \"426663e4-70cb-44b7-a7cb-444ecdbae049\") "
Jul 24 15:57:58.364956 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:58.364789    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bckhp\" (UniqueName: \"kubernetes.io/projected/426663e4-70cb-44b7-a7cb-444ecdbae049-kube-api-access-bckhp\") pod \"426663e4-70cb-44b7-a7cb-444ecdbae049\" (UID: \"426663e4-70cb-44b7-a7cb-444ecdbae049\") "
Jul 24 15:57:58.364956 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:58.364873    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/426663e4-70cb-44b7-a7cb-444ecdbae049-dshm\") pod \"426663e4-70cb-44b7-a7cb-444ecdbae049\" (UID: \"426663e4-70cb-44b7-a7cb-444ecdbae049\") "
Jul 24 15:57:58.364956 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:58.364907    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/426663e4-70cb-44b7-a7cb-444ecdbae049-home\") pod \"426663e4-70cb-44b7-a7cb-444ecdbae049\" (UID: \"426663e4-70cb-44b7-a7cb-444ecdbae049\") "
Jul 24 15:57:58.364956 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:58.364929    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/426663e4-70cb-44b7-a7cb-444ecdbae049-tls-certs\") pod \"426663e4-70cb-44b7-a7cb-444ecdbae049\" (UID: \"426663e4-70cb-44b7-a7cb-444ecdbae049\") "
Jul 24 15:57:58.365255 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:58.365043    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426663e4-70cb-44b7-a7cb-444ecdbae049-model-cache" (OuterVolumeSpecName: "model-cache") pod "426663e4-70cb-44b7-a7cb-444ecdbae049" (UID: "426663e4-70cb-44b7-a7cb-444ecdbae049"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:57:58.365255 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:58.365052    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426663e4-70cb-44b7-a7cb-444ecdbae049-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "426663e4-70cb-44b7-a7cb-444ecdbae049" (UID: "426663e4-70cb-44b7-a7cb-444ecdbae049"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:57:58.365255 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:58.365192    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426663e4-70cb-44b7-a7cb-444ecdbae049-home" (OuterVolumeSpecName: "home") pod "426663e4-70cb-44b7-a7cb-444ecdbae049" (UID: "426663e4-70cb-44b7-a7cb-444ecdbae049"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:57:58.365424 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:58.365314    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/426663e4-70cb-44b7-a7cb-444ecdbae049-tmp-dir\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:57:58.365424 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:58.365332    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/426663e4-70cb-44b7-a7cb-444ecdbae049-model-cache\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:57:58.365424 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:58.365344    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/426663e4-70cb-44b7-a7cb-444ecdbae049-home\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:57:58.367260 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:58.367234    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426663e4-70cb-44b7-a7cb-444ecdbae049-dshm" (OuterVolumeSpecName: "dshm") pod "426663e4-70cb-44b7-a7cb-444ecdbae049" (UID: "426663e4-70cb-44b7-a7cb-444ecdbae049"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:57:58.367367 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:58.367267    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/426663e4-70cb-44b7-a7cb-444ecdbae049-kube-api-access-bckhp" (OuterVolumeSpecName: "kube-api-access-bckhp") pod "426663e4-70cb-44b7-a7cb-444ecdbae049" (UID: "426663e4-70cb-44b7-a7cb-444ecdbae049"). InnerVolumeSpecName "kube-api-access-bckhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:57:58.367367 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:58.367267    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/426663e4-70cb-44b7-a7cb-444ecdbae049-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "426663e4-70cb-44b7-a7cb-444ecdbae049" (UID: "426663e4-70cb-44b7-a7cb-444ecdbae049"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 15:57:58.466476 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:58.466099    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bckhp\" (UniqueName: \"kubernetes.io/projected/426663e4-70cb-44b7-a7cb-444ecdbae049-kube-api-access-bckhp\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:57:58.466476 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:58.466129    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/426663e4-70cb-44b7-a7cb-444ecdbae049-dshm\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:57:58.466476 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:58.466143    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/426663e4-70cb-44b7-a7cb-444ecdbae049-tls-certs\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:57:59.133330 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:59.133299    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a64ba4/ipp-route-a64ba4-kserve-5d56955d7-pfj8s"
Jul 24 15:57:59.133791 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:59.133304    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a64ba4/ipp-route-a64ba4-kserve-5d56955d7-pfj8s" event={"ID":"426663e4-70cb-44b7-a7cb-444ecdbae049","Type":"ContainerDied","Data":"d9c399252540efe41920c7d27faf0f3efafc7e1b3bf770293d7815fc8c649e9f"}
Jul 24 15:57:59.133791 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:59.133417    2579 scope.go:117] "RemoveContainer" containerID="76b31db7e08e514fbf19bcbbc33d4dfd02da2bb10bf1ddd4d78b9efa25b436fe"
Jul 24 15:57:59.153932 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:59.153895    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-a64ba4/ipp-route-a64ba4-kserve-5d56955d7-pfj8s"]
Jul 24 15:57:59.159431 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:59.159411    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-a64ba4/ipp-route-a64ba4-kserve-5d56955d7-pfj8s"]
Jul 24 15:57:59.717194 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:57:59.717156    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="426663e4-70cb-44b7-a7cb-444ecdbae049" path="/var/lib/kubelet/pods/426663e4-70cb-44b7-a7cb-444ecdbae049/volumes"
Jul 24 15:58:51.215029 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:58:51.214994    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4dda13-7b9796cb6b-w8fgx"]
Jul 24 15:58:51.215576 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:58:51.215263    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-4dda13-7b9796cb6b-w8fgx" podUID="a3c304a1-65d0-4a80-8868-90d5471ecf4b" containerName="payload-processing" containerID="cri-o://801b2da9fc66a38995cfedd10ab5ec03e51e454cdff17cf5b383de1c95d77e99" gracePeriod=30
Jul 24 15:58:51.222421 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:58:51.222025    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4dda13-5fff879bb4-ndshl"]
Jul 24 15:58:51.222421 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:58:51.222242    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4dda13-5fff879bb4-ndshl" podUID="8cd34332-3dd4-4350-9fc0-26f1fee180f0" containerName="payload-pre-processing" containerID="cri-o://2dc98b6a39483d6b28bbb6cd79baf4603cb8091135e6689e7d69e30911efb7ca" gracePeriod=30
Jul 24 15:58:51.395756 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:58:51.395716    2579 generic.go:356] "Generic (PLEG): container finished" podID="a3c304a1-65d0-4a80-8868-90d5471ecf4b" containerID="801b2da9fc66a38995cfedd10ab5ec03e51e454cdff17cf5b383de1c95d77e99" exitCode=0
Jul 24 15:58:51.395934 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:58:51.395905    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4dda13-7b9796cb6b-w8fgx" event={"ID":"a3c304a1-65d0-4a80-8868-90d5471ecf4b","Type":"ContainerDied","Data":"801b2da9fc66a38995cfedd10ab5ec03e51e454cdff17cf5b383de1c95d77e99"}
Jul 24 15:58:51.397804 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:58:51.397780    2579 generic.go:356] "Generic (PLEG): container finished" podID="8cd34332-3dd4-4350-9fc0-26f1fee180f0" containerID="2dc98b6a39483d6b28bbb6cd79baf4603cb8091135e6689e7d69e30911efb7ca" exitCode=0
Jul 24 15:58:51.397947 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:58:51.397885    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4dda13-5fff879bb4-ndshl" event={"ID":"8cd34332-3dd4-4350-9fc0-26f1fee180f0","Type":"ContainerDied","Data":"2dc98b6a39483d6b28bbb6cd79baf4603cb8091135e6689e7d69e30911efb7ca"}
Jul 24 15:58:51.478092 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:58:51.478031    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4dda13-7b9796cb6b-w8fgx"
Jul 24 15:58:51.507792 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:58:51.507764    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4dda13-5fff879bb4-ndshl"
Jul 24 15:58:51.576338 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:58:51.576307    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkvh7\" (UniqueName: \"kubernetes.io/projected/8cd34332-3dd4-4350-9fc0-26f1fee180f0-kube-api-access-jkvh7\") pod \"8cd34332-3dd4-4350-9fc0-26f1fee180f0\" (UID: \"8cd34332-3dd4-4350-9fc0-26f1fee180f0\") "
Jul 24 15:58:51.576513 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:58:51.576374    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m47cv\" (UniqueName: \"kubernetes.io/projected/a3c304a1-65d0-4a80-8868-90d5471ecf4b-kube-api-access-m47cv\") pod \"a3c304a1-65d0-4a80-8868-90d5471ecf4b\" (UID: \"a3c304a1-65d0-4a80-8868-90d5471ecf4b\") "
Jul 24 15:58:51.576513 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:58:51.576485    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3c304a1-65d0-4a80-8868-90d5471ecf4b-plugins-config-volume\") pod \"a3c304a1-65d0-4a80-8868-90d5471ecf4b\" (UID: \"a3c304a1-65d0-4a80-8868-90d5471ecf4b\") "
Jul 24 15:58:51.576615 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:58:51.576595    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cd34332-3dd4-4350-9fc0-26f1fee180f0-plugins-config-volume\") pod \"8cd34332-3dd4-4350-9fc0-26f1fee180f0\" (UID: \"8cd34332-3dd4-4350-9fc0-26f1fee180f0\") "
Jul 24 15:58:51.576930 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:58:51.576899    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3c304a1-65d0-4a80-8868-90d5471ecf4b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a3c304a1-65d0-4a80-8868-90d5471ecf4b" (UID: "a3c304a1-65d0-4a80-8868-90d5471ecf4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:58:51.577079 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:58:51.577034    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cd34332-3dd4-4350-9fc0-26f1fee180f0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8cd34332-3dd4-4350-9fc0-26f1fee180f0" (UID: "8cd34332-3dd4-4350-9fc0-26f1fee180f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:58:51.578485 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:58:51.578453    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3c304a1-65d0-4a80-8868-90d5471ecf4b-kube-api-access-m47cv" (OuterVolumeSpecName: "kube-api-access-m47cv") pod "a3c304a1-65d0-4a80-8868-90d5471ecf4b" (UID: "a3c304a1-65d0-4a80-8868-90d5471ecf4b"). InnerVolumeSpecName "kube-api-access-m47cv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:58:51.578710 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:58:51.578693    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cd34332-3dd4-4350-9fc0-26f1fee180f0-kube-api-access-jkvh7" (OuterVolumeSpecName: "kube-api-access-jkvh7") pod "8cd34332-3dd4-4350-9fc0-26f1fee180f0" (UID: "8cd34332-3dd4-4350-9fc0-26f1fee180f0"). InnerVolumeSpecName "kube-api-access-jkvh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:58:51.678296 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:58:51.678264    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkvh7\" (UniqueName: \"kubernetes.io/projected/8cd34332-3dd4-4350-9fc0-26f1fee180f0-kube-api-access-jkvh7\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:58:51.678296 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:58:51.678290    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m47cv\" (UniqueName: \"kubernetes.io/projected/a3c304a1-65d0-4a80-8868-90d5471ecf4b-kube-api-access-m47cv\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:58:51.678296 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:58:51.678299    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3c304a1-65d0-4a80-8868-90d5471ecf4b-plugins-config-volume\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:58:51.678533 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:58:51.678312    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cd34332-3dd4-4350-9fc0-26f1fee180f0-plugins-config-volume\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:58:52.404266 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:58:52.404234    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4dda13-7b9796cb6b-w8fgx"
Jul 24 15:58:52.404726 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:58:52.404287    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4dda13-7b9796cb6b-w8fgx" event={"ID":"a3c304a1-65d0-4a80-8868-90d5471ecf4b","Type":"ContainerDied","Data":"b1536eb4d48ab9eb9cccf720b5762d10366ef4235e5f79b0c9558467b9bc3bb7"}
Jul 24 15:58:52.404726 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:58:52.404350    2579 scope.go:117] "RemoveContainer" containerID="801b2da9fc66a38995cfedd10ab5ec03e51e454cdff17cf5b383de1c95d77e99"
Jul 24 15:58:52.405644 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:58:52.405619    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4dda13-5fff879bb4-ndshl"
Jul 24 15:58:52.405722 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:58:52.405662    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4dda13-5fff879bb4-ndshl" event={"ID":"8cd34332-3dd4-4350-9fc0-26f1fee180f0","Type":"ContainerDied","Data":"280e1fc5c4f02988655f1364b1302a6c91c9ef67144a5063b7d1953c32436c07"}
Jul 24 15:58:52.413205 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:58:52.413189    2579 scope.go:117] "RemoveContainer" containerID="2dc98b6a39483d6b28bbb6cd79baf4603cb8091135e6689e7d69e30911efb7ca"
Jul 24 15:58:52.421478 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:58:52.421458    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4dda13-5fff879bb4-ndshl"]
Jul 24 15:58:52.424741 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:58:52.424720    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4dda13-5fff879bb4-ndshl"]
Jul 24 15:58:52.434282 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:58:52.434260    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4dda13-7b9796cb6b-w8fgx"]
Jul 24 15:58:52.436429 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:58:52.436409    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4dda13-7b9796cb6b-w8fgx"]
Jul 24 15:58:53.717987 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:58:53.717951    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cd34332-3dd4-4350-9fc0-26f1fee180f0" path="/var/lib/kubelet/pods/8cd34332-3dd4-4350-9fc0-26f1fee180f0/volumes"
Jul 24 15:58:53.718375 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:58:53.718331    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3c304a1-65d0-4a80-8868-90d5471ecf4b" path="/var/lib/kubelet/pods/a3c304a1-65d0-4a80-8868-90d5471ecf4b/volumes"
Jul 24 15:58:54.526663 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:58:54.526628    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:59:13.306420 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:13.306384    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-2619ff-67c944cf6-56txk"]
Jul 24 15:59:13.307017 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:13.306649    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2619ff-67c944cf6-56txk" podUID="af885fa0-3295-4818-83b1-d38e540f292e" containerName="maas-api" containerID="cri-o://b81d9de77b4d57db37787762169b7f26e35023079a9a2a1dd7f6885b03d204cc" gracePeriod=30
Jul 24 15:59:13.380003 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:13.379956    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-2619ff-57fdbfb486-5gkbp"]
Jul 24 15:59:13.380246 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:13.380220    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-2619ff-57fdbfb486-5gkbp" podUID="71111d85-3b11-48fe-a587-54c105cb74d7" containerName="payload-processing" containerID="cri-o://28bc077ddc0c812f6a19e96473c54390618f06b7437d645aaf6227a303d4e01b" gracePeriod=30
Jul 24 15:59:13.407274 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:13.407233    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-2619ff-677c9878b4-fqr7x"]
Jul 24 15:59:13.407530 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:13.407500    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2619ff-677c9878b4-fqr7x" podUID="f2b13488-f677-4987-bafe-df99527776e7" containerName="payload-pre-processing" containerID="cri-o://7523380cc5a739a76488709b5124045a9d96ce8f8e9a07ccce2f3e5dbbff6190" gracePeriod=30
Jul 24 15:59:13.519996 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:13.519967    2579 generic.go:356] "Generic (PLEG): container finished" podID="af885fa0-3295-4818-83b1-d38e540f292e" containerID="b81d9de77b4d57db37787762169b7f26e35023079a9a2a1dd7f6885b03d204cc" exitCode=0
Jul 24 15:59:13.520127 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:13.520043    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2619ff-67c944cf6-56txk" event={"ID":"af885fa0-3295-4818-83b1-d38e540f292e","Type":"ContainerDied","Data":"b81d9de77b4d57db37787762169b7f26e35023079a9a2a1dd7f6885b03d204cc"}
Jul 24 15:59:13.522031 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:13.522009    2579 generic.go:356] "Generic (PLEG): container finished" podID="71111d85-3b11-48fe-a587-54c105cb74d7" containerID="28bc077ddc0c812f6a19e96473c54390618f06b7437d645aaf6227a303d4e01b" exitCode=0
Jul 24 15:59:13.522136 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:13.522056    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-2619ff-57fdbfb486-5gkbp" event={"ID":"71111d85-3b11-48fe-a587-54c105cb74d7","Type":"ContainerDied","Data":"28bc077ddc0c812f6a19e96473c54390618f06b7437d645aaf6227a303d4e01b"}
Jul 24 15:59:13.640240 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:13.640213    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2619ff-67c944cf6-56txk"
Jul 24 15:59:13.684648 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:13.684621    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2619ff-677c9878b4-fqr7x"
Jul 24 15:59:13.694845 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:13.694805    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-2619ff-57fdbfb486-5gkbp"
Jul 24 15:59:13.706741 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:13.706697    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xhbr\" (UniqueName: \"kubernetes.io/projected/71111d85-3b11-48fe-a587-54c105cb74d7-kube-api-access-9xhbr\") pod \"71111d85-3b11-48fe-a587-54c105cb74d7\" (UID: \"71111d85-3b11-48fe-a587-54c105cb74d7\") "
Jul 24 15:59:13.706894 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:13.706771    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2b13488-f677-4987-bafe-df99527776e7-plugins-config-volume\") pod \"f2b13488-f677-4987-bafe-df99527776e7\" (UID: \"f2b13488-f677-4987-bafe-df99527776e7\") "
Jul 24 15:59:13.706988 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:13.706927    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8plp\" (UniqueName: \"kubernetes.io/projected/f2b13488-f677-4987-bafe-df99527776e7-kube-api-access-f8plp\") pod \"f2b13488-f677-4987-bafe-df99527776e7\" (UID: \"f2b13488-f677-4987-bafe-df99527776e7\") "
Jul 24 15:59:13.706988 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:13.706958    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kz95\" (UniqueName: \"kubernetes.io/projected/af885fa0-3295-4818-83b1-d38e540f292e-kube-api-access-4kz95\") pod \"af885fa0-3295-4818-83b1-d38e540f292e\" (UID: \"af885fa0-3295-4818-83b1-d38e540f292e\") "
Jul 24 15:59:13.706988 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:13.706986    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af885fa0-3295-4818-83b1-d38e540f292e-maas-api-tls\") pod \"af885fa0-3295-4818-83b1-d38e540f292e\" (UID: \"af885fa0-3295-4818-83b1-d38e540f292e\") "
Jul 24 15:59:13.707227 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:13.707011    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71111d85-3b11-48fe-a587-54c105cb74d7-plugins-config-volume\") pod \"71111d85-3b11-48fe-a587-54c105cb74d7\" (UID: \"71111d85-3b11-48fe-a587-54c105cb74d7\") "
Jul 24 15:59:13.707227 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:13.707060    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af885fa0-3295-4818-83b1-d38e540f292e-openshift-service-ca\") pod \"af885fa0-3295-4818-83b1-d38e540f292e\" (UID: \"af885fa0-3295-4818-83b1-d38e540f292e\") "
Jul 24 15:59:13.707227 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:13.707187    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2b13488-f677-4987-bafe-df99527776e7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f2b13488-f677-4987-bafe-df99527776e7" (UID: "f2b13488-f677-4987-bafe-df99527776e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:59:13.707479 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:13.707439    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2b13488-f677-4987-bafe-df99527776e7-plugins-config-volume\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:59:13.707839 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:13.707774    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71111d85-3b11-48fe-a587-54c105cb74d7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "71111d85-3b11-48fe-a587-54c105cb74d7" (UID: "71111d85-3b11-48fe-a587-54c105cb74d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:59:13.707929 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:13.707891    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af885fa0-3295-4818-83b1-d38e540f292e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "af885fa0-3295-4818-83b1-d38e540f292e" (UID: "af885fa0-3295-4818-83b1-d38e540f292e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:59:13.709609 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:13.709577    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71111d85-3b11-48fe-a587-54c105cb74d7-kube-api-access-9xhbr" (OuterVolumeSpecName: "kube-api-access-9xhbr") pod "71111d85-3b11-48fe-a587-54c105cb74d7" (UID: "71111d85-3b11-48fe-a587-54c105cb74d7"). InnerVolumeSpecName "kube-api-access-9xhbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:59:13.710413 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:13.710381    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af885fa0-3295-4818-83b1-d38e540f292e-kube-api-access-4kz95" (OuterVolumeSpecName: "kube-api-access-4kz95") pod "af885fa0-3295-4818-83b1-d38e540f292e" (UID: "af885fa0-3295-4818-83b1-d38e540f292e"). InnerVolumeSpecName "kube-api-access-4kz95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:59:13.710638 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:13.710610    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2b13488-f677-4987-bafe-df99527776e7-kube-api-access-f8plp" (OuterVolumeSpecName: "kube-api-access-f8plp") pod "f2b13488-f677-4987-bafe-df99527776e7" (UID: "f2b13488-f677-4987-bafe-df99527776e7"). InnerVolumeSpecName "kube-api-access-f8plp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:59:13.711142 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:13.711114    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af885fa0-3295-4818-83b1-d38e540f292e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "af885fa0-3295-4818-83b1-d38e540f292e" (UID: "af885fa0-3295-4818-83b1-d38e540f292e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 15:59:13.808397 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:13.808366    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af885fa0-3295-4818-83b1-d38e540f292e-maas-api-tls\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:59:13.808397 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:13.808395    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71111d85-3b11-48fe-a587-54c105cb74d7-plugins-config-volume\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:59:13.808611 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:13.808409    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af885fa0-3295-4818-83b1-d38e540f292e-openshift-service-ca\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:59:13.808611 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:13.808422    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xhbr\" (UniqueName: \"kubernetes.io/projected/71111d85-3b11-48fe-a587-54c105cb74d7-kube-api-access-9xhbr\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:59:13.808611 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:13.808434    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8plp\" (UniqueName: \"kubernetes.io/projected/f2b13488-f677-4987-bafe-df99527776e7-kube-api-access-f8plp\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:59:13.808611 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:13.808447    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kz95\" (UniqueName: \"kubernetes.io/projected/af885fa0-3295-4818-83b1-d38e540f292e-kube-api-access-4kz95\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:59:14.528126 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:14.528095    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-2619ff-57fdbfb486-5gkbp"
Jul 24 15:59:14.528650 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:14.528102    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-2619ff-57fdbfb486-5gkbp" event={"ID":"71111d85-3b11-48fe-a587-54c105cb74d7","Type":"ContainerDied","Data":"88f78970bd5e71efa6c30321ae8ca12cd1fd3effbcf06344b8988a5bbe2fae2d"}
Jul 24 15:59:14.528650 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:14.528234    2579 scope.go:117] "RemoveContainer" containerID="28bc077ddc0c812f6a19e96473c54390618f06b7437d645aaf6227a303d4e01b"
Jul 24 15:59:14.529613 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:14.529538    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2619ff-67c944cf6-56txk"
Jul 24 15:59:14.529613 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:14.529585    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2619ff-67c944cf6-56txk" event={"ID":"af885fa0-3295-4818-83b1-d38e540f292e","Type":"ContainerDied","Data":"f7517882b4a4d427b221c3f4029acd03ba07a72266892a9a9c97106ab95e683e"}
Jul 24 15:59:14.531031 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:14.530994    2579 generic.go:356] "Generic (PLEG): container finished" podID="f2b13488-f677-4987-bafe-df99527776e7" containerID="7523380cc5a739a76488709b5124045a9d96ce8f8e9a07ccce2f3e5dbbff6190" exitCode=0
Jul 24 15:59:14.531196 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:14.531032    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2619ff-677c9878b4-fqr7x"
Jul 24 15:59:14.531196 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:14.531031    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2619ff-677c9878b4-fqr7x" event={"ID":"f2b13488-f677-4987-bafe-df99527776e7","Type":"ContainerDied","Data":"7523380cc5a739a76488709b5124045a9d96ce8f8e9a07ccce2f3e5dbbff6190"}
Jul 24 15:59:14.531196 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:14.531152    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2619ff-677c9878b4-fqr7x" event={"ID":"f2b13488-f677-4987-bafe-df99527776e7","Type":"ContainerDied","Data":"212965c41468658f00ab07bae64192b272d1ad5625e0fe3abfb423a4e2e103bc"}
Jul 24 15:59:14.538640 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:14.538623    2579 scope.go:117] "RemoveContainer" containerID="b81d9de77b4d57db37787762169b7f26e35023079a9a2a1dd7f6885b03d204cc"
Jul 24 15:59:14.546936 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:14.546914    2579 scope.go:117] "RemoveContainer" containerID="7523380cc5a739a76488709b5124045a9d96ce8f8e9a07ccce2f3e5dbbff6190"
Jul 24 15:59:14.551173 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:14.551150    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-2619ff-67c944cf6-56txk"]
Jul 24 15:59:14.555598 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:14.555569    2579 scope.go:117] "RemoveContainer" containerID="7523380cc5a739a76488709b5124045a9d96ce8f8e9a07ccce2f3e5dbbff6190"
Jul 24 15:59:14.555941 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:59:14.555914    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7523380cc5a739a76488709b5124045a9d96ce8f8e9a07ccce2f3e5dbbff6190\": container with ID starting with 7523380cc5a739a76488709b5124045a9d96ce8f8e9a07ccce2f3e5dbbff6190 not found: ID does not exist" containerID="7523380cc5a739a76488709b5124045a9d96ce8f8e9a07ccce2f3e5dbbff6190"
Jul 24 15:59:14.556013 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:14.555954    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7523380cc5a739a76488709b5124045a9d96ce8f8e9a07ccce2f3e5dbbff6190"} err="failed to get container status \"7523380cc5a739a76488709b5124045a9d96ce8f8e9a07ccce2f3e5dbbff6190\": rpc error: code = NotFound desc = could not find container \"7523380cc5a739a76488709b5124045a9d96ce8f8e9a07ccce2f3e5dbbff6190\": container with ID starting with 7523380cc5a739a76488709b5124045a9d96ce8f8e9a07ccce2f3e5dbbff6190 not found: ID does not exist"
Jul 24 15:59:14.556518 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:14.556497    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-2619ff-67c944cf6-56txk"]
Jul 24 15:59:14.569409 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:14.569380    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-2619ff-677c9878b4-fqr7x"]
Jul 24 15:59:14.572789 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:14.572705    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-2619ff-677c9878b4-fqr7x"]
Jul 24 15:59:14.584938 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:14.584912    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-2619ff-57fdbfb486-5gkbp"]
Jul 24 15:59:14.590454 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:14.590427    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-2619ff-57fdbfb486-5gkbp"]
Jul 24 15:59:15.717430 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:15.717397    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71111d85-3b11-48fe-a587-54c105cb74d7" path="/var/lib/kubelet/pods/71111d85-3b11-48fe-a587-54c105cb74d7/volumes"
Jul 24 15:59:15.717799 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:15.717782    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af885fa0-3295-4818-83b1-d38e540f292e" path="/var/lib/kubelet/pods/af885fa0-3295-4818-83b1-d38e540f292e/volumes"
Jul 24 15:59:15.718130 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:15.718117    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2b13488-f677-4987-bafe-df99527776e7" path="/var/lib/kubelet/pods/f2b13488-f677-4987-bafe-df99527776e7/volumes"
Jul 24 15:59:16.736441 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:16.736407    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:59:35.952528 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:35.952494    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1578f9-c7bbcd9c-mhqkm"]
Jul 24 15:59:35.953129 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:35.952737    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1578f9-c7bbcd9c-mhqkm" podUID="3bb038d2-9151-42db-b6a1-9e4b6286d220" containerName="maas-api" containerID="cri-o://0dd6ee28feef6e4ea03ee08d9576a1f3059ce9dc2f9ec99cfc4b1d0a5b1cda21" gracePeriod=30
Jul 24 15:59:36.005332 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.005296    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-1578f9-7fcfcd5566-hkxm4"]
Jul 24 15:59:36.005595 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.005545    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-1578f9-7fcfcd5566-hkxm4" podUID="d447f9d4-1a69-41a2-a6f0-7d4d6b2b61c0" containerName="payload-processing" containerID="cri-o://acbffae8806269dd6ced1985d12344b6d4825639a8e8daeb732564057d6eba83" gracePeriod=30
Jul 24 15:59:36.011951 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.011927    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-1578f9-647bdc8c88-fqq67"]
Jul 24 15:59:36.012119 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.012100    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1578f9-647bdc8c88-fqq67" podUID="5f3c7c3e-9695-4364-8ede-5274813096a5" containerName="payload-pre-processing" containerID="cri-o://431d64ee4f2d69a05bb6dd58fccedb2c86f56d0d9c1c7f5d48368db78f667981" gracePeriod=30
Jul 24 15:59:36.269246 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.269215    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1578f9-c7bbcd9c-mhqkm"
Jul 24 15:59:36.301045 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.301018    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-1578f9-7fcfcd5566-hkxm4"
Jul 24 15:59:36.304210 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.304187    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1578f9-647bdc8c88-fqq67"
Jul 24 15:59:36.423124 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.423091    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d447f9d4-1a69-41a2-a6f0-7d4d6b2b61c0-plugins-config-volume\") pod \"d447f9d4-1a69-41a2-a6f0-7d4d6b2b61c0\" (UID: \"d447f9d4-1a69-41a2-a6f0-7d4d6b2b61c0\") "
Jul 24 15:59:36.423329 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.423138    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f3c7c3e-9695-4364-8ede-5274813096a5-plugins-config-volume\") pod \"5f3c7c3e-9695-4364-8ede-5274813096a5\" (UID: \"5f3c7c3e-9695-4364-8ede-5274813096a5\") "
Jul 24 15:59:36.423329 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.423170    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4g5b\" (UniqueName: \"kubernetes.io/projected/3bb038d2-9151-42db-b6a1-9e4b6286d220-kube-api-access-j4g5b\") pod \"3bb038d2-9151-42db-b6a1-9e4b6286d220\" (UID: \"3bb038d2-9151-42db-b6a1-9e4b6286d220\") "
Jul 24 15:59:36.423329 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.423237    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bb038d2-9151-42db-b6a1-9e4b6286d220-maas-api-tls\") pod \"3bb038d2-9151-42db-b6a1-9e4b6286d220\" (UID: \"3bb038d2-9151-42db-b6a1-9e4b6286d220\") "
Jul 24 15:59:36.423329 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.423252    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtj27\" (UniqueName: \"kubernetes.io/projected/5f3c7c3e-9695-4364-8ede-5274813096a5-kube-api-access-dtj27\") pod \"5f3c7c3e-9695-4364-8ede-5274813096a5\" (UID: \"5f3c7c3e-9695-4364-8ede-5274813096a5\") "
Jul 24 15:59:36.423329 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.423271    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8r2g\" (UniqueName: \"kubernetes.io/projected/d447f9d4-1a69-41a2-a6f0-7d4d6b2b61c0-kube-api-access-f8r2g\") pod \"d447f9d4-1a69-41a2-a6f0-7d4d6b2b61c0\" (UID: \"d447f9d4-1a69-41a2-a6f0-7d4d6b2b61c0\") "
Jul 24 15:59:36.423329 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.423317    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bb038d2-9151-42db-b6a1-9e4b6286d220-openshift-service-ca\") pod \"3bb038d2-9151-42db-b6a1-9e4b6286d220\" (UID: \"3bb038d2-9151-42db-b6a1-9e4b6286d220\") "
Jul 24 15:59:36.423663 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.423548    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d447f9d4-1a69-41a2-a6f0-7d4d6b2b61c0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d447f9d4-1a69-41a2-a6f0-7d4d6b2b61c0" (UID: "d447f9d4-1a69-41a2-a6f0-7d4d6b2b61c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:59:36.423774 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.423743    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f3c7c3e-9695-4364-8ede-5274813096a5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5f3c7c3e-9695-4364-8ede-5274813096a5" (UID: "5f3c7c3e-9695-4364-8ede-5274813096a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:59:36.423931 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.423771    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bb038d2-9151-42db-b6a1-9e4b6286d220-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3bb038d2-9151-42db-b6a1-9e4b6286d220" (UID: "3bb038d2-9151-42db-b6a1-9e4b6286d220"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:59:36.425517 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.425494    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bb038d2-9151-42db-b6a1-9e4b6286d220-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3bb038d2-9151-42db-b6a1-9e4b6286d220" (UID: "3bb038d2-9151-42db-b6a1-9e4b6286d220"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 15:59:36.425616 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.425514    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d447f9d4-1a69-41a2-a6f0-7d4d6b2b61c0-kube-api-access-f8r2g" (OuterVolumeSpecName: "kube-api-access-f8r2g") pod "d447f9d4-1a69-41a2-a6f0-7d4d6b2b61c0" (UID: "d447f9d4-1a69-41a2-a6f0-7d4d6b2b61c0"). InnerVolumeSpecName "kube-api-access-f8r2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:59:36.425616 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.425557    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bb038d2-9151-42db-b6a1-9e4b6286d220-kube-api-access-j4g5b" (OuterVolumeSpecName: "kube-api-access-j4g5b") pod "3bb038d2-9151-42db-b6a1-9e4b6286d220" (UID: "3bb038d2-9151-42db-b6a1-9e4b6286d220"). InnerVolumeSpecName "kube-api-access-j4g5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:59:36.425616 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.425565    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f3c7c3e-9695-4364-8ede-5274813096a5-kube-api-access-dtj27" (OuterVolumeSpecName: "kube-api-access-dtj27") pod "5f3c7c3e-9695-4364-8ede-5274813096a5" (UID: "5f3c7c3e-9695-4364-8ede-5274813096a5"). InnerVolumeSpecName "kube-api-access-dtj27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:59:36.524463 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.524434    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bb038d2-9151-42db-b6a1-9e4b6286d220-maas-api-tls\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:59:36.524463 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.524465    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtj27\" (UniqueName: \"kubernetes.io/projected/5f3c7c3e-9695-4364-8ede-5274813096a5-kube-api-access-dtj27\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:59:36.524652 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.524475    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8r2g\" (UniqueName: \"kubernetes.io/projected/d447f9d4-1a69-41a2-a6f0-7d4d6b2b61c0-kube-api-access-f8r2g\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:59:36.524652 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.524483    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bb038d2-9151-42db-b6a1-9e4b6286d220-openshift-service-ca\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:59:36.524652 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.524493    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d447f9d4-1a69-41a2-a6f0-7d4d6b2b61c0-plugins-config-volume\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:59:36.524652 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.524501    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f3c7c3e-9695-4364-8ede-5274813096a5-plugins-config-volume\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:59:36.524652 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.524511    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4g5b\" (UniqueName: \"kubernetes.io/projected/3bb038d2-9151-42db-b6a1-9e4b6286d220-kube-api-access-j4g5b\") on node \"ip-10-0-140-160.ec2.internal\" DevicePath \"\""
Jul 24 15:59:36.641419 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.641380    2579 generic.go:356] "Generic (PLEG): container finished" podID="3bb038d2-9151-42db-b6a1-9e4b6286d220" containerID="0dd6ee28feef6e4ea03ee08d9576a1f3059ce9dc2f9ec99cfc4b1d0a5b1cda21" exitCode=0
Jul 24 15:59:36.641419 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.641424    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1578f9-c7bbcd9c-mhqkm"
Jul 24 15:59:36.641701 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.641464    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1578f9-c7bbcd9c-mhqkm" event={"ID":"3bb038d2-9151-42db-b6a1-9e4b6286d220","Type":"ContainerDied","Data":"0dd6ee28feef6e4ea03ee08d9576a1f3059ce9dc2f9ec99cfc4b1d0a5b1cda21"}
Jul 24 15:59:36.641701 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.641522    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1578f9-c7bbcd9c-mhqkm" event={"ID":"3bb038d2-9151-42db-b6a1-9e4b6286d220","Type":"ContainerDied","Data":"0b0701295d5f375f2419075dec6e6c81303812b57ab4bd9d259fa9cc5ce86ab6"}
Jul 24 15:59:36.641701 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.641556    2579 scope.go:117] "RemoveContainer" containerID="0dd6ee28feef6e4ea03ee08d9576a1f3059ce9dc2f9ec99cfc4b1d0a5b1cda21"
Jul 24 15:59:36.642959 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.642934    2579 generic.go:356] "Generic (PLEG): container finished" podID="5f3c7c3e-9695-4364-8ede-5274813096a5" containerID="431d64ee4f2d69a05bb6dd58fccedb2c86f56d0d9c1c7f5d48368db78f667981" exitCode=0
Jul 24 15:59:36.643069 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.643017    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1578f9-647bdc8c88-fqq67" event={"ID":"5f3c7c3e-9695-4364-8ede-5274813096a5","Type":"ContainerDied","Data":"431d64ee4f2d69a05bb6dd58fccedb2c86f56d0d9c1c7f5d48368db78f667981"}
Jul 24 15:59:36.643136 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.643079    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1578f9-647bdc8c88-fqq67" event={"ID":"5f3c7c3e-9695-4364-8ede-5274813096a5","Type":"ContainerDied","Data":"5ccf6813566e0a48c8726c7392af0046be3314ea55bb472a16d7f09826b29eca"}
Jul 24 15:59:36.643136 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.643028    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1578f9-647bdc8c88-fqq67"
Jul 24 15:59:36.644470 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.644445    2579 generic.go:356] "Generic (PLEG): container finished" podID="d447f9d4-1a69-41a2-a6f0-7d4d6b2b61c0" containerID="acbffae8806269dd6ced1985d12344b6d4825639a8e8daeb732564057d6eba83" exitCode=0
Jul 24 15:59:36.644590 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.644483    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-1578f9-7fcfcd5566-hkxm4"
Jul 24 15:59:36.644590 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.644565    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-1578f9-7fcfcd5566-hkxm4" event={"ID":"d447f9d4-1a69-41a2-a6f0-7d4d6b2b61c0","Type":"ContainerDied","Data":"acbffae8806269dd6ced1985d12344b6d4825639a8e8daeb732564057d6eba83"}
Jul 24 15:59:36.644722 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.644610    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-1578f9-7fcfcd5566-hkxm4" event={"ID":"d447f9d4-1a69-41a2-a6f0-7d4d6b2b61c0","Type":"ContainerDied","Data":"f88cd8d6f3807fba8265cd5cdb0901e01895a15191d46829d0569206a77d920c"}
Jul 24 15:59:36.657570 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.657553    2579 scope.go:117] "RemoveContainer" containerID="0dd6ee28feef6e4ea03ee08d9576a1f3059ce9dc2f9ec99cfc4b1d0a5b1cda21"
Jul 24 15:59:36.657998 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:59:36.657870    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dd6ee28feef6e4ea03ee08d9576a1f3059ce9dc2f9ec99cfc4b1d0a5b1cda21\": container with ID starting with 0dd6ee28feef6e4ea03ee08d9576a1f3059ce9dc2f9ec99cfc4b1d0a5b1cda21 not found: ID does not exist" containerID="0dd6ee28feef6e4ea03ee08d9576a1f3059ce9dc2f9ec99cfc4b1d0a5b1cda21"
Jul 24 15:59:36.657998 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.657903    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dd6ee28feef6e4ea03ee08d9576a1f3059ce9dc2f9ec99cfc4b1d0a5b1cda21"} err="failed to get container status \"0dd6ee28feef6e4ea03ee08d9576a1f3059ce9dc2f9ec99cfc4b1d0a5b1cda21\": rpc error: code = NotFound desc = could not find container \"0dd6ee28feef6e4ea03ee08d9576a1f3059ce9dc2f9ec99cfc4b1d0a5b1cda21\": container with ID starting with 0dd6ee28feef6e4ea03ee08d9576a1f3059ce9dc2f9ec99cfc4b1d0a5b1cda21 not found: ID does not exist"
Jul 24 15:59:36.657998 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.657924    2579 scope.go:117] "RemoveContainer" containerID="431d64ee4f2d69a05bb6dd58fccedb2c86f56d0d9c1c7f5d48368db78f667981"
Jul 24 15:59:36.665713 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.665695    2579 scope.go:117] "RemoveContainer" containerID="431d64ee4f2d69a05bb6dd58fccedb2c86f56d0d9c1c7f5d48368db78f667981"
Jul 24 15:59:36.666065 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:59:36.666048    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"431d64ee4f2d69a05bb6dd58fccedb2c86f56d0d9c1c7f5d48368db78f667981\": container with ID starting with 431d64ee4f2d69a05bb6dd58fccedb2c86f56d0d9c1c7f5d48368db78f667981 not found: ID does not exist" containerID="431d64ee4f2d69a05bb6dd58fccedb2c86f56d0d9c1c7f5d48368db78f667981"
Jul 24 15:59:36.666139 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.666070    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"431d64ee4f2d69a05bb6dd58fccedb2c86f56d0d9c1c7f5d48368db78f667981"} err="failed to get container status \"431d64ee4f2d69a05bb6dd58fccedb2c86f56d0d9c1c7f5d48368db78f667981\": rpc error: code = NotFound desc = could not find container \"431d64ee4f2d69a05bb6dd58fccedb2c86f56d0d9c1c7f5d48368db78f667981\": container with ID starting with 431d64ee4f2d69a05bb6dd58fccedb2c86f56d0d9c1c7f5d48368db78f667981 not found: ID does not exist"
Jul 24 15:59:36.666139 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.666087    2579 scope.go:117] "RemoveContainer" containerID="acbffae8806269dd6ced1985d12344b6d4825639a8e8daeb732564057d6eba83"
Jul 24 15:59:36.670519 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.670497    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1578f9-c7bbcd9c-mhqkm"]
Jul 24 15:59:36.675312 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.675293    2579 scope.go:117] "RemoveContainer" containerID="acbffae8806269dd6ced1985d12344b6d4825639a8e8daeb732564057d6eba83"
Jul 24 15:59:36.675508 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.675482    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1578f9-c7bbcd9c-mhqkm"]
Jul 24 15:59:36.675623 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:59:36.675593    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acbffae8806269dd6ced1985d12344b6d4825639a8e8daeb732564057d6eba83\": container with ID starting with acbffae8806269dd6ced1985d12344b6d4825639a8e8daeb732564057d6eba83 not found: ID does not exist" containerID="acbffae8806269dd6ced1985d12344b6d4825639a8e8daeb732564057d6eba83"
Jul 24 15:59:36.675695 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.675613    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acbffae8806269dd6ced1985d12344b6d4825639a8e8daeb732564057d6eba83"} err="failed to get container status \"acbffae8806269dd6ced1985d12344b6d4825639a8e8daeb732564057d6eba83\": rpc error: code = NotFound desc = could not find container \"acbffae8806269dd6ced1985d12344b6d4825639a8e8daeb732564057d6eba83\": container with ID starting with acbffae8806269dd6ced1985d12344b6d4825639a8e8daeb732564057d6eba83 not found: ID does not exist"
Jul 24 15:59:36.684757 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.684734    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-1578f9-647bdc8c88-fqq67"]
Jul 24 15:59:36.689509 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.689487    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-1578f9-647bdc8c88-fqq67"]
Jul 24 15:59:36.691529 ip-10-0-140-160 kubenswrapper[2579]: E0724 15:59:36.691498    2579 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bb038d2_9151_42db_b6a1_9e4b6286d220.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd447f9d4_1a69_41a2_a6f0_7d4d6b2b61c0.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd447f9d4_1a69_41a2_a6f0_7d4d6b2b61c0.slice/crio-f88cd8d6f3807fba8265cd5cdb0901e01895a15191d46829d0569206a77d920c\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f3c7c3e_9695_4364_8ede_5274813096a5.slice/crio-5ccf6813566e0a48c8726c7392af0046be3314ea55bb472a16d7f09826b29eca\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bb038d2_9151_42db_b6a1_9e4b6286d220.slice/crio-0b0701295d5f375f2419075dec6e6c81303812b57ab4bd9d259fa9cc5ce86ab6\": RecentStats: unable to find data in memory cache]"
Jul 24 15:59:36.697611 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.697582    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-1578f9-7fcfcd5566-hkxm4"]
Jul 24 15:59:36.699553 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:36.699530    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-1578f9-7fcfcd5566-hkxm4"]
Jul 24 15:59:37.718472 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:37.718433    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bb038d2-9151-42db-b6a1-9e4b6286d220" path="/var/lib/kubelet/pods/3bb038d2-9151-42db-b6a1-9e4b6286d220/volumes"
Jul 24 15:59:37.718860 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:37.718797    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f3c7c3e-9695-4364-8ede-5274813096a5" path="/var/lib/kubelet/pods/5f3c7c3e-9695-4364-8ede-5274813096a5/volumes"
Jul 24 15:59:37.719145 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:37.719132    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d447f9d4-1a69-41a2-a6f0-7d4d6b2b61c0" path="/var/lib/kubelet/pods/d447f9d4-1a69-41a2-a6f0-7d4d6b2b61c0/volumes"
Jul 24 15:59:39.019248 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:39.019215    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dd79h"]
Jul 24 15:59:48.935051 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:48.935018    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-55bbff9d46-fjk7k_1f7f8458-c4d3-4f25-aa7c-b134af1c64cd/authorino/0.log"
Jul 24 15:59:55.150406 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:55.150371    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cff5b748-n772p_5ffd42f8-a238-4b6f-b1fd-80cecfaac0f8/manager/0.log"
Jul 24 15:59:55.161744 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:55.161719    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cff5b748-n772p_5ffd42f8-a238-4b6f-b1fd-80cecfaac0f8/copy-manifests/0.log"
Jul 24 15:59:55.733565 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:55.733531    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29748450-h2lwk_288486bd-44c1-46ec-8b2f-de18bb743daa/cleanup/0.log"
Jul 24 15:59:57.179297 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:57.179261    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-55bbff9d46-fjk7k_1f7f8458-c4d3-4f25-aa7c-b134af1c64cd/authorino/0.log"
Jul 24 15:59:57.842609 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:57.842583    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-dd79h_20959482-70ef-4376-a727-48d94348946c/limitador/0.log"
Jul 24 15:59:58.619994 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:58.619967    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-587457bbc9-tz554_b25451af-04f1-4aeb-88ad-ffc1d422c54d/kube-auth-proxy/0.log"
Jul 24 15:59:59.076542 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:59.076515    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-55db6d66b7-2bssd_7eb671c8-eae8-409d-943a-c8f76ddede9f/router/0.log"
Jul 24 15:59:59.522496 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:59.522467    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-89b5c_b246993f-a343-4117-9ed9-7ef370e2e8df/main/0.log"
Jul 24 15:59:59.962587 ip-10-0-140-160 kubenswrapper[2579]: I0724 15:59:59.962509    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-tk5h2_ad3fa403-b926-4005-bf62-95542d6c32c2/main/0.log"
Jul 24 16:00:04.900884 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:04.900850    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-w5b48/must-gather-gkt9b"]
Jul 24 16:00:04.901277 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:04.901237    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bb038d2-9151-42db-b6a1-9e4b6286d220" containerName="maas-api"
Jul 24 16:00:04.901277 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:04.901248    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bb038d2-9151-42db-b6a1-9e4b6286d220" containerName="maas-api"
Jul 24 16:00:04.901277 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:04.901258    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f3c7c3e-9695-4364-8ede-5274813096a5" containerName="payload-pre-processing"
Jul 24 16:00:04.901277 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:04.901264    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f3c7c3e-9695-4364-8ede-5274813096a5" containerName="payload-pre-processing"
Jul 24 16:00:04.901277 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:04.901272    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cd34332-3dd4-4350-9fc0-26f1fee180f0" containerName="payload-pre-processing"
Jul 24 16:00:04.901277 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:04.901279    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cd34332-3dd4-4350-9fc0-26f1fee180f0" containerName="payload-pre-processing"
Jul 24 16:00:04.901480 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:04.901290    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3c304a1-65d0-4a80-8868-90d5471ecf4b" containerName="payload-processing"
Jul 24 16:00:04.901480 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:04.901295    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3c304a1-65d0-4a80-8868-90d5471ecf4b" containerName="payload-processing"
Jul 24 16:00:04.901480 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:04.901302    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d447f9d4-1a69-41a2-a6f0-7d4d6b2b61c0" containerName="payload-processing"
Jul 24 16:00:04.901480 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:04.901308    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d447f9d4-1a69-41a2-a6f0-7d4d6b2b61c0" containerName="payload-processing"
Jul 24 16:00:04.901480 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:04.901313    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63faa5c3-0db1-4f31-bd0a-b6f73e0f710c" containerName="payload-processing"
Jul 24 16:00:04.901480 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:04.901318    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="63faa5c3-0db1-4f31-bd0a-b6f73e0f710c" containerName="payload-processing"
Jul 24 16:00:04.901480 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:04.901332    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2b13488-f677-4987-bafe-df99527776e7" containerName="payload-pre-processing"
Jul 24 16:00:04.901480 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:04.901337    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2b13488-f677-4987-bafe-df99527776e7" containerName="payload-pre-processing"
Jul 24 16:00:04.901480 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:04.901344    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19cffcc0-896f-436f-a962-b6291852f1b3" containerName="payload-processing"
Jul 24 16:00:04.901480 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:04.901349    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="19cffcc0-896f-436f-a962-b6291852f1b3" containerName="payload-processing"
Jul 24 16:00:04.901480 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:04.901356    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71111d85-3b11-48fe-a587-54c105cb74d7" containerName="payload-processing"
Jul 24 16:00:04.901480 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:04.901362    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="71111d85-3b11-48fe-a587-54c105cb74d7" containerName="payload-processing"
Jul 24 16:00:04.901480 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:04.901370    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af885fa0-3295-4818-83b1-d38e540f292e" containerName="maas-api"
Jul 24 16:00:04.901480 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:04.901376    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="af885fa0-3295-4818-83b1-d38e540f292e" containerName="maas-api"
Jul 24 16:00:04.901480 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:04.901381    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="426663e4-70cb-44b7-a7cb-444ecdbae049" containerName="main"
Jul 24 16:00:04.901480 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:04.901385    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="426663e4-70cb-44b7-a7cb-444ecdbae049" containerName="main"
Jul 24 16:00:04.901480 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:04.901445    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="426663e4-70cb-44b7-a7cb-444ecdbae049" containerName="main"
Jul 24 16:00:04.901480 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:04.901453    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="af885fa0-3295-4818-83b1-d38e540f292e" containerName="maas-api"
Jul 24 16:00:04.901480 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:04.901458    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bb038d2-9151-42db-b6a1-9e4b6286d220" containerName="maas-api"
Jul 24 16:00:04.901480 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:04.901464    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f3c7c3e-9695-4364-8ede-5274813096a5" containerName="payload-pre-processing"
Jul 24 16:00:04.901480 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:04.901470    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d447f9d4-1a69-41a2-a6f0-7d4d6b2b61c0" containerName="payload-processing"
Jul 24 16:00:04.901480 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:04.901474    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3c304a1-65d0-4a80-8868-90d5471ecf4b" containerName="payload-processing"
Jul 24 16:00:04.901480 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:04.901479    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2b13488-f677-4987-bafe-df99527776e7" containerName="payload-pre-processing"
Jul 24 16:00:04.901480 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:04.901485    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="19cffcc0-896f-436f-a962-b6291852f1b3" containerName="payload-processing"
Jul 24 16:00:04.901480 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:04.901491    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="71111d85-3b11-48fe-a587-54c105cb74d7" containerName="payload-processing"
Jul 24 16:00:04.902257 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:04.901497    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cd34332-3dd4-4350-9fc0-26f1fee180f0" containerName="payload-pre-processing"
Jul 24 16:00:04.902257 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:04.901503    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="63faa5c3-0db1-4f31-bd0a-b6f73e0f710c" containerName="payload-processing"
Jul 24 16:00:04.904579 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:04.904562    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w5b48/must-gather-gkt9b"
Jul 24 16:00:04.907121 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:04.907100    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-w5b48\"/\"default-dockercfg-qcxq2\""
Jul 24 16:00:04.907356 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:04.907340    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w5b48\"/\"kube-root-ca.crt\""
Jul 24 16:00:04.907615 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:04.907603    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w5b48\"/\"openshift-service-ca.crt\""
Jul 24 16:00:04.920000 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:04.919969    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w5b48/must-gather-gkt9b"]
Jul 24 16:00:04.992016 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:04.991974    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5lpc\" (UniqueName: \"kubernetes.io/projected/881a1e29-1704-416e-88db-57fa190bd684-kube-api-access-n5lpc\") pod \"must-gather-gkt9b\" (UID: \"881a1e29-1704-416e-88db-57fa190bd684\") " pod="openshift-must-gather-w5b48/must-gather-gkt9b"
Jul 24 16:00:04.992203 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:04.992031    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/881a1e29-1704-416e-88db-57fa190bd684-must-gather-output\") pod \"must-gather-gkt9b\" (UID: \"881a1e29-1704-416e-88db-57fa190bd684\") " pod="openshift-must-gather-w5b48/must-gather-gkt9b"
Jul 24 16:00:05.093041 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:05.092991    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5lpc\" (UniqueName: \"kubernetes.io/projected/881a1e29-1704-416e-88db-57fa190bd684-kube-api-access-n5lpc\") pod \"must-gather-gkt9b\" (UID: \"881a1e29-1704-416e-88db-57fa190bd684\") " pod="openshift-must-gather-w5b48/must-gather-gkt9b"
Jul 24 16:00:05.093235 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:05.093059    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/881a1e29-1704-416e-88db-57fa190bd684-must-gather-output\") pod \"must-gather-gkt9b\" (UID: \"881a1e29-1704-416e-88db-57fa190bd684\") " pod="openshift-must-gather-w5b48/must-gather-gkt9b"
Jul 24 16:00:05.093443 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:05.093421    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/881a1e29-1704-416e-88db-57fa190bd684-must-gather-output\") pod \"must-gather-gkt9b\" (UID: \"881a1e29-1704-416e-88db-57fa190bd684\") " pod="openshift-must-gather-w5b48/must-gather-gkt9b"
Jul 24 16:00:05.104150 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:05.104115    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5lpc\" (UniqueName: \"kubernetes.io/projected/881a1e29-1704-416e-88db-57fa190bd684-kube-api-access-n5lpc\") pod \"must-gather-gkt9b\" (UID: \"881a1e29-1704-416e-88db-57fa190bd684\") " pod="openshift-must-gather-w5b48/must-gather-gkt9b"
Jul 24 16:00:05.216213 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:05.216115    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w5b48/must-gather-gkt9b"
Jul 24 16:00:05.349738 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:05.349700    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w5b48/must-gather-gkt9b"]
Jul 24 16:00:05.354093 ip-10-0-140-160 kubenswrapper[2579]: W0724 16:00:05.354057    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod881a1e29_1704_416e_88db_57fa190bd684.slice/crio-7905ac5bb33c65b1ff530f1c2caa932dc4e42f9bfd8887be87500b8c2b392fe8 WatchSource:0}: Error finding container 7905ac5bb33c65b1ff530f1c2caa932dc4e42f9bfd8887be87500b8c2b392fe8: Status 404 returned error can't find the container with id 7905ac5bb33c65b1ff530f1c2caa932dc4e42f9bfd8887be87500b8c2b392fe8
Jul 24 16:00:05.356029 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:05.356008    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 16:00:05.792506 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:05.792474    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w5b48/must-gather-gkt9b" event={"ID":"881a1e29-1704-416e-88db-57fa190bd684","Type":"ContainerStarted","Data":"7905ac5bb33c65b1ff530f1c2caa932dc4e42f9bfd8887be87500b8c2b392fe8"}
Jul 24 16:00:06.801664 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:06.801573    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w5b48/must-gather-gkt9b" event={"ID":"881a1e29-1704-416e-88db-57fa190bd684","Type":"ContainerStarted","Data":"0c2756259a08d5f9e065df09d9ae81603f8d8dff69097b067532a13c505d94ba"}
Jul 24 16:00:06.801664 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:06.801630    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w5b48/must-gather-gkt9b" event={"ID":"881a1e29-1704-416e-88db-57fa190bd684","Type":"ContainerStarted","Data":"20e1c3068769117c06d1c764c00a51c59a5c2e1495352381017fd901b19349d0"}
Jul 24 16:00:06.822404 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:06.822344    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-w5b48/must-gather-gkt9b" podStartSLOduration=1.964803153 podStartE2EDuration="2.822328668s" totalImagesPullingTime="857.525515ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:00:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:00:06.818499187 +0000 UTC m=+3099.630851366" watchObservedRunningTime="2026-07-24 16:00:06.822328668 +0000 UTC m=+3099.634680847"
Jul 24 16:00:07.879230 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:07.879201    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7qc8n_5a7d4e75-bfc5-4fe8-8ced-eb1ba5ba2a70/global-pull-secret-syncer/0.log"
Jul 24 16:00:08.073368 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:08.073334    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-d7d7z_e4dfc4ee-1d7a-4e52-a295-efc820c560bd/konnectivity-agent/0.log"
Jul 24 16:00:08.191174 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:08.191097    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-160.ec2.internal_6aa67c3d9e8eb538931b1417f3d45520/haproxy/0.log"
Jul 24 16:00:13.260958 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:13.260370    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-55bbff9d46-fjk7k_1f7f8458-c4d3-4f25-aa7c-b134af1c64cd/authorino/0.log"
Jul 24 16:00:13.559713 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:13.559685    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-dd79h_20959482-70ef-4376-a727-48d94348946c/limitador/0.log"
Jul 24 16:00:15.381156 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:15.381108    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00/alertmanager/0.log"
Jul 24 16:00:15.405037 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:15.404972    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00/config-reloader/0.log"
Jul 24 16:00:15.437652 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:15.437618    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00/kube-rbac-proxy-web/0.log"
Jul 24 16:00:15.470652 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:15.470597    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00/kube-rbac-proxy/0.log"
Jul 24 16:00:15.495728 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:15.495652    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00/kube-rbac-proxy-metric/0.log"
Jul 24 16:00:15.525958 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:15.525928    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00/prom-label-proxy/0.log"
Jul 24 16:00:15.554085 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:15.554044    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b41ede1a-6b93-42d4-a4bd-3bf5f1a3ef00/init-config-reloader/0.log"
Jul 24 16:00:15.707086 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:15.707052    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-lwh8n_c6bc6e76-0a71-4834-9c9b-9f81846580ea/kube-state-metrics/0.log"
Jul 24 16:00:15.731981 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:15.731948    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-lwh8n_c6bc6e76-0a71-4834-9c9b-9f81846580ea/kube-rbac-proxy-main/0.log"
Jul 24 16:00:15.756661 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:15.756578    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-lwh8n_c6bc6e76-0a71-4834-9c9b-9f81846580ea/kube-rbac-proxy-self/0.log"
Jul 24 16:00:15.931714 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:15.931689    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q79n7_280a572b-d0e3-4074-9c59-51792e5adb32/node-exporter/0.log"
Jul 24 16:00:15.974315 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:15.974281    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q79n7_280a572b-d0e3-4074-9c59-51792e5adb32/kube-rbac-proxy/0.log"
Jul 24 16:00:15.999702 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:15.999674    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q79n7_280a572b-d0e3-4074-9c59-51792e5adb32/init-textfile/0.log"
Jul 24 16:00:16.125673 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:16.125640    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-hb5nq_7623b444-a307-4704-8ee5-0a53778a716f/kube-rbac-proxy-main/0.log"
Jul 24 16:00:16.151148 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:16.151120    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-hb5nq_7623b444-a307-4704-8ee5-0a53778a716f/kube-rbac-proxy-self/0.log"
Jul 24 16:00:16.175476 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:16.175446    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-hb5nq_7623b444-a307-4704-8ee5-0a53778a716f/openshift-state-metrics/0.log"
Jul 24 16:00:16.219497 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:16.219463    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ff4c5089-e2d7-4345-9761-9d96a4c41a0f/prometheus/0.log"
Jul 24 16:00:16.242908 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:16.242874    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ff4c5089-e2d7-4345-9761-9d96a4c41a0f/config-reloader/0.log"
Jul 24 16:00:16.266110 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:16.266085    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ff4c5089-e2d7-4345-9761-9d96a4c41a0f/thanos-sidecar/0.log"
Jul 24 16:00:16.288270 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:16.288191    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ff4c5089-e2d7-4345-9761-9d96a4c41a0f/kube-rbac-proxy-web/0.log"
Jul 24 16:00:16.311265 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:16.311240    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ff4c5089-e2d7-4345-9761-9d96a4c41a0f/kube-rbac-proxy/0.log"
Jul 24 16:00:16.341290 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:16.341262    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ff4c5089-e2d7-4345-9761-9d96a4c41a0f/kube-rbac-proxy-thanos/0.log"
Jul 24 16:00:16.364799 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:16.364771    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ff4c5089-e2d7-4345-9761-9d96a4c41a0f/init-config-reloader/0.log"
Jul 24 16:00:16.445373 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:16.445338    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-2n2ms_483059f3-b83b-4def-90d9-d42dd79d20e1/prometheus-operator-admission-webhook/0.log"
Jul 24 16:00:16.553387 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:16.553315    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67c4bd47f4-pt275_cfebcd8d-9ec5-400b-ad42-d5fc438d8e15/thanos-query/0.log"
Jul 24 16:00:16.577954 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:16.577923    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67c4bd47f4-pt275_cfebcd8d-9ec5-400b-ad42-d5fc438d8e15/kube-rbac-proxy-web/0.log"
Jul 24 16:00:16.600968 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:16.600934    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67c4bd47f4-pt275_cfebcd8d-9ec5-400b-ad42-d5fc438d8e15/kube-rbac-proxy/0.log"
Jul 24 16:00:16.628961 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:16.628931    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67c4bd47f4-pt275_cfebcd8d-9ec5-400b-ad42-d5fc438d8e15/prom-label-proxy/0.log"
Jul 24 16:00:16.651928 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:16.651889    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67c4bd47f4-pt275_cfebcd8d-9ec5-400b-ad42-d5fc438d8e15/kube-rbac-proxy-rules/0.log"
Jul 24 16:00:16.673797 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:16.673765    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67c4bd47f4-pt275_cfebcd8d-9ec5-400b-ad42-d5fc438d8e15/kube-rbac-proxy-metrics/0.log"
Jul 24 16:00:17.733687 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:17.733656    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-w5b48/perf-node-gather-daemonset-7674t"]
Jul 24 16:00:17.742840 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:17.742792    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-7674t"
Jul 24 16:00:17.747527 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:17.747494    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w5b48/perf-node-gather-daemonset-7674t"]
Jul 24 16:00:17.837211 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:17.837166    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/913b71ad-5018-4cb0-a94b-c52384d1587e-podres\") pod \"perf-node-gather-daemonset-7674t\" (UID: \"913b71ad-5018-4cb0-a94b-c52384d1587e\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-7674t"
Jul 24 16:00:17.837211 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:17.837214    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/913b71ad-5018-4cb0-a94b-c52384d1587e-proc\") pod \"perf-node-gather-daemonset-7674t\" (UID: \"913b71ad-5018-4cb0-a94b-c52384d1587e\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-7674t"
Jul 24 16:00:17.837503 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:17.837282    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sxb2\" (UniqueName: \"kubernetes.io/projected/913b71ad-5018-4cb0-a94b-c52384d1587e-kube-api-access-2sxb2\") pod \"perf-node-gather-daemonset-7674t\" (UID: \"913b71ad-5018-4cb0-a94b-c52384d1587e\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-7674t"
Jul 24 16:00:17.837503 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:17.837344    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/913b71ad-5018-4cb0-a94b-c52384d1587e-lib-modules\") pod \"perf-node-gather-daemonset-7674t\" (UID: \"913b71ad-5018-4cb0-a94b-c52384d1587e\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-7674t"
Jul 24 16:00:17.837503 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:17.837404    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/913b71ad-5018-4cb0-a94b-c52384d1587e-sys\") pod \"perf-node-gather-daemonset-7674t\" (UID: \"913b71ad-5018-4cb0-a94b-c52384d1587e\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-7674t"
Jul 24 16:00:17.938481 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:17.938438    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/913b71ad-5018-4cb0-a94b-c52384d1587e-lib-modules\") pod \"perf-node-gather-daemonset-7674t\" (UID: \"913b71ad-5018-4cb0-a94b-c52384d1587e\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-7674t"
Jul 24 16:00:17.938681 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:17.938527    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/913b71ad-5018-4cb0-a94b-c52384d1587e-sys\") pod \"perf-node-gather-daemonset-7674t\" (UID: \"913b71ad-5018-4cb0-a94b-c52384d1587e\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-7674t"
Jul 24 16:00:17.938681 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:17.938583    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/913b71ad-5018-4cb0-a94b-c52384d1587e-podres\") pod \"perf-node-gather-daemonset-7674t\" (UID: \"913b71ad-5018-4cb0-a94b-c52384d1587e\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-7674t"
Jul 24 16:00:17.938681 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:17.938612    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/913b71ad-5018-4cb0-a94b-c52384d1587e-proc\") pod \"perf-node-gather-daemonset-7674t\" (UID: \"913b71ad-5018-4cb0-a94b-c52384d1587e\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-7674t"
Jul 24 16:00:17.938902 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:17.938681    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sxb2\" (UniqueName: \"kubernetes.io/projected/913b71ad-5018-4cb0-a94b-c52384d1587e-kube-api-access-2sxb2\") pod \"perf-node-gather-daemonset-7674t\" (UID: \"913b71ad-5018-4cb0-a94b-c52384d1587e\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-7674t"
Jul 24 16:00:17.939236 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:17.939208    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/913b71ad-5018-4cb0-a94b-c52384d1587e-lib-modules\") pod \"perf-node-gather-daemonset-7674t\" (UID: \"913b71ad-5018-4cb0-a94b-c52384d1587e\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-7674t"
Jul 24 16:00:17.939323 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:17.939287    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/913b71ad-5018-4cb0-a94b-c52384d1587e-sys\") pod \"perf-node-gather-daemonset-7674t\" (UID: \"913b71ad-5018-4cb0-a94b-c52384d1587e\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-7674t"
Jul 24 16:00:17.939394 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:17.939377    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/913b71ad-5018-4cb0-a94b-c52384d1587e-podres\") pod \"perf-node-gather-daemonset-7674t\" (UID: \"913b71ad-5018-4cb0-a94b-c52384d1587e\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-7674t"
Jul 24 16:00:17.939457 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:17.939433    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/913b71ad-5018-4cb0-a94b-c52384d1587e-proc\") pod \"perf-node-gather-daemonset-7674t\" (UID: \"913b71ad-5018-4cb0-a94b-c52384d1587e\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-7674t"
Jul 24 16:00:17.949516 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:17.949484    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sxb2\" (UniqueName: \"kubernetes.io/projected/913b71ad-5018-4cb0-a94b-c52384d1587e-kube-api-access-2sxb2\") pod \"perf-node-gather-daemonset-7674t\" (UID: \"913b71ad-5018-4cb0-a94b-c52384d1587e\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-7674t"
Jul 24 16:00:18.060211 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:18.060171    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-7674t"
Jul 24 16:00:18.238838 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:18.238790    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w5b48/perf-node-gather-daemonset-7674t"]
Jul 24 16:00:18.243988 ip-10-0-140-160 kubenswrapper[2579]: W0724 16:00:18.243948    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod913b71ad_5018_4cb0_a94b_c52384d1587e.slice/crio-93b4f18e598f0fe35a82177a7dc0feb5ba422a7520022e4fd4dcca0eb8e424a7 WatchSource:0}: Error finding container 93b4f18e598f0fe35a82177a7dc0feb5ba422a7520022e4fd4dcca0eb8e424a7: Status 404 returned error can't find the container with id 93b4f18e598f0fe35a82177a7dc0feb5ba422a7520022e4fd4dcca0eb8e424a7
Jul 24 16:00:18.895138 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:18.895094    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-7674t" event={"ID":"913b71ad-5018-4cb0-a94b-c52384d1587e","Type":"ContainerStarted","Data":"4da5996746e83bbccca008b7a76cf01dc19c0996a5a22cd50d986de557878fc2"}
Jul 24 16:00:18.895637 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:18.895183    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-7674t" event={"ID":"913b71ad-5018-4cb0-a94b-c52384d1587e","Type":"ContainerStarted","Data":"93b4f18e598f0fe35a82177a7dc0feb5ba422a7520022e4fd4dcca0eb8e424a7"}
Jul 24 16:00:18.895637 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:18.895226    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-7674t"
Jul 24 16:00:18.917165 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:18.917098    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-7674t" podStartSLOduration=1.917076269 podStartE2EDuration="1.917076269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 16:00:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:00:18.916067433 +0000 UTC m=+3111.728419612" watchObservedRunningTime="2026-07-24 16:00:18.917076269 +0000 UTC m=+3111.729428449"
Jul 24 16:00:19.103449 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:19.103410    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6649c54c98-ct7sx_9b8027fe-6846-486d-8288-e4842ba1b2d5/console/0.log"
Jul 24 16:00:19.716250 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:19.716212    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-2vhjm_443116d2-ada8-4261-abcb-113fa99e38ae/volume-data-source-validator/0.log"
Jul 24 16:00:20.758574 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:20.758541    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4vmc8_d4427fb4-a65e-4505-9329-88c2c1c4e812/dns/0.log"
Jul 24 16:00:20.779749 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:20.779723    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4vmc8_d4427fb4-a65e-4505-9329-88c2c1c4e812/kube-rbac-proxy/0.log"
Jul 24 16:00:20.947414 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:20.947386    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-s5nzq_ca98516e-79dc-4972-86de-8ff390bc0de2/dns-node-resolver/0.log"
Jul 24 16:00:21.568369 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:21.568335    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n9snp_2216188a-b1f1-4ddf-8f9e-346e93860df2/node-ca/0.log"
Jul 24 16:00:22.737988 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:22.737961    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-587457bbc9-tz554_b25451af-04f1-4aeb-88ad-ffc1d422c54d/kube-auth-proxy/0.log"
Jul 24 16:00:22.881528 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:22.881500    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-55db6d66b7-2bssd_7eb671c8-eae8-409d-943a-c8f76ddede9f/router/0.log"
Jul 24 16:00:23.701421 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:23.701374    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bks2n_53659e71-d188-4315-81ab-7b53e4701a4f/serve-healthcheck-canary/0.log"
Jul 24 16:00:24.310547 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:24.310512    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ncf4z_8910cead-2436-44b7-b425-d42a7063d4b8/kube-rbac-proxy/0.log"
Jul 24 16:00:24.333551 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:24.333518    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ncf4z_8910cead-2436-44b7-b425-d42a7063d4b8/exporter/0.log"
Jul 24 16:00:24.357621 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:24.357594    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ncf4z_8910cead-2436-44b7-b425-d42a7063d4b8/extractor/0.log"
Jul 24 16:00:24.912903 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:24.912446    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-7674t"
Jul 24 16:00:27.255940 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:27.254730    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cff5b748-n772p_5ffd42f8-a238-4b6f-b1fd-80cecfaac0f8/manager/0.log"
Jul 24 16:00:27.274033 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:27.273984    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cff5b748-n772p_5ffd42f8-a238-4b6f-b1fd-80cecfaac0f8/copy-manifests/0.log"
Jul 24 16:00:28.667080 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:28.667033    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-657dd8bf9c-lw5k2_47bce575-938f-44c6-9435-412e6a825e2c/manager/0.log"
Jul 24 16:00:28.718028 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:28.717995    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-587457bbc9-tz554_b25451af-04f1-4aeb-88ad-ffc1d422c54d/kube-auth-proxy/0.log"
Jul 24 16:00:28.831586 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:28.831552    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-55db6d66b7-2bssd_7eb671c8-eae8-409d-943a-c8f76ddede9f/router/0.log"
Jul 24 16:00:30.354577 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:30.354544    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-587457bbc9-tz554_b25451af-04f1-4aeb-88ad-ffc1d422c54d/kube-auth-proxy/0.log"
Jul 24 16:00:30.464136 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:30.464100    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-55db6d66b7-2bssd_7eb671c8-eae8-409d-943a-c8f76ddede9f/router/0.log"
Jul 24 16:00:33.899509 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:33.899478    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-xxl8c_d9cecba9-741c-4afe-8dc3-9bc7790c6278/migrator/0.log"
Jul 24 16:00:33.925227 ip-10-0-140-160 kubenswrapper[2579]: I0724 16:00:33.925200    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-xxl8c_d9cecba9-741c-4afe-8dc3-9bc7790c6278/graceful-termination/0.log"