Sep 09 05:49:33.307711 ip-10-0-131-142 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 05:49:33.748143 ip-10-0-131-142 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 05:49:33.748143 ip-10-0-131-142 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 05:49:33.748143 ip-10-0-131-142 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 05:49:33.748143 ip-10-0-131-142 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 05:49:33.748143 ip-10-0-131-142 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 05:49:33.750924 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.750826    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 05:49:33.758403 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758374    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 05:49:33.758403 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758403    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 05:49:33.758471 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758408    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 05:49:33.758471 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758411    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 05:49:33.758471 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758414    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 05:49:33.758471 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758417    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 05:49:33.758471 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758420    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 05:49:33.758471 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758422    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 05:49:33.758471 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758425    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 05:49:33.758471 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758433    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 05:49:33.758471 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758437    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 05:49:33.758471 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758440    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 05:49:33.758471 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758443    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 05:49:33.758471 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758446    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 05:49:33.758471 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758449    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 05:49:33.758471 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758452    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 05:49:33.758471 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758455    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 05:49:33.758471 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758458    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 05:49:33.758471 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758460    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 05:49:33.758471 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758463    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 05:49:33.758471 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758466    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 05:49:33.758471 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758468    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 05:49:33.758948 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758472    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 05:49:33.758948 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758475    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 05:49:33.758948 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758478    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 05:49:33.758948 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758481    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 05:49:33.758948 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758484    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 05:49:33.758948 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758486    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 05:49:33.758948 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758489    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 05:49:33.758948 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758492    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 05:49:33.758948 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758494    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 05:49:33.758948 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758497    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 05:49:33.758948 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758500    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 05:49:33.758948 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758503    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 05:49:33.758948 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758505    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 05:49:33.758948 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758508    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 05:49:33.758948 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758510    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 05:49:33.758948 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758512    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 05:49:33.758948 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758515    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 05:49:33.758948 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758518    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 05:49:33.758948 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758520    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 05:49:33.759447 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758523    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 05:49:33.759447 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758529    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 05:49:33.759447 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758531    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 05:49:33.759447 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758534    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 05:49:33.759447 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758536    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 05:49:33.759447 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758538    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 05:49:33.759447 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758541    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 05:49:33.759447 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758543    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 05:49:33.759447 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758545    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 05:49:33.759447 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758548    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 05:49:33.759447 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758550    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 05:49:33.759447 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758553    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 05:49:33.759447 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758555    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 05:49:33.759447 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758557    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 05:49:33.759447 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758561    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 05:49:33.759447 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758564    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 05:49:33.759447 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758566    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 05:49:33.759447 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758569    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 05:49:33.759447 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758572    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 05:49:33.759447 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758574    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 05:49:33.759941 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758577    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 05:49:33.759941 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758579    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 05:49:33.759941 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758582    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 05:49:33.759941 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758584    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 05:49:33.759941 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758589    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 05:49:33.759941 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758593    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 05:49:33.759941 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758596    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 05:49:33.759941 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758599    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 05:49:33.759941 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758603    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 05:49:33.759941 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758606    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 05:49:33.759941 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758609    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 05:49:33.759941 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758612    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 05:49:33.759941 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758615    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 05:49:33.759941 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758618    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 05:49:33.759941 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758620    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 05:49:33.759941 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758623    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 05:49:33.759941 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758625    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 05:49:33.759941 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758627    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 05:49:33.759941 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758630    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 05:49:33.760426 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758632    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 05:49:33.760426 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758635    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 05:49:33.760426 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758637    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 05:49:33.760426 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758640    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 05:49:33.760426 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758642    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 05:49:33.760426 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758644    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 05:49:33.760426 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758647    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 05:49:33.760426 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758650    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 05:49:33.760426 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758653    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 05:49:33.760426 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758656    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 05:49:33.760426 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758658    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 05:49:33.760426 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758662    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 05:49:33.760426 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758666    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 05:49:33.760426 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758669    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 05:49:33.760426 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758672    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 05:49:33.760426 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758674    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 05:49:33.760426 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758677    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 05:49:33.760426 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758679    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 05:49:33.760426 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.758682    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 05:49:33.760426 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759339    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 05:49:33.760938 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759345    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 05:49:33.760938 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759348    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 05:49:33.760938 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759351    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 05:49:33.760938 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759354    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 05:49:33.760938 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759357    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 05:49:33.760938 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759359    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 05:49:33.760938 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759362    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 05:49:33.760938 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759365    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 05:49:33.760938 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759368    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 05:49:33.760938 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759370    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 05:49:33.760938 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759373    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 05:49:33.760938 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759375    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 05:49:33.760938 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759378    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 05:49:33.760938 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759395    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 05:49:33.760938 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759400    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 05:49:33.760938 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759404    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 05:49:33.760938 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759407    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 05:49:33.760938 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759410    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 05:49:33.760938 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759413    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 05:49:33.760938 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759416    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 05:49:33.761458 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759419    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 05:49:33.761458 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759421    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 05:49:33.761458 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759424    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 05:49:33.761458 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759426    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 05:49:33.761458 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759429    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 05:49:33.761458 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759432    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 05:49:33.761458 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759435    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 05:49:33.761458 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759439    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 05:49:33.761458 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759441    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 05:49:33.761458 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759444    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 05:49:33.761458 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759446    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 05:49:33.761458 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759449    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 05:49:33.761458 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759453    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 05:49:33.761458 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759455    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 05:49:33.761458 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759458    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 05:49:33.761458 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759461    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 05:49:33.761458 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759463    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 05:49:33.761458 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759472    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 05:49:33.761458 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759475    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 05:49:33.761932 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759477    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 05:49:33.761932 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759480    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 05:49:33.761932 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759482    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 05:49:33.761932 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759485    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 05:49:33.761932 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759487    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 05:49:33.761932 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759490    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 05:49:33.761932 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759492    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 05:49:33.761932 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759494    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 05:49:33.761932 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759497    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 05:49:33.761932 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759500    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 05:49:33.761932 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759502    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 05:49:33.761932 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759505    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 05:49:33.761932 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759507    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 05:49:33.761932 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759510    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 05:49:33.761932 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759513    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 05:49:33.761932 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759515    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 05:49:33.761932 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759518    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 05:49:33.761932 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759520    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 05:49:33.761932 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759523    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 05:49:33.761932 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759525    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 05:49:33.762444 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759527    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 05:49:33.762444 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759530    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 05:49:33.762444 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759532    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 05:49:33.762444 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759535    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 05:49:33.762444 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759537    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 05:49:33.762444 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759540    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 05:49:33.762444 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759542    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 05:49:33.762444 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759544    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 05:49:33.762444 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759547    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 05:49:33.762444 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759549    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 05:49:33.762444 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759551    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 05:49:33.762444 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759554    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 05:49:33.762444 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759557    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 05:49:33.762444 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759559    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 05:49:33.762444 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759562    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 05:49:33.762444 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759564    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 05:49:33.762444 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759567    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 05:49:33.762444 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759569    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 05:49:33.762444 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759571    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 05:49:33.762903 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759575    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 05:49:33.762903 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759578    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 05:49:33.762903 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759584    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 05:49:33.762903 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759587    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 05:49:33.762903 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759590    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 05:49:33.762903 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759592    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 05:49:33.762903 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759595    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 05:49:33.762903 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759598    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 05:49:33.762903 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759600    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 05:49:33.762903 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759603    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 05:49:33.762903 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759605    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 05:49:33.762903 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759608    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 05:49:33.762903 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759612    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 05:49:33.762903 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759614    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 05:49:33.762903 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759617    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 05:49:33.762903 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759619    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 05:49:33.762903 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759621    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 05:49:33.762903 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759624    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 05:49:33.762903 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759626    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 05:49:33.763375 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.759629    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 05:49:33.763375 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760009    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 05:49:33.763375 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760013    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 05:49:33.763375 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760016    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 05:49:33.763375 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760019    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 05:49:33.763375 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760022    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 05:49:33.763375 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760024    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 05:49:33.763375 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760027    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 05:49:33.763375 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760029    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 05:49:33.763375 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760032    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 05:49:33.763375 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760035    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 05:49:33.763375 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760038    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 05:49:33.763375 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760042    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 05:49:33.763375 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760044    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 05:49:33.763375 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760047    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 05:49:33.763375 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760051    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 05:49:33.763375 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760054    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 05:49:33.763375 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760057    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 05:49:33.763375 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760060    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 05:49:33.763375 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760062    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 05:49:33.763880 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760064    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 05:49:33.763880 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760067    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 05:49:33.763880 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760069    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 05:49:33.763880 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760072    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 05:49:33.763880 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760075    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 05:49:33.763880 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760077    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 05:49:33.763880 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760080    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 05:49:33.763880 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760082    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 05:49:33.763880 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760084    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 05:49:33.763880 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760087    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 05:49:33.763880 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760089    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 05:49:33.763880 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760092    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 05:49:33.763880 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760094    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 05:49:33.763880 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760097    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 05:49:33.763880 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760099    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 05:49:33.763880 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760102    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 05:49:33.763880 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760104    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 05:49:33.763880 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760106    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 05:49:33.763880 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760110    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 05:49:33.763880 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760112    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 05:49:33.764365 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760115    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 05:49:33.764365 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760118    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 05:49:33.764365 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760121    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 05:49:33.764365 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760124    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 05:49:33.764365 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760126    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 05:49:33.764365 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760128    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 05:49:33.764365 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760131    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 05:49:33.764365 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760135    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 05:49:33.764365 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760137    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 05:49:33.764365 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760139    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 05:49:33.764365 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760144    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 05:49:33.764365 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760147    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 05:49:33.764365 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760150    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 05:49:33.764365 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760153    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 05:49:33.764365 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760156    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 05:49:33.764365 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760159    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 05:49:33.764365 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760162    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 05:49:33.764365 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760165    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 05:49:33.764365 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760168    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 05:49:33.764841 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760170    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 05:49:33.764841 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760173    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 05:49:33.764841 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760175    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 05:49:33.764841 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760178    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 05:49:33.764841 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760180    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 05:49:33.764841 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760183    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 05:49:33.764841 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760185    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 05:49:33.764841 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760187    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 05:49:33.764841 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760190    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 05:49:33.764841 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760192    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 05:49:33.764841 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760195    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 05:49:33.764841 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760197    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 05:49:33.764841 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760200    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 05:49:33.764841 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760203    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 05:49:33.764841 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760205    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 05:49:33.764841 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760209    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 05:49:33.764841 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760211    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 05:49:33.764841 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760214    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 05:49:33.764841 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760216    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 05:49:33.764841 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760219    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 05:49:33.765319 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760224    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 05:49:33.765319 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760226    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 05:49:33.765319 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760228    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 05:49:33.765319 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760231    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 05:49:33.765319 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760233    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 05:49:33.765319 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760236    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 05:49:33.765319 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760238    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 05:49:33.765319 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760241    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 05:49:33.765319 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760243    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 05:49:33.765319 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760246    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 05:49:33.765319 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760248    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 05:49:33.765319 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760251    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 05:49:33.765319 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760253    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 05:49:33.765319 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760256    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 05:49:33.765319 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760258    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 05:49:33.765319 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760261    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 05:49:33.765319 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760264    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 05:49:33.765319 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760266    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 05:49:33.765319 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760268    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 05:49:33.765319 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760271    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 05:49:33.765870 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.760273    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 05:49:33.765870 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760347    2579 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 05:49:33.765870 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760407    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 05:49:33.765870 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760418    2579 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 05:49:33.765870 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760423    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 05:49:33.765870 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760428    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 05:49:33.765870 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760431    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 05:49:33.765870 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760436    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 05:49:33.765870 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760440    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 05:49:33.765870 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760443    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 05:49:33.765870 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760446    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 05:49:33.765870 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760450    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 05:49:33.765870 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760454    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 05:49:33.765870 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760457    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 05:49:33.765870 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760459    2579 flags.go:64] FLAG: --cgroup-root=""
Sep 09 05:49:33.765870 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760462    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 05:49:33.765870 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760465    2579 flags.go:64] FLAG: --client-ca-file=""
Sep 09 05:49:33.765870 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760468    2579 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 05:49:33.765870 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760471    2579 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 05:49:33.765870 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760474    2579 flags.go:64] FLAG: --cluster-domain=""
Sep 09 05:49:33.765870 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760477    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 05:49:33.765870 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760480    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 05:49:33.765870 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760483    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 05:49:33.765870 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760486    2579 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 05:49:33.766467 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760490    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 05:49:33.766467 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760493    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 05:49:33.766467 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760495    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 05:49:33.766467 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760499    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 05:49:33.766467 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760501    2579 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 05:49:33.766467 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760504    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 05:49:33.766467 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760507    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 05:49:33.766467 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760510    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 05:49:33.766467 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760513    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 05:49:33.766467 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760517    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 05:49:33.766467 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760520    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 05:49:33.766467 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760522    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 05:49:33.766467 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760525    2579 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 05:49:33.766467 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760528    2579 flags.go:64] FLAG: --enable-server="true"
Sep 09 05:49:33.766467 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760532    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 05:49:33.766467 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760536    2579 flags.go:64] FLAG: --event-burst="100"
Sep 09 05:49:33.766467 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760539    2579 flags.go:64] FLAG: --event-qps="50"
Sep 09 05:49:33.766467 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760542    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 05:49:33.766467 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760545    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 05:49:33.766467 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760547    2579 flags.go:64] FLAG: --eviction-hard=""
Sep 09 05:49:33.766467 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760552    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 05:49:33.766467 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760555    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 05:49:33.766467 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760558    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 05:49:33.766467 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760560    2579 flags.go:64] FLAG: --eviction-soft=""
Sep 09 05:49:33.766467 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760563    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 05:49:33.767071 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760566    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 05:49:33.767071 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760569    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 05:49:33.767071 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760572    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 05:49:33.767071 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760575    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 05:49:33.767071 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760577    2579 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 05:49:33.767071 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760580    2579 flags.go:64] FLAG: --feature-gates=""
Sep 09 05:49:33.767071 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760584    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 05:49:33.767071 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760587    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 05:49:33.767071 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760590    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 05:49:33.767071 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760593    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 05:49:33.767071 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760596    2579 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 05:49:33.767071 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760598    2579 flags.go:64] FLAG: --help="false"
Sep 09 05:49:33.767071 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760601    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-131-142.ec2.internal"
Sep 09 05:49:33.767071 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760604    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 05:49:33.767071 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760607    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 05:49:33.767071 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760627    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 05:49:33.767071 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760631    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 05:49:33.767071 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760635    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 05:49:33.767071 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760638    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 05:49:33.767071 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760640    2579 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 05:49:33.767071 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760643    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 05:49:33.767071 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760648    2579 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 05:49:33.767071 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760651    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 05:49:33.767680 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760655    2579 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 05:49:33.767680 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760657    2579 flags.go:64] FLAG: --kube-reserved=""
Sep 09 05:49:33.767680 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760660    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 05:49:33.767680 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760663    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 05:49:33.767680 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760667    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 05:49:33.767680 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760669    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 05:49:33.767680 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760672    2579 flags.go:64] FLAG: --lock-file=""
Sep 09 05:49:33.767680 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760675    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 05:49:33.767680 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760678    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 05:49:33.767680 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760680    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 05:49:33.767680 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760685    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 05:49:33.767680 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760688    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 05:49:33.767680 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760691    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 05:49:33.767680 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760694    2579 flags.go:64] FLAG: --logging-format="text"
Sep 09 05:49:33.767680 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760696    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 05:49:33.767680 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760699    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 05:49:33.767680 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760703    2579 flags.go:64] FLAG: --manifest-url=""
Sep 09 05:49:33.767680 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760706    2579 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 05:49:33.767680 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760710    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 05:49:33.767680 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760713    2579 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 05:49:33.767680 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760716    2579 flags.go:64] FLAG: --max-pods="110"
Sep 09 05:49:33.767680 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760719    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 05:49:33.767680 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760722    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 05:49:33.767680 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760725    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 05:49:33.767680 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760728    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 05:49:33.768296 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760730    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 05:49:33.768296 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760733    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 05:49:33.768296 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760736    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 05:49:33.768296 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760743    2579 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 05:49:33.768296 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760745    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 05:49:33.768296 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760749    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 05:49:33.768296 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760752    2579 flags.go:64] FLAG: --pod-cidr=""
Sep 09 05:49:33.768296 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760755    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 05:49:33.768296 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760760    2579 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 05:49:33.768296 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760763    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 05:49:33.768296 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760766    2579 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 05:49:33.768296 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760769    2579 flags.go:64] FLAG: --port="10250"
Sep 09 05:49:33.768296 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760773    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 05:49:33.768296 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760776    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-008e94b6619bebddf"
Sep 09 05:49:33.768296 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760779    2579 flags.go:64] FLAG: --qos-reserved=""
Sep 09 05:49:33.768296 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760782    2579 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 05:49:33.768296 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760784    2579 flags.go:64] FLAG: --register-node="true"
Sep 09 05:49:33.768296 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760787    2579 flags.go:64] FLAG: --register-with-taints=""
Sep 09 05:49:33.768296 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760791    2579 flags.go:64] FLAG: --registry-burst="10"
Sep 09 05:49:33.768296 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760793    2579 flags.go:64] FLAG: --registry-qps="5"
Sep 09 05:49:33.768296 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760796    2579 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 05:49:33.768296 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760799    2579 flags.go:64] FLAG: --reserved-memory=""
Sep 09 05:49:33.768296 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760802    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 05:49:33.768296 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760805    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 05:49:33.768296 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760808    2579 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 05:49:33.768961 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760811    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 05:49:33.768961 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760813    2579 flags.go:64] FLAG: --runonce="false"
Sep 09 05:49:33.768961 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760816    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 05:49:33.768961 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760819    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 05:49:33.768961 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760822    2579 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 05:49:33.768961 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760825    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 05:49:33.768961 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760828    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 05:49:33.768961 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760831    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 05:49:33.768961 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760834    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 05:49:33.768961 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760837    2579 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 05:49:33.768961 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760840    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 05:49:33.768961 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760842    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 05:49:33.768961 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760847    2579 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 05:49:33.768961 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760849    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 05:49:33.768961 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760852    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 05:49:33.768961 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760856    2579 flags.go:64] FLAG: --system-cgroups=""
Sep 09 05:49:33.768961 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760859    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 05:49:33.768961 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760864    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 05:49:33.768961 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760866    2579 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 05:49:33.768961 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760871    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 05:49:33.768961 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760875    2579 flags.go:64] FLAG: --tls-min-version=""
Sep 09 05:49:33.768961 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760878    2579 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 05:49:33.768961 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760881    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 05:49:33.768961 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760883    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 05:49:33.768961 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760886    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 05:49:33.769623 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760889    2579 flags.go:64] FLAG: --v="2"
Sep 09 05:49:33.769623 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760893    2579 flags.go:64] FLAG: --version="false"
Sep 09 05:49:33.769623 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760896    2579 flags.go:64] FLAG: --vmodule=""
Sep 09 05:49:33.769623 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760900    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 05:49:33.769623 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.760903    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 05:49:33.769623 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761018    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 05:49:33.769623 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761022    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 05:49:33.769623 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761025    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 05:49:33.769623 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761029    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 05:49:33.769623 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761031    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 05:49:33.769623 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761034    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 05:49:33.769623 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761037    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 05:49:33.769623 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761039    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 05:49:33.769623 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761042    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 05:49:33.769623 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761045    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 05:49:33.769623 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761047    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 05:49:33.769623 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761050    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 05:49:33.769623 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761052    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 05:49:33.769623 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761055    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 05:49:33.769623 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761059    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 05:49:33.769623 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761062    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 05:49:33.770143 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761064    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 05:49:33.770143 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761067    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 05:49:33.770143 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761070    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 05:49:33.770143 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761072    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 05:49:33.770143 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761075    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 05:49:33.770143 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761078    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 05:49:33.770143 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761080    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 05:49:33.770143 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761083    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 05:49:33.770143 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761085    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 05:49:33.770143 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761088    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 05:49:33.770143 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761090    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 05:49:33.770143 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761092    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 05:49:33.770143 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761095    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 05:49:33.770143 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761097    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 05:49:33.770143 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761100    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 05:49:33.770143 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761102    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 05:49:33.770143 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761105    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 05:49:33.770143 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761107    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 05:49:33.770143 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761111    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 05:49:33.770143 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761114    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 05:49:33.770643 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761117    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 05:49:33.770643 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761119    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 05:49:33.770643 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761122    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 05:49:33.770643 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761124    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 05:49:33.770643 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761128    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 05:49:33.770643 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761131    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 05:49:33.770643 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761134    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 05:49:33.770643 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761136    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 05:49:33.770643 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761139    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 05:49:33.770643 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761141    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 05:49:33.770643 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761145    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 05:49:33.770643 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761148    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 05:49:33.770643 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761151    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 05:49:33.770643 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761153    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 05:49:33.770643 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761156    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 05:49:33.770643 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761159    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 05:49:33.770643 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761161    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 05:49:33.770643 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761164    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 05:49:33.770643 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761166    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 05:49:33.771108 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761169    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 05:49:33.771108 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761171    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 05:49:33.771108 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761174    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 05:49:33.771108 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761176    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 05:49:33.771108 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761179    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 05:49:33.771108 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761181    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 05:49:33.771108 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761183    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 05:49:33.771108 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761186    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 05:49:33.771108 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761188    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 05:49:33.771108 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761190    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 05:49:33.771108 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761193    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 05:49:33.771108 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761195    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 05:49:33.771108 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761198    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 05:49:33.771108 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761200    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 05:49:33.771108 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761202    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 05:49:33.771108 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761205    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 05:49:33.771108 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761209    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 05:49:33.771108 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761212    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 05:49:33.771108 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761215    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 05:49:33.771602 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761217    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 05:49:33.771602 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761220    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 05:49:33.771602 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761223    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 05:49:33.771602 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761226    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 05:49:33.771602 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761230    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 05:49:33.771602 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761233    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 05:49:33.771602 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761235    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 05:49:33.771602 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761238    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 05:49:33.771602 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761240    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 05:49:33.771602 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761243    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 05:49:33.771602 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761246    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 05:49:33.771602 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761248    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 05:49:33.771602 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761251    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 05:49:33.771602 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761253    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 05:49:33.771602 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761255    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 05:49:33.771602 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761258    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 05:49:33.771602 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761260    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 05:49:33.771602 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761263    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 05:49:33.771602 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761265    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 05:49:33.771602 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761268    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 05:49:33.772086 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761270    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 05:49:33.772086 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761272    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 05:49:33.772086 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761275    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 05:49:33.772086 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761277    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 05:49:33.772086 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.761280    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 05:49:33.772086 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.762207    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 05:49:33.772086 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.769932    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 05:49:33.772086 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.769946    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 05:49:33.772086 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.769965    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 05:49:33.772086 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.769974    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 05:49:33.772086 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770034    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 05:49:33.772086 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770039    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 05:49:33.772086 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770042    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 05:49:33.772086 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770045    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 05:49:33.772086 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770049    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 05:49:33.772491 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770052    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 05:49:33.772491 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770054    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 05:49:33.772491 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770057    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 05:49:33.772491 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770060    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 05:49:33.772491 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770062    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 05:49:33.772491 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770065    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 05:49:33.772491 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770067    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 05:49:33.772491 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770076    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 05:49:33.772491 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770078    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 05:49:33.772491 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770081    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 05:49:33.772491 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770084    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 05:49:33.772491 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770086    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 05:49:33.772491 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770089    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 05:49:33.772491 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770091    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 05:49:33.772491 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770094    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 05:49:33.772491 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770097    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 05:49:33.772491 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770099    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 05:49:33.772491 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770102    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 05:49:33.772491 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770104    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 05:49:33.772491 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770107    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 05:49:33.773007 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770109    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 05:49:33.773007 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770112    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 05:49:33.773007 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770114    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 05:49:33.773007 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770118    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 05:49:33.773007 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770123    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 05:49:33.773007 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770126    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 05:49:33.773007 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770129    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 05:49:33.773007 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770132    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 05:49:33.773007 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770135    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 05:49:33.773007 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770138    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 05:49:33.773007 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770140    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 05:49:33.773007 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770143    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 05:49:33.773007 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770146    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 05:49:33.773007 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770149    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 05:49:33.773007 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770151    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 05:49:33.773007 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770155    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 05:49:33.773007 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770159    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 05:49:33.773007 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770162    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 05:49:33.773007 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770167    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 05:49:33.773596 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770170    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 05:49:33.773596 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770178    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 05:49:33.773596 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770181    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 05:49:33.773596 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770184    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 05:49:33.773596 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770186    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 05:49:33.773596 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770189    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 05:49:33.773596 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770192    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 05:49:33.773596 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770194    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 05:49:33.773596 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770196    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 05:49:33.773596 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770199    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 05:49:33.773596 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770202    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 05:49:33.773596 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770204    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 05:49:33.773596 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770207    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 05:49:33.773596 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770210    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 05:49:33.773596 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770212    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 05:49:33.773596 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770215    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 05:49:33.773596 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770217    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 05:49:33.773596 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770219    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 05:49:33.773596 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770222    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 05:49:33.773596 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770224    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 05:49:33.774127 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770227    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 05:49:33.774127 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770229    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 05:49:33.774127 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770232    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 05:49:33.774127 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770234    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 05:49:33.774127 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770237    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 05:49:33.774127 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770240    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 05:49:33.774127 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770243    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 05:49:33.774127 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770245    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 05:49:33.774127 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770248    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 05:49:33.774127 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770250    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 05:49:33.774127 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770252    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 05:49:33.774127 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770255    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 05:49:33.774127 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770257    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 05:49:33.774127 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770262    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 05:49:33.774127 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770270    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 05:49:33.774127 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770273    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 05:49:33.774127 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770277    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 05:49:33.774127 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770280    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 05:49:33.774127 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770282    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 05:49:33.774620 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770285    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 05:49:33.774620 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770287    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 05:49:33.774620 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770290    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 05:49:33.774620 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770292    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 05:49:33.774620 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770295    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 05:49:33.774620 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770297    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 05:49:33.774620 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770300    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 05:49:33.774620 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770302    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 05:49:33.774620 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770305    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 05:49:33.774620 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770307    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 05:49:33.774620 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770310    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 05:49:33.774620 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770312    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 05:49:33.774620 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770314    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 05:49:33.774620 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770317    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 05:49:33.774620 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770319    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 05:49:33.774620 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770322    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 05:49:33.775037 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.770327    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 05:49:33.775037 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770500    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 05:49:33.775037 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770506    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 05:49:33.775037 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770509    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 05:49:33.775037 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770512    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 05:49:33.775037 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770515    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 05:49:33.775037 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770517    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 05:49:33.775037 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770520    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 05:49:33.775037 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770522    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 05:49:33.775037 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770530    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 05:49:33.775037 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770533    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 05:49:33.775037 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770535    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 05:49:33.775037 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770543    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 05:49:33.775037 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770546    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 05:49:33.775037 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770548    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 05:49:33.775438 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770551    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 05:49:33.775438 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770553    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 05:49:33.775438 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770555    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 05:49:33.775438 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770558    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 05:49:33.775438 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770560    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 05:49:33.775438 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770563    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 05:49:33.775438 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770565    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 05:49:33.775438 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770568    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 05:49:33.775438 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770570    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 05:49:33.775438 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770573    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 05:49:33.775438 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770575    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 05:49:33.775438 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770577    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 05:49:33.775438 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770580    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 05:49:33.775438 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770582    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 05:49:33.775438 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770585    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 05:49:33.775438 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770587    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 05:49:33.775438 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770589    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 05:49:33.775438 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770592    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 05:49:33.775438 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770594    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 05:49:33.775438 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770596    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 05:49:33.775920 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770599    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 05:49:33.775920 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770601    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 05:49:33.775920 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770604    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 05:49:33.775920 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770606    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 05:49:33.775920 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770608    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 05:49:33.775920 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770611    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 05:49:33.775920 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770613    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 05:49:33.775920 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770616    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 05:49:33.775920 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770618    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 05:49:33.775920 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770620    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 05:49:33.775920 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770628    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 05:49:33.775920 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770632    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 05:49:33.775920 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770634    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 05:49:33.775920 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770636    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 05:49:33.775920 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770639    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 05:49:33.775920 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770641    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 05:49:33.775920 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770644    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 05:49:33.775920 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770647    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 05:49:33.775920 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770649    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 05:49:33.775920 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770652    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 05:49:33.776417 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770654    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 05:49:33.776417 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770657    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 05:49:33.776417 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770659    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 05:49:33.776417 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770662    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 05:49:33.776417 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770664    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 05:49:33.776417 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770667    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 05:49:33.776417 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770669    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 05:49:33.776417 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770671    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 05:49:33.776417 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770675    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 05:49:33.776417 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770679    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 05:49:33.776417 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770681    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 05:49:33.776417 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770684    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 05:49:33.776417 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770686    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 05:49:33.776417 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770688    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 05:49:33.776417 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770691    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 05:49:33.776417 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770695    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 05:49:33.776417 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770698    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 05:49:33.776417 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770701    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 05:49:33.776417 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770704    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 05:49:33.776876 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770708    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 05:49:33.776876 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770711    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 05:49:33.776876 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770714    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 05:49:33.776876 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770716    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 05:49:33.776876 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770724    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 05:49:33.776876 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770727    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 05:49:33.776876 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770729    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 05:49:33.776876 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770732    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 05:49:33.776876 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770735    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 05:49:33.776876 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770737    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 05:49:33.776876 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770739    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 05:49:33.776876 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770742    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 05:49:33.776876 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770744    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 05:49:33.776876 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770747    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 05:49:33.776876 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770749    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 05:49:33.776876 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770751    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 05:49:33.776876 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770754    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 05:49:33.776876 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770756    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 05:49:33.776876 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770759    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 05:49:33.776876 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770761    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 05:49:33.777422 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770763    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 05:49:33.777422 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770766    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 05:49:33.777422 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770768    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 05:49:33.777422 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770771    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 05:49:33.777422 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770773    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 05:49:33.777422 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:33.770775    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 05:49:33.777422 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.770780    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 05:49:33.777422 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.770885    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 05:49:33.777422 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.773370    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 05:49:33.777422 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.774514    2579 server.go:1019] "Starting client certificate rotation"
Sep 09 05:49:33.777422 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.774615    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 05:49:33.777422 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.774650    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 05:49:33.777422 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.777355    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 05:49:33.779142 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.779091    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 05:49:33.791647 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.791630    2579 log.go:25] "Validated CRI v1 runtime API"
Sep 09 05:49:33.796346 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.796330    2579 log.go:25] "Validated CRI v1 image API"
Sep 09 05:49:33.797269 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.797246    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 05:49:33.804586 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.804561    2579 fs.go:135] Filesystem UUIDs: map[7063ebf2-87b1-467c-bffd-c0b404c5981b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 91dc09bd-d6f8-4aa4-9400-3cd86ef6f960:/dev/nvme0n1p4]
Sep 09 05:49:33.804646 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.804586    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 05:49:33.804854 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.804838    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 05:49:33.811129 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.811027    2579 manager.go:216] Machine: {Timestamp:2026-09-09 05:49:33.809600579 +0000 UTC m=+0.375329281 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099347 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec276dff650ccb864f2d92badcdbeeda SystemUUID:ec276dff-650c-cb86-4f2d-92badcdbeeda BootID:126abb5a-1724-446d-91f9-3f5db40d3244 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:8c:88:3c:36:f5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:8c:88:3c:36:f5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:22:80:1c:79:2b:62 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 05:49:33.811129 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.811125    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 05:49:33.811252 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.811189    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 05:49:33.817399 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.817363    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 05:49:33.817520 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.817400    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-142.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 05:49:33.818256 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.818247    2579 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 05:49:33.818287 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.818260    2579 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 05:49:33.818287 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.818268    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 05:49:33.818918 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.818908    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 05:49:33.820735 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.820724    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 05:49:33.825287 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.825276    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 05:49:33.825409 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.825400    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 05:49:33.825447 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.825428    2579 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 05:49:33.825447 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.825444    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 05:49:33.825502 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.825459    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 05:49:33.825502 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.825469    2579 kubelet.go:398] "Adding apiserver pod source"
Sep 09 05:49:33.825502 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.825479    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 05:49:33.825858 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.825843    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 05:49:33.825904 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.825870    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 05:49:33.828238 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.828218    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 05:49:33.828561 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.828547    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hhvgv"
Sep 09 05:49:33.831306 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.831290    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 05:49:33.831398 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.831329    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 05:49:33.831532 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.831519    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 05:49:33.831594 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.831540    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 05:49:33.831594 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.831550    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 05:49:33.831594 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.831558    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 05:49:33.831594 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.831569    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 05:49:33.831594 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.831577    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 05:49:33.832230 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.832219    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 05:49:33.832279 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.832232    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 05:49:33.832279 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.832243    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 05:49:33.832279 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.832252    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 05:49:33.832279 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.832271    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 05:49:33.832474 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.832284    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 05:49:33.833216 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.833205    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 05:49:33.833265 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.833219    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 05:49:33.836065 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:33.835901    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 05:49:33.836065 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.836052    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-142.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 05:49:33.836224 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:33.836112    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-142.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 05:49:33.836563 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.836547    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hhvgv"
Sep 09 05:49:33.837832 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.837811    2579 server.go:1268] "Started kubelet"
Sep 09 05:49:33.837935 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.837900    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 05:49:33.837991 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.837921    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 05:49:33.838045 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.838001    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 05:49:33.838605 ip-10-0-131-142 systemd[1]: Started Kubernetes Kubelet.
Sep 09 05:49:33.842190 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.842155    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 05:49:33.843531 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:33.843498    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 05:49:33.844010 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.843995    2579 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 05:49:33.846825 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:33.845698    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-142.ec2.internal.18d3910fc095808e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-142.ec2.internal,UID:ip-10-0-131-142.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-142.ec2.internal,},FirstTimestamp:2026-09-09 05:49:33.837779086 +0000 UTC m=+0.403507788,LastTimestamp:2026-09-09 05:49:33.837779086 +0000 UTC m=+0.403507788,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-142.ec2.internal,}"
Sep 09 05:49:33.848099 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.848078    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 05:49:33.848791 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.848776    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 05:49:33.848903 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:33.848884    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-142.ec2.internal\" not found"
Sep 09 05:49:33.849522 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.849503    2579 factory.go:55] Registering systemd factory
Sep 09 05:49:33.849522 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.849522    2579 factory.go:223] Registration of the systemd container factory successfully
Sep 09 05:49:33.849678 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.849581    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 05:49:33.849678 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.849584    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 05:49:33.849678 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.849606    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 05:49:33.849997 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.849682    2579 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 05:49:33.849997 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.849688    2579 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 05:49:33.850606 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.850579    2579 factory.go:153] Registering CRI-O factory
Sep 09 05:49:33.850606 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.850598    2579 factory.go:223] Registration of the crio container factory successfully
Sep 09 05:49:33.850746 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.850645    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 05:49:33.850746 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.850661    2579 factory.go:103] Registering Raw factory
Sep 09 05:49:33.850746 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.850673    2579 manager.go:1196] Started watching for new ooms in manager
Sep 09 05:49:33.851019 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.851006    2579 manager.go:349] Starting recovery of all containers
Sep 09 05:49:33.860123 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.860105    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 05:49:33.860750 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.860725    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 05:49:33.860856 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.860780    2579 manager.go:354] Recovery completed
Sep 09 05:49:33.865032 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.865018    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 05:49:33.866065 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:33.866050    2579 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-131-142.ec2.internal\" not found" node="ip-10-0-131-142.ec2.internal"
Sep 09 05:49:33.866891 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.866876    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-142.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 05:49:33.866958 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.866909    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-142.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 05:49:33.866958 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.866924    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-142.ec2.internal" event="NodeHasSufficientPID"
Sep 09 05:49:33.867490 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.867479    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 05:49:33.867490 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.867489    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 05:49:33.867568 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.867490    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 05:49:33.867568 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.867524    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 05:49:33.868164 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.868148    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-142.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 05:49:33.868199 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.868176    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-142.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 05:49:33.868199 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.868186    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-142.ec2.internal" event="NodeHasSufficientPID"
Sep 09 05:49:33.869656 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.869644    2579 policy_none.go:49] "None policy: Start"
Sep 09 05:49:33.869701 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.869660    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 05:49:33.869701 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.869669    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 05:49:33.871619 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.871609    2579 policy_none.go:47] "Start"
Sep 09 05:49:33.927687 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.906530    2579 manager.go:337] "Starting Device Plugin manager"
Sep 09 05:49:33.927687 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:33.906558    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 05:49:33.927687 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.906567    2579 server.go:87] "Starting device plugin registration server"
Sep 09 05:49:33.927687 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.906827    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 05:49:33.927687 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.906839    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 05:49:33.927687 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.906934    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 05:49:33.927687 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.907017    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 05:49:33.927687 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.907027    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 05:49:33.927687 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:33.907444    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 05:49:33.927687 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:33.907490    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-142.ec2.internal\" not found"
Sep 09 05:49:33.927687 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.916613    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 05:49:33.927687 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.916659    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 05:49:33.927687 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.916680    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 05:49:33.927687 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:33.916721    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 05:49:33.927687 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:33.918612    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 05:49:34.007322 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.007264    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 05:49:34.008141 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.008125    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-142.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 05:49:34.008218 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.008154    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-142.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 05:49:34.008218 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.008165    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-142.ec2.internal" event="NodeHasSufficientPID"
Sep 09 05:49:34.008218 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.008186    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-142.ec2.internal"
Sep 09 05:49:34.015775 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.015762    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-142.ec2.internal"
Sep 09 05:49:34.015843 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:34.015780    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-142.ec2.internal\": node \"ip-10-0-131-142.ec2.internal\" not found"
Sep 09 05:49:34.016967 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.016955    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-142.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-142.ec2.internal"]
Sep 09 05:49:34.017011 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.017002    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 05:49:34.017970 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.017949    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-142.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 05:49:34.018048 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.017977    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-142.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 05:49:34.018048 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.017988    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-142.ec2.internal" event="NodeHasSufficientPID"
Sep 09 05:49:34.020238 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.020224    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 05:49:34.020401 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.020372    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-142.ec2.internal"
Sep 09 05:49:34.020464 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.020413    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 05:49:34.020950 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.020936    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-142.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 05:49:34.021032 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.020963    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-142.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 05:49:34.021032 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.020978    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-142.ec2.internal" event="NodeHasSufficientPID"
Sep 09 05:49:34.021032 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.020997    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-142.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 05:49:34.021032 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.021027    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-142.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 05:49:34.021166 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.021038    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-142.ec2.internal" event="NodeHasSufficientPID"
Sep 09 05:49:34.023174 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.023161    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-142.ec2.internal"
Sep 09 05:49:34.023241 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.023184    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 05:49:34.023963 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.023945    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-142.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 05:49:34.024016 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.023978    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-142.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 05:49:34.024016 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.023995    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-142.ec2.internal" event="NodeHasSufficientPID"
Sep 09 05:49:34.033847 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:34.033831    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-142.ec2.internal\" not found"
Sep 09 05:49:34.044896 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:34.044874    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-142.ec2.internal\" not found" node="ip-10-0-131-142.ec2.internal"
Sep 09 05:49:34.048958 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:34.048942    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-142.ec2.internal\" not found" node="ip-10-0-131-142.ec2.internal"
Sep 09 05:49:34.051070 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.051053    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/59ee296aad7c4e56d27fc66bd1b97d18-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-142.ec2.internal\" (UID: \"59ee296aad7c4e56d27fc66bd1b97d18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-142.ec2.internal"
Sep 09 05:49:34.051136 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.051079    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/59ee296aad7c4e56d27fc66bd1b97d18-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-142.ec2.internal\" (UID: \"59ee296aad7c4e56d27fc66bd1b97d18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-142.ec2.internal"
Sep 09 05:49:34.051136 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.051095    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/58007dd99525496ea22ff71e4a80cd63-config\") pod \"kube-apiserver-proxy-ip-10-0-131-142.ec2.internal\" (UID: \"58007dd99525496ea22ff71e4a80cd63\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-142.ec2.internal"
Sep 09 05:49:34.134032 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:34.134013    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-142.ec2.internal\" not found"
Sep 09 05:49:34.151759 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.151741    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/59ee296aad7c4e56d27fc66bd1b97d18-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-142.ec2.internal\" (UID: \"59ee296aad7c4e56d27fc66bd1b97d18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-142.ec2.internal"
Sep 09 05:49:34.151808 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.151765    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/59ee296aad7c4e56d27fc66bd1b97d18-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-142.ec2.internal\" (UID: \"59ee296aad7c4e56d27fc66bd1b97d18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-142.ec2.internal"
Sep 09 05:49:34.151808 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.151788    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/58007dd99525496ea22ff71e4a80cd63-config\") pod \"kube-apiserver-proxy-ip-10-0-131-142.ec2.internal\" (UID: \"58007dd99525496ea22ff71e4a80cd63\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-142.ec2.internal"
Sep 09 05:49:34.151908 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.151841    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/58007dd99525496ea22ff71e4a80cd63-config\") pod \"kube-apiserver-proxy-ip-10-0-131-142.ec2.internal\" (UID: \"58007dd99525496ea22ff71e4a80cd63\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-142.ec2.internal"
Sep 09 05:49:34.151908 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.151846    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/59ee296aad7c4e56d27fc66bd1b97d18-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-142.ec2.internal\" (UID: \"59ee296aad7c4e56d27fc66bd1b97d18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-142.ec2.internal"
Sep 09 05:49:34.151908 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.151865    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/59ee296aad7c4e56d27fc66bd1b97d18-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-142.ec2.internal\" (UID: \"59ee296aad7c4e56d27fc66bd1b97d18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-142.ec2.internal"
Sep 09 05:49:34.234585 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:34.234562    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-142.ec2.internal\" not found"
Sep 09 05:49:34.335218 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:34.335202    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-142.ec2.internal\" not found"
Sep 09 05:49:34.348358 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.348341    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-142.ec2.internal"
Sep 09 05:49:34.352669 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.352653    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-142.ec2.internal"
Sep 09 05:49:34.435880 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:34.435857    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-142.ec2.internal\" not found"
Sep 09 05:49:34.536313 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:34.536298    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-142.ec2.internal\" not found"
Sep 09 05:49:34.636898 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:34.636843    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-142.ec2.internal\" not found"
Sep 09 05:49:34.737415 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:34.737379    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-142.ec2.internal\" not found"
Sep 09 05:49:34.773905 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.773888    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 05:49:34.774234 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.774012    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 05:49:34.774234 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.774029    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 05:49:34.837710 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:34.837683    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-142.ec2.internal\" not found"
Sep 09 05:49:34.843846 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.843817    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 05:44:35 +0000 UTC" deadline="2028-04-28 06:30:00.007171515 +0000 UTC"
Sep 09 05:49:34.843900 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.843845    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14328h40m25.163329364s"
Sep 09 05:49:34.904607 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.904553    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 05:49:34.949439 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.949417    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-142.ec2.internal"
Sep 09 05:49:34.957430 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.957410    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 05:49:34.959300 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.959287    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-142.ec2.internal"
Sep 09 05:49:34.968776 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:34.968764    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 05:49:34.999022 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:34.998996    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59ee296aad7c4e56d27fc66bd1b97d18.slice/crio-2d58f05f5bc0e195ecdbbc521c71f5b063504f7a76398d3e029db0820404a2c8 WatchSource:0}: Error finding container 2d58f05f5bc0e195ecdbbc521c71f5b063504f7a76398d3e029db0820404a2c8: Status 404 returned error can't find the container with id 2d58f05f5bc0e195ecdbbc521c71f5b063504f7a76398d3e029db0820404a2c8
Sep 09 05:49:34.999343 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:34.999324    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58007dd99525496ea22ff71e4a80cd63.slice/crio-93a179e1c97f2efaa1ddba4f5a72a5d3fd4538b8d91b93a20d50da8991645903 WatchSource:0}: Error finding container 93a179e1c97f2efaa1ddba4f5a72a5d3fd4538b8d91b93a20d50da8991645903: Status 404 returned error can't find the container with id 93a179e1c97f2efaa1ddba4f5a72a5d3fd4538b8d91b93a20d50da8991645903
Sep 09 05:49:35.005024 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:35.005009    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 05:49:35.217593 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:35.217541    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 05:49:36.881732 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:36.881700    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 05:49:37.052264 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.052234    2579 apiserver.go:52] "Watching apiserver"
Sep 09 05:49:37.066246 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.066221    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 05:49:37.067140 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.067117    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-6z9q7","openshift-multus/multus-additional-cni-plugins-q4pdb","openshift-network-operator/iptables-alerter-r7gnm","openshift-ovn-kubernetes/ovnkube-node-s7jp7","kube-system/global-pull-secret-syncer-fs4n6","kube-system/kube-apiserver-proxy-ip-10-0-131-142.ec2.internal","openshift-cluster-node-tuning-operator/tuned-4r2gq","openshift-dns/node-resolver-b9lbm","openshift-image-registry/node-ca-brqzd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-142.ec2.internal","openshift-multus/network-metrics-daemon-s7r5d","openshift-network-diagnostics/network-check-target-b78pk","kube-system/konnectivity-agent-fv99k","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7lkv"]
Sep 09 05:49:37.070207 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.070188    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4r2gq"
Sep 09 05:49:37.072570 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.072542    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 05:49:37.072834 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.072686    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 05:49:37.072834 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.072790    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hckmc\""
Sep 09 05:49:37.074886 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.073356    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fv99k"
Sep 09 05:49:37.075049 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.075030    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 05:49:37.076866 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.076849    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r7gnm"
Sep 09 05:49:37.077378 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.077358    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 05:49:37.077464 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.077392    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 05:49:37.077585 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.077570    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8nhk5\""
Sep 09 05:49:37.079055 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.078723    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.079055 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.078953    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 05:49:37.079462 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.079358    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 05:49:37.079462 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.079381    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8l7lw\""
Sep 09 05:49:37.079668 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.079561    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 05:49:37.081268 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.081034    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 05:49:37.081894 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.081567    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sdkmc\""
Sep 09 05:49:37.081894 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.081810    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 05:49:37.081894 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.081823    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 05:49:37.081894 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.081833    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 05:49:37.082142 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.081989    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fs4n6"
Sep 09 05:49:37.082142 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.082059    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 05:49:37.082142 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:37.082057    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fs4n6" podUID="26cac444-abfa-4060-8f52-4e5756866f3f"
Sep 09 05:49:37.082317 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.082284    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 05:49:37.084874 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.083935    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7lkv"
Sep 09 05:49:37.084874 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.084724    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 05:49:37.086092 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.086073    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-brqzd"
Sep 09 05:49:37.086224 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.086206    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 05:49:37.086408 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.086392    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 05:49:37.086690 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.086671    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9864f736-2e71-4723-bdb5-a54abeedecbb-etc-sysconfig\") pod \"tuned-4r2gq\" (UID: \"9864f736-2e71-4723-bdb5-a54abeedecbb\") " pod="openshift-cluster-node-tuning-operator/tuned-4r2gq"
Sep 09 05:49:37.086768 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.086705    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9864f736-2e71-4723-bdb5-a54abeedecbb-run\") pod \"tuned-4r2gq\" (UID: \"9864f736-2e71-4723-bdb5-a54abeedecbb\") " pod="openshift-cluster-node-tuning-operator/tuned-4r2gq"
Sep 09 05:49:37.086768 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.086725    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9864f736-2e71-4723-bdb5-a54abeedecbb-sys\") pod \"tuned-4r2gq\" (UID: \"9864f736-2e71-4723-bdb5-a54abeedecbb\") " pod="openshift-cluster-node-tuning-operator/tuned-4r2gq"
Sep 09 05:49:37.086768 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.086740    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9864f736-2e71-4723-bdb5-a54abeedecbb-host\") pod \"tuned-4r2gq\" (UID: \"9864f736-2e71-4723-bdb5-a54abeedecbb\") " pod="openshift-cluster-node-tuning-operator/tuned-4r2gq"
Sep 09 05:49:37.086768 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.086765    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9864f736-2e71-4723-bdb5-a54abeedecbb-tmp\") pod \"tuned-4r2gq\" (UID: \"9864f736-2e71-4723-bdb5-a54abeedecbb\") " pod="openshift-cluster-node-tuning-operator/tuned-4r2gq"
Sep 09 05:49:37.086920 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.086788    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b5f162ab-38ab-4515-81a3-f46c8f2c0e4c-konnectivity-ca\") pod \"konnectivity-agent-fv99k\" (UID: \"b5f162ab-38ab-4515-81a3-f46c8f2c0e4c\") " pod="kube-system/konnectivity-agent-fv99k"
Sep 09 05:49:37.086920 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.086810    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9864f736-2e71-4723-bdb5-a54abeedecbb-etc-sysctl-conf\") pod \"tuned-4r2gq\" (UID: \"9864f736-2e71-4723-bdb5-a54abeedecbb\") " pod="openshift-cluster-node-tuning-operator/tuned-4r2gq"
Sep 09 05:49:37.086920 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.086820    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 05:49:37.086920 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.086834    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tcsv\" (UniqueName: \"kubernetes.io/projected/9864f736-2e71-4723-bdb5-a54abeedecbb-kube-api-access-2tcsv\") pod \"tuned-4r2gq\" (UID: \"9864f736-2e71-4723-bdb5-a54abeedecbb\") " pod="openshift-cluster-node-tuning-operator/tuned-4r2gq"
Sep 09 05:49:37.086920 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.086857    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b5f162ab-38ab-4515-81a3-f46c8f2c0e4c-agent-certs\") pod \"konnectivity-agent-fv99k\" (UID: \"b5f162ab-38ab-4515-81a3-f46c8f2c0e4c\") " pod="kube-system/konnectivity-agent-fv99k"
Sep 09 05:49:37.086920 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.086878    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6388487-c897-41c8-b8db-30a860ad66e3-host-slash\") pod \"iptables-alerter-r7gnm\" (UID: \"f6388487-c897-41c8-b8db-30a860ad66e3\") " pod="openshift-network-operator/iptables-alerter-r7gnm"
Sep 09 05:49:37.086920 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.086900    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxvtx\" (UniqueName: \"kubernetes.io/projected/f6388487-c897-41c8-b8db-30a860ad66e3-kube-api-access-gxvtx\") pod \"iptables-alerter-r7gnm\" (UID: \"f6388487-c897-41c8-b8db-30a860ad66e3\") " pod="openshift-network-operator/iptables-alerter-r7gnm"
Sep 09 05:49:37.087157 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.086924    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9864f736-2e71-4723-bdb5-a54abeedecbb-etc-kubernetes\") pod \"tuned-4r2gq\" (UID: \"9864f736-2e71-4723-bdb5-a54abeedecbb\") " pod="openshift-cluster-node-tuning-operator/tuned-4r2gq"
Sep 09 05:49:37.087157 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.086939    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9864f736-2e71-4723-bdb5-a54abeedecbb-lib-modules\") pod \"tuned-4r2gq\" (UID: \"9864f736-2e71-4723-bdb5-a54abeedecbb\") " pod="openshift-cluster-node-tuning-operator/tuned-4r2gq"
Sep 09 05:49:37.087157 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.086955    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9864f736-2e71-4723-bdb5-a54abeedecbb-etc-sysctl-d\") pod \"tuned-4r2gq\" (UID: \"9864f736-2e71-4723-bdb5-a54abeedecbb\") " pod="openshift-cluster-node-tuning-operator/tuned-4r2gq"
Sep 09 05:49:37.087157 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.086975    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9864f736-2e71-4723-bdb5-a54abeedecbb-etc-systemd\") pod \"tuned-4r2gq\" (UID: \"9864f736-2e71-4723-bdb5-a54abeedecbb\") " pod="openshift-cluster-node-tuning-operator/tuned-4r2gq"
Sep 09 05:49:37.087157 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.086998    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9864f736-2e71-4723-bdb5-a54abeedecbb-var-lib-kubelet\") pod \"tuned-4r2gq\" (UID: \"9864f736-2e71-4723-bdb5-a54abeedecbb\") " pod="openshift-cluster-node-tuning-operator/tuned-4r2gq"
Sep 09 05:49:37.087157 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.087019    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9864f736-2e71-4723-bdb5-a54abeedecbb-etc-tuned\") pod \"tuned-4r2gq\" (UID: \"9864f736-2e71-4723-bdb5-a54abeedecbb\") " pod="openshift-cluster-node-tuning-operator/tuned-4r2gq"
Sep 09 05:49:37.087157 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.087040    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wbfsk\""
Sep 09 05:49:37.087157 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.087050    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f6388487-c897-41c8-b8db-30a860ad66e3-iptables-alerter-script\") pod \"iptables-alerter-r7gnm\" (UID: \"f6388487-c897-41c8-b8db-30a860ad66e3\") " pod="openshift-network-operator/iptables-alerter-r7gnm"
Sep 09 05:49:37.087157 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.087074    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9864f736-2e71-4723-bdb5-a54abeedecbb-etc-modprobe-d\") pod \"tuned-4r2gq\" (UID: \"9864f736-2e71-4723-bdb5-a54abeedecbb\") " pod="openshift-cluster-node-tuning-operator/tuned-4r2gq"
Sep 09 05:49:37.088293 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.088261    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 05:49:37.088446 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.088427    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9j88h\""
Sep 09 05:49:37.088446 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.088442    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 05:49:37.088581 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.088474    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.088581 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.088511    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 05:49:37.091409 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.091391    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q4pdb"
Sep 09 05:49:37.091653 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.091636    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 05:49:37.091916 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.091897    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 05:49:37.092088 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.092066    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2kj79\""
Sep 09 05:49:37.092170 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.092155    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 05:49:37.092423 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.092407    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 05:49:37.093776 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.093761    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 05:49:37.093882 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.093807    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-h2966\""
Sep 09 05:49:37.094255 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.094085    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 05:49:37.094458 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.094445    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-b9lbm"
Sep 09 05:49:37.096620 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.096603    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 05:49:37.096815 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.096796    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 05:49:37.096896 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.096860    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nm762\""
Sep 09 05:49:37.098825 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.098798    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7r5d"
Sep 09 05:49:37.098915 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:37.098892    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7r5d" podUID="b179b936-0cca-4a9e-86ce-775441316808"
Sep 09 05:49:37.101247 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.101183    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b78pk"
Sep 09 05:49:37.101335 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:37.101315    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b78pk" podUID="5459053f-e126-4327-a1e2-7676a5cc107b"
Sep 09 05:49:37.108223 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.108200    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-59vxh"
Sep 09 05:49:37.117119 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.117102    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-59vxh"
Sep 09 05:49:37.149330 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.149275    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-142.ec2.internal" event={"ID":"59ee296aad7c4e56d27fc66bd1b97d18","Type":"ContainerStarted","Data":"2d58f05f5bc0e195ecdbbc521c71f5b063504f7a76398d3e029db0820404a2c8"}
Sep 09 05:49:37.150908 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.150881    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-142.ec2.internal" event={"ID":"58007dd99525496ea22ff71e4a80cd63","Type":"ContainerStarted","Data":"93a179e1c97f2efaa1ddba4f5a72a5d3fd4538b8d91b93a20d50da8991645903"}
Sep 09 05:49:37.176393 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.176370    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 05:49:37.187857 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.187827    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84455\" (UniqueName: \"kubernetes.io/projected/48323112-dc28-4bd0-825d-9fa9e43304c0-kube-api-access-84455\") pod \"aws-ebs-csi-driver-node-z7lkv\" (UID: \"48323112-dc28-4bd0-825d-9fa9e43304c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7lkv"
Sep 09 05:49:37.187944 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.187873    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9864f736-2e71-4723-bdb5-a54abeedecbb-var-lib-kubelet\") pod \"tuned-4r2gq\" (UID: \"9864f736-2e71-4723-bdb5-a54abeedecbb\") " pod="openshift-cluster-node-tuning-operator/tuned-4r2gq"
Sep 09 05:49:37.187944 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.187907    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-host-run-netns\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.187944 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.187932    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7596b80-c365-41fb-97e8-e99cbad3e412-ovn-node-metrics-cert\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.188100 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.187951    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bbab69e9-3123-418a-92e7-88e2fb389fb5-host-run-k8s-cni-cncf-io\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.188100 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.187965    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bbab69e9-3123-418a-92e7-88e2fb389fb5-host-var-lib-cni-bin\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.188100 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.187990    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9864f736-2e71-4723-bdb5-a54abeedecbb-var-lib-kubelet\") pod \"tuned-4r2gq\" (UID: \"9864f736-2e71-4723-bdb5-a54abeedecbb\") " pod="openshift-cluster-node-tuning-operator/tuned-4r2gq"
Sep 09 05:49:37.188100 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.188004    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b5f162ab-38ab-4515-81a3-f46c8f2c0e4c-konnectivity-ca\") pod \"konnectivity-agent-fv99k\" (UID: \"b5f162ab-38ab-4515-81a3-f46c8f2c0e4c\") " pod="kube-system/konnectivity-agent-fv99k"
Sep 09 05:49:37.188100 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.188042    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-run-openvswitch\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.188278 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.188117    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.188278 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.188144    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/48323112-dc28-4bd0-825d-9fa9e43304c0-registration-dir\") pod \"aws-ebs-csi-driver-node-z7lkv\" (UID: \"48323112-dc28-4bd0-825d-9fa9e43304c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7lkv"
Sep 09 05:49:37.188278 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.188173    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bbab69e9-3123-418a-92e7-88e2fb389fb5-etc-kubernetes\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.188278 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.188203    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tcsv\" (UniqueName: \"kubernetes.io/projected/9864f736-2e71-4723-bdb5-a54abeedecbb-kube-api-access-2tcsv\") pod \"tuned-4r2gq\" (UID: \"9864f736-2e71-4723-bdb5-a54abeedecbb\") " pod="openshift-cluster-node-tuning-operator/tuned-4r2gq"
Sep 09 05:49:37.188278 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.188226    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cd282f88-a442-4ad3-931a-18e86703ece9-serviceca\") pod \"node-ca-brqzd\" (UID: \"cd282f88-a442-4ad3-931a-18e86703ece9\") " pod="openshift-image-registry/node-ca-brqzd"
Sep 09 05:49:37.188278 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.188249    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmjjs\" (UniqueName: \"kubernetes.io/projected/cd282f88-a442-4ad3-931a-18e86703ece9-kube-api-access-wmjjs\") pod \"node-ca-brqzd\" (UID: \"cd282f88-a442-4ad3-931a-18e86703ece9\") " pod="openshift-image-registry/node-ca-brqzd"
Sep 09 05:49:37.188278 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.188271    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9864f736-2e71-4723-bdb5-a54abeedecbb-lib-modules\") pod \"tuned-4r2gq\" (UID: \"9864f736-2e71-4723-bdb5-a54abeedecbb\") " pod="openshift-cluster-node-tuning-operator/tuned-4r2gq"
Sep 09 05:49:37.188552 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.188293    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-systemd-units\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.188552 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.188329    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-host-run-ovn-kubernetes\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.188552 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.188344    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/48323112-dc28-4bd0-825d-9fa9e43304c0-device-dir\") pod \"aws-ebs-csi-driver-node-z7lkv\" (UID: \"48323112-dc28-4bd0-825d-9fa9e43304c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7lkv"
Sep 09 05:49:37.188552 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.188357    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bbab69e9-3123-418a-92e7-88e2fb389fb5-cni-binary-copy\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.188552 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.188386    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bbab69e9-3123-418a-92e7-88e2fb389fb5-host-var-lib-cni-multus\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.188552 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.188422    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9864f736-2e71-4723-bdb5-a54abeedecbb-etc-sysctl-d\") pod \"tuned-4r2gq\" (UID: \"9864f736-2e71-4723-bdb5-a54abeedecbb\") " pod="openshift-cluster-node-tuning-operator/tuned-4r2gq"
Sep 09 05:49:37.188552 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.188436    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9864f736-2e71-4723-bdb5-a54abeedecbb-etc-tuned\") pod \"tuned-4r2gq\" (UID: \"9864f736-2e71-4723-bdb5-a54abeedecbb\") " pod="openshift-cluster-node-tuning-operator/tuned-4r2gq"
Sep 09 05:49:37.188552 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.188455    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9864f736-2e71-4723-bdb5-a54abeedecbb-etc-modprobe-d\") pod \"tuned-4r2gq\" (UID: \"9864f736-2e71-4723-bdb5-a54abeedecbb\") " pod="openshift-cluster-node-tuning-operator/tuned-4r2gq"
Sep 09 05:49:37.188552 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.188469    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9864f736-2e71-4723-bdb5-a54abeedecbb-etc-sysconfig\") pod \"tuned-4r2gq\" (UID: \"9864f736-2e71-4723-bdb5-a54abeedecbb\") " pod="openshift-cluster-node-tuning-operator/tuned-4r2gq"
Sep 09 05:49:37.188552 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.188466    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9864f736-2e71-4723-bdb5-a54abeedecbb-lib-modules\") pod \"tuned-4r2gq\" (UID: \"9864f736-2e71-4723-bdb5-a54abeedecbb\") " pod="openshift-cluster-node-tuning-operator/tuned-4r2gq"
Sep 09 05:49:37.188552 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.188493    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9864f736-2e71-4723-bdb5-a54abeedecbb-host\") pod \"tuned-4r2gq\" (UID: \"9864f736-2e71-4723-bdb5-a54abeedecbb\") " pod="openshift-cluster-node-tuning-operator/tuned-4r2gq"
Sep 09 05:49:37.188552 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.188530    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/48323112-dc28-4bd0-825d-9fa9e43304c0-sys-fs\") pod \"aws-ebs-csi-driver-node-z7lkv\" (UID: \"48323112-dc28-4bd0-825d-9fa9e43304c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7lkv"
Sep 09 05:49:37.188552 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.188543    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b5f162ab-38ab-4515-81a3-f46c8f2c0e4c-konnectivity-ca\") pod \"konnectivity-agent-fv99k\" (UID: \"b5f162ab-38ab-4515-81a3-f46c8f2c0e4c\") " pod="kube-system/konnectivity-agent-fv99k"
Sep 09 05:49:37.189046 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.188563    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bbab69e9-3123-418a-92e7-88e2fb389fb5-system-cni-dir\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.189046 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.188572    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9864f736-2e71-4723-bdb5-a54abeedecbb-etc-sysctl-d\") pod \"tuned-4r2gq\" (UID: \"9864f736-2e71-4723-bdb5-a54abeedecbb\") " pod="openshift-cluster-node-tuning-operator/tuned-4r2gq"
Sep 09 05:49:37.189046 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.188584    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d6d0d7b7-703b-439d-9aae-064c217b875a-hosts-file\") pod \"node-resolver-b9lbm\" (UID: \"d6d0d7b7-703b-439d-9aae-064c217b875a\") " pod="openshift-dns/node-resolver-b9lbm"
Sep 09 05:49:37.189046 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.188587    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9864f736-2e71-4723-bdb5-a54abeedecbb-etc-modprobe-d\") pod \"tuned-4r2gq\" (UID: \"9864f736-2e71-4723-bdb5-a54abeedecbb\") " pod="openshift-cluster-node-tuning-operator/tuned-4r2gq"
Sep 09 05:49:37.189046 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.188536    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9864f736-2e71-4723-bdb5-a54abeedecbb-host\") pod \"tuned-4r2gq\" (UID: \"9864f736-2e71-4723-bdb5-a54abeedecbb\") " pod="openshift-cluster-node-tuning-operator/tuned-4r2gq"
Sep 09 05:49:37.189046 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.188639    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxvtx\" (UniqueName: \"kubernetes.io/projected/f6388487-c897-41c8-b8db-30a860ad66e3-kube-api-access-gxvtx\") pod \"iptables-alerter-r7gnm\" (UID: \"f6388487-c897-41c8-b8db-30a860ad66e3\") " pod="openshift-network-operator/iptables-alerter-r7gnm"
Sep 09 05:49:37.189046 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.188671    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/48323112-dc28-4bd0-825d-9fa9e43304c0-socket-dir\") pod \"aws-ebs-csi-driver-node-z7lkv\" (UID: \"48323112-dc28-4bd0-825d-9fa9e43304c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7lkv"
Sep 09 05:49:37.189046 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.188702    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9864f736-2e71-4723-bdb5-a54abeedecbb-etc-sysconfig\") pod \"tuned-4r2gq\" (UID: \"9864f736-2e71-4723-bdb5-a54abeedecbb\") " pod="openshift-cluster-node-tuning-operator/tuned-4r2gq"
Sep 09 05:49:37.189046 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.188769    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/08607c27-ea7f-4487-9e8c-d84f000464d7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q4pdb\" (UID: \"08607c27-ea7f-4487-9e8c-d84f000464d7\") " pod="openshift-multus/multus-additional-cni-plugins-q4pdb"
Sep 09 05:49:37.189046 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.188794    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-run-systemd\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.189046 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.188828    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-node-log\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.189046 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.188851    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bbab69e9-3123-418a-92e7-88e2fb389fb5-multus-conf-dir\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.189046 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.188902    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7kp7\" (UniqueName: \"kubernetes.io/projected/bbab69e9-3123-418a-92e7-88e2fb389fb5-kube-api-access-c7kp7\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.189046 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.188924    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2ww5\" (UniqueName: \"kubernetes.io/projected/d6d0d7b7-703b-439d-9aae-064c217b875a-kube-api-access-w2ww5\") pod \"node-resolver-b9lbm\" (UID: \"d6d0d7b7-703b-439d-9aae-064c217b875a\") " pod="openshift-dns/node-resolver-b9lbm"
Sep 09 05:49:37.189046 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.188938    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-etc-openvswitch\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.189046 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.188987    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-host-cni-bin\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.189046 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.189020    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-host-cni-netd\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.189665 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.189059    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 05:49:37.189665 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.189051    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/48323112-dc28-4bd0-825d-9fa9e43304c0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z7lkv\" (UID: \"48323112-dc28-4bd0-825d-9fa9e43304c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7lkv"
Sep 09 05:49:37.189665 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.189120    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9864f736-2e71-4723-bdb5-a54abeedecbb-sys\") pod \"tuned-4r2gq\" (UID: \"9864f736-2e71-4723-bdb5-a54abeedecbb\") " pod="openshift-cluster-node-tuning-operator/tuned-4r2gq"
Sep 09 05:49:37.189665 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.189140    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-host-kubelet\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.189665 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.189180    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26cac444-abfa-4060-8f52-4e5756866f3f-original-pull-secret\") pod \"global-pull-secret-syncer-fs4n6\" (UID: \"26cac444-abfa-4060-8f52-4e5756866f3f\") " pod="kube-system/global-pull-secret-syncer-fs4n6"
Sep 09 05:49:37.189665 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.189201    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bbab69e9-3123-418a-92e7-88e2fb389fb5-cnibin\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.189665 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.189417    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08607c27-ea7f-4487-9e8c-d84f000464d7-cni-binary-copy\") pod \"multus-additional-cni-plugins-q4pdb\" (UID: \"08607c27-ea7f-4487-9e8c-d84f000464d7\") " pod="openshift-multus/multus-additional-cni-plugins-q4pdb"
Sep 09 05:49:37.189665 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.189442    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/08607c27-ea7f-4487-9e8c-d84f000464d7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q4pdb\" (UID: \"08607c27-ea7f-4487-9e8c-d84f000464d7\") " pod="openshift-multus/multus-additional-cni-plugins-q4pdb"
Sep 09 05:49:37.189665 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.189468    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9864f736-2e71-4723-bdb5-a54abeedecbb-etc-sysctl-conf\") pod \"tuned-4r2gq\" (UID: \"9864f736-2e71-4723-bdb5-a54abeedecbb\") " pod="openshift-cluster-node-tuning-operator/tuned-4r2gq"
Sep 09 05:49:37.189665 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.189275    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9864f736-2e71-4723-bdb5-a54abeedecbb-sys\") pod \"tuned-4r2gq\" (UID: \"9864f736-2e71-4723-bdb5-a54abeedecbb\") " pod="openshift-cluster-node-tuning-operator/tuned-4r2gq"
Sep 09 05:49:37.189665 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.189510    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b5f162ab-38ab-4515-81a3-f46c8f2c0e4c-agent-certs\") pod \"konnectivity-agent-fv99k\" (UID: \"b5f162ab-38ab-4515-81a3-f46c8f2c0e4c\") " pod="kube-system/konnectivity-agent-fv99k"
Sep 09 05:49:37.189665 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.189569    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6388487-c897-41c8-b8db-30a860ad66e3-host-slash\") pod \"iptables-alerter-r7gnm\" (UID: \"f6388487-c897-41c8-b8db-30a860ad66e3\") " pod="openshift-network-operator/iptables-alerter-r7gnm"
Sep 09 05:49:37.190149 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.189673    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/26cac444-abfa-4060-8f52-4e5756866f3f-dbus\") pod \"global-pull-secret-syncer-fs4n6\" (UID: \"26cac444-abfa-4060-8f52-4e5756866f3f\") " pod="kube-system/global-pull-secret-syncer-fs4n6"
Sep 09 05:49:37.190149 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.189679    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9864f736-2e71-4723-bdb5-a54abeedecbb-etc-sysctl-conf\") pod \"tuned-4r2gq\" (UID: \"9864f736-2e71-4723-bdb5-a54abeedecbb\") " pod="openshift-cluster-node-tuning-operator/tuned-4r2gq"
Sep 09 05:49:37.190149 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.189683    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6388487-c897-41c8-b8db-30a860ad66e3-host-slash\") pod \"iptables-alerter-r7gnm\" (UID: \"f6388487-c897-41c8-b8db-30a860ad66e3\") " pod="openshift-network-operator/iptables-alerter-r7gnm"
Sep 09 05:49:37.190149 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.189717    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b179b936-0cca-4a9e-86ce-775441316808-metrics-certs\") pod \"network-metrics-daemon-s7r5d\" (UID: \"b179b936-0cca-4a9e-86ce-775441316808\") " pod="openshift-multus/network-metrics-daemon-s7r5d"
Sep 09 05:49:37.190149 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.189755    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/48323112-dc28-4bd0-825d-9fa9e43304c0-etc-selinux\") pod \"aws-ebs-csi-driver-node-z7lkv\" (UID: \"48323112-dc28-4bd0-825d-9fa9e43304c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7lkv"
Sep 09 05:49:37.190149 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.189780    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08607c27-ea7f-4487-9e8c-d84f000464d7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q4pdb\" (UID: \"08607c27-ea7f-4487-9e8c-d84f000464d7\") " pod="openshift-multus/multus-additional-cni-plugins-q4pdb"
Sep 09 05:49:37.190149 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.189802    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6d0d7b7-703b-439d-9aae-064c217b875a-tmp-dir\") pod \"node-resolver-b9lbm\" (UID: \"d6d0d7b7-703b-439d-9aae-064c217b875a\") " pod="openshift-dns/node-resolver-b9lbm"
Sep 09 05:49:37.190149 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.189826    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9864f736-2e71-4723-bdb5-a54abeedecbb-etc-kubernetes\") pod \"tuned-4r2gq\" (UID: \"9864f736-2e71-4723-bdb5-a54abeedecbb\") " pod="openshift-cluster-node-tuning-operator/tuned-4r2gq"
Sep 09 05:49:37.190149 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.189850    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-log-socket\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.190149 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.189872    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bbab69e9-3123-418a-92e7-88e2fb389fb5-host-run-netns\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.190149 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.189914    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9864f736-2e71-4723-bdb5-a54abeedecbb-etc-kubernetes\") pod \"tuned-4r2gq\" (UID: \"9864f736-2e71-4723-bdb5-a54abeedecbb\") " pod="openshift-cluster-node-tuning-operator/tuned-4r2gq"
Sep 09 05:49:37.190149 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.189914    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bbab69e9-3123-418a-92e7-88e2fb389fb5-hostroot\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.190149 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.189953    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08607c27-ea7f-4487-9e8c-d84f000464d7-system-cni-dir\") pod \"multus-additional-cni-plugins-q4pdb\" (UID: \"08607c27-ea7f-4487-9e8c-d84f000464d7\") " pod="openshift-multus/multus-additional-cni-plugins-q4pdb"
Sep 09 05:49:37.190149 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.189981    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55mpw\" (UniqueName: \"kubernetes.io/projected/08607c27-ea7f-4487-9e8c-d84f000464d7-kube-api-access-55mpw\") pod \"multus-additional-cni-plugins-q4pdb\" (UID: \"08607c27-ea7f-4487-9e8c-d84f000464d7\") " pod="openshift-multus/multus-additional-cni-plugins-q4pdb"
Sep 09 05:49:37.190149 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.190007    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqt5c\" (UniqueName: \"kubernetes.io/projected/5459053f-e126-4327-a1e2-7676a5cc107b-kube-api-access-rqt5c\") pod \"network-check-target-b78pk\" (UID: \"5459053f-e126-4327-a1e2-7676a5cc107b\") " pod="openshift-network-diagnostics/network-check-target-b78pk"
Sep 09 05:49:37.190149 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.190032    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9864f736-2e71-4723-bdb5-a54abeedecbb-etc-systemd\") pod \"tuned-4r2gq\" (UID: \"9864f736-2e71-4723-bdb5-a54abeedecbb\") " pod="openshift-cluster-node-tuning-operator/tuned-4r2gq"
Sep 09 05:49:37.190149 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.190056    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f6388487-c897-41c8-b8db-30a860ad66e3-iptables-alerter-script\") pod \"iptables-alerter-r7gnm\" (UID: \"f6388487-c897-41c8-b8db-30a860ad66e3\") " pod="openshift-network-operator/iptables-alerter-r7gnm"
Sep 09 05:49:37.190834 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.190079    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-run-ovn\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.190834 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.190100    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7596b80-c365-41fb-97e8-e99cbad3e412-env-overrides\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.190834 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.190124    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7596b80-c365-41fb-97e8-e99cbad3e412-ovnkube-script-lib\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.190834 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.190147    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bbab69e9-3123-418a-92e7-88e2fb389fb5-multus-cni-dir\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.190834 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.190183    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9864f736-2e71-4723-bdb5-a54abeedecbb-run\") pod \"tuned-4r2gq\" (UID: \"9864f736-2e71-4723-bdb5-a54abeedecbb\") " pod="openshift-cluster-node-tuning-operator/tuned-4r2gq"
Sep 09 05:49:37.190834 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.190205    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9864f736-2e71-4723-bdb5-a54abeedecbb-tmp\") pod \"tuned-4r2gq\" (UID: \"9864f736-2e71-4723-bdb5-a54abeedecbb\") " pod="openshift-cluster-node-tuning-operator/tuned-4r2gq"
Sep 09 05:49:37.190834 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.190227    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-var-lib-openvswitch\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.190834 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.190254    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7596b80-c365-41fb-97e8-e99cbad3e412-ovnkube-config\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.190834 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.190586    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f6388487-c897-41c8-b8db-30a860ad66e3-iptables-alerter-script\") pod \"iptables-alerter-r7gnm\" (UID: \"f6388487-c897-41c8-b8db-30a860ad66e3\") " pod="openshift-network-operator/iptables-alerter-r7gnm"
Sep 09 05:49:37.190834 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.190708    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9864f736-2e71-4723-bdb5-a54abeedecbb-run\") pod \"tuned-4r2gq\" (UID: \"9864f736-2e71-4723-bdb5-a54abeedecbb\") " pod="openshift-cluster-node-tuning-operator/tuned-4r2gq"
Sep 09 05:49:37.190834 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.190157    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9864f736-2e71-4723-bdb5-a54abeedecbb-etc-systemd\") pod \"tuned-4r2gq\" (UID: \"9864f736-2e71-4723-bdb5-a54abeedecbb\") " pod="openshift-cluster-node-tuning-operator/tuned-4r2gq"
Sep 09 05:49:37.190834 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.190729    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76bsk\" (UniqueName: \"kubernetes.io/projected/f7596b80-c365-41fb-97e8-e99cbad3e412-kube-api-access-76bsk\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.190834 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.190767    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/26cac444-abfa-4060-8f52-4e5756866f3f-kubelet-config\") pod \"global-pull-secret-syncer-fs4n6\" (UID: \"26cac444-abfa-4060-8f52-4e5756866f3f\") " pod="kube-system/global-pull-secret-syncer-fs4n6"
Sep 09 05:49:37.190834 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.190800    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bbab69e9-3123-418a-92e7-88e2fb389fb5-multus-socket-dir-parent\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.190834 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.190824    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bbab69e9-3123-418a-92e7-88e2fb389fb5-multus-daemon-config\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.191490 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.190846    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-host-slash\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.191490 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.190869    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jczdf\" (UniqueName: \"kubernetes.io/projected/b179b936-0cca-4a9e-86ce-775441316808-kube-api-access-jczdf\") pod \"network-metrics-daemon-s7r5d\" (UID: \"b179b936-0cca-4a9e-86ce-775441316808\") " pod="openshift-multus/network-metrics-daemon-s7r5d"
Sep 09 05:49:37.191490 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.190898    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbab69e9-3123-418a-92e7-88e2fb389fb5-host-var-lib-kubelet\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.191490 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.190920    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bbab69e9-3123-418a-92e7-88e2fb389fb5-host-run-multus-certs\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.191490 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.190943    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08607c27-ea7f-4487-9e8c-d84f000464d7-cnibin\") pod \"multus-additional-cni-plugins-q4pdb\" (UID: \"08607c27-ea7f-4487-9e8c-d84f000464d7\") " pod="openshift-multus/multus-additional-cni-plugins-q4pdb"
Sep 09 05:49:37.191490 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.190989    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cd282f88-a442-4ad3-931a-18e86703ece9-host\") pod \"node-ca-brqzd\" (UID: \"cd282f88-a442-4ad3-931a-18e86703ece9\") " pod="openshift-image-registry/node-ca-brqzd"
Sep 09 05:49:37.194209 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.194143    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9864f736-2e71-4723-bdb5-a54abeedecbb-etc-tuned\") pod \"tuned-4r2gq\" (UID: \"9864f736-2e71-4723-bdb5-a54abeedecbb\") " pod="openshift-cluster-node-tuning-operator/tuned-4r2gq"
Sep 09 05:49:37.194209 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.194153    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9864f736-2e71-4723-bdb5-a54abeedecbb-tmp\") pod \"tuned-4r2gq\" (UID: \"9864f736-2e71-4723-bdb5-a54abeedecbb\") " pod="openshift-cluster-node-tuning-operator/tuned-4r2gq"
Sep 09 05:49:37.194388 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.194371    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b5f162ab-38ab-4515-81a3-f46c8f2c0e4c-agent-certs\") pod \"konnectivity-agent-fv99k\" (UID: \"b5f162ab-38ab-4515-81a3-f46c8f2c0e4c\") " pod="kube-system/konnectivity-agent-fv99k"
Sep 09 05:49:37.199234 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.199198    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tcsv\" (UniqueName: \"kubernetes.io/projected/9864f736-2e71-4723-bdb5-a54abeedecbb-kube-api-access-2tcsv\") pod \"tuned-4r2gq\" (UID: \"9864f736-2e71-4723-bdb5-a54abeedecbb\") " pod="openshift-cluster-node-tuning-operator/tuned-4r2gq"
Sep 09 05:49:37.200543 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.200513    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxvtx\" (UniqueName: \"kubernetes.io/projected/f6388487-c897-41c8-b8db-30a860ad66e3-kube-api-access-gxvtx\") pod \"iptables-alerter-r7gnm\" (UID: \"f6388487-c897-41c8-b8db-30a860ad66e3\") " pod="openshift-network-operator/iptables-alerter-r7gnm"
Sep 09 05:49:37.291860 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.291827    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7596b80-c365-41fb-97e8-e99cbad3e412-ovnkube-config\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.291984 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.291868    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76bsk\" (UniqueName: \"kubernetes.io/projected/f7596b80-c365-41fb-97e8-e99cbad3e412-kube-api-access-76bsk\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.291984 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.291886    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/26cac444-abfa-4060-8f52-4e5756866f3f-kubelet-config\") pod \"global-pull-secret-syncer-fs4n6\" (UID: \"26cac444-abfa-4060-8f52-4e5756866f3f\") " pod="kube-system/global-pull-secret-syncer-fs4n6"
Sep 09 05:49:37.291984 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.291903    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bbab69e9-3123-418a-92e7-88e2fb389fb5-multus-socket-dir-parent\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.291984 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.291924    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bbab69e9-3123-418a-92e7-88e2fb389fb5-multus-daemon-config\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.291984 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.291946    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-host-slash\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.292243 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.291983    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/26cac444-abfa-4060-8f52-4e5756866f3f-kubelet-config\") pod \"global-pull-secret-syncer-fs4n6\" (UID: \"26cac444-abfa-4060-8f52-4e5756866f3f\") " pod="kube-system/global-pull-secret-syncer-fs4n6"
Sep 09 05:49:37.292243 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.291985    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bbab69e9-3123-418a-92e7-88e2fb389fb5-multus-socket-dir-parent\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.292243 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.291990    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-host-slash\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.292243 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292046    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jczdf\" (UniqueName: \"kubernetes.io/projected/b179b936-0cca-4a9e-86ce-775441316808-kube-api-access-jczdf\") pod \"network-metrics-daemon-s7r5d\" (UID: \"b179b936-0cca-4a9e-86ce-775441316808\") " pod="openshift-multus/network-metrics-daemon-s7r5d"
Sep 09 05:49:37.292243 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292072    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbab69e9-3123-418a-92e7-88e2fb389fb5-host-var-lib-kubelet\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.292243 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292089    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bbab69e9-3123-418a-92e7-88e2fb389fb5-host-run-multus-certs\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.292243 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292103    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08607c27-ea7f-4487-9e8c-d84f000464d7-cnibin\") pod \"multus-additional-cni-plugins-q4pdb\" (UID: \"08607c27-ea7f-4487-9e8c-d84f000464d7\") " pod="openshift-multus/multus-additional-cni-plugins-q4pdb"
Sep 09 05:49:37.292243 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292120    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cd282f88-a442-4ad3-931a-18e86703ece9-host\") pod \"node-ca-brqzd\" (UID: \"cd282f88-a442-4ad3-931a-18e86703ece9\") " pod="openshift-image-registry/node-ca-brqzd"
Sep 09 05:49:37.292243 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292142    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbab69e9-3123-418a-92e7-88e2fb389fb5-host-var-lib-kubelet\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.292243 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292153    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bbab69e9-3123-418a-92e7-88e2fb389fb5-host-run-multus-certs\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.292243 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292144    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84455\" (UniqueName: \"kubernetes.io/projected/48323112-dc28-4bd0-825d-9fa9e43304c0-kube-api-access-84455\") pod \"aws-ebs-csi-driver-node-z7lkv\" (UID: \"48323112-dc28-4bd0-825d-9fa9e43304c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7lkv"
Sep 09 05:49:37.292243 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292203    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-host-run-netns\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.292243 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292200    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08607c27-ea7f-4487-9e8c-d84f000464d7-cnibin\") pod \"multus-additional-cni-plugins-q4pdb\" (UID: \"08607c27-ea7f-4487-9e8c-d84f000464d7\") " pod="openshift-multus/multus-additional-cni-plugins-q4pdb"
Sep 09 05:49:37.292243 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292230    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7596b80-c365-41fb-97e8-e99cbad3e412-ovn-node-metrics-cert\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.292955 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292278    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bbab69e9-3123-418a-92e7-88e2fb389fb5-host-run-k8s-cni-cncf-io\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.292955 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292283    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-host-run-netns\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.292955 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292309    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bbab69e9-3123-418a-92e7-88e2fb389fb5-host-var-lib-cni-bin\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.292955 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292337    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-run-openvswitch\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.292955 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292342    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bbab69e9-3123-418a-92e7-88e2fb389fb5-host-run-k8s-cni-cncf-io\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.292955 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292252    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cd282f88-a442-4ad3-931a-18e86703ece9-host\") pod \"node-ca-brqzd\" (UID: \"cd282f88-a442-4ad3-931a-18e86703ece9\") " pod="openshift-image-registry/node-ca-brqzd"
Sep 09 05:49:37.292955 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292361    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.292955 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292388    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/48323112-dc28-4bd0-825d-9fa9e43304c0-registration-dir\") pod \"aws-ebs-csi-driver-node-z7lkv\" (UID: \"48323112-dc28-4bd0-825d-9fa9e43304c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7lkv"
Sep 09 05:49:37.292955 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292402    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bbab69e9-3123-418a-92e7-88e2fb389fb5-host-var-lib-cni-bin\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.292955 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292411    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bbab69e9-3123-418a-92e7-88e2fb389fb5-etc-kubernetes\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.292955 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292438    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cd282f88-a442-4ad3-931a-18e86703ece9-serviceca\") pod \"node-ca-brqzd\" (UID: \"cd282f88-a442-4ad3-931a-18e86703ece9\") " pod="openshift-image-registry/node-ca-brqzd"
Sep 09 05:49:37.292955 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292444    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.292955 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292451    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7596b80-c365-41fb-97e8-e99cbad3e412-ovnkube-config\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.292955 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292461    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/48323112-dc28-4bd0-825d-9fa9e43304c0-registration-dir\") pod \"aws-ebs-csi-driver-node-z7lkv\" (UID: \"48323112-dc28-4bd0-825d-9fa9e43304c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7lkv"
Sep 09 05:49:37.292955 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292462    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmjjs\" (UniqueName: \"kubernetes.io/projected/cd282f88-a442-4ad3-931a-18e86703ece9-kube-api-access-wmjjs\") pod \"node-ca-brqzd\" (UID: \"cd282f88-a442-4ad3-931a-18e86703ece9\") " pod="openshift-image-registry/node-ca-brqzd"
Sep 09 05:49:37.292955 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292403    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-run-openvswitch\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.292955 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292488    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bbab69e9-3123-418a-92e7-88e2fb389fb5-etc-kubernetes\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.293758 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292501    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-systemd-units\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.293758 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292533    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-host-run-ovn-kubernetes\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.293758 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292534    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bbab69e9-3123-418a-92e7-88e2fb389fb5-multus-daemon-config\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.293758 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292557    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/48323112-dc28-4bd0-825d-9fa9e43304c0-device-dir\") pod \"aws-ebs-csi-driver-node-z7lkv\" (UID: \"48323112-dc28-4bd0-825d-9fa9e43304c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7lkv"
Sep 09 05:49:37.293758 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292581    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bbab69e9-3123-418a-92e7-88e2fb389fb5-cni-binary-copy\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.293758 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292591    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-systemd-units\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.293758 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292602    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bbab69e9-3123-418a-92e7-88e2fb389fb5-host-var-lib-cni-multus\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.293758 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292611    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/48323112-dc28-4bd0-825d-9fa9e43304c0-device-dir\") pod \"aws-ebs-csi-driver-node-z7lkv\" (UID: \"48323112-dc28-4bd0-825d-9fa9e43304c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7lkv"
Sep 09 05:49:37.293758 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292661    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/48323112-dc28-4bd0-825d-9fa9e43304c0-sys-fs\") pod \"aws-ebs-csi-driver-node-z7lkv\" (UID: \"48323112-dc28-4bd0-825d-9fa9e43304c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7lkv"
Sep 09 05:49:37.293758 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292603    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-host-run-ovn-kubernetes\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.293758 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292688    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bbab69e9-3123-418a-92e7-88e2fb389fb5-system-cni-dir\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.293758 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292709    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d6d0d7b7-703b-439d-9aae-064c217b875a-hosts-file\") pod \"node-resolver-b9lbm\" (UID: \"d6d0d7b7-703b-439d-9aae-064c217b875a\") " pod="openshift-dns/node-resolver-b9lbm"
Sep 09 05:49:37.293758 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292723    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/48323112-dc28-4bd0-825d-9fa9e43304c0-sys-fs\") pod \"aws-ebs-csi-driver-node-z7lkv\" (UID: \"48323112-dc28-4bd0-825d-9fa9e43304c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7lkv"
Sep 09 05:49:37.293758 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292735    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/48323112-dc28-4bd0-825d-9fa9e43304c0-socket-dir\") pod \"aws-ebs-csi-driver-node-z7lkv\" (UID: \"48323112-dc28-4bd0-825d-9fa9e43304c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7lkv"
Sep 09 05:49:37.293758 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292663    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bbab69e9-3123-418a-92e7-88e2fb389fb5-host-var-lib-cni-multus\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.293758 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292757    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bbab69e9-3123-418a-92e7-88e2fb389fb5-system-cni-dir\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.293758 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292776    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d6d0d7b7-703b-439d-9aae-064c217b875a-hosts-file\") pod \"node-resolver-b9lbm\" (UID: \"d6d0d7b7-703b-439d-9aae-064c217b875a\") " pod="openshift-dns/node-resolver-b9lbm"
Sep 09 05:49:37.294540 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292767    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/08607c27-ea7f-4487-9e8c-d84f000464d7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q4pdb\" (UID: \"08607c27-ea7f-4487-9e8c-d84f000464d7\") " pod="openshift-multus/multus-additional-cni-plugins-q4pdb"
Sep 09 05:49:37.294540 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292816    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-run-systemd\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.294540 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292840    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-node-log\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.294540 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292843    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/48323112-dc28-4bd0-825d-9fa9e43304c0-socket-dir\") pod \"aws-ebs-csi-driver-node-z7lkv\" (UID: \"48323112-dc28-4bd0-825d-9fa9e43304c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7lkv"
Sep 09 05:49:37.294540 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292875    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bbab69e9-3123-418a-92e7-88e2fb389fb5-multus-conf-dir\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.294540 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292880    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-node-log\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.294540 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292898    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cd282f88-a442-4ad3-931a-18e86703ece9-serviceca\") pod \"node-ca-brqzd\" (UID: \"cd282f88-a442-4ad3-931a-18e86703ece9\") " pod="openshift-image-registry/node-ca-brqzd"
Sep 09 05:49:37.294540 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292901    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7kp7\" (UniqueName: \"kubernetes.io/projected/bbab69e9-3123-418a-92e7-88e2fb389fb5-kube-api-access-c7kp7\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.294540 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292916    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-run-systemd\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.294540 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292926    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2ww5\" (UniqueName: \"kubernetes.io/projected/d6d0d7b7-703b-439d-9aae-064c217b875a-kube-api-access-w2ww5\") pod \"node-resolver-b9lbm\" (UID: \"d6d0d7b7-703b-439d-9aae-064c217b875a\") " pod="openshift-dns/node-resolver-b9lbm"
Sep 09 05:49:37.294540 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292950    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-etc-openvswitch\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.294540 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292942    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bbab69e9-3123-418a-92e7-88e2fb389fb5-multus-conf-dir\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.294540 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.292975    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-host-cni-bin\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.294540 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.293000    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-host-cni-netd\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.294540 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.293023    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/48323112-dc28-4bd0-825d-9fa9e43304c0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z7lkv\" (UID: \"48323112-dc28-4bd0-825d-9fa9e43304c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7lkv"
Sep 09 05:49:37.294540 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.293046    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-host-kubelet\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.294540 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.293068    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26cac444-abfa-4060-8f52-4e5756866f3f-original-pull-secret\") pod \"global-pull-secret-syncer-fs4n6\" (UID: \"26cac444-abfa-4060-8f52-4e5756866f3f\") " pod="kube-system/global-pull-secret-syncer-fs4n6"
Sep 09 05:49:37.294540 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.293090    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bbab69e9-3123-418a-92e7-88e2fb389fb5-cnibin\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.295187 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.293105    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bbab69e9-3123-418a-92e7-88e2fb389fb5-cni-binary-copy\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.295187 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.293113    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08607c27-ea7f-4487-9e8c-d84f000464d7-cni-binary-copy\") pod \"multus-additional-cni-plugins-q4pdb\" (UID: \"08607c27-ea7f-4487-9e8c-d84f000464d7\") " pod="openshift-multus/multus-additional-cni-plugins-q4pdb"
Sep 09 05:49:37.295187 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.293124    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/48323112-dc28-4bd0-825d-9fa9e43304c0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z7lkv\" (UID: \"48323112-dc28-4bd0-825d-9fa9e43304c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7lkv"
Sep 09 05:49:37.295187 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.293129    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-host-cni-bin\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.295187 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.293137    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/08607c27-ea7f-4487-9e8c-d84f000464d7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q4pdb\" (UID: \"08607c27-ea7f-4487-9e8c-d84f000464d7\") " pod="openshift-multus/multus-additional-cni-plugins-q4pdb"
Sep 09 05:49:37.295187 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.293092    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-host-cni-netd\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.295187 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.293112    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-etc-openvswitch\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.295187 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.293180    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/26cac444-abfa-4060-8f52-4e5756866f3f-dbus\") pod \"global-pull-secret-syncer-fs4n6\" (UID: \"26cac444-abfa-4060-8f52-4e5756866f3f\") " pod="kube-system/global-pull-secret-syncer-fs4n6"
Sep 09 05:49:37.295187 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.293187    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-host-kubelet\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.295187 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.293237    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/26cac444-abfa-4060-8f52-4e5756866f3f-dbus\") pod \"global-pull-secret-syncer-fs4n6\" (UID: \"26cac444-abfa-4060-8f52-4e5756866f3f\") " pod="kube-system/global-pull-secret-syncer-fs4n6"
Sep 09 05:49:37.295187 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.293240    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bbab69e9-3123-418a-92e7-88e2fb389fb5-cnibin\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.295187 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:37.293280    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 05:49:37.295187 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.293348    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b179b936-0cca-4a9e-86ce-775441316808-metrics-certs\") pod \"network-metrics-daemon-s7r5d\" (UID: \"b179b936-0cca-4a9e-86ce-775441316808\") " pod="openshift-multus/network-metrics-daemon-s7r5d"
Sep 09 05:49:37.295187 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:37.293377    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26cac444-abfa-4060-8f52-4e5756866f3f-original-pull-secret podName:26cac444-abfa-4060-8f52-4e5756866f3f nodeName:}" failed. No retries permitted until 2026-09-09 05:49:37.793335319 +0000 UTC m=+3.133823220 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/26cac444-abfa-4060-8f52-4e5756866f3f-original-pull-secret") pod "global-pull-secret-syncer-fs4n6" (UID: "26cac444-abfa-4060-8f52-4e5756866f3f") : object "kube-system"/"original-pull-secret" not registered
Sep 09 05:49:37.295187 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:37.293407    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 05:49:37.295187 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.293410    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/48323112-dc28-4bd0-825d-9fa9e43304c0-etc-selinux\") pod \"aws-ebs-csi-driver-node-z7lkv\" (UID: \"48323112-dc28-4bd0-825d-9fa9e43304c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7lkv"
Sep 09 05:49:37.295187 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.293438    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08607c27-ea7f-4487-9e8c-d84f000464d7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q4pdb\" (UID: \"08607c27-ea7f-4487-9e8c-d84f000464d7\") " pod="openshift-multus/multus-additional-cni-plugins-q4pdb"
Sep 09 05:49:37.295775 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:37.293458    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b179b936-0cca-4a9e-86ce-775441316808-metrics-certs podName:b179b936-0cca-4a9e-86ce-775441316808 nodeName:}" failed. No retries permitted until 2026-09-09 05:49:37.793444282 +0000 UTC m=+3.133932172 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b179b936-0cca-4a9e-86ce-775441316808-metrics-certs") pod "network-metrics-daemon-s7r5d" (UID: "b179b936-0cca-4a9e-86ce-775441316808") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 05:49:37.295775 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.293484    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6d0d7b7-703b-439d-9aae-064c217b875a-tmp-dir\") pod \"node-resolver-b9lbm\" (UID: \"d6d0d7b7-703b-439d-9aae-064c217b875a\") " pod="openshift-dns/node-resolver-b9lbm"
Sep 09 05:49:37.295775 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.293490    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/08607c27-ea7f-4487-9e8c-d84f000464d7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q4pdb\" (UID: \"08607c27-ea7f-4487-9e8c-d84f000464d7\") " pod="openshift-multus/multus-additional-cni-plugins-q4pdb"
Sep 09 05:49:37.295775 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.293509    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-log-socket\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.295775 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.293532    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bbab69e9-3123-418a-92e7-88e2fb389fb5-host-run-netns\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.295775 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.293554    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bbab69e9-3123-418a-92e7-88e2fb389fb5-hostroot\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.295775 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.293594    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bbab69e9-3123-418a-92e7-88e2fb389fb5-hostroot\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.295775 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.293591    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/08607c27-ea7f-4487-9e8c-d84f000464d7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q4pdb\" (UID: \"08607c27-ea7f-4487-9e8c-d84f000464d7\") " pod="openshift-multus/multus-additional-cni-plugins-q4pdb"
Sep 09 05:49:37.295775 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.293616    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08607c27-ea7f-4487-9e8c-d84f000464d7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q4pdb\" (UID: \"08607c27-ea7f-4487-9e8c-d84f000464d7\") " pod="openshift-multus/multus-additional-cni-plugins-q4pdb"
Sep 09 05:49:37.295775 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.293599    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/48323112-dc28-4bd0-825d-9fa9e43304c0-etc-selinux\") pod \"aws-ebs-csi-driver-node-z7lkv\" (UID: \"48323112-dc28-4bd0-825d-9fa9e43304c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7lkv"
Sep 09 05:49:37.295775 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.293607    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08607c27-ea7f-4487-9e8c-d84f000464d7-cni-binary-copy\") pod \"multus-additional-cni-plugins-q4pdb\" (UID: \"08607c27-ea7f-4487-9e8c-d84f000464d7\") " pod="openshift-multus/multus-additional-cni-plugins-q4pdb"
Sep 09 05:49:37.295775 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.293666    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bbab69e9-3123-418a-92e7-88e2fb389fb5-host-run-netns\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.295775 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.293661    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-log-socket\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.295775 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.293660    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08607c27-ea7f-4487-9e8c-d84f000464d7-system-cni-dir\") pod \"multus-additional-cni-plugins-q4pdb\" (UID: \"08607c27-ea7f-4487-9e8c-d84f000464d7\") " pod="openshift-multus/multus-additional-cni-plugins-q4pdb"
Sep 09 05:49:37.295775 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.293689    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08607c27-ea7f-4487-9e8c-d84f000464d7-system-cni-dir\") pod \"multus-additional-cni-plugins-q4pdb\" (UID: \"08607c27-ea7f-4487-9e8c-d84f000464d7\") " pod="openshift-multus/multus-additional-cni-plugins-q4pdb"
Sep 09 05:49:37.295775 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.293712    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55mpw\" (UniqueName: \"kubernetes.io/projected/08607c27-ea7f-4487-9e8c-d84f000464d7-kube-api-access-55mpw\") pod \"multus-additional-cni-plugins-q4pdb\" (UID: \"08607c27-ea7f-4487-9e8c-d84f000464d7\") " pod="openshift-multus/multus-additional-cni-plugins-q4pdb"
Sep 09 05:49:37.296245 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.293763    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqt5c\" (UniqueName: \"kubernetes.io/projected/5459053f-e126-4327-a1e2-7676a5cc107b-kube-api-access-rqt5c\") pod \"network-check-target-b78pk\" (UID: \"5459053f-e126-4327-a1e2-7676a5cc107b\") " pod="openshift-network-diagnostics/network-check-target-b78pk"
Sep 09 05:49:37.296245 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.293794    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-run-ovn\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.296245 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.293813    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6d0d7b7-703b-439d-9aae-064c217b875a-tmp-dir\") pod \"node-resolver-b9lbm\" (UID: \"d6d0d7b7-703b-439d-9aae-064c217b875a\") " pod="openshift-dns/node-resolver-b9lbm"
Sep 09 05:49:37.296245 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.293820    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7596b80-c365-41fb-97e8-e99cbad3e412-env-overrides\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.296245 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.293844    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7596b80-c365-41fb-97e8-e99cbad3e412-ovnkube-script-lib\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.296245 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.293867    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bbab69e9-3123-418a-92e7-88e2fb389fb5-multus-cni-dir\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.296245 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.293892    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-var-lib-openvswitch\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.296245 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.293958    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-var-lib-openvswitch\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.296245 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.294000    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7596b80-c365-41fb-97e8-e99cbad3e412-run-ovn\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.296245 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.294061    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bbab69e9-3123-418a-92e7-88e2fb389fb5-multus-cni-dir\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.296245 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.294151    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7596b80-c365-41fb-97e8-e99cbad3e412-env-overrides\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.296245 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.294475    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7596b80-c365-41fb-97e8-e99cbad3e412-ovnkube-script-lib\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.296245 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.295087    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7596b80-c365-41fb-97e8-e99cbad3e412-ovn-node-metrics-cert\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.301838 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:37.301818    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 05:49:37.301838 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:37.301839    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 05:49:37.302029 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:37.301852    2579 projected.go:196] Error preparing data for projected volume kube-api-access-rqt5c for pod openshift-network-diagnostics/network-check-target-b78pk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 05:49:37.302029 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:37.301918    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5459053f-e126-4327-a1e2-7676a5cc107b-kube-api-access-rqt5c podName:5459053f-e126-4327-a1e2-7676a5cc107b nodeName:}" failed. No retries permitted until 2026-09-09 05:49:37.801902175 +0000 UTC m=+3.142390078 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rqt5c" (UniqueName: "kubernetes.io/projected/5459053f-e126-4327-a1e2-7676a5cc107b-kube-api-access-rqt5c") pod "network-check-target-b78pk" (UID: "5459053f-e126-4327-a1e2-7676a5cc107b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 05:49:37.303882 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.303858    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2ww5\" (UniqueName: \"kubernetes.io/projected/d6d0d7b7-703b-439d-9aae-064c217b875a-kube-api-access-w2ww5\") pod \"node-resolver-b9lbm\" (UID: \"d6d0d7b7-703b-439d-9aae-064c217b875a\") " pod="openshift-dns/node-resolver-b9lbm"
Sep 09 05:49:37.304366 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.304341    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55mpw\" (UniqueName: \"kubernetes.io/projected/08607c27-ea7f-4487-9e8c-d84f000464d7-kube-api-access-55mpw\") pod \"multus-additional-cni-plugins-q4pdb\" (UID: \"08607c27-ea7f-4487-9e8c-d84f000464d7\") " pod="openshift-multus/multus-additional-cni-plugins-q4pdb"
Sep 09 05:49:37.305360 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.305318    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jczdf\" (UniqueName: \"kubernetes.io/projected/b179b936-0cca-4a9e-86ce-775441316808-kube-api-access-jczdf\") pod \"network-metrics-daemon-s7r5d\" (UID: \"b179b936-0cca-4a9e-86ce-775441316808\") " pod="openshift-multus/network-metrics-daemon-s7r5d"
Sep 09 05:49:37.305488 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.305464    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84455\" (UniqueName: \"kubernetes.io/projected/48323112-dc28-4bd0-825d-9fa9e43304c0-kube-api-access-84455\") pod \"aws-ebs-csi-driver-node-z7lkv\" (UID: \"48323112-dc28-4bd0-825d-9fa9e43304c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7lkv"
Sep 09 05:49:37.305866 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.305844    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7kp7\" (UniqueName: \"kubernetes.io/projected/bbab69e9-3123-418a-92e7-88e2fb389fb5-kube-api-access-c7kp7\") pod \"multus-6z9q7\" (UID: \"bbab69e9-3123-418a-92e7-88e2fb389fb5\") " pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.305935 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.305873    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76bsk\" (UniqueName: \"kubernetes.io/projected/f7596b80-c365-41fb-97e8-e99cbad3e412-kube-api-access-76bsk\") pod \"ovnkube-node-s7jp7\" (UID: \"f7596b80-c365-41fb-97e8-e99cbad3e412\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.307235 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.307192    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmjjs\" (UniqueName: \"kubernetes.io/projected/cd282f88-a442-4ad3-931a-18e86703ece9-kube-api-access-wmjjs\") pod \"node-ca-brqzd\" (UID: \"cd282f88-a442-4ad3-931a-18e86703ece9\") " pod="openshift-image-registry/node-ca-brqzd"
Sep 09 05:49:37.386354 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.386283    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4r2gq"
Sep 09 05:49:37.397479 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.397449    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fv99k"
Sep 09 05:49:37.411315 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.411289    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r7gnm"
Sep 09 05:49:37.416944 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.416923    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:49:37.423497 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.423479    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7lkv"
Sep 09 05:49:37.429028 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.429009    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-brqzd"
Sep 09 05:49:37.436555 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.436536    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6z9q7"
Sep 09 05:49:37.444189 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.444173    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q4pdb"
Sep 09 05:49:37.449766 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.449749    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-b9lbm"
Sep 09 05:49:37.496562 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.496540    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 05:49:37.797029 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.796962    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26cac444-abfa-4060-8f52-4e5756866f3f-original-pull-secret\") pod \"global-pull-secret-syncer-fs4n6\" (UID: \"26cac444-abfa-4060-8f52-4e5756866f3f\") " pod="kube-system/global-pull-secret-syncer-fs4n6"
Sep 09 05:49:37.797029 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.796993    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b179b936-0cca-4a9e-86ce-775441316808-metrics-certs\") pod \"network-metrics-daemon-s7r5d\" (UID: \"b179b936-0cca-4a9e-86ce-775441316808\") " pod="openshift-multus/network-metrics-daemon-s7r5d"
Sep 09 05:49:37.797181 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:37.797091    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 05:49:37.797181 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:37.797098    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 05:49:37.797181 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:37.797139    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b179b936-0cca-4a9e-86ce-775441316808-metrics-certs podName:b179b936-0cca-4a9e-86ce-775441316808 nodeName:}" failed. No retries permitted until 2026-09-09 05:49:38.797126441 +0000 UTC m=+4.137614327 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b179b936-0cca-4a9e-86ce-775441316808-metrics-certs") pod "network-metrics-daemon-s7r5d" (UID: "b179b936-0cca-4a9e-86ce-775441316808") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 05:49:37.797181 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:37.797153    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26cac444-abfa-4060-8f52-4e5756866f3f-original-pull-secret podName:26cac444-abfa-4060-8f52-4e5756866f3f nodeName:}" failed. No retries permitted until 2026-09-09 05:49:38.797146744 +0000 UTC m=+4.137634631 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/26cac444-abfa-4060-8f52-4e5756866f3f-original-pull-secret") pod "global-pull-secret-syncer-fs4n6" (UID: "26cac444-abfa-4060-8f52-4e5756866f3f") : object "kube-system"/"original-pull-secret" not registered
Sep 09 05:49:37.897995 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:37.897964    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqt5c\" (UniqueName: \"kubernetes.io/projected/5459053f-e126-4327-a1e2-7676a5cc107b-kube-api-access-rqt5c\") pod \"network-check-target-b78pk\" (UID: \"5459053f-e126-4327-a1e2-7676a5cc107b\") " pod="openshift-network-diagnostics/network-check-target-b78pk"
Sep 09 05:49:37.898158 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:37.898149    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 05:49:37.898215 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:37.898164    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 05:49:37.898215 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:37.898173    2579 projected.go:196] Error preparing data for projected volume kube-api-access-rqt5c for pod openshift-network-diagnostics/network-check-target-b78pk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 05:49:37.898323 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:37.898226    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5459053f-e126-4327-a1e2-7676a5cc107b-kube-api-access-rqt5c podName:5459053f-e126-4327-a1e2-7676a5cc107b nodeName:}" failed. No retries permitted until 2026-09-09 05:49:38.898209622 +0000 UTC m=+4.238697528 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rqt5c" (UniqueName: "kubernetes.io/projected/5459053f-e126-4327-a1e2-7676a5cc107b-kube-api-access-rqt5c") pod "network-check-target-b78pk" (UID: "5459053f-e126-4327-a1e2-7676a5cc107b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 05:49:38.102999 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:38.102974    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd282f88_a442_4ad3_931a_18e86703ece9.slice/crio-6faabf474953d304889842c96a632cfe44f645262e00b25a9bf62835f000cf52 WatchSource:0}: Error finding container 6faabf474953d304889842c96a632cfe44f645262e00b25a9bf62835f000cf52: Status 404 returned error can't find the container with id 6faabf474953d304889842c96a632cfe44f645262e00b25a9bf62835f000cf52
Sep 09 05:49:38.104808 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:38.104719    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6d0d7b7_703b_439d_9aae_064c217b875a.slice/crio-0d0ac0518c50fc7c9e4266f1d2e7160d239960148ed59898a019bcb4cf9bb27e WatchSource:0}: Error finding container 0d0ac0518c50fc7c9e4266f1d2e7160d239960148ed59898a019bcb4cf9bb27e: Status 404 returned error can't find the container with id 0d0ac0518c50fc7c9e4266f1d2e7160d239960148ed59898a019bcb4cf9bb27e
Sep 09 05:49:38.105366 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:38.105344    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6388487_c897_41c8_b8db_30a860ad66e3.slice/crio-924b12f9dd07847e59b3cbb8dc3e0b4f2152a98ba80da378a4a751d9431a85de WatchSource:0}: Error finding container 924b12f9dd07847e59b3cbb8dc3e0b4f2152a98ba80da378a4a751d9431a85de: Status 404 returned error can't find the container with id 924b12f9dd07847e59b3cbb8dc3e0b4f2152a98ba80da378a4a751d9431a85de
Sep 09 05:49:38.106362 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:38.106337    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9864f736_2e71_4723_bdb5_a54abeedecbb.slice/crio-49a60587cfe332396a227f0f8f0ea5c2721aa886642ee95cb44b1b8c8c1bce70 WatchSource:0}: Error finding container 49a60587cfe332396a227f0f8f0ea5c2721aa886642ee95cb44b1b8c8c1bce70: Status 404 returned error can't find the container with id 49a60587cfe332396a227f0f8f0ea5c2721aa886642ee95cb44b1b8c8c1bce70
Sep 09 05:49:38.107285 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:38.107156    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbab69e9_3123_418a_92e7_88e2fb389fb5.slice/crio-d1a153d2f6b71aac34a48ccc297bd36738d3345b690694ee60139714f4b0e678 WatchSource:0}: Error finding container d1a153d2f6b71aac34a48ccc297bd36738d3345b690694ee60139714f4b0e678: Status 404 returned error can't find the container with id d1a153d2f6b71aac34a48ccc297bd36738d3345b690694ee60139714f4b0e678
Sep 09 05:49:38.108207 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:38.108082    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08607c27_ea7f_4487_9e8c_d84f000464d7.slice/crio-1a3b5249e45b48e18c58eb2aff44b9f59dcd09af1aebb1b3b0b423cf469b6a1b WatchSource:0}: Error finding container 1a3b5249e45b48e18c58eb2aff44b9f59dcd09af1aebb1b3b0b423cf469b6a1b: Status 404 returned error can't find the container with id 1a3b5249e45b48e18c58eb2aff44b9f59dcd09af1aebb1b3b0b423cf469b6a1b
Sep 09 05:49:38.109490 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:38.109469    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5f162ab_38ab_4515_81a3_f46c8f2c0e4c.slice/crio-c5734cb73f226d4e0c6848574639bd1e907be97042a2ab6438fe0e2d958f2d63 WatchSource:0}: Error finding container c5734cb73f226d4e0c6848574639bd1e907be97042a2ab6438fe0e2d958f2d63: Status 404 returned error can't find the container with id c5734cb73f226d4e0c6848574639bd1e907be97042a2ab6438fe0e2d958f2d63
Sep 09 05:49:38.120491 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:38.120462    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 05:44:37 +0000 UTC" deadline="2028-05-31 18:21:19.508298192 +0000 UTC"
Sep 09 05:49:38.120491 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:38.120489    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15132h31m41.387813025s"
Sep 09 05:49:38.137949 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:38.137920    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48323112_dc28_4bd0_825d_9fa9e43304c0.slice/crio-ccc6b338628bf0b08d6271e743bcccebf8de7513d573e3e6e6be8fdf2bd42943 WatchSource:0}: Error finding container ccc6b338628bf0b08d6271e743bcccebf8de7513d573e3e6e6be8fdf2bd42943: Status 404 returned error can't find the container with id ccc6b338628bf0b08d6271e743bcccebf8de7513d573e3e6e6be8fdf2bd42943
Sep 09 05:49:38.138688 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:49:38.138661    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7596b80_c365_41fb_97e8_e99cbad3e412.slice/crio-e55a81205e8778a6ffac45498aea724497b4e75afc0e7d95a129633ab2360a9d WatchSource:0}: Error finding container e55a81205e8778a6ffac45498aea724497b4e75afc0e7d95a129633ab2360a9d: Status 404 returned error can't find the container with id e55a81205e8778a6ffac45498aea724497b4e75afc0e7d95a129633ab2360a9d
Sep 09 05:49:38.155018 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:38.154991    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fv99k" event={"ID":"b5f162ab-38ab-4515-81a3-f46c8f2c0e4c","Type":"ContainerStarted","Data":"c5734cb73f226d4e0c6848574639bd1e907be97042a2ab6438fe0e2d958f2d63"}
Sep 09 05:49:38.157373 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:38.157310    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4pdb" event={"ID":"08607c27-ea7f-4487-9e8c-d84f000464d7","Type":"ContainerStarted","Data":"1a3b5249e45b48e18c58eb2aff44b9f59dcd09af1aebb1b3b0b423cf469b6a1b"}
Sep 09 05:49:38.160363 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:38.160327    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r7gnm" event={"ID":"f6388487-c897-41c8-b8db-30a860ad66e3","Type":"ContainerStarted","Data":"924b12f9dd07847e59b3cbb8dc3e0b4f2152a98ba80da378a4a751d9431a85de"}
Sep 09 05:49:38.162968 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:38.162949    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-brqzd" event={"ID":"cd282f88-a442-4ad3-931a-18e86703ece9","Type":"ContainerStarted","Data":"6faabf474953d304889842c96a632cfe44f645262e00b25a9bf62835f000cf52"}
Sep 09 05:49:38.165278 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:38.165256    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7lkv" event={"ID":"48323112-dc28-4bd0-825d-9fa9e43304c0","Type":"ContainerStarted","Data":"ccc6b338628bf0b08d6271e743bcccebf8de7513d573e3e6e6be8fdf2bd42943"}
Sep 09 05:49:38.166433 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:38.166409    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6z9q7" event={"ID":"bbab69e9-3123-418a-92e7-88e2fb389fb5","Type":"ContainerStarted","Data":"d1a153d2f6b71aac34a48ccc297bd36738d3345b690694ee60139714f4b0e678"}
Sep 09 05:49:38.167728 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:38.167703    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4r2gq" event={"ID":"9864f736-2e71-4723-bdb5-a54abeedecbb","Type":"ContainerStarted","Data":"49a60587cfe332396a227f0f8f0ea5c2721aa886642ee95cb44b1b8c8c1bce70"}
Sep 09 05:49:38.168853 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:38.168830    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-b9lbm" event={"ID":"d6d0d7b7-703b-439d-9aae-064c217b875a","Type":"ContainerStarted","Data":"0d0ac0518c50fc7c9e4266f1d2e7160d239960148ed59898a019bcb4cf9bb27e"}
Sep 09 05:49:38.169990 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:38.169953    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7" event={"ID":"f7596b80-c365-41fb-97e8-e99cbad3e412","Type":"ContainerStarted","Data":"e55a81205e8778a6ffac45498aea724497b4e75afc0e7d95a129633ab2360a9d"}
Sep 09 05:49:38.806517 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:38.805762    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26cac444-abfa-4060-8f52-4e5756866f3f-original-pull-secret\") pod \"global-pull-secret-syncer-fs4n6\" (UID: \"26cac444-abfa-4060-8f52-4e5756866f3f\") " pod="kube-system/global-pull-secret-syncer-fs4n6"
Sep 09 05:49:38.806517 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:38.805816    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b179b936-0cca-4a9e-86ce-775441316808-metrics-certs\") pod \"network-metrics-daemon-s7r5d\" (UID: \"b179b936-0cca-4a9e-86ce-775441316808\") " pod="openshift-multus/network-metrics-daemon-s7r5d"
Sep 09 05:49:38.806517 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:38.805993    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 05:49:38.806517 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:38.806054    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b179b936-0cca-4a9e-86ce-775441316808-metrics-certs podName:b179b936-0cca-4a9e-86ce-775441316808 nodeName:}" failed. No retries permitted until 2026-09-09 05:49:40.806034865 +0000 UTC m=+6.146522757 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b179b936-0cca-4a9e-86ce-775441316808-metrics-certs") pod "network-metrics-daemon-s7r5d" (UID: "b179b936-0cca-4a9e-86ce-775441316808") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 05:49:38.806517 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:38.806422    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 05:49:38.806517 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:38.806468    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26cac444-abfa-4060-8f52-4e5756866f3f-original-pull-secret podName:26cac444-abfa-4060-8f52-4e5756866f3f nodeName:}" failed. No retries permitted until 2026-09-09 05:49:40.806452205 +0000 UTC m=+6.146940108 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/26cac444-abfa-4060-8f52-4e5756866f3f-original-pull-secret") pod "global-pull-secret-syncer-fs4n6" (UID: "26cac444-abfa-4060-8f52-4e5756866f3f") : object "kube-system"/"original-pull-secret" not registered
Sep 09 05:49:38.906495 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:38.906459    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqt5c\" (UniqueName: \"kubernetes.io/projected/5459053f-e126-4327-a1e2-7676a5cc107b-kube-api-access-rqt5c\") pod \"network-check-target-b78pk\" (UID: \"5459053f-e126-4327-a1e2-7676a5cc107b\") " pod="openshift-network-diagnostics/network-check-target-b78pk"
Sep 09 05:49:38.906694 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:38.906665    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 05:49:38.906694 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:38.906688    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 05:49:38.906823 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:38.906701    2579 projected.go:196] Error preparing data for projected volume kube-api-access-rqt5c for pod openshift-network-diagnostics/network-check-target-b78pk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 05:49:38.906823 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:38.906776    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5459053f-e126-4327-a1e2-7676a5cc107b-kube-api-access-rqt5c podName:5459053f-e126-4327-a1e2-7676a5cc107b nodeName:}" failed. No retries permitted until 2026-09-09 05:49:40.906758178 +0000 UTC m=+6.247246068 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rqt5c" (UniqueName: "kubernetes.io/projected/5459053f-e126-4327-a1e2-7676a5cc107b-kube-api-access-rqt5c") pod "network-check-target-b78pk" (UID: "5459053f-e126-4327-a1e2-7676a5cc107b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 05:49:39.120987 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:39.120903    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 05:44:37 +0000 UTC" deadline="2028-03-14 04:57:12.274789182 +0000 UTC"
Sep 09 05:49:39.120987 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:39.120986    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13247h7m33.153807532s"
Sep 09 05:49:39.143709 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:39.142729    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fs4n6"
Sep 09 05:49:39.143709 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:39.142872    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fs4n6" podUID="26cac444-abfa-4060-8f52-4e5756866f3f"
Sep 09 05:49:39.143709 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:39.143328    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7r5d"
Sep 09 05:49:39.143709 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:39.143433    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7r5d" podUID="b179b936-0cca-4a9e-86ce-775441316808"
Sep 09 05:49:39.143709 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:39.143502    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b78pk"
Sep 09 05:49:39.143709 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:39.143668    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b78pk" podUID="5459053f-e126-4327-a1e2-7676a5cc107b"
Sep 09 05:49:39.190308 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:39.190241    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-142.ec2.internal" event={"ID":"58007dd99525496ea22ff71e4a80cd63","Type":"ContainerStarted","Data":"1042014eb0937de698cf8024ad84b35583d96de9312225b128cf06b5986b8a49"}
Sep 09 05:49:39.193147 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:39.193118    2579 generic.go:356] "Generic (PLEG): container finished" podID="59ee296aad7c4e56d27fc66bd1b97d18" containerID="78297f682722f15dfa2be0c5926115d608f293b0c2930b27dd2c3882e27a0b45" exitCode=0
Sep 09 05:49:39.193256 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:39.193161    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-142.ec2.internal" event={"ID":"59ee296aad7c4e56d27fc66bd1b97d18","Type":"ContainerDied","Data":"78297f682722f15dfa2be0c5926115d608f293b0c2930b27dd2c3882e27a0b45"}
Sep 09 05:49:39.225047 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:39.224143    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-142.ec2.internal" podStartSLOduration=3.224125121 podStartE2EDuration="3.224125121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 05:49:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:49:39.206357805 +0000 UTC m=+4.546845715" watchObservedRunningTime="2026-09-09 05:49:39.224125121 +0000 UTC m=+4.564613033"
Sep 09 05:49:40.207580 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:40.207546    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-142.ec2.internal" event={"ID":"59ee296aad7c4e56d27fc66bd1b97d18","Type":"ContainerStarted","Data":"935c106fcef6e9300bfc1b9f965ccf2592decbcd691d23fca8415034f1637aee"}
Sep 09 05:49:40.820939 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:40.820902    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26cac444-abfa-4060-8f52-4e5756866f3f-original-pull-secret\") pod \"global-pull-secret-syncer-fs4n6\" (UID: \"26cac444-abfa-4060-8f52-4e5756866f3f\") " pod="kube-system/global-pull-secret-syncer-fs4n6"
Sep 09 05:49:40.821117 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:40.820956    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b179b936-0cca-4a9e-86ce-775441316808-metrics-certs\") pod \"network-metrics-daemon-s7r5d\" (UID: \"b179b936-0cca-4a9e-86ce-775441316808\") " pod="openshift-multus/network-metrics-daemon-s7r5d"
Sep 09 05:49:40.821117 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:40.821070    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 05:49:40.821117 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:40.821072    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 05:49:40.821273 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:40.821145    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b179b936-0cca-4a9e-86ce-775441316808-metrics-certs podName:b179b936-0cca-4a9e-86ce-775441316808 nodeName:}" failed. No retries permitted until 2026-09-09 05:49:44.821125809 +0000 UTC m=+10.161613702 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b179b936-0cca-4a9e-86ce-775441316808-metrics-certs") pod "network-metrics-daemon-s7r5d" (UID: "b179b936-0cca-4a9e-86ce-775441316808") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 05:49:40.821273 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:40.821167    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26cac444-abfa-4060-8f52-4e5756866f3f-original-pull-secret podName:26cac444-abfa-4060-8f52-4e5756866f3f nodeName:}" failed. No retries permitted until 2026-09-09 05:49:44.821155135 +0000 UTC m=+10.161643027 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/26cac444-abfa-4060-8f52-4e5756866f3f-original-pull-secret") pod "global-pull-secret-syncer-fs4n6" (UID: "26cac444-abfa-4060-8f52-4e5756866f3f") : object "kube-system"/"original-pull-secret" not registered
Sep 09 05:49:40.921818 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:40.921775    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqt5c\" (UniqueName: \"kubernetes.io/projected/5459053f-e126-4327-a1e2-7676a5cc107b-kube-api-access-rqt5c\") pod \"network-check-target-b78pk\" (UID: \"5459053f-e126-4327-a1e2-7676a5cc107b\") " pod="openshift-network-diagnostics/network-check-target-b78pk"
Sep 09 05:49:40.921991 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:40.921936    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 05:49:40.921991 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:40.921954    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 05:49:40.921991 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:40.921967    2579 projected.go:196] Error preparing data for projected volume kube-api-access-rqt5c for pod openshift-network-diagnostics/network-check-target-b78pk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 05:49:40.922142 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:40.922023    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5459053f-e126-4327-a1e2-7676a5cc107b-kube-api-access-rqt5c podName:5459053f-e126-4327-a1e2-7676a5cc107b nodeName:}" failed. No retries permitted until 2026-09-09 05:49:44.922005849 +0000 UTC m=+10.262493743 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rqt5c" (UniqueName: "kubernetes.io/projected/5459053f-e126-4327-a1e2-7676a5cc107b-kube-api-access-rqt5c") pod "network-check-target-b78pk" (UID: "5459053f-e126-4327-a1e2-7676a5cc107b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 05:49:41.144510 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:41.144406    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fs4n6"
Sep 09 05:49:41.144708 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:41.144529    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fs4n6" podUID="26cac444-abfa-4060-8f52-4e5756866f3f"
Sep 09 05:49:41.144708 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:41.144688    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b78pk"
Sep 09 05:49:41.144832 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:41.144770    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b78pk" podUID="5459053f-e126-4327-a1e2-7676a5cc107b"
Sep 09 05:49:41.144832 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:41.144815    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7r5d"
Sep 09 05:49:41.144927 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:41.144888    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7r5d" podUID="b179b936-0cca-4a9e-86ce-775441316808"
Sep 09 05:49:43.143449 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:43.142739    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fs4n6"
Sep 09 05:49:43.143449 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:43.142873    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fs4n6" podUID="26cac444-abfa-4060-8f52-4e5756866f3f"
Sep 09 05:49:43.143449 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:43.143285    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7r5d"
Sep 09 05:49:43.143449 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:43.143390    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7r5d" podUID="b179b936-0cca-4a9e-86ce-775441316808"
Sep 09 05:49:43.144082 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:43.143554    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b78pk"
Sep 09 05:49:43.144082 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:43.143665    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b78pk" podUID="5459053f-e126-4327-a1e2-7676a5cc107b"
Sep 09 05:49:44.854347 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:44.854314    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26cac444-abfa-4060-8f52-4e5756866f3f-original-pull-secret\") pod \"global-pull-secret-syncer-fs4n6\" (UID: \"26cac444-abfa-4060-8f52-4e5756866f3f\") " pod="kube-system/global-pull-secret-syncer-fs4n6"
Sep 09 05:49:44.854772 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:44.854368    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b179b936-0cca-4a9e-86ce-775441316808-metrics-certs\") pod \"network-metrics-daemon-s7r5d\" (UID: \"b179b936-0cca-4a9e-86ce-775441316808\") " pod="openshift-multus/network-metrics-daemon-s7r5d"
Sep 09 05:49:44.854772 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:44.854502    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 05:49:44.854772 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:44.854558    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b179b936-0cca-4a9e-86ce-775441316808-metrics-certs podName:b179b936-0cca-4a9e-86ce-775441316808 nodeName:}" failed. No retries permitted until 2026-09-09 05:49:52.854539403 +0000 UTC m=+18.195027330 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b179b936-0cca-4a9e-86ce-775441316808-metrics-certs") pod "network-metrics-daemon-s7r5d" (UID: "b179b936-0cca-4a9e-86ce-775441316808") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 05:49:44.855013 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:44.854995    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 05:49:44.855072 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:44.855039    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26cac444-abfa-4060-8f52-4e5756866f3f-original-pull-secret podName:26cac444-abfa-4060-8f52-4e5756866f3f nodeName:}" failed. No retries permitted until 2026-09-09 05:49:52.855029335 +0000 UTC m=+18.195517221 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/26cac444-abfa-4060-8f52-4e5756866f3f-original-pull-secret") pod "global-pull-secret-syncer-fs4n6" (UID: "26cac444-abfa-4060-8f52-4e5756866f3f") : object "kube-system"/"original-pull-secret" not registered
Sep 09 05:49:44.955469 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:44.954911    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqt5c\" (UniqueName: \"kubernetes.io/projected/5459053f-e126-4327-a1e2-7676a5cc107b-kube-api-access-rqt5c\") pod \"network-check-target-b78pk\" (UID: \"5459053f-e126-4327-a1e2-7676a5cc107b\") " pod="openshift-network-diagnostics/network-check-target-b78pk"
Sep 09 05:49:44.955469 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:44.955077    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 05:49:44.955469 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:44.955096    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 05:49:44.955469 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:44.955110    2579 projected.go:196] Error preparing data for projected volume kube-api-access-rqt5c for pod openshift-network-diagnostics/network-check-target-b78pk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 05:49:44.955469 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:44.955172    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5459053f-e126-4327-a1e2-7676a5cc107b-kube-api-access-rqt5c podName:5459053f-e126-4327-a1e2-7676a5cc107b nodeName:}" failed. No retries permitted until 2026-09-09 05:49:52.955154784 +0000 UTC m=+18.295642676 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rqt5c" (UniqueName: "kubernetes.io/projected/5459053f-e126-4327-a1e2-7676a5cc107b-kube-api-access-rqt5c") pod "network-check-target-b78pk" (UID: "5459053f-e126-4327-a1e2-7676a5cc107b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 05:49:45.143936 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:45.143696    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fs4n6"
Sep 09 05:49:45.144071 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:45.143799    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7r5d"
Sep 09 05:49:45.144071 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:45.143975    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fs4n6" podUID="26cac444-abfa-4060-8f52-4e5756866f3f"
Sep 09 05:49:45.144167 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:45.144090    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7r5d" podUID="b179b936-0cca-4a9e-86ce-775441316808"
Sep 09 05:49:45.144167 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:45.143822    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b78pk"
Sep 09 05:49:45.144275 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:45.144176    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b78pk" podUID="5459053f-e126-4327-a1e2-7676a5cc107b"
Sep 09 05:49:47.143173 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:47.143138    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7r5d"
Sep 09 05:49:47.143535 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:47.143138    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fs4n6"
Sep 09 05:49:47.143535 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:47.143275    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7r5d" podUID="b179b936-0cca-4a9e-86ce-775441316808"
Sep 09 05:49:47.143535 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:47.143291    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b78pk"
Sep 09 05:49:47.143535 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:47.143411    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fs4n6" podUID="26cac444-abfa-4060-8f52-4e5756866f3f"
Sep 09 05:49:47.143535 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:47.143476    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b78pk" podUID="5459053f-e126-4327-a1e2-7676a5cc107b"
Sep 09 05:49:49.142732 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:49.142696    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fs4n6"
Sep 09 05:49:49.143174 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:49.142696    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b78pk"
Sep 09 05:49:49.143174 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:49.142831    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fs4n6" podUID="26cac444-abfa-4060-8f52-4e5756866f3f"
Sep 09 05:49:49.143174 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:49.142913    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7r5d"
Sep 09 05:49:49.143174 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:49.142908    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b78pk" podUID="5459053f-e126-4327-a1e2-7676a5cc107b"
Sep 09 05:49:49.143174 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:49.143005    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7r5d" podUID="b179b936-0cca-4a9e-86ce-775441316808"
Sep 09 05:49:51.144020 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:51.143749    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fs4n6"
Sep 09 05:49:51.144020 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:51.143762    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7r5d"
Sep 09 05:49:51.144020 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:51.143878    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fs4n6" podUID="26cac444-abfa-4060-8f52-4e5756866f3f"
Sep 09 05:49:51.144912 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:51.144871    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7r5d" podUID="b179b936-0cca-4a9e-86ce-775441316808"
Sep 09 05:49:51.145029 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:51.145003    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b78pk"
Sep 09 05:49:51.145297 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:51.145273    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b78pk" podUID="5459053f-e126-4327-a1e2-7676a5cc107b"
Sep 09 05:49:52.915143 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:52.915107    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26cac444-abfa-4060-8f52-4e5756866f3f-original-pull-secret\") pod \"global-pull-secret-syncer-fs4n6\" (UID: \"26cac444-abfa-4060-8f52-4e5756866f3f\") " pod="kube-system/global-pull-secret-syncer-fs4n6"
Sep 09 05:49:52.915143 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:52.915150    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b179b936-0cca-4a9e-86ce-775441316808-metrics-certs\") pod \"network-metrics-daemon-s7r5d\" (UID: \"b179b936-0cca-4a9e-86ce-775441316808\") " pod="openshift-multus/network-metrics-daemon-s7r5d"
Sep 09 05:49:52.915739 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:52.915279    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 05:49:52.915739 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:52.915282    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 05:49:52.915739 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:52.915335    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b179b936-0cca-4a9e-86ce-775441316808-metrics-certs podName:b179b936-0cca-4a9e-86ce-775441316808 nodeName:}" failed. No retries permitted until 2026-09-09 05:50:08.915316197 +0000 UTC m=+34.255804119 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b179b936-0cca-4a9e-86ce-775441316808-metrics-certs") pod "network-metrics-daemon-s7r5d" (UID: "b179b936-0cca-4a9e-86ce-775441316808") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 05:49:52.915739 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:52.915356    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26cac444-abfa-4060-8f52-4e5756866f3f-original-pull-secret podName:26cac444-abfa-4060-8f52-4e5756866f3f nodeName:}" failed. No retries permitted until 2026-09-09 05:50:08.915345052 +0000 UTC m=+34.255832942 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/26cac444-abfa-4060-8f52-4e5756866f3f-original-pull-secret") pod "global-pull-secret-syncer-fs4n6" (UID: "26cac444-abfa-4060-8f52-4e5756866f3f") : object "kube-system"/"original-pull-secret" not registered
Sep 09 05:49:53.016382 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:53.016348    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqt5c\" (UniqueName: \"kubernetes.io/projected/5459053f-e126-4327-a1e2-7676a5cc107b-kube-api-access-rqt5c\") pod \"network-check-target-b78pk\" (UID: \"5459053f-e126-4327-a1e2-7676a5cc107b\") " pod="openshift-network-diagnostics/network-check-target-b78pk"
Sep 09 05:49:53.016575 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:53.016530    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 05:49:53.016575 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:53.016556    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 05:49:53.016575 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:53.016570    2579 projected.go:196] Error preparing data for projected volume kube-api-access-rqt5c for pod openshift-network-diagnostics/network-check-target-b78pk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 05:49:53.016763 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:53.016640    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5459053f-e126-4327-a1e2-7676a5cc107b-kube-api-access-rqt5c podName:5459053f-e126-4327-a1e2-7676a5cc107b nodeName:}" failed. No retries permitted until 2026-09-09 05:50:09.016611997 +0000 UTC m=+34.357099884 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rqt5c" (UniqueName: "kubernetes.io/projected/5459053f-e126-4327-a1e2-7676a5cc107b-kube-api-access-rqt5c") pod "network-check-target-b78pk" (UID: "5459053f-e126-4327-a1e2-7676a5cc107b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 05:49:53.142644 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:53.142601    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fs4n6"
Sep 09 05:49:53.142814 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:53.142657    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b78pk"
Sep 09 05:49:53.142814 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:53.142606    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7r5d"
Sep 09 05:49:53.142814 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:53.142744    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fs4n6" podUID="26cac444-abfa-4060-8f52-4e5756866f3f"
Sep 09 05:49:53.142814 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:53.142804    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7r5d" podUID="b179b936-0cca-4a9e-86ce-775441316808"
Sep 09 05:49:53.143012 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:53.142868    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b78pk" podUID="5459053f-e126-4327-a1e2-7676a5cc107b"
Sep 09 05:49:55.143022 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:55.142993    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7r5d"
Sep 09 05:49:55.143481 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:55.143104    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7r5d" podUID="b179b936-0cca-4a9e-86ce-775441316808"
Sep 09 05:49:55.143481 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:55.143187    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b78pk"
Sep 09 05:49:55.143481 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:55.143275    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b78pk" podUID="5459053f-e126-4327-a1e2-7676a5cc107b"
Sep 09 05:49:55.143481 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:55.143319    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fs4n6"
Sep 09 05:49:55.143481 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:55.143389    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fs4n6" podUID="26cac444-abfa-4060-8f52-4e5756866f3f"
Sep 09 05:49:56.247961 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:56.247934    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fv99k" event={"ID":"b5f162ab-38ab-4515-81a3-f46c8f2c0e4c","Type":"ContainerStarted","Data":"cb68947e985c68df6a7232c07d06d1e2bec5122405344f0bbabfac811c23f5a2"}
Sep 09 05:49:56.249713 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:56.249682    2579 generic.go:356] "Generic (PLEG): container finished" podID="08607c27-ea7f-4487-9e8c-d84f000464d7" containerID="091525fcee691960c81d928b8aa4cf089be72eb1e47afa4ad434a4e6d4fae17e" exitCode=0
Sep 09 05:49:56.249826 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:56.249758    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4pdb" event={"ID":"08607c27-ea7f-4487-9e8c-d84f000464d7","Type":"ContainerDied","Data":"091525fcee691960c81d928b8aa4cf089be72eb1e47afa4ad434a4e6d4fae17e"}
Sep 09 05:49:56.252459 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:56.252398    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-brqzd" event={"ID":"cd282f88-a442-4ad3-931a-18e86703ece9","Type":"ContainerStarted","Data":"19c8e82afb015ca039fb4b94d2526f7de762eb259fbe165ee606badcc5fa2c20"}
Sep 09 05:49:56.254172 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:56.254112    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7lkv" event={"ID":"48323112-dc28-4bd0-825d-9fa9e43304c0","Type":"ContainerStarted","Data":"4476f97fe79fdcdf02917be3b7fc3731949161f78b5aa94c273afa245cf0f421"}
Sep 09 05:49:56.256020 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:56.255992    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6z9q7" event={"ID":"bbab69e9-3123-418a-92e7-88e2fb389fb5","Type":"ContainerStarted","Data":"b23294cd544ed9296ce30918cf2a77553a0d78ca9cd2beff34c91eb9c02a9f5d"}
Sep 09 05:49:56.257668 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:56.257649    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4r2gq" event={"ID":"9864f736-2e71-4723-bdb5-a54abeedecbb","Type":"ContainerStarted","Data":"b8993608977b991afc74b1b501f9164f77b52528929e6eae7248fa8035b3145e"}
Sep 09 05:49:56.259287 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:56.259271    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-b9lbm" event={"ID":"d6d0d7b7-703b-439d-9aae-064c217b875a","Type":"ContainerStarted","Data":"db1bd5ff5a84251c287ff847f6aa61ff6f91d435dc7a49400bfaf2fab86cc009"}
Sep 09 05:49:56.261186 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:56.261167    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7" event={"ID":"f7596b80-c365-41fb-97e8-e99cbad3e412","Type":"ContainerStarted","Data":"6e9eb287525bc446bbeb855777a2583c76302159335d6772b1ee59534684cbd8"}
Sep 09 05:49:56.261237 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:56.261221    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7" event={"ID":"f7596b80-c365-41fb-97e8-e99cbad3e412","Type":"ContainerStarted","Data":"ccba46670bc31a3aae17fe3fe49b2e4535622f81de086c9eedae1cfc14d4cdcb"}
Sep 09 05:49:56.262778 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:56.262742    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-142.ec2.internal" podStartSLOduration=20.262731867 podStartE2EDuration="20.262731867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 05:49:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:49:40.223029216 +0000 UTC m=+5.563517129" watchObservedRunningTime="2026-09-09 05:49:56.262731867 +0000 UTC m=+21.603219778"
Sep 09 05:49:56.262889 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:56.262871    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-fv99k" podStartSLOduration=7.369556612 podStartE2EDuration="21.262866383s" totalImagesPullingTime="13.893309771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:49:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:49:56.262862424 +0000 UTC m=+21.603350323" watchObservedRunningTime="2026-09-09 05:49:56.262866383 +0000 UTC m=+21.603354288"
Sep 09 05:49:56.276261 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:56.276223    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-brqzd" podStartSLOduration=3.423814083 podStartE2EDuration="21.276211729s" totalImagesPullingTime="17.852397646s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 05:49:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:49:56.27620676 +0000 UTC m=+21.616694668" watchObservedRunningTime="2026-09-09 05:49:56.276211729 +0000 UTC m=+21.616699638"
Sep 09 05:49:56.291909 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:56.291864    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6z9q7" podStartSLOduration=3.412921968 podStartE2EDuration="21.29184598s" totalImagesPullingTime="17.878924012s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 05:49:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:49:56.291670754 +0000 UTC m=+21.632158659" watchObservedRunningTime="2026-09-09 05:49:56.29184598 +0000 UTC m=+21.632333890"
Sep 09 05:49:56.324808 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:56.324767    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-b9lbm" podStartSLOduration=3.473756703 podStartE2EDuration="21.32475245s" totalImagesPullingTime="17.850995747s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 05:49:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:49:56.324509953 +0000 UTC m=+21.664997863" watchObservedRunningTime="2026-09-09 05:49:56.32475245 +0000 UTC m=+21.665240359"
Sep 09 05:49:56.341721 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:56.340987    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4r2gq" podStartSLOduration=3.490658478 podStartE2EDuration="21.340950862s" totalImagesPullingTime="17.850292384s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 05:49:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:49:56.340659161 +0000 UTC m=+21.681147072" watchObservedRunningTime="2026-09-09 05:49:56.340950862 +0000 UTC m=+21.681438772"
Sep 09 05:49:57.142952 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:57.142921    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7r5d"
Sep 09 05:49:57.143152 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:57.143072    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7r5d" podUID="b179b936-0cca-4a9e-86ce-775441316808"
Sep 09 05:49:57.143513 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:57.143493    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b78pk"
Sep 09 05:49:57.143650 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:57.143610    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b78pk" podUID="5459053f-e126-4327-a1e2-7676a5cc107b"
Sep 09 05:49:57.143746 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:57.143732    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fs4n6"
Sep 09 05:49:57.143826 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:57.143810    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fs4n6" podUID="26cac444-abfa-4060-8f52-4e5756866f3f"
Sep 09 05:49:57.267196 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:57.267129    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7" event={"ID":"f7596b80-c365-41fb-97e8-e99cbad3e412","Type":"ContainerStarted","Data":"8bfeb5c562db49e98c8a5a121f07b0a391532e7582da270aca2c0b6110dfe75a"}
Sep 09 05:49:57.267678 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:57.267207    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7" event={"ID":"f7596b80-c365-41fb-97e8-e99cbad3e412","Type":"ContainerStarted","Data":"d4db9b39d39c6cf917b48f001ad25738bae56f525eb59e7c312e76a343d899b5"}
Sep 09 05:49:57.267678 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:57.267238    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7" event={"ID":"f7596b80-c365-41fb-97e8-e99cbad3e412","Type":"ContainerStarted","Data":"17edfe7fa1e1ae34b816b49b1ee99874127790f3d72e82c50a8dfd6afe51e172"}
Sep 09 05:49:57.267678 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:57.267275    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7" event={"ID":"f7596b80-c365-41fb-97e8-e99cbad3e412","Type":"ContainerStarted","Data":"d2aa911ab57a3c8733f2439da60f2e1143a0dbc1082461c187a26d6feeab09e0"}
Sep 09 05:49:57.414910 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:57.414684    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-fv99k"
Sep 09 05:49:57.415528 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:57.415505    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-fv99k"
Sep 09 05:49:57.499399 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:57.499378    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 05:49:58.147049 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:58.146932    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T05:49:57.499395143Z","UUID":"a28d8cdc-fe67-4ca3-b561-91232acdb545","Handler":null,"Name":"","Endpoint":""}
Sep 09 05:49:58.148919 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:58.148705    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 05:49:58.148919 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:58.148738    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 05:49:58.275754 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:58.275722    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r7gnm" event={"ID":"f6388487-c897-41c8-b8db-30a860ad66e3","Type":"ContainerStarted","Data":"323d4cf8b3c95dbb50a71492c90f2958e3409f7bd6d4d052aa2ba8ff395a0783"}
Sep 09 05:49:58.278120 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:58.278091    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7lkv" event={"ID":"48323112-dc28-4bd0-825d-9fa9e43304c0","Type":"ContainerStarted","Data":"b374c351656f40613c3f0661eb485606ec8875057dab8c2783ba2d5d0ab11723"}
Sep 09 05:49:58.278519 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:58.278501    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-fv99k"
Sep 09 05:49:58.278941 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:58.278925    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fv99k"
Sep 09 05:49:58.305398 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:58.305353    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-r7gnm" podStartSLOduration=5.457897463 podStartE2EDuration="23.305336812s" totalImagesPullingTime="17.847439349s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 05:49:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:49:58.291297243 +0000 UTC m=+23.631785154" watchObservedRunningTime="2026-09-09 05:49:58.305336812 +0000 UTC m=+23.645824722"
Sep 09 05:49:59.142946 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:59.142914    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b78pk"
Sep 09 05:49:59.142946 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:59.142932    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fs4n6"
Sep 09 05:49:59.143215 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:59.143078    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b78pk" podUID="5459053f-e126-4327-a1e2-7676a5cc107b"
Sep 09 05:49:59.143403 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:59.143369    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7r5d"
Sep 09 05:49:59.143538 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:59.143477    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fs4n6" podUID="26cac444-abfa-4060-8f52-4e5756866f3f"
Sep 09 05:49:59.143605 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:49:59.143550    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7r5d" podUID="b179b936-0cca-4a9e-86ce-775441316808"
Sep 09 05:49:59.284295 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:59.284215    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7" event={"ID":"f7596b80-c365-41fb-97e8-e99cbad3e412","Type":"ContainerStarted","Data":"7aca41ffc737ec4335b1dab29b1c84639a955db06a720371a702cf6bb99c85cc"}
Sep 09 05:49:59.286585 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:59.286552    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7lkv" event={"ID":"48323112-dc28-4bd0-825d-9fa9e43304c0","Type":"ContainerStarted","Data":"ec606c80ee91121071afb18285d96bb708e410f77046c256d561e56006fb9c4f"}
Sep 09 05:49:59.305187 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:49:59.305147    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7lkv" podStartSLOduration=4.19433158 podStartE2EDuration="24.305133634s" totalImagesPullingTime="20.110802054s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 05:49:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:49:59.304792657 +0000 UTC m=+24.645280567" watchObservedRunningTime="2026-09-09 05:49:59.305133634 +0000 UTC m=+24.645621543"
Sep 09 05:50:01.142614 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:01.142397    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7r5d"
Sep 09 05:50:01.143032 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:01.142397    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fs4n6"
Sep 09 05:50:01.143032 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:01.142397    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b78pk"
Sep 09 05:50:01.143032 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:01.142749    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7r5d" podUID="b179b936-0cca-4a9e-86ce-775441316808"
Sep 09 05:50:01.143032 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:01.142840    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b78pk" podUID="5459053f-e126-4327-a1e2-7676a5cc107b"
Sep 09 05:50:01.143032 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:01.142918    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fs4n6" podUID="26cac444-abfa-4060-8f52-4e5756866f3f"
Sep 09 05:50:01.295254 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:01.295223    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7" event={"ID":"f7596b80-c365-41fb-97e8-e99cbad3e412","Type":"ContainerStarted","Data":"c2f7646ee46ff25021530a8d42b52e968d5c61b3cbe01ff6844a8e9f8ee24a9b"}
Sep 09 05:50:01.295475 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:01.295456    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:50:01.297175 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:01.297152    2579 generic.go:356] "Generic (PLEG): container finished" podID="08607c27-ea7f-4487-9e8c-d84f000464d7" containerID="9008f0b18f6a06037e8520b64e727db23075321a08a68e17a8783c6b387624c4" exitCode=0
Sep 09 05:50:01.297283 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:01.297185    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4pdb" event={"ID":"08607c27-ea7f-4487-9e8c-d84f000464d7","Type":"ContainerDied","Data":"9008f0b18f6a06037e8520b64e727db23075321a08a68e17a8783c6b387624c4"}
Sep 09 05:50:01.315467 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:01.315447    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:50:01.352472 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:01.352419    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7" podStartSLOduration=8.477641849 podStartE2EDuration="26.352406663s" totalImagesPullingTime="17.874764814s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 05:49:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:50:01.352279144 +0000 UTC m=+26.692767054" watchObservedRunningTime="2026-09-09 05:50:01.352406663 +0000 UTC m=+26.692894572"
Sep 09 05:50:02.300182 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:02.300146    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:50:02.300727 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:02.300284    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:50:02.314876 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:02.314852    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:50:02.660751 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:02.660545    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fs4n6"]
Sep 09 05:50:02.660906 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:02.660852    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fs4n6"
Sep 09 05:50:02.660962 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:02.660938    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fs4n6" podUID="26cac444-abfa-4060-8f52-4e5756866f3f"
Sep 09 05:50:02.663886 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:02.663859    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b78pk"]
Sep 09 05:50:02.664038 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:02.663962    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b78pk"
Sep 09 05:50:02.664127 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:02.664067    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b78pk" podUID="5459053f-e126-4327-a1e2-7676a5cc107b"
Sep 09 05:50:02.664686 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:02.664661    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s7r5d"]
Sep 09 05:50:02.664803 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:02.664765    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7r5d"
Sep 09 05:50:02.664870 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:02.664852    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7r5d" podUID="b179b936-0cca-4a9e-86ce-775441316808"
Sep 09 05:50:03.304009 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:03.303975    2579 generic.go:356] "Generic (PLEG): container finished" podID="08607c27-ea7f-4487-9e8c-d84f000464d7" containerID="c0739df935efcb8c03eba0dd43f32811391968da25cbd6d9d52a7274eb6de096" exitCode=0
Sep 09 05:50:03.304601 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:03.304062    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4pdb" event={"ID":"08607c27-ea7f-4487-9e8c-d84f000464d7","Type":"ContainerDied","Data":"c0739df935efcb8c03eba0dd43f32811391968da25cbd6d9d52a7274eb6de096"}
Sep 09 05:50:04.142363 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:04.142329    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fs4n6"
Sep 09 05:50:04.142542 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:04.142330    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7r5d"
Sep 09 05:50:04.142542 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:04.142338    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b78pk"
Sep 09 05:50:04.142663 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:04.142533    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7r5d" podUID="b179b936-0cca-4a9e-86ce-775441316808"
Sep 09 05:50:04.142663 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:04.142418    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fs4n6" podUID="26cac444-abfa-4060-8f52-4e5756866f3f"
Sep 09 05:50:04.142663 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:04.142586    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b78pk" podUID="5459053f-e126-4327-a1e2-7676a5cc107b"
Sep 09 05:50:05.310740 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:05.310704    2579 generic.go:356] "Generic (PLEG): container finished" podID="08607c27-ea7f-4487-9e8c-d84f000464d7" containerID="6331e1014a84ddbd293c61c672cfc57ddf6a0d0950368e59bdef3bac21b77049" exitCode=0
Sep 09 05:50:05.311073 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:05.310768    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4pdb" event={"ID":"08607c27-ea7f-4487-9e8c-d84f000464d7","Type":"ContainerDied","Data":"6331e1014a84ddbd293c61c672cfc57ddf6a0d0950368e59bdef3bac21b77049"}
Sep 09 05:50:06.143046 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:06.143009    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b78pk"
Sep 09 05:50:06.143046 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:06.143043    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fs4n6"
Sep 09 05:50:06.143331 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:06.143138    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b78pk" podUID="5459053f-e126-4327-a1e2-7676a5cc107b"
Sep 09 05:50:06.143331 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:06.143150    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7r5d"
Sep 09 05:50:06.143331 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:06.143251    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fs4n6" podUID="26cac444-abfa-4060-8f52-4e5756866f3f"
Sep 09 05:50:06.143331 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:06.143320    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7r5d" podUID="b179b936-0cca-4a9e-86ce-775441316808"
Sep 09 05:50:08.143132 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:08.143095    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fs4n6"
Sep 09 05:50:08.143132 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:08.143137    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7r5d"
Sep 09 05:50:08.143600 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:08.143225    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fs4n6" podUID="26cac444-abfa-4060-8f52-4e5756866f3f"
Sep 09 05:50:08.143600 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:08.143272    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7r5d" podUID="b179b936-0cca-4a9e-86ce-775441316808"
Sep 09 05:50:08.143600 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:08.143307    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b78pk"
Sep 09 05:50:08.143600 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:08.143393    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b78pk" podUID="5459053f-e126-4327-a1e2-7676a5cc107b"
Sep 09 05:50:08.935433 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:08.935399    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26cac444-abfa-4060-8f52-4e5756866f3f-original-pull-secret\") pod \"global-pull-secret-syncer-fs4n6\" (UID: \"26cac444-abfa-4060-8f52-4e5756866f3f\") " pod="kube-system/global-pull-secret-syncer-fs4n6"
Sep 09 05:50:08.935607 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:08.935447    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b179b936-0cca-4a9e-86ce-775441316808-metrics-certs\") pod \"network-metrics-daemon-s7r5d\" (UID: \"b179b936-0cca-4a9e-86ce-775441316808\") " pod="openshift-multus/network-metrics-daemon-s7r5d"
Sep 09 05:50:08.935607 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:08.935561    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 05:50:08.935607 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:08.935566    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 05:50:08.935739 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:08.935619    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b179b936-0cca-4a9e-86ce-775441316808-metrics-certs podName:b179b936-0cca-4a9e-86ce-775441316808 nodeName:}" failed. No retries permitted until 2026-09-09 05:50:40.935603751 +0000 UTC m=+66.276091639 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b179b936-0cca-4a9e-86ce-775441316808-metrics-certs") pod "network-metrics-daemon-s7r5d" (UID: "b179b936-0cca-4a9e-86ce-775441316808") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 05:50:08.935739 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:08.935652    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26cac444-abfa-4060-8f52-4e5756866f3f-original-pull-secret podName:26cac444-abfa-4060-8f52-4e5756866f3f nodeName:}" failed. No retries permitted until 2026-09-09 05:50:40.935642088 +0000 UTC m=+66.276129979 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/26cac444-abfa-4060-8f52-4e5756866f3f-original-pull-secret") pod "global-pull-secret-syncer-fs4n6" (UID: "26cac444-abfa-4060-8f52-4e5756866f3f") : object "kube-system"/"original-pull-secret" not registered
Sep 09 05:50:09.007480 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.007448    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-142.ec2.internal" event="NodeReady"
Sep 09 05:50:09.007619 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.007590    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 05:50:09.036148 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.036113    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqt5c\" (UniqueName: \"kubernetes.io/projected/5459053f-e126-4327-a1e2-7676a5cc107b-kube-api-access-rqt5c\") pod \"network-check-target-b78pk\" (UID: \"5459053f-e126-4327-a1e2-7676a5cc107b\") " pod="openshift-network-diagnostics/network-check-target-b78pk"
Sep 09 05:50:09.036295 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:09.036241    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 05:50:09.036295 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:09.036264    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 05:50:09.036295 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:09.036276    2579 projected.go:196] Error preparing data for projected volume kube-api-access-rqt5c for pod openshift-network-diagnostics/network-check-target-b78pk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 05:50:09.036424 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:09.036329    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5459053f-e126-4327-a1e2-7676a5cc107b-kube-api-access-rqt5c podName:5459053f-e126-4327-a1e2-7676a5cc107b nodeName:}" failed. No retries permitted until 2026-09-09 05:50:41.036314586 +0000 UTC m=+66.376802473 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rqt5c" (UniqueName: "kubernetes.io/projected/5459053f-e126-4327-a1e2-7676a5cc107b-kube-api-access-rqt5c") pod "network-check-target-b78pk" (UID: "5459053f-e126-4327-a1e2-7676a5cc107b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 05:50:09.039203 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.039179    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-56cfc6cd49-45b2m"]
Sep 09 05:50:09.044202 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.044186    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56cfc6cd49-45b2m"
Sep 09 05:50:09.047424 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.047193    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 05:50:09.048000 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.047536    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 05:50:09.048000 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.047719    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gpg8k\""
Sep 09 05:50:09.048531 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.048511    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 05:50:09.052169 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.052142    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56cfc6cd49-45b2m"]
Sep 09 05:50:09.053898 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.053875    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 05:50:09.056806 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.056217    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-sddl4"]
Sep 09 05:50:09.060094 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.060078    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-v95w5"]
Sep 09 05:50:09.060228 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.060210    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sddl4"
Sep 09 05:50:09.063510 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.063431    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sfr25\""
Sep 09 05:50:09.063510 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.063441    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 05:50:09.064043 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.063702    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 05:50:09.064043 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.063820    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-v95w5"
Sep 09 05:50:09.067021 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.066852    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-lskth\""
Sep 09 05:50:09.067710 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.067693    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 05:50:09.067962 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.067944    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 05:50:09.075829 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.075811    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 05:50:09.081998 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.081973    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-v95w5"]
Sep 09 05:50:09.089755 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.089737    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sddl4"]
Sep 09 05:50:09.136596 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.136568    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c75th\" (UniqueName: \"kubernetes.io/projected/bc357870-f940-491c-be5f-3fbe8626631b-kube-api-access-c75th\") pod \"dns-default-sddl4\" (UID: \"bc357870-f940-491c-be5f-3fbe8626631b\") " pod="openshift-dns/dns-default-sddl4"
Sep 09 05:50:09.136596 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.136597    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e11f0c02-51d7-479c-a288-45dc4a766c7a-bound-sa-token\") pod \"image-registry-56cfc6cd49-45b2m\" (UID: \"e11f0c02-51d7-479c-a288-45dc4a766c7a\") " pod="openshift-image-registry/image-registry-56cfc6cd49-45b2m"
Sep 09 05:50:09.136763 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.136641    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e11f0c02-51d7-479c-a288-45dc4a766c7a-trusted-ca\") pod \"image-registry-56cfc6cd49-45b2m\" (UID: \"e11f0c02-51d7-479c-a288-45dc4a766c7a\") " pod="openshift-image-registry/image-registry-56cfc6cd49-45b2m"
Sep 09 05:50:09.136763 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.136666    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvwq9\" (UniqueName: \"kubernetes.io/projected/cde197f1-1feb-4110-970e-53389ad986ae-kube-api-access-lvwq9\") pod \"ingress-canary-v95w5\" (UID: \"cde197f1-1feb-4110-970e-53389ad986ae\") " pod="openshift-ingress-canary/ingress-canary-v95w5"
Sep 09 05:50:09.136763 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.136699    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e11f0c02-51d7-479c-a288-45dc4a766c7a-registry-tls\") pod \"image-registry-56cfc6cd49-45b2m\" (UID: \"e11f0c02-51d7-479c-a288-45dc4a766c7a\") " pod="openshift-image-registry/image-registry-56cfc6cd49-45b2m"
Sep 09 05:50:09.136763 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.136715    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e11f0c02-51d7-479c-a288-45dc4a766c7a-ca-trust-extracted\") pod \"image-registry-56cfc6cd49-45b2m\" (UID: \"e11f0c02-51d7-479c-a288-45dc4a766c7a\") " pod="openshift-image-registry/image-registry-56cfc6cd49-45b2m"
Sep 09 05:50:09.136763 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.136738    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfptm\" (UniqueName: \"kubernetes.io/projected/e11f0c02-51d7-479c-a288-45dc4a766c7a-kube-api-access-lfptm\") pod \"image-registry-56cfc6cd49-45b2m\" (UID: \"e11f0c02-51d7-479c-a288-45dc4a766c7a\") " pod="openshift-image-registry/image-registry-56cfc6cd49-45b2m"
Sep 09 05:50:09.136974 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.136780    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e11f0c02-51d7-479c-a288-45dc4a766c7a-installation-pull-secrets\") pod \"image-registry-56cfc6cd49-45b2m\" (UID: \"e11f0c02-51d7-479c-a288-45dc4a766c7a\") " pod="openshift-image-registry/image-registry-56cfc6cd49-45b2m"
Sep 09 05:50:09.136974 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.136864    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc357870-f940-491c-be5f-3fbe8626631b-metrics-tls\") pod \"dns-default-sddl4\" (UID: \"bc357870-f940-491c-be5f-3fbe8626631b\") " pod="openshift-dns/dns-default-sddl4"
Sep 09 05:50:09.136974 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.136901    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e11f0c02-51d7-479c-a288-45dc4a766c7a-registry-certificates\") pod \"image-registry-56cfc6cd49-45b2m\" (UID: \"e11f0c02-51d7-479c-a288-45dc4a766c7a\") " pod="openshift-image-registry/image-registry-56cfc6cd49-45b2m"
Sep 09 05:50:09.136974 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.136930    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bc357870-f940-491c-be5f-3fbe8626631b-config-volume\") pod \"dns-default-sddl4\" (UID: \"bc357870-f940-491c-be5f-3fbe8626631b\") " pod="openshift-dns/dns-default-sddl4"
Sep 09 05:50:09.136974 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.136948    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc357870-f940-491c-be5f-3fbe8626631b-tmp-dir\") pod \"dns-default-sddl4\" (UID: \"bc357870-f940-491c-be5f-3fbe8626631b\") " pod="openshift-dns/dns-default-sddl4"
Sep 09 05:50:09.136974 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.136967    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e11f0c02-51d7-479c-a288-45dc4a766c7a-image-registry-private-configuration\") pod \"image-registry-56cfc6cd49-45b2m\" (UID: \"e11f0c02-51d7-479c-a288-45dc4a766c7a\") " pod="openshift-image-registry/image-registry-56cfc6cd49-45b2m"
Sep 09 05:50:09.137167 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.136992    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cde197f1-1feb-4110-970e-53389ad986ae-cert\") pod \"ingress-canary-v95w5\" (UID: \"cde197f1-1feb-4110-970e-53389ad986ae\") " pod="openshift-ingress-canary/ingress-canary-v95w5"
Sep 09 05:50:09.238069 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.238048    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e11f0c02-51d7-479c-a288-45dc4a766c7a-installation-pull-secrets\") pod \"image-registry-56cfc6cd49-45b2m\" (UID: \"e11f0c02-51d7-479c-a288-45dc4a766c7a\") " pod="openshift-image-registry/image-registry-56cfc6cd49-45b2m"
Sep 09 05:50:09.238432 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.238108    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc357870-f940-491c-be5f-3fbe8626631b-metrics-tls\") pod \"dns-default-sddl4\" (UID: \"bc357870-f940-491c-be5f-3fbe8626631b\") " pod="openshift-dns/dns-default-sddl4"
Sep 09 05:50:09.238432 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.238139    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e11f0c02-51d7-479c-a288-45dc4a766c7a-registry-certificates\") pod \"image-registry-56cfc6cd49-45b2m\" (UID: \"e11f0c02-51d7-479c-a288-45dc4a766c7a\") " pod="openshift-image-registry/image-registry-56cfc6cd49-45b2m"
Sep 09 05:50:09.238432 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.238159    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bc357870-f940-491c-be5f-3fbe8626631b-config-volume\") pod \"dns-default-sddl4\" (UID: \"bc357870-f940-491c-be5f-3fbe8626631b\") " pod="openshift-dns/dns-default-sddl4"
Sep 09 05:50:09.238432 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.238172    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc357870-f940-491c-be5f-3fbe8626631b-tmp-dir\") pod \"dns-default-sddl4\" (UID: \"bc357870-f940-491c-be5f-3fbe8626631b\") " pod="openshift-dns/dns-default-sddl4"
Sep 09 05:50:09.238432 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.238196    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e11f0c02-51d7-479c-a288-45dc4a766c7a-image-registry-private-configuration\") pod \"image-registry-56cfc6cd49-45b2m\" (UID: \"e11f0c02-51d7-479c-a288-45dc4a766c7a\") " pod="openshift-image-registry/image-registry-56cfc6cd49-45b2m"
Sep 09 05:50:09.238432 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:09.238292    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 05:50:09.238432 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:09.238357    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc357870-f940-491c-be5f-3fbe8626631b-metrics-tls podName:bc357870-f940-491c-be5f-3fbe8626631b nodeName:}" failed. No retries permitted until 2026-09-09 05:50:09.738337176 +0000 UTC m=+35.078825082 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc357870-f940-491c-be5f-3fbe8626631b-metrics-tls") pod "dns-default-sddl4" (UID: "bc357870-f940-491c-be5f-3fbe8626631b") : secret "dns-default-metrics-tls" not found
Sep 09 05:50:09.238432 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.238289    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cde197f1-1feb-4110-970e-53389ad986ae-cert\") pod \"ingress-canary-v95w5\" (UID: \"cde197f1-1feb-4110-970e-53389ad986ae\") " pod="openshift-ingress-canary/ingress-canary-v95w5"
Sep 09 05:50:09.238432 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.238403    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c75th\" (UniqueName: \"kubernetes.io/projected/bc357870-f940-491c-be5f-3fbe8626631b-kube-api-access-c75th\") pod \"dns-default-sddl4\" (UID: \"bc357870-f940-491c-be5f-3fbe8626631b\") " pod="openshift-dns/dns-default-sddl4"
Sep 09 05:50:09.238432 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.238430    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e11f0c02-51d7-479c-a288-45dc4a766c7a-bound-sa-token\") pod \"image-registry-56cfc6cd49-45b2m\" (UID: \"e11f0c02-51d7-479c-a288-45dc4a766c7a\") " pod="openshift-image-registry/image-registry-56cfc6cd49-45b2m"
Sep 09 05:50:09.238934 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.238459    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e11f0c02-51d7-479c-a288-45dc4a766c7a-trusted-ca\") pod \"image-registry-56cfc6cd49-45b2m\" (UID: \"e11f0c02-51d7-479c-a288-45dc4a766c7a\") " pod="openshift-image-registry/image-registry-56cfc6cd49-45b2m"
Sep 09 05:50:09.238934 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.238482    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvwq9\" (UniqueName: \"kubernetes.io/projected/cde197f1-1feb-4110-970e-53389ad986ae-kube-api-access-lvwq9\") pod \"ingress-canary-v95w5\" (UID: \"cde197f1-1feb-4110-970e-53389ad986ae\") " pod="openshift-ingress-canary/ingress-canary-v95w5"
Sep 09 05:50:09.238934 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.238511    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc357870-f940-491c-be5f-3fbe8626631b-tmp-dir\") pod \"dns-default-sddl4\" (UID: \"bc357870-f940-491c-be5f-3fbe8626631b\") " pod="openshift-dns/dns-default-sddl4"
Sep 09 05:50:09.238934 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.238511    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e11f0c02-51d7-479c-a288-45dc4a766c7a-registry-tls\") pod \"image-registry-56cfc6cd49-45b2m\" (UID: \"e11f0c02-51d7-479c-a288-45dc4a766c7a\") " pod="openshift-image-registry/image-registry-56cfc6cd49-45b2m"
Sep 09 05:50:09.238934 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.238553    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e11f0c02-51d7-479c-a288-45dc4a766c7a-ca-trust-extracted\") pod \"image-registry-56cfc6cd49-45b2m\" (UID: \"e11f0c02-51d7-479c-a288-45dc4a766c7a\") " pod="openshift-image-registry/image-registry-56cfc6cd49-45b2m"
Sep 09 05:50:09.238934 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.238571    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfptm\" (UniqueName: \"kubernetes.io/projected/e11f0c02-51d7-479c-a288-45dc4a766c7a-kube-api-access-lfptm\") pod \"image-registry-56cfc6cd49-45b2m\" (UID: \"e11f0c02-51d7-479c-a288-45dc4a766c7a\") " pod="openshift-image-registry/image-registry-56cfc6cd49-45b2m"
Sep 09 05:50:09.238934 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.238808    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bc357870-f940-491c-be5f-3fbe8626631b-config-volume\") pod \"dns-default-sddl4\" (UID: \"bc357870-f940-491c-be5f-3fbe8626631b\") " pod="openshift-dns/dns-default-sddl4"
Sep 09 05:50:09.238934 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:09.238924    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 05:50:09.238934 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:09.238939    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56cfc6cd49-45b2m: secret "image-registry-tls" not found
Sep 09 05:50:09.239326 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:09.239000    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e11f0c02-51d7-479c-a288-45dc4a766c7a-registry-tls podName:e11f0c02-51d7-479c-a288-45dc4a766c7a nodeName:}" failed. No retries permitted until 2026-09-09 05:50:09.738980763 +0000 UTC m=+35.079468665 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e11f0c02-51d7-479c-a288-45dc4a766c7a-registry-tls") pod "image-registry-56cfc6cd49-45b2m" (UID: "e11f0c02-51d7-479c-a288-45dc4a766c7a") : secret "image-registry-tls" not found
Sep 09 05:50:09.239381 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.239336    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e11f0c02-51d7-479c-a288-45dc4a766c7a-ca-trust-extracted\") pod \"image-registry-56cfc6cd49-45b2m\" (UID: \"e11f0c02-51d7-479c-a288-45dc4a766c7a\") " pod="openshift-image-registry/image-registry-56cfc6cd49-45b2m"
Sep 09 05:50:09.239823 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.239802    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e11f0c02-51d7-479c-a288-45dc4a766c7a-registry-certificates\") pod \"image-registry-56cfc6cd49-45b2m\" (UID: \"e11f0c02-51d7-479c-a288-45dc4a766c7a\") " pod="openshift-image-registry/image-registry-56cfc6cd49-45b2m"
Sep 09 05:50:09.240507 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.240481    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e11f0c02-51d7-479c-a288-45dc4a766c7a-trusted-ca\") pod \"image-registry-56cfc6cd49-45b2m\" (UID: \"e11f0c02-51d7-479c-a288-45dc4a766c7a\") " pod="openshift-image-registry/image-registry-56cfc6cd49-45b2m"
Sep 09 05:50:09.243053 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.243034    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e11f0c02-51d7-479c-a288-45dc4a766c7a-installation-pull-secrets\") pod \"image-registry-56cfc6cd49-45b2m\" (UID: \"e11f0c02-51d7-479c-a288-45dc4a766c7a\") " pod="openshift-image-registry/image-registry-56cfc6cd49-45b2m"
Sep 09 05:50:09.243169 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.243140    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e11f0c02-51d7-479c-a288-45dc4a766c7a-image-registry-private-configuration\") pod \"image-registry-56cfc6cd49-45b2m\" (UID: \"e11f0c02-51d7-479c-a288-45dc4a766c7a\") " pod="openshift-image-registry/image-registry-56cfc6cd49-45b2m"
Sep 09 05:50:09.243230 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.243202    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cde197f1-1feb-4110-970e-53389ad986ae-cert\") pod \"ingress-canary-v95w5\" (UID: \"cde197f1-1feb-4110-970e-53389ad986ae\") " pod="openshift-ingress-canary/ingress-canary-v95w5"
Sep 09 05:50:09.247158 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.247134    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e11f0c02-51d7-479c-a288-45dc4a766c7a-bound-sa-token\") pod \"image-registry-56cfc6cd49-45b2m\" (UID: \"e11f0c02-51d7-479c-a288-45dc4a766c7a\") " pod="openshift-image-registry/image-registry-56cfc6cd49-45b2m"
Sep 09 05:50:09.248648 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.248473    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfptm\" (UniqueName: \"kubernetes.io/projected/e11f0c02-51d7-479c-a288-45dc4a766c7a-kube-api-access-lfptm\") pod \"image-registry-56cfc6cd49-45b2m\" (UID: \"e11f0c02-51d7-479c-a288-45dc4a766c7a\") " pod="openshift-image-registry/image-registry-56cfc6cd49-45b2m"
Sep 09 05:50:09.248853 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.248832    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c75th\" (UniqueName: \"kubernetes.io/projected/bc357870-f940-491c-be5f-3fbe8626631b-kube-api-access-c75th\") pod \"dns-default-sddl4\" (UID: \"bc357870-f940-491c-be5f-3fbe8626631b\") " pod="openshift-dns/dns-default-sddl4"
Sep 09 05:50:09.248971 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.248949    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvwq9\" (UniqueName: \"kubernetes.io/projected/cde197f1-1feb-4110-970e-53389ad986ae-kube-api-access-lvwq9\") pod \"ingress-canary-v95w5\" (UID: \"cde197f1-1feb-4110-970e-53389ad986ae\") " pod="openshift-ingress-canary/ingress-canary-v95w5"
Sep 09 05:50:09.323888 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.323852    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4pdb" event={"ID":"08607c27-ea7f-4487-9e8c-d84f000464d7","Type":"ContainerStarted","Data":"ae41fa480f9882a157f73cd441b4883c7422ba949d310b4de046193ee3cc4c21"}
Sep 09 05:50:09.375868 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.375834    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-v95w5"
Sep 09 05:50:09.660196 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.660116    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-v95w5"]
Sep 09 05:50:09.663980 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:50:09.663956    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcde197f1_1feb_4110_970e_53389ad986ae.slice/crio-4aca433fda1359e3f03cb04779b0a3c3794628bfad0cfa33de86d2c1e5560f96 WatchSource:0}: Error finding container 4aca433fda1359e3f03cb04779b0a3c3794628bfad0cfa33de86d2c1e5560f96: Status 404 returned error can't find the container with id 4aca433fda1359e3f03cb04779b0a3c3794628bfad0cfa33de86d2c1e5560f96
Sep 09 05:50:09.743236 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.743204    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc357870-f940-491c-be5f-3fbe8626631b-metrics-tls\") pod \"dns-default-sddl4\" (UID: \"bc357870-f940-491c-be5f-3fbe8626631b\") " pod="openshift-dns/dns-default-sddl4"
Sep 09 05:50:09.743311 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:09.743251    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e11f0c02-51d7-479c-a288-45dc4a766c7a-registry-tls\") pod \"image-registry-56cfc6cd49-45b2m\" (UID: \"e11f0c02-51d7-479c-a288-45dc4a766c7a\") " pod="openshift-image-registry/image-registry-56cfc6cd49-45b2m"
Sep 09 05:50:09.743366 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:09.743354    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 05:50:09.743398 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:09.743369    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56cfc6cd49-45b2m: secret "image-registry-tls" not found
Sep 09 05:50:09.743398 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:09.743370    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 05:50:09.743456 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:09.743431    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e11f0c02-51d7-479c-a288-45dc4a766c7a-registry-tls podName:e11f0c02-51d7-479c-a288-45dc4a766c7a nodeName:}" failed. No retries permitted until 2026-09-09 05:50:10.743412745 +0000 UTC m=+36.083900633 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e11f0c02-51d7-479c-a288-45dc4a766c7a-registry-tls") pod "image-registry-56cfc6cd49-45b2m" (UID: "e11f0c02-51d7-479c-a288-45dc4a766c7a") : secret "image-registry-tls" not found
Sep 09 05:50:09.743456 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:09.743443    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc357870-f940-491c-be5f-3fbe8626631b-metrics-tls podName:bc357870-f940-491c-be5f-3fbe8626631b nodeName:}" failed. No retries permitted until 2026-09-09 05:50:10.743437729 +0000 UTC m=+36.083925616 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc357870-f940-491c-be5f-3fbe8626631b-metrics-tls") pod "dns-default-sddl4" (UID: "bc357870-f940-491c-be5f-3fbe8626631b") : secret "dns-default-metrics-tls" not found
Sep 09 05:50:10.142777 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:10.142736    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7r5d"
Sep 09 05:50:10.142777 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:10.142764    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b78pk"
Sep 09 05:50:10.142777 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:10.142780    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fs4n6"
Sep 09 05:50:10.145667 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:10.145644    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 05:50:10.145802 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:10.145693    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 05:50:10.146482 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:10.146443    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hh975\""
Sep 09 05:50:10.146482 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:10.146445    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wnkgt\""
Sep 09 05:50:10.146696 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:10.146511    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 05:50:10.146696 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:10.146576    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 05:50:10.329933 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:10.329904    2579 generic.go:356] "Generic (PLEG): container finished" podID="08607c27-ea7f-4487-9e8c-d84f000464d7" containerID="ae41fa480f9882a157f73cd441b4883c7422ba949d310b4de046193ee3cc4c21" exitCode=0
Sep 09 05:50:10.330710 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:10.329975    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4pdb" event={"ID":"08607c27-ea7f-4487-9e8c-d84f000464d7","Type":"ContainerDied","Data":"ae41fa480f9882a157f73cd441b4883c7422ba949d310b4de046193ee3cc4c21"}
Sep 09 05:50:10.331477 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:10.331452    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-v95w5" event={"ID":"cde197f1-1feb-4110-970e-53389ad986ae","Type":"ContainerStarted","Data":"4aca433fda1359e3f03cb04779b0a3c3794628bfad0cfa33de86d2c1e5560f96"}
Sep 09 05:50:10.752040 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:10.751942    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc357870-f940-491c-be5f-3fbe8626631b-metrics-tls\") pod \"dns-default-sddl4\" (UID: \"bc357870-f940-491c-be5f-3fbe8626631b\") " pod="openshift-dns/dns-default-sddl4"
Sep 09 05:50:10.752040 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:10.752022    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e11f0c02-51d7-479c-a288-45dc4a766c7a-registry-tls\") pod \"image-registry-56cfc6cd49-45b2m\" (UID: \"e11f0c02-51d7-479c-a288-45dc4a766c7a\") " pod="openshift-image-registry/image-registry-56cfc6cd49-45b2m"
Sep 09 05:50:10.752260 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:10.752102    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 05:50:10.752260 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:10.752155    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 05:50:10.752260 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:10.752167    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56cfc6cd49-45b2m: secret "image-registry-tls" not found
Sep 09 05:50:10.752260 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:10.752184    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc357870-f940-491c-be5f-3fbe8626631b-metrics-tls podName:bc357870-f940-491c-be5f-3fbe8626631b nodeName:}" failed. No retries permitted until 2026-09-09 05:50:12.752165226 +0000 UTC m=+38.092653130 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc357870-f940-491c-be5f-3fbe8626631b-metrics-tls") pod "dns-default-sddl4" (UID: "bc357870-f940-491c-be5f-3fbe8626631b") : secret "dns-default-metrics-tls" not found
Sep 09 05:50:10.752260 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:10.752210    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e11f0c02-51d7-479c-a288-45dc4a766c7a-registry-tls podName:e11f0c02-51d7-479c-a288-45dc4a766c7a nodeName:}" failed. No retries permitted until 2026-09-09 05:50:12.752193617 +0000 UTC m=+38.092681524 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e11f0c02-51d7-479c-a288-45dc4a766c7a-registry-tls") pod "image-registry-56cfc6cd49-45b2m" (UID: "e11f0c02-51d7-479c-a288-45dc4a766c7a") : secret "image-registry-tls" not found
Sep 09 05:50:11.337950 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:11.337911    2579 generic.go:356] "Generic (PLEG): container finished" podID="08607c27-ea7f-4487-9e8c-d84f000464d7" containerID="097beb62a8b5a0969962461b02c27cd5378c44f2596930d7be4e9d2782918241" exitCode=0
Sep 09 05:50:11.338415 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:11.337975    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4pdb" event={"ID":"08607c27-ea7f-4487-9e8c-d84f000464d7","Type":"ContainerDied","Data":"097beb62a8b5a0969962461b02c27cd5378c44f2596930d7be4e9d2782918241"}
Sep 09 05:50:12.343938 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:12.343715    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-v95w5" event={"ID":"cde197f1-1feb-4110-970e-53389ad986ae","Type":"ContainerStarted","Data":"0cadd13fcdc0f916e65da2fc85e61ad507ecbb4e3c3abff1d14178219fbe262c"}
Sep 09 05:50:12.347368 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:12.347343    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4pdb" event={"ID":"08607c27-ea7f-4487-9e8c-d84f000464d7","Type":"ContainerStarted","Data":"016e691ccb5943ebd73f6c3c022655e3738095a4204930c5b78007f2885fb0f5"}
Sep 09 05:50:12.357868 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:12.357829    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-v95w5" podStartSLOduration=0.909402871 podStartE2EDuration="3.357815299s" totalImagesPullingTime="2.448412428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:50:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:50:12.357259011 +0000 UTC m=+37.697746919" watchObservedRunningTime="2026-09-09 05:50:12.357815299 +0000 UTC m=+37.698303207"
Sep 09 05:50:12.376051 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:12.376003    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-q4pdb" podStartSLOduration=8.956309822 podStartE2EDuration="37.375985985s" totalImagesPullingTime="28.419676163s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 05:49:35 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:50:12.375153314 +0000 UTC m=+37.715641224" watchObservedRunningTime="2026-09-09 05:50:12.375985985 +0000 UTC m=+37.716473895"
Sep 09 05:50:12.767494 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:12.767452    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc357870-f940-491c-be5f-3fbe8626631b-metrics-tls\") pod \"dns-default-sddl4\" (UID: \"bc357870-f940-491c-be5f-3fbe8626631b\") " pod="openshift-dns/dns-default-sddl4"
Sep 09 05:50:12.767771 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:12.767514    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e11f0c02-51d7-479c-a288-45dc4a766c7a-registry-tls\") pod \"image-registry-56cfc6cd49-45b2m\" (UID: \"e11f0c02-51d7-479c-a288-45dc4a766c7a\") " pod="openshift-image-registry/image-registry-56cfc6cd49-45b2m"
Sep 09 05:50:12.767771 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:12.767614    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 05:50:12.767771 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:12.767652    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 05:50:12.767771 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:12.767664    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56cfc6cd49-45b2m: secret "image-registry-tls" not found
Sep 09 05:50:12.767771 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:12.767709    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e11f0c02-51d7-479c-a288-45dc4a766c7a-registry-tls podName:e11f0c02-51d7-479c-a288-45dc4a766c7a nodeName:}" failed. No retries permitted until 2026-09-09 05:50:16.767690552 +0000 UTC m=+42.108178446 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e11f0c02-51d7-479c-a288-45dc4a766c7a-registry-tls") pod "image-registry-56cfc6cd49-45b2m" (UID: "e11f0c02-51d7-479c-a288-45dc4a766c7a") : secret "image-registry-tls" not found
Sep 09 05:50:12.767771 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:12.767725    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc357870-f940-491c-be5f-3fbe8626631b-metrics-tls podName:bc357870-f940-491c-be5f-3fbe8626631b nodeName:}" failed. No retries permitted until 2026-09-09 05:50:16.76771724 +0000 UTC m=+42.108205127 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc357870-f940-491c-be5f-3fbe8626631b-metrics-tls") pod "dns-default-sddl4" (UID: "bc357870-f940-491c-be5f-3fbe8626631b") : secret "dns-default-metrics-tls" not found
Sep 09 05:50:16.796098 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:16.796054    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e11f0c02-51d7-479c-a288-45dc4a766c7a-registry-tls\") pod \"image-registry-56cfc6cd49-45b2m\" (UID: \"e11f0c02-51d7-479c-a288-45dc4a766c7a\") " pod="openshift-image-registry/image-registry-56cfc6cd49-45b2m"
Sep 09 05:50:16.796553 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:16.796131    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc357870-f940-491c-be5f-3fbe8626631b-metrics-tls\") pod \"dns-default-sddl4\" (UID: \"bc357870-f940-491c-be5f-3fbe8626631b\") " pod="openshift-dns/dns-default-sddl4"
Sep 09 05:50:16.796553 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:16.796235    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 05:50:16.796553 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:16.796239    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 05:50:16.796553 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:16.796263    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56cfc6cd49-45b2m: secret "image-registry-tls" not found
Sep 09 05:50:16.796553 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:16.796290    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc357870-f940-491c-be5f-3fbe8626631b-metrics-tls podName:bc357870-f940-491c-be5f-3fbe8626631b nodeName:}" failed. No retries permitted until 2026-09-09 05:50:24.796275416 +0000 UTC m=+50.136763307 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc357870-f940-491c-be5f-3fbe8626631b-metrics-tls") pod "dns-default-sddl4" (UID: "bc357870-f940-491c-be5f-3fbe8626631b") : secret "dns-default-metrics-tls" not found
Sep 09 05:50:16.796553 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:16.796320    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e11f0c02-51d7-479c-a288-45dc4a766c7a-registry-tls podName:e11f0c02-51d7-479c-a288-45dc4a766c7a nodeName:}" failed. No retries permitted until 2026-09-09 05:50:24.79630243 +0000 UTC m=+50.136790319 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e11f0c02-51d7-479c-a288-45dc4a766c7a-registry-tls") pod "image-registry-56cfc6cd49-45b2m" (UID: "e11f0c02-51d7-479c-a288-45dc4a766c7a") : secret "image-registry-tls" not found
Sep 09 05:50:17.424425 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:17.424388    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5vkbv"]
Sep 09 05:50:17.456725 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:17.456698    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5vkbv"]
Sep 09 05:50:17.456883 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:17.456836    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5vkbv"
Sep 09 05:50:17.459519 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:17.459464    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 05:50:17.459519 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:17.459482    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 05:50:17.459519 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:17.459503    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 05:50:17.459845 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:17.459482    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xqjvr\""
Sep 09 05:50:17.459845 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:17.459579    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 05:50:17.601067 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:17.601036    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f15c4b09-8d4c-4662-b9df-ef40e5ccc281-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5vkbv\" (UID: \"f15c4b09-8d4c-4662-b9df-ef40e5ccc281\") " pod="openshift-insights/insights-runtime-extractor-5vkbv"
Sep 09 05:50:17.601231 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:17.601150    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f15c4b09-8d4c-4662-b9df-ef40e5ccc281-data-volume\") pod \"insights-runtime-extractor-5vkbv\" (UID: \"f15c4b09-8d4c-4662-b9df-ef40e5ccc281\") " pod="openshift-insights/insights-runtime-extractor-5vkbv"
Sep 09 05:50:17.601231 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:17.601184    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx4q4\" (UniqueName: \"kubernetes.io/projected/f15c4b09-8d4c-4662-b9df-ef40e5ccc281-kube-api-access-jx4q4\") pod \"insights-runtime-extractor-5vkbv\" (UID: \"f15c4b09-8d4c-4662-b9df-ef40e5ccc281\") " pod="openshift-insights/insights-runtime-extractor-5vkbv"
Sep 09 05:50:17.601231 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:17.601219    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f15c4b09-8d4c-4662-b9df-ef40e5ccc281-crio-socket\") pod \"insights-runtime-extractor-5vkbv\" (UID: \"f15c4b09-8d4c-4662-b9df-ef40e5ccc281\") " pod="openshift-insights/insights-runtime-extractor-5vkbv"
Sep 09 05:50:17.601330 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:17.601297    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f15c4b09-8d4c-4662-b9df-ef40e5ccc281-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5vkbv\" (UID: \"f15c4b09-8d4c-4662-b9df-ef40e5ccc281\") " pod="openshift-insights/insights-runtime-extractor-5vkbv"
Sep 09 05:50:17.701662 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:17.701583    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f15c4b09-8d4c-4662-b9df-ef40e5ccc281-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5vkbv\" (UID: \"f15c4b09-8d4c-4662-b9df-ef40e5ccc281\") " pod="openshift-insights/insights-runtime-extractor-5vkbv"
Sep 09 05:50:17.701803 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:17.701660    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f15c4b09-8d4c-4662-b9df-ef40e5ccc281-data-volume\") pod \"insights-runtime-extractor-5vkbv\" (UID: \"f15c4b09-8d4c-4662-b9df-ef40e5ccc281\") " pod="openshift-insights/insights-runtime-extractor-5vkbv"
Sep 09 05:50:17.701803 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:17.701690    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx4q4\" (UniqueName: \"kubernetes.io/projected/f15c4b09-8d4c-4662-b9df-ef40e5ccc281-kube-api-access-jx4q4\") pod \"insights-runtime-extractor-5vkbv\" (UID: \"f15c4b09-8d4c-4662-b9df-ef40e5ccc281\") " pod="openshift-insights/insights-runtime-extractor-5vkbv"
Sep 09 05:50:17.701803 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:17.701715    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f15c4b09-8d4c-4662-b9df-ef40e5ccc281-crio-socket\") pod \"insights-runtime-extractor-5vkbv\" (UID: \"f15c4b09-8d4c-4662-b9df-ef40e5ccc281\") " pod="openshift-insights/insights-runtime-extractor-5vkbv"
Sep 09 05:50:17.701803 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:17.701719    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 05:50:17.701803 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:17.701755    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f15c4b09-8d4c-4662-b9df-ef40e5ccc281-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5vkbv\" (UID: \"f15c4b09-8d4c-4662-b9df-ef40e5ccc281\") " pod="openshift-insights/insights-runtime-extractor-5vkbv"
Sep 09 05:50:17.701803 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:17.701792    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f15c4b09-8d4c-4662-b9df-ef40e5ccc281-kube-rbac-proxy-cm podName:f15c4b09-8d4c-4662-b9df-ef40e5ccc281 nodeName:}" failed. No retries permitted until 2026-09-09 05:50:18.201772357 +0000 UTC m=+43.542260248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f15c4b09-8d4c-4662-b9df-ef40e5ccc281-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5vkbv" (UID: "f15c4b09-8d4c-4662-b9df-ef40e5ccc281") : configmap "kube-rbac-proxy" not found
Sep 09 05:50:17.702052 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:17.701856    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 05:50:17.702052 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:17.701904    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f15c4b09-8d4c-4662-b9df-ef40e5ccc281-insights-runtime-extractor-tls podName:f15c4b09-8d4c-4662-b9df-ef40e5ccc281 nodeName:}" failed. No retries permitted until 2026-09-09 05:50:18.201888397 +0000 UTC m=+43.542376299 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f15c4b09-8d4c-4662-b9df-ef40e5ccc281-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5vkbv" (UID: "f15c4b09-8d4c-4662-b9df-ef40e5ccc281") : secret "insights-runtime-extractor-tls" not found
Sep 09 05:50:17.702052 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:17.701940    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f15c4b09-8d4c-4662-b9df-ef40e5ccc281-crio-socket\") pod \"insights-runtime-extractor-5vkbv\" (UID: \"f15c4b09-8d4c-4662-b9df-ef40e5ccc281\") " pod="openshift-insights/insights-runtime-extractor-5vkbv"
Sep 09 05:50:17.702052 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:17.702014    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f15c4b09-8d4c-4662-b9df-ef40e5ccc281-data-volume\") pod \"insights-runtime-extractor-5vkbv\" (UID: \"f15c4b09-8d4c-4662-b9df-ef40e5ccc281\") " pod="openshift-insights/insights-runtime-extractor-5vkbv"
Sep 09 05:50:17.714086 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:17.714066    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx4q4\" (UniqueName: \"kubernetes.io/projected/f15c4b09-8d4c-4662-b9df-ef40e5ccc281-kube-api-access-jx4q4\") pod \"insights-runtime-extractor-5vkbv\" (UID: \"f15c4b09-8d4c-4662-b9df-ef40e5ccc281\") " pod="openshift-insights/insights-runtime-extractor-5vkbv"
Sep 09 05:50:18.205943 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:18.205911    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f15c4b09-8d4c-4662-b9df-ef40e5ccc281-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5vkbv\" (UID: \"f15c4b09-8d4c-4662-b9df-ef40e5ccc281\") " pod="openshift-insights/insights-runtime-extractor-5vkbv"
Sep 09 05:50:18.206303 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:18.206016    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f15c4b09-8d4c-4662-b9df-ef40e5ccc281-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5vkbv\" (UID: \"f15c4b09-8d4c-4662-b9df-ef40e5ccc281\") " pod="openshift-insights/insights-runtime-extractor-5vkbv"
Sep 09 05:50:18.206303 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:18.206044    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 05:50:18.206303 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:18.206130    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f15c4b09-8d4c-4662-b9df-ef40e5ccc281-kube-rbac-proxy-cm podName:f15c4b09-8d4c-4662-b9df-ef40e5ccc281 nodeName:}" failed. No retries permitted until 2026-09-09 05:50:19.206106198 +0000 UTC m=+44.546594106 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f15c4b09-8d4c-4662-b9df-ef40e5ccc281-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5vkbv" (UID: "f15c4b09-8d4c-4662-b9df-ef40e5ccc281") : configmap "kube-rbac-proxy" not found
Sep 09 05:50:18.206303 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:18.206157    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 05:50:18.206303 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:18.206212    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f15c4b09-8d4c-4662-b9df-ef40e5ccc281-insights-runtime-extractor-tls podName:f15c4b09-8d4c-4662-b9df-ef40e5ccc281 nodeName:}" failed. No retries permitted until 2026-09-09 05:50:19.206196925 +0000 UTC m=+44.546684828 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f15c4b09-8d4c-4662-b9df-ef40e5ccc281-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5vkbv" (UID: "f15c4b09-8d4c-4662-b9df-ef40e5ccc281") : secret "insights-runtime-extractor-tls" not found
Sep 09 05:50:19.182053 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:19.182028    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-b9lbm_d6d0d7b7-703b-439d-9aae-064c217b875a/dns-node-resolver/0.log"
Sep 09 05:50:19.213461 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:19.213432    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f15c4b09-8d4c-4662-b9df-ef40e5ccc281-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5vkbv\" (UID: \"f15c4b09-8d4c-4662-b9df-ef40e5ccc281\") " pod="openshift-insights/insights-runtime-extractor-5vkbv"
Sep 09 05:50:19.213832 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:19.213481    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f15c4b09-8d4c-4662-b9df-ef40e5ccc281-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5vkbv\" (UID: \"f15c4b09-8d4c-4662-b9df-ef40e5ccc281\") " pod="openshift-insights/insights-runtime-extractor-5vkbv"
Sep 09 05:50:19.213832 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:19.213583    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 05:50:19.213832 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:19.213588    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 05:50:19.213832 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:19.213659    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f15c4b09-8d4c-4662-b9df-ef40e5ccc281-insights-runtime-extractor-tls podName:f15c4b09-8d4c-4662-b9df-ef40e5ccc281 nodeName:}" failed. No retries permitted until 2026-09-09 05:50:21.213641563 +0000 UTC m=+46.554129464 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f15c4b09-8d4c-4662-b9df-ef40e5ccc281-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5vkbv" (UID: "f15c4b09-8d4c-4662-b9df-ef40e5ccc281") : secret "insights-runtime-extractor-tls" not found
Sep 09 05:50:19.213832 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:19.213676    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f15c4b09-8d4c-4662-b9df-ef40e5ccc281-kube-rbac-proxy-cm podName:f15c4b09-8d4c-4662-b9df-ef40e5ccc281 nodeName:}" failed. No retries permitted until 2026-09-09 05:50:21.21366776 +0000 UTC m=+46.554155646 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f15c4b09-8d4c-4662-b9df-ef40e5ccc281-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5vkbv" (UID: "f15c4b09-8d4c-4662-b9df-ef40e5ccc281") : configmap "kube-rbac-proxy" not found
Sep 09 05:50:20.382641 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:20.382596    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-brqzd_cd282f88-a442-4ad3-931a-18e86703ece9/node-ca/0.log"
Sep 09 05:50:21.230348 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:21.230270    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f15c4b09-8d4c-4662-b9df-ef40e5ccc281-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5vkbv\" (UID: \"f15c4b09-8d4c-4662-b9df-ef40e5ccc281\") " pod="openshift-insights/insights-runtime-extractor-5vkbv"
Sep 09 05:50:21.230348 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:21.230318    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f15c4b09-8d4c-4662-b9df-ef40e5ccc281-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5vkbv\" (UID: \"f15c4b09-8d4c-4662-b9df-ef40e5ccc281\") " pod="openshift-insights/insights-runtime-extractor-5vkbv"
Sep 09 05:50:21.230541 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:21.230416    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 05:50:21.230541 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:21.230425    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 05:50:21.230541 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:21.230467    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f15c4b09-8d4c-4662-b9df-ef40e5ccc281-kube-rbac-proxy-cm podName:f15c4b09-8d4c-4662-b9df-ef40e5ccc281 nodeName:}" failed. No retries permitted until 2026-09-09 05:50:25.230453754 +0000 UTC m=+50.570941642 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f15c4b09-8d4c-4662-b9df-ef40e5ccc281-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5vkbv" (UID: "f15c4b09-8d4c-4662-b9df-ef40e5ccc281") : configmap "kube-rbac-proxy" not found
Sep 09 05:50:21.230541 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:21.230480    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f15c4b09-8d4c-4662-b9df-ef40e5ccc281-insights-runtime-extractor-tls podName:f15c4b09-8d4c-4662-b9df-ef40e5ccc281 nodeName:}" failed. No retries permitted until 2026-09-09 05:50:25.230474508 +0000 UTC m=+50.570962395 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f15c4b09-8d4c-4662-b9df-ef40e5ccc281-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5vkbv" (UID: "f15c4b09-8d4c-4662-b9df-ef40e5ccc281") : secret "insights-runtime-extractor-tls" not found
Sep 09 05:50:21.382654 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:21.382612    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-v95w5_cde197f1-1feb-4110-970e-53389ad986ae/serve-healthcheck-canary/0.log"
Sep 09 05:50:24.859074 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:24.859046    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e11f0c02-51d7-479c-a288-45dc4a766c7a-registry-tls\") pod \"image-registry-56cfc6cd49-45b2m\" (UID: \"e11f0c02-51d7-479c-a288-45dc4a766c7a\") " pod="openshift-image-registry/image-registry-56cfc6cd49-45b2m"
Sep 09 05:50:24.859449 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:24.859145    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc357870-f940-491c-be5f-3fbe8626631b-metrics-tls\") pod \"dns-default-sddl4\" (UID: \"bc357870-f940-491c-be5f-3fbe8626631b\") " pod="openshift-dns/dns-default-sddl4"
Sep 09 05:50:24.859449 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:24.859218    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 05:50:24.859449 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:24.859232    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56cfc6cd49-45b2m: secret "image-registry-tls" not found
Sep 09 05:50:24.859449 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:24.859259    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 05:50:24.859449 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:24.859289    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e11f0c02-51d7-479c-a288-45dc4a766c7a-registry-tls podName:e11f0c02-51d7-479c-a288-45dc4a766c7a nodeName:}" failed. No retries permitted until 2026-09-09 05:50:40.859272457 +0000 UTC m=+66.199760361 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e11f0c02-51d7-479c-a288-45dc4a766c7a-registry-tls") pod "image-registry-56cfc6cd49-45b2m" (UID: "e11f0c02-51d7-479c-a288-45dc4a766c7a") : secret "image-registry-tls" not found
Sep 09 05:50:24.859449 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:24.859308    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc357870-f940-491c-be5f-3fbe8626631b-metrics-tls podName:bc357870-f940-491c-be5f-3fbe8626631b nodeName:}" failed. No retries permitted until 2026-09-09 05:50:40.859294556 +0000 UTC m=+66.199782443 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc357870-f940-491c-be5f-3fbe8626631b-metrics-tls") pod "dns-default-sddl4" (UID: "bc357870-f940-491c-be5f-3fbe8626631b") : secret "dns-default-metrics-tls" not found
Sep 09 05:50:25.261709 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:25.261606    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f15c4b09-8d4c-4662-b9df-ef40e5ccc281-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5vkbv\" (UID: \"f15c4b09-8d4c-4662-b9df-ef40e5ccc281\") " pod="openshift-insights/insights-runtime-extractor-5vkbv"
Sep 09 05:50:25.261709 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:25.261675    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f15c4b09-8d4c-4662-b9df-ef40e5ccc281-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5vkbv\" (UID: \"f15c4b09-8d4c-4662-b9df-ef40e5ccc281\") " pod="openshift-insights/insights-runtime-extractor-5vkbv"
Sep 09 05:50:25.261896 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:25.261776    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 05:50:25.261896 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:25.261841    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f15c4b09-8d4c-4662-b9df-ef40e5ccc281-insights-runtime-extractor-tls podName:f15c4b09-8d4c-4662-b9df-ef40e5ccc281 nodeName:}" failed. No retries permitted until 2026-09-09 05:50:33.261826658 +0000 UTC m=+58.602314545 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f15c4b09-8d4c-4662-b9df-ef40e5ccc281-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5vkbv" (UID: "f15c4b09-8d4c-4662-b9df-ef40e5ccc281") : secret "insights-runtime-extractor-tls" not found
Sep 09 05:50:25.262223 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:25.262205    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f15c4b09-8d4c-4662-b9df-ef40e5ccc281-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5vkbv\" (UID: \"f15c4b09-8d4c-4662-b9df-ef40e5ccc281\") " pod="openshift-insights/insights-runtime-extractor-5vkbv"
Sep 09 05:50:33.322016 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:33.321980    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f15c4b09-8d4c-4662-b9df-ef40e5ccc281-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5vkbv\" (UID: \"f15c4b09-8d4c-4662-b9df-ef40e5ccc281\") " pod="openshift-insights/insights-runtime-extractor-5vkbv"
Sep 09 05:50:33.324274 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:33.324254    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f15c4b09-8d4c-4662-b9df-ef40e5ccc281-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5vkbv\" (UID: \"f15c4b09-8d4c-4662-b9df-ef40e5ccc281\") " pod="openshift-insights/insights-runtime-extractor-5vkbv"
Sep 09 05:50:33.365400 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:33.365375    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5vkbv"
Sep 09 05:50:33.661046 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:33.660966    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5vkbv"]
Sep 09 05:50:33.664836 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:50:33.664801    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf15c4b09_8d4c_4662_b9df_ef40e5ccc281.slice/crio-8cb4fa4516701e51da72ee59625c7c59a6e88b66e45fb3ee94c9a55cfc942f02 WatchSource:0}: Error finding container 8cb4fa4516701e51da72ee59625c7c59a6e88b66e45fb3ee94c9a55cfc942f02: Status 404 returned error can't find the container with id 8cb4fa4516701e51da72ee59625c7c59a6e88b66e45fb3ee94c9a55cfc942f02
Sep 09 05:50:34.318206 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:34.318169    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s7jp7"
Sep 09 05:50:34.401584 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:34.401554    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5vkbv" event={"ID":"f15c4b09-8d4c-4662-b9df-ef40e5ccc281","Type":"ContainerStarted","Data":"7400e9eea64787e1788d0e14d3e25cf69b7f9b5386369377f79480542f69fd1d"}
Sep 09 05:50:34.401934 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:34.401603    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5vkbv" event={"ID":"f15c4b09-8d4c-4662-b9df-ef40e5ccc281","Type":"ContainerStarted","Data":"8cb4fa4516701e51da72ee59625c7c59a6e88b66e45fb3ee94c9a55cfc942f02"}
Sep 09 05:50:35.405807 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:35.405769    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5vkbv" event={"ID":"f15c4b09-8d4c-4662-b9df-ef40e5ccc281","Type":"ContainerStarted","Data":"fda11a4ea724c984e9f671d0387274cbfb93430de47948cb17d231d7850f2c7a"}
Sep 09 05:50:36.410659 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:36.410613    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5vkbv" event={"ID":"f15c4b09-8d4c-4662-b9df-ef40e5ccc281","Type":"ContainerStarted","Data":"36406d8f1df19a3b0dba3a8dcd5088d8a30a79b20cd55eda726d1a7a5cd4b66f"}
Sep 09 05:50:36.429671 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:36.429610    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5vkbv" podStartSLOduration=17.284250654 podStartE2EDuration="19.429577196s" totalImagesPullingTime="2.145326542s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 05:50:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:50:36.428487437 +0000 UTC m=+61.768975347" watchObservedRunningTime="2026-09-09 05:50:36.429577196 +0000 UTC m=+61.770065104"
Sep 09 05:50:39.361742 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.361709    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74d5bf7b44-qjxsf"]
Sep 09 05:50:39.364869 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.364846    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-658bd8c66c-769vd"]
Sep 09 05:50:39.365004 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.364972    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74d5bf7b44-qjxsf"
Sep 09 05:50:39.367404 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.367385    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658bd8c66c-769vd"
Sep 09 05:50:39.367763 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.367744    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 09 05:50:39.367863 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.367763    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 09 05:50:39.368083 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.367880    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 09 05:50:39.368083 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.368061    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 05:50:39.368492 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.368476    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 05:50:39.368554 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.368533    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 09 05:50:39.368608 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.368556    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 05:50:39.369408 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.369391    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 09 05:50:39.379976 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.379957    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-658bd8c66c-769vd"]
Sep 09 05:50:39.380609 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.380592    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74d5bf7b44-qjxsf"]
Sep 09 05:50:39.466056 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.466028    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/69c537d9-501e-42e1-956d-0f55386365da-klusterlet-config\") pod \"klusterlet-addon-workmgr-658bd8c66c-769vd\" (UID: \"69c537d9-501e-42e1-956d-0f55386365da\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658bd8c66c-769vd"
Sep 09 05:50:39.466186 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.466069    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69c537d9-501e-42e1-956d-0f55386365da-tmp\") pod \"klusterlet-addon-workmgr-658bd8c66c-769vd\" (UID: \"69c537d9-501e-42e1-956d-0f55386365da\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658bd8c66c-769vd"
Sep 09 05:50:39.466186 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.466092    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/064cde6b-fbc0-446b-8912-c3f81be50a9a-hub\") pod \"cluster-proxy-proxy-agent-74d5bf7b44-qjxsf\" (UID: \"064cde6b-fbc0-446b-8912-c3f81be50a9a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74d5bf7b44-qjxsf"
Sep 09 05:50:39.466186 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.466144    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/064cde6b-fbc0-446b-8912-c3f81be50a9a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-74d5bf7b44-qjxsf\" (UID: \"064cde6b-fbc0-446b-8912-c3f81be50a9a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74d5bf7b44-qjxsf"
Sep 09 05:50:39.466290 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.466252    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc47l\" (UniqueName: \"kubernetes.io/projected/69c537d9-501e-42e1-956d-0f55386365da-kube-api-access-hc47l\") pod \"klusterlet-addon-workmgr-658bd8c66c-769vd\" (UID: \"69c537d9-501e-42e1-956d-0f55386365da\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658bd8c66c-769vd"
Sep 09 05:50:39.466290 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.466282    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkl4t\" (UniqueName: \"kubernetes.io/projected/064cde6b-fbc0-446b-8912-c3f81be50a9a-kube-api-access-mkl4t\") pod \"cluster-proxy-proxy-agent-74d5bf7b44-qjxsf\" (UID: \"064cde6b-fbc0-446b-8912-c3f81be50a9a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74d5bf7b44-qjxsf"
Sep 09 05:50:39.466355 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.466300    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/064cde6b-fbc0-446b-8912-c3f81be50a9a-ca\") pod \"cluster-proxy-proxy-agent-74d5bf7b44-qjxsf\" (UID: \"064cde6b-fbc0-446b-8912-c3f81be50a9a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74d5bf7b44-qjxsf"
Sep 09 05:50:39.466355 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.466321    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/064cde6b-fbc0-446b-8912-c3f81be50a9a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-74d5bf7b44-qjxsf\" (UID: \"064cde6b-fbc0-446b-8912-c3f81be50a9a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74d5bf7b44-qjxsf"
Sep 09 05:50:39.466355 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.466340    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/064cde6b-fbc0-446b-8912-c3f81be50a9a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-74d5bf7b44-qjxsf\" (UID: \"064cde6b-fbc0-446b-8912-c3f81be50a9a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74d5bf7b44-qjxsf"
Sep 09 05:50:39.479409 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.479388    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-56cfc6cd49-45b2m"]
Sep 09 05:50:39.479603 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:50:39.479582    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-56cfc6cd49-45b2m" podUID="e11f0c02-51d7-479c-a288-45dc4a766c7a"
Sep 09 05:50:39.566918 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.566898    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/69c537d9-501e-42e1-956d-0f55386365da-klusterlet-config\") pod \"klusterlet-addon-workmgr-658bd8c66c-769vd\" (UID: \"69c537d9-501e-42e1-956d-0f55386365da\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658bd8c66c-769vd"
Sep 09 05:50:39.567034 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.566931    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69c537d9-501e-42e1-956d-0f55386365da-tmp\") pod \"klusterlet-addon-workmgr-658bd8c66c-769vd\" (UID: \"69c537d9-501e-42e1-956d-0f55386365da\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658bd8c66c-769vd"
Sep 09 05:50:39.567034 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.566952    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/064cde6b-fbc0-446b-8912-c3f81be50a9a-hub\") pod \"cluster-proxy-proxy-agent-74d5bf7b44-qjxsf\" (UID: \"064cde6b-fbc0-446b-8912-c3f81be50a9a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74d5bf7b44-qjxsf"
Sep 09 05:50:39.567034 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.566980    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/064cde6b-fbc0-446b-8912-c3f81be50a9a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-74d5bf7b44-qjxsf\" (UID: \"064cde6b-fbc0-446b-8912-c3f81be50a9a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74d5bf7b44-qjxsf"
Sep 09 05:50:39.567192 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.567073    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc47l\" (UniqueName: \"kubernetes.io/projected/69c537d9-501e-42e1-956d-0f55386365da-kube-api-access-hc47l\") pod \"klusterlet-addon-workmgr-658bd8c66c-769vd\" (UID: \"69c537d9-501e-42e1-956d-0f55386365da\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658bd8c66c-769vd"
Sep 09 05:50:39.567192 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.567100    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkl4t\" (UniqueName: \"kubernetes.io/projected/064cde6b-fbc0-446b-8912-c3f81be50a9a-kube-api-access-mkl4t\") pod \"cluster-proxy-proxy-agent-74d5bf7b44-qjxsf\" (UID: \"064cde6b-fbc0-446b-8912-c3f81be50a9a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74d5bf7b44-qjxsf"
Sep 09 05:50:39.567192 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.567128    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/064cde6b-fbc0-446b-8912-c3f81be50a9a-ca\") pod \"cluster-proxy-proxy-agent-74d5bf7b44-qjxsf\" (UID: \"064cde6b-fbc0-446b-8912-c3f81be50a9a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74d5bf7b44-qjxsf"
Sep 09 05:50:39.567192 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.567155    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/064cde6b-fbc0-446b-8912-c3f81be50a9a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-74d5bf7b44-qjxsf\" (UID: \"064cde6b-fbc0-446b-8912-c3f81be50a9a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74d5bf7b44-qjxsf"
Sep 09 05:50:39.567192 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.567187    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/064cde6b-fbc0-446b-8912-c3f81be50a9a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-74d5bf7b44-qjxsf\" (UID: \"064cde6b-fbc0-446b-8912-c3f81be50a9a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74d5bf7b44-qjxsf"
Sep 09 05:50:39.567429 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.567396    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69c537d9-501e-42e1-956d-0f55386365da-tmp\") pod \"klusterlet-addon-workmgr-658bd8c66c-769vd\" (UID: \"69c537d9-501e-42e1-956d-0f55386365da\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658bd8c66c-769vd"
Sep 09 05:50:39.567941 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.567917    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/064cde6b-fbc0-446b-8912-c3f81be50a9a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-74d5bf7b44-qjxsf\" (UID: \"064cde6b-fbc0-446b-8912-c3f81be50a9a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74d5bf7b44-qjxsf"
Sep 09 05:50:39.569957 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.569929    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/69c537d9-501e-42e1-956d-0f55386365da-klusterlet-config\") pod \"klusterlet-addon-workmgr-658bd8c66c-769vd\" (UID: \"69c537d9-501e-42e1-956d-0f55386365da\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658bd8c66c-769vd"
Sep 09 05:50:39.570064 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.570032    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/064cde6b-fbc0-446b-8912-c3f81be50a9a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-74d5bf7b44-qjxsf\" (UID: \"064cde6b-fbc0-446b-8912-c3f81be50a9a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74d5bf7b44-qjxsf"
Sep 09 05:50:39.570203 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.570186    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/064cde6b-fbc0-446b-8912-c3f81be50a9a-hub\") pod \"cluster-proxy-proxy-agent-74d5bf7b44-qjxsf\" (UID: \"064cde6b-fbc0-446b-8912-c3f81be50a9a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74d5bf7b44-qjxsf"
Sep 09 05:50:39.570272 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.570213    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/064cde6b-fbc0-446b-8912-c3f81be50a9a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-74d5bf7b44-qjxsf\" (UID: \"064cde6b-fbc0-446b-8912-c3f81be50a9a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74d5bf7b44-qjxsf"
Sep 09 05:50:39.570272 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.570240    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/064cde6b-fbc0-446b-8912-c3f81be50a9a-ca\") pod \"cluster-proxy-proxy-agent-74d5bf7b44-qjxsf\" (UID: \"064cde6b-fbc0-446b-8912-c3f81be50a9a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74d5bf7b44-qjxsf"
Sep 09 05:50:39.583987 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.583968    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc47l\" (UniqueName: \"kubernetes.io/projected/69c537d9-501e-42e1-956d-0f55386365da-kube-api-access-hc47l\") pod \"klusterlet-addon-workmgr-658bd8c66c-769vd\" (UID: \"69c537d9-501e-42e1-956d-0f55386365da\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658bd8c66c-769vd"
Sep 09 05:50:39.586806 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.586772    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkl4t\" (UniqueName: \"kubernetes.io/projected/064cde6b-fbc0-446b-8912-c3f81be50a9a-kube-api-access-mkl4t\") pod \"cluster-proxy-proxy-agent-74d5bf7b44-qjxsf\" (UID: \"064cde6b-fbc0-446b-8912-c3f81be50a9a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74d5bf7b44-qjxsf"
Sep 09 05:50:39.682136 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.682068    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74d5bf7b44-qjxsf"
Sep 09 05:50:39.688078 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.688060    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658bd8c66c-769vd"
Sep 09 05:50:39.959388 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.959327    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74d5bf7b44-qjxsf"]
Sep 09 05:50:39.961708 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:39.961689    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-658bd8c66c-769vd"]
Sep 09 05:50:39.963775 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:50:39.963750    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod064cde6b_fbc0_446b_8912_c3f81be50a9a.slice/crio-abc67665ee0c807f3da2ec3769b7f12a6681cba68e31c0c5dda9b7f70554500e WatchSource:0}: Error finding container abc67665ee0c807f3da2ec3769b7f12a6681cba68e31c0c5dda9b7f70554500e: Status 404 returned error can't find the container with id abc67665ee0c807f3da2ec3769b7f12a6681cba68e31c0c5dda9b7f70554500e
Sep 09 05:50:39.965122 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:50:39.965100    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c537d9_501e_42e1_956d_0f55386365da.slice/crio-88afb42bf1d072ce6e0cd22a1dd7eb3cb5a5b0fa52347f114941e05987246899 WatchSource:0}: Error finding container 88afb42bf1d072ce6e0cd22a1dd7eb3cb5a5b0fa52347f114941e05987246899: Status 404 returned error can't find the container with id 88afb42bf1d072ce6e0cd22a1dd7eb3cb5a5b0fa52347f114941e05987246899
Sep 09 05:50:40.421238 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:40.421208    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658bd8c66c-769vd" event={"ID":"69c537d9-501e-42e1-956d-0f55386365da","Type":"ContainerStarted","Data":"88afb42bf1d072ce6e0cd22a1dd7eb3cb5a5b0fa52347f114941e05987246899"}
Sep 09 05:50:40.422361 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:40.422344    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56cfc6cd49-45b2m"
Sep 09 05:50:40.422453 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:40.422360    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74d5bf7b44-qjxsf" event={"ID":"064cde6b-fbc0-446b-8912-c3f81be50a9a","Type":"ContainerStarted","Data":"abc67665ee0c807f3da2ec3769b7f12a6681cba68e31c0c5dda9b7f70554500e"}
Sep 09 05:50:40.454152 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:40.454123    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56cfc6cd49-45b2m"
Sep 09 05:50:40.474779 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:40.474755    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e11f0c02-51d7-479c-a288-45dc4a766c7a-trusted-ca\") pod \"e11f0c02-51d7-479c-a288-45dc4a766c7a\" (UID: \"e11f0c02-51d7-479c-a288-45dc4a766c7a\") "
Sep 09 05:50:40.474872 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:40.474782    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfptm\" (UniqueName: \"kubernetes.io/projected/e11f0c02-51d7-479c-a288-45dc4a766c7a-kube-api-access-lfptm\") pod \"e11f0c02-51d7-479c-a288-45dc4a766c7a\" (UID: \"e11f0c02-51d7-479c-a288-45dc4a766c7a\") "
Sep 09 05:50:40.474872 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:40.474820    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e11f0c02-51d7-479c-a288-45dc4a766c7a-bound-sa-token\") pod \"e11f0c02-51d7-479c-a288-45dc4a766c7a\" (UID: \"e11f0c02-51d7-479c-a288-45dc4a766c7a\") "
Sep 09 05:50:40.474872 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:40.474837    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e11f0c02-51d7-479c-a288-45dc4a766c7a-installation-pull-secrets\") pod \"e11f0c02-51d7-479c-a288-45dc4a766c7a\" (UID: \"e11f0c02-51d7-479c-a288-45dc4a766c7a\") "
Sep 09 05:50:40.474872 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:40.474860    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e11f0c02-51d7-479c-a288-45dc4a766c7a-registry-certificates\") pod \"e11f0c02-51d7-479c-a288-45dc4a766c7a\" (UID: \"e11f0c02-51d7-479c-a288-45dc4a766c7a\") "
Sep 09 05:50:40.475054 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:40.474991    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e11f0c02-51d7-479c-a288-45dc4a766c7a-ca-trust-extracted\") pod \"e11f0c02-51d7-479c-a288-45dc4a766c7a\" (UID: \"e11f0c02-51d7-479c-a288-45dc4a766c7a\") "
Sep 09 05:50:40.475054 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:40.475032    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e11f0c02-51d7-479c-a288-45dc4a766c7a-image-registry-private-configuration\") pod \"e11f0c02-51d7-479c-a288-45dc4a766c7a\" (UID: \"e11f0c02-51d7-479c-a288-45dc4a766c7a\") "
Sep 09 05:50:40.475240 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:40.475210    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e11f0c02-51d7-479c-a288-45dc4a766c7a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "e11f0c02-51d7-479c-a288-45dc4a766c7a" (UID: "e11f0c02-51d7-479c-a288-45dc4a766c7a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 05:50:40.475343 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:40.475283    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e11f0c02-51d7-479c-a288-45dc4a766c7a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "e11f0c02-51d7-479c-a288-45dc4a766c7a" (UID: "e11f0c02-51d7-479c-a288-45dc4a766c7a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 05:50:40.475343 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:40.475219    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e11f0c02-51d7-479c-a288-45dc4a766c7a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "e11f0c02-51d7-479c-a288-45dc4a766c7a" (UID: "e11f0c02-51d7-479c-a288-45dc4a766c7a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 05:50:40.477193 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:40.477166    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e11f0c02-51d7-479c-a288-45dc4a766c7a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "e11f0c02-51d7-479c-a288-45dc4a766c7a" (UID: "e11f0c02-51d7-479c-a288-45dc4a766c7a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 05:50:40.477294 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:40.477219    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e11f0c02-51d7-479c-a288-45dc4a766c7a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "e11f0c02-51d7-479c-a288-45dc4a766c7a" (UID: "e11f0c02-51d7-479c-a288-45dc4a766c7a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 05:50:40.477294 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:40.477237    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e11f0c02-51d7-479c-a288-45dc4a766c7a-kube-api-access-lfptm" (OuterVolumeSpecName: "kube-api-access-lfptm") pod "e11f0c02-51d7-479c-a288-45dc4a766c7a" (UID: "e11f0c02-51d7-479c-a288-45dc4a766c7a"). InnerVolumeSpecName "kube-api-access-lfptm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 05:50:40.477294 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:40.477246    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e11f0c02-51d7-479c-a288-45dc4a766c7a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "e11f0c02-51d7-479c-a288-45dc4a766c7a" (UID: "e11f0c02-51d7-479c-a288-45dc4a766c7a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 05:50:40.576293 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:40.576260    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e11f0c02-51d7-479c-a288-45dc4a766c7a-trusted-ca\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 05:50:40.576293 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:40.576291    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfptm\" (UniqueName: \"kubernetes.io/projected/e11f0c02-51d7-479c-a288-45dc4a766c7a-kube-api-access-lfptm\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 05:50:40.576443 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:40.576306    2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e11f0c02-51d7-479c-a288-45dc4a766c7a-bound-sa-token\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 05:50:40.576443 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:40.576320    2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e11f0c02-51d7-479c-a288-45dc4a766c7a-installation-pull-secrets\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 05:50:40.576443 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:40.576333    2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e11f0c02-51d7-479c-a288-45dc4a766c7a-registry-certificates\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 05:50:40.576443 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:40.576346    2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e11f0c02-51d7-479c-a288-45dc4a766c7a-ca-trust-extracted\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 05:50:40.576443 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:40.576360    2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e11f0c02-51d7-479c-a288-45dc4a766c7a-image-registry-private-configuration\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 05:50:40.879696 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:40.879662    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e11f0c02-51d7-479c-a288-45dc4a766c7a-registry-tls\") pod \"image-registry-56cfc6cd49-45b2m\" (UID: \"e11f0c02-51d7-479c-a288-45dc4a766c7a\") " pod="openshift-image-registry/image-registry-56cfc6cd49-45b2m"
Sep 09 05:50:40.879948 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:40.879757    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc357870-f940-491c-be5f-3fbe8626631b-metrics-tls\") pod \"dns-default-sddl4\" (UID: \"bc357870-f940-491c-be5f-3fbe8626631b\") " pod="openshift-dns/dns-default-sddl4"
Sep 09 05:50:40.882903 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:40.882875    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc357870-f940-491c-be5f-3fbe8626631b-metrics-tls\") pod \"dns-default-sddl4\" (UID: \"bc357870-f940-491c-be5f-3fbe8626631b\") " pod="openshift-dns/dns-default-sddl4"
Sep 09 05:50:40.884521 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:40.884496    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e11f0c02-51d7-479c-a288-45dc4a766c7a-registry-tls\") pod \"image-registry-56cfc6cd49-45b2m\" (UID: \"e11f0c02-51d7-479c-a288-45dc4a766c7a\") " pod="openshift-image-registry/image-registry-56cfc6cd49-45b2m"
Sep 09 05:50:40.980544 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:40.980500    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e11f0c02-51d7-479c-a288-45dc4a766c7a-registry-tls\") pod \"e11f0c02-51d7-479c-a288-45dc4a766c7a\" (UID: \"e11f0c02-51d7-479c-a288-45dc4a766c7a\") "
Sep 09 05:50:40.980740 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:40.980717    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26cac444-abfa-4060-8f52-4e5756866f3f-original-pull-secret\") pod \"global-pull-secret-syncer-fs4n6\" (UID: \"26cac444-abfa-4060-8f52-4e5756866f3f\") " pod="kube-system/global-pull-secret-syncer-fs4n6"
Sep 09 05:50:40.980808 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:40.980760    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b179b936-0cca-4a9e-86ce-775441316808-metrics-certs\") pod \"network-metrics-daemon-s7r5d\" (UID: \"b179b936-0cca-4a9e-86ce-775441316808\") " pod="openshift-multus/network-metrics-daemon-s7r5d"
Sep 09 05:50:40.983824 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:40.983282    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e11f0c02-51d7-479c-a288-45dc4a766c7a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e11f0c02-51d7-479c-a288-45dc4a766c7a" (UID: "e11f0c02-51d7-479c-a288-45dc4a766c7a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 05:50:40.983824 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:40.983576    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 05:50:40.989053 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:40.988868    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 05:50:40.995959 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:40.995919    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b179b936-0cca-4a9e-86ce-775441316808-metrics-certs\") pod \"network-metrics-daemon-s7r5d\" (UID: \"b179b936-0cca-4a9e-86ce-775441316808\") " pod="openshift-multus/network-metrics-daemon-s7r5d"
Sep 09 05:50:41.004379 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:41.004333    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26cac444-abfa-4060-8f52-4e5756866f3f-original-pull-secret\") pod \"global-pull-secret-syncer-fs4n6\" (UID: \"26cac444-abfa-4060-8f52-4e5756866f3f\") " pod="kube-system/global-pull-secret-syncer-fs4n6"
Sep 09 05:50:41.057832 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:41.057726    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wnkgt\""
Sep 09 05:50:41.065872 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:41.065826    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7r5d"
Sep 09 05:50:41.068197 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:41.068149    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fs4n6"
Sep 09 05:50:41.081547 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:41.081335    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqt5c\" (UniqueName: \"kubernetes.io/projected/5459053f-e126-4327-a1e2-7676a5cc107b-kube-api-access-rqt5c\") pod \"network-check-target-b78pk\" (UID: \"5459053f-e126-4327-a1e2-7676a5cc107b\") " pod="openshift-network-diagnostics/network-check-target-b78pk"
Sep 09 05:50:41.081547 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:41.081391    2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e11f0c02-51d7-479c-a288-45dc4a766c7a-registry-tls\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 05:50:41.083865 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:41.083605    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 05:50:41.094616 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:41.094452    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 05:50:41.108650 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:41.108590    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqt5c\" (UniqueName: \"kubernetes.io/projected/5459053f-e126-4327-a1e2-7676a5cc107b-kube-api-access-rqt5c\") pod \"network-check-target-b78pk\" (UID: \"5459053f-e126-4327-a1e2-7676a5cc107b\") " pod="openshift-network-diagnostics/network-check-target-b78pk"
Sep 09 05:50:41.173701 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:41.173675    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sfr25\""
Sep 09 05:50:41.182060 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:41.181856    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sddl4"
Sep 09 05:50:41.365644 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:41.365602    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hh975\""
Sep 09 05:50:41.373548 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:41.373525    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b78pk"
Sep 09 05:50:41.386834 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:41.386517    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s7r5d"]
Sep 09 05:50:41.408297 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:41.407655    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fs4n6"]
Sep 09 05:50:41.416820 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:50:41.416786    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26cac444_abfa_4060_8f52_4e5756866f3f.slice/crio-773c7228e69efcc07e5b90feac7829610ab440eea0a18a0dc16b6ffcf25fc73d WatchSource:0}: Error finding container 773c7228e69efcc07e5b90feac7829610ab440eea0a18a0dc16b6ffcf25fc73d: Status 404 returned error can't find the container with id 773c7228e69efcc07e5b90feac7829610ab440eea0a18a0dc16b6ffcf25fc73d
Sep 09 05:50:41.429192 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:41.429163    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fs4n6" event={"ID":"26cac444-abfa-4060-8f52-4e5756866f3f","Type":"ContainerStarted","Data":"773c7228e69efcc07e5b90feac7829610ab440eea0a18a0dc16b6ffcf25fc73d"}
Sep 09 05:50:41.431286 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:41.431257    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56cfc6cd49-45b2m"
Sep 09 05:50:41.431736 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:41.431716    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s7r5d" event={"ID":"b179b936-0cca-4a9e-86ce-775441316808","Type":"ContainerStarted","Data":"e484b8df6a52289c318a2b2f42fe3349d887948b55f45d7a5944d0717a71c83d"}
Sep 09 05:50:41.463498 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:41.461397    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-56cfc6cd49-45b2m"]
Sep 09 05:50:41.467078 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:41.467025    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-56cfc6cd49-45b2m"]
Sep 09 05:50:41.499982 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:41.499829    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sddl4"]
Sep 09 05:50:41.506265 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:50:41.506236    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc357870_f940_491c_be5f_3fbe8626631b.slice/crio-d211137b8ad127d10fbcd584e72dabcf7d1e97b05744e8ff6193631fdc838658 WatchSource:0}: Error finding container d211137b8ad127d10fbcd584e72dabcf7d1e97b05744e8ff6193631fdc838658: Status 404 returned error can't find the container with id d211137b8ad127d10fbcd584e72dabcf7d1e97b05744e8ff6193631fdc838658
Sep 09 05:50:41.712065 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:41.711950    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b78pk"]
Sep 09 05:50:42.436753 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:42.436690    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sddl4" event={"ID":"bc357870-f940-491c-be5f-3fbe8626631b","Type":"ContainerStarted","Data":"d211137b8ad127d10fbcd584e72dabcf7d1e97b05744e8ff6193631fdc838658"}
Sep 09 05:50:43.149372 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:43.149331    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e11f0c02-51d7-479c-a288-45dc4a766c7a" path="/var/lib/kubelet/pods/e11f0c02-51d7-479c-a288-45dc4a766c7a/volumes"
Sep 09 05:50:43.441188 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:43.441095    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b78pk" event={"ID":"5459053f-e126-4327-a1e2-7676a5cc107b","Type":"ContainerStarted","Data":"ea81e515a67ec9e606c1f8f16126954793918671feb6d2ea94d3a366521c8372"}
Sep 09 05:50:50.466672 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:50.466615    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fs4n6" event={"ID":"26cac444-abfa-4060-8f52-4e5756866f3f","Type":"ContainerStarted","Data":"a3681ba4c600108e29f0a19f8b22af35dfcf2323909676539e964b03294406fc"}
Sep 09 05:50:50.468513 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:50.468485    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s7r5d" event={"ID":"b179b936-0cca-4a9e-86ce-775441316808","Type":"ContainerStarted","Data":"91765656cf33d2bdac8500eaf36d77733e534e1ee5ed37d8a2f043e3df187722"}
Sep 09 05:50:50.468662 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:50.468532    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s7r5d" event={"ID":"b179b936-0cca-4a9e-86ce-775441316808","Type":"ContainerStarted","Data":"0b4916dceb96f402aab29cc54ecfea8249267e421d13949aea9d2bb87d23f8cd"}
Sep 09 05:50:50.470014 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:50.469992    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b78pk" event={"ID":"5459053f-e126-4327-a1e2-7676a5cc107b","Type":"ContainerStarted","Data":"3b6010c2d6d5372bf9baf80596e996e6c85d0fe691ee4b15306a165ff4bc112c"}
Sep 09 05:50:50.470096 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:50.470062    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-b78pk"
Sep 09 05:50:50.471430 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:50.471407    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658bd8c66c-769vd" event={"ID":"69c537d9-501e-42e1-956d-0f55386365da","Type":"ContainerStarted","Data":"465c0c5d067b0abaf34f31cd07177a5ea87f31e155281ce9cf0222a252fd97c4"}
Sep 09 05:50:50.471522 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:50.471453    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658bd8c66c-769vd"
Sep 09 05:50:50.473336 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:50.473318    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658bd8c66c-769vd"
Sep 09 05:50:50.473683 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:50.473655    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74d5bf7b44-qjxsf" event={"ID":"064cde6b-fbc0-446b-8912-c3f81be50a9a","Type":"ContainerStarted","Data":"8f2c6a3aa4bdcf6e4f602a4a08142684e5824f1e127168b58136e10585129824"}
Sep 09 05:50:50.473806 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:50.473698    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74d5bf7b44-qjxsf" event={"ID":"064cde6b-fbc0-446b-8912-c3f81be50a9a","Type":"ContainerStarted","Data":"fe3bd4b8b6abf2847d1cee64e3670cf68c749372e3349882895939952f7d6182"}
Sep 09 05:50:50.473806 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:50.473712    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74d5bf7b44-qjxsf" event={"ID":"064cde6b-fbc0-446b-8912-c3f81be50a9a","Type":"ContainerStarted","Data":"31e664ac747a6ecb2edf0ebbb3b313fe469304814b84ede1e0dad64bec447a18"}
Sep 09 05:50:50.475231 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:50.475211    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sddl4" event={"ID":"bc357870-f940-491c-be5f-3fbe8626631b","Type":"ContainerStarted","Data":"9525c70be459b2ab2d9de199d2049699f8f69a2e8ec6ed879856c102766c2239"}
Sep 09 05:50:50.475306 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:50.475252    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sddl4" event={"ID":"bc357870-f940-491c-be5f-3fbe8626631b","Type":"ContainerStarted","Data":"4a2d0fcb90c6f18d9e5e3ff497c0ef4a31dd2f59044756ccc74f8b6f96e9fa64"}
Sep 09 05:50:50.475347 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:50.475332    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-sddl4"
Sep 09 05:50:50.482432 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:50.482395    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-fs4n6" podStartSLOduration=67.320459646 podStartE2EDuration="1m15.482382901s" totalImagesPullingTime="8.161923255s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 05:49:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:50:50.481473647 +0000 UTC m=+75.821961555" watchObservedRunningTime="2026-09-09 05:50:50.482382901 +0000 UTC m=+75.822870806"
Sep 09 05:50:50.494920 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:50.494879    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-s7r5d" podStartSLOduration=67.3062864 podStartE2EDuration="1m15.494862917s" totalImagesPullingTime="8.188576517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:49:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:50:50.494448297 +0000 UTC m=+75.834936206" watchObservedRunningTime="2026-09-09 05:50:50.494862917 +0000 UTC m=+75.835350826"
Sep 09 05:50:50.510927 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:50.510886    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74d5bf7b44-qjxsf" podStartSLOduration=1.90560376 podStartE2EDuration="11.510873243s" totalImagesPullingTime="9.605269483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:50:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:50:50.510515745 +0000 UTC m=+75.851003667" watchObservedRunningTime="2026-09-09 05:50:50.510873243 +0000 UTC m=+75.851361151"
Sep 09 05:50:50.526125 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:50.526089    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-sddl4" podStartSLOduration=33.457837768 podStartE2EDuration="41.526078648s" totalImagesPullingTime="8.06824088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:50:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:50:50.524997107 +0000 UTC m=+75.865485041" watchObservedRunningTime="2026-09-09 05:50:50.526078648 +0000 UTC m=+75.866566557"
Sep 09 05:50:50.540865 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:50.540833    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658bd8c66c-769vd" podStartSLOduration=1.9368317240000001 podStartE2EDuration="11.540821898s" totalImagesPullingTime="9.603990174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:50:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:50:50.540221503 +0000 UTC m=+75.880709411" watchObservedRunningTime="2026-09-09 05:50:50.540821898 +0000 UTC m=+75.881309805"
Sep 09 05:50:50.556918 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:50:50.554850    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-b78pk" podStartSLOduration=68.655591335 podStartE2EDuration="1m15.554834633s" totalImagesPullingTime="6.899243298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:49:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:50:50.554030408 +0000 UTC m=+75.894518317" watchObservedRunningTime="2026-09-09 05:50:50.554834633 +0000 UTC m=+75.895322544"
Sep 09 05:51:00.482853 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:00.482750    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sddl4"
Sep 09 05:51:04.593296 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:04.593262    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lrzrl"]
Sep 09 05:51:04.597847 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:04.597827    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lrzrl"
Sep 09 05:51:04.603477 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:04.603456    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 05:51:04.603560 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:04.603458    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 05:51:04.604054 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:04.603925    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 05:51:04.604054 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:04.603974    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 05:51:04.604054 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:04.604007    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 05:51:04.604221 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:04.604174    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8mwq7\""
Sep 09 05:51:04.604757 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:04.604739    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 05:51:04.647920 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:04.647895    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fbb14eb0-b938-4d9a-a16e-9c9857c978ef-node-exporter-tls\") pod \"node-exporter-lrzrl\" (UID: \"fbb14eb0-b938-4d9a-a16e-9c9857c978ef\") " pod="openshift-monitoring/node-exporter-lrzrl"
Sep 09 05:51:04.648014 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:04.647925    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc975\" (UniqueName: \"kubernetes.io/projected/fbb14eb0-b938-4d9a-a16e-9c9857c978ef-kube-api-access-mc975\") pod \"node-exporter-lrzrl\" (UID: \"fbb14eb0-b938-4d9a-a16e-9c9857c978ef\") " pod="openshift-monitoring/node-exporter-lrzrl"
Sep 09 05:51:04.648014 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:04.647949    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fbb14eb0-b938-4d9a-a16e-9c9857c978ef-metrics-client-ca\") pod \"node-exporter-lrzrl\" (UID: \"fbb14eb0-b938-4d9a-a16e-9c9857c978ef\") " pod="openshift-monitoring/node-exporter-lrzrl"
Sep 09 05:51:04.648014 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:04.647984    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fbb14eb0-b938-4d9a-a16e-9c9857c978ef-node-exporter-textfile\") pod \"node-exporter-lrzrl\" (UID: \"fbb14eb0-b938-4d9a-a16e-9c9857c978ef\") " pod="openshift-monitoring/node-exporter-lrzrl"
Sep 09 05:51:04.648161 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:04.648035    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fbb14eb0-b938-4d9a-a16e-9c9857c978ef-sys\") pod \"node-exporter-lrzrl\" (UID: \"fbb14eb0-b938-4d9a-a16e-9c9857c978ef\") " pod="openshift-monitoring/node-exporter-lrzrl"
Sep 09 05:51:04.648161 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:04.648089    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fbb14eb0-b938-4d9a-a16e-9c9857c978ef-node-exporter-wtmp\") pod \"node-exporter-lrzrl\" (UID: \"fbb14eb0-b938-4d9a-a16e-9c9857c978ef\") " pod="openshift-monitoring/node-exporter-lrzrl"
Sep 09 05:51:04.648243 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:04.648157    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fbb14eb0-b938-4d9a-a16e-9c9857c978ef-node-exporter-accelerators-collector-config\") pod \"node-exporter-lrzrl\" (UID: \"fbb14eb0-b938-4d9a-a16e-9c9857c978ef\") " pod="openshift-monitoring/node-exporter-lrzrl"
Sep 09 05:51:04.648243 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:04.648210    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fbb14eb0-b938-4d9a-a16e-9c9857c978ef-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lrzrl\" (UID: \"fbb14eb0-b938-4d9a-a16e-9c9857c978ef\") " pod="openshift-monitoring/node-exporter-lrzrl"
Sep 09 05:51:04.648243 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:04.648230    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fbb14eb0-b938-4d9a-a16e-9c9857c978ef-root\") pod \"node-exporter-lrzrl\" (UID: \"fbb14eb0-b938-4d9a-a16e-9c9857c978ef\") " pod="openshift-monitoring/node-exporter-lrzrl"
Sep 09 05:51:04.749347 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:04.749325    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fbb14eb0-b938-4d9a-a16e-9c9857c978ef-metrics-client-ca\") pod \"node-exporter-lrzrl\" (UID: \"fbb14eb0-b938-4d9a-a16e-9c9857c978ef\") " pod="openshift-monitoring/node-exporter-lrzrl"
Sep 09 05:51:04.749468 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:04.749353    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fbb14eb0-b938-4d9a-a16e-9c9857c978ef-node-exporter-textfile\") pod \"node-exporter-lrzrl\" (UID: \"fbb14eb0-b938-4d9a-a16e-9c9857c978ef\") " pod="openshift-monitoring/node-exporter-lrzrl"
Sep 09 05:51:04.749468 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:04.749376    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fbb14eb0-b938-4d9a-a16e-9c9857c978ef-sys\") pod \"node-exporter-lrzrl\" (UID: \"fbb14eb0-b938-4d9a-a16e-9c9857c978ef\") " pod="openshift-monitoring/node-exporter-lrzrl"
Sep 09 05:51:04.749468 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:04.749393    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fbb14eb0-b938-4d9a-a16e-9c9857c978ef-node-exporter-wtmp\") pod \"node-exporter-lrzrl\" (UID: \"fbb14eb0-b938-4d9a-a16e-9c9857c978ef\") " pod="openshift-monitoring/node-exporter-lrzrl"
Sep 09 05:51:04.749468 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:04.749412    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fbb14eb0-b938-4d9a-a16e-9c9857c978ef-node-exporter-accelerators-collector-config\") pod \"node-exporter-lrzrl\" (UID: \"fbb14eb0-b938-4d9a-a16e-9c9857c978ef\") " pod="openshift-monitoring/node-exporter-lrzrl"
Sep 09 05:51:04.749468 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:04.749432    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fbb14eb0-b938-4d9a-a16e-9c9857c978ef-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lrzrl\" (UID: \"fbb14eb0-b938-4d9a-a16e-9c9857c978ef\") " pod="openshift-monitoring/node-exporter-lrzrl"
Sep 09 05:51:04.749726 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:04.749466    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fbb14eb0-b938-4d9a-a16e-9c9857c978ef-sys\") pod \"node-exporter-lrzrl\" (UID: \"fbb14eb0-b938-4d9a-a16e-9c9857c978ef\") " pod="openshift-monitoring/node-exporter-lrzrl"
Sep 09 05:51:04.749726 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:04.749483    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fbb14eb0-b938-4d9a-a16e-9c9857c978ef-root\") pod \"node-exporter-lrzrl\" (UID: \"fbb14eb0-b938-4d9a-a16e-9c9857c978ef\") " pod="openshift-monitoring/node-exporter-lrzrl"
Sep 09 05:51:04.749726 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:04.749528    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fbb14eb0-b938-4d9a-a16e-9c9857c978ef-root\") pod \"node-exporter-lrzrl\" (UID: \"fbb14eb0-b938-4d9a-a16e-9c9857c978ef\") " pod="openshift-monitoring/node-exporter-lrzrl"
Sep 09 05:51:04.749726 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:04.749536    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fbb14eb0-b938-4d9a-a16e-9c9857c978ef-node-exporter-tls\") pod \"node-exporter-lrzrl\" (UID: \"fbb14eb0-b938-4d9a-a16e-9c9857c978ef\") " pod="openshift-monitoring/node-exporter-lrzrl"
Sep 09 05:51:04.749726 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:04.749547    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fbb14eb0-b938-4d9a-a16e-9c9857c978ef-node-exporter-wtmp\") pod \"node-exporter-lrzrl\" (UID: \"fbb14eb0-b938-4d9a-a16e-9c9857c978ef\") " pod="openshift-monitoring/node-exporter-lrzrl"
Sep 09 05:51:04.749726 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:04.749565    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc975\" (UniqueName: \"kubernetes.io/projected/fbb14eb0-b938-4d9a-a16e-9c9857c978ef-kube-api-access-mc975\") pod \"node-exporter-lrzrl\" (UID: \"fbb14eb0-b938-4d9a-a16e-9c9857c978ef\") " pod="openshift-monitoring/node-exporter-lrzrl"
Sep 09 05:51:04.749726 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:51:04.749660    2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 09 05:51:04.749726 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:51:04.749720    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbb14eb0-b938-4d9a-a16e-9c9857c978ef-node-exporter-tls podName:fbb14eb0-b938-4d9a-a16e-9c9857c978ef nodeName:}" failed. No retries permitted until 2026-09-09 05:51:05.249699221 +0000 UTC m=+90.590187128 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/fbb14eb0-b938-4d9a-a16e-9c9857c978ef-node-exporter-tls") pod "node-exporter-lrzrl" (UID: "fbb14eb0-b938-4d9a-a16e-9c9857c978ef") : secret "node-exporter-tls" not found
Sep 09 05:51:04.750024 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:04.749767    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fbb14eb0-b938-4d9a-a16e-9c9857c978ef-node-exporter-textfile\") pod \"node-exporter-lrzrl\" (UID: \"fbb14eb0-b938-4d9a-a16e-9c9857c978ef\") " pod="openshift-monitoring/node-exporter-lrzrl"
Sep 09 05:51:04.750024 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:04.750009    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fbb14eb0-b938-4d9a-a16e-9c9857c978ef-metrics-client-ca\") pod \"node-exporter-lrzrl\" (UID: \"fbb14eb0-b938-4d9a-a16e-9c9857c978ef\") " pod="openshift-monitoring/node-exporter-lrzrl"
Sep 09 05:51:04.750097 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:04.750081    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fbb14eb0-b938-4d9a-a16e-9c9857c978ef-node-exporter-accelerators-collector-config\") pod \"node-exporter-lrzrl\" (UID: \"fbb14eb0-b938-4d9a-a16e-9c9857c978ef\") " pod="openshift-monitoring/node-exporter-lrzrl"
Sep 09 05:51:04.751793 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:04.751778    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fbb14eb0-b938-4d9a-a16e-9c9857c978ef-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lrzrl\" (UID: \"fbb14eb0-b938-4d9a-a16e-9c9857c978ef\") " pod="openshift-monitoring/node-exporter-lrzrl"
Sep 09 05:51:04.758925 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:04.758905    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc975\" (UniqueName: \"kubernetes.io/projected/fbb14eb0-b938-4d9a-a16e-9c9857c978ef-kube-api-access-mc975\") pod \"node-exporter-lrzrl\" (UID: \"fbb14eb0-b938-4d9a-a16e-9c9857c978ef\") " pod="openshift-monitoring/node-exporter-lrzrl"
Sep 09 05:51:05.252219 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:05.252176    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fbb14eb0-b938-4d9a-a16e-9c9857c978ef-node-exporter-tls\") pod \"node-exporter-lrzrl\" (UID: \"fbb14eb0-b938-4d9a-a16e-9c9857c978ef\") " pod="openshift-monitoring/node-exporter-lrzrl"
Sep 09 05:51:05.254593 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:05.254568    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fbb14eb0-b938-4d9a-a16e-9c9857c978ef-node-exporter-tls\") pod \"node-exporter-lrzrl\" (UID: \"fbb14eb0-b938-4d9a-a16e-9c9857c978ef\") " pod="openshift-monitoring/node-exporter-lrzrl"
Sep 09 05:51:05.506501 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:05.506426    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lrzrl"
Sep 09 05:51:05.518573 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:51:05.518544    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbb14eb0_b938_4d9a_a16e_9c9857c978ef.slice/crio-48bd82ea05fb728280320e4da34e38e7498c28d1ffe9d2945951d70cf0b08f6a WatchSource:0}: Error finding container 48bd82ea05fb728280320e4da34e38e7498c28d1ffe9d2945951d70cf0b08f6a: Status 404 returned error can't find the container with id 48bd82ea05fb728280320e4da34e38e7498c28d1ffe9d2945951d70cf0b08f6a
Sep 09 05:51:05.524161 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:05.524134    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lrzrl" event={"ID":"fbb14eb0-b938-4d9a-a16e-9c9857c978ef","Type":"ContainerStarted","Data":"48bd82ea05fb728280320e4da34e38e7498c28d1ffe9d2945951d70cf0b08f6a"}
Sep 09 05:51:07.532147 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:07.532112    2579 generic.go:356] "Generic (PLEG): container finished" podID="fbb14eb0-b938-4d9a-a16e-9c9857c978ef" containerID="b40b878c69b246e1df5b9aca44fb87de81d37ab9a4f34d867895658c77aa2207" exitCode=0
Sep 09 05:51:07.532501 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:07.532157    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lrzrl" event={"ID":"fbb14eb0-b938-4d9a-a16e-9c9857c978ef","Type":"ContainerDied","Data":"b40b878c69b246e1df5b9aca44fb87de81d37ab9a4f34d867895658c77aa2207"}
Sep 09 05:51:08.537602 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:08.537564    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lrzrl" event={"ID":"fbb14eb0-b938-4d9a-a16e-9c9857c978ef","Type":"ContainerStarted","Data":"dd897dab0db54a76dd5c2f8dc4b1b3227473880b463a075837b9e91e79a3b00a"}
Sep 09 05:51:08.537971 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:08.537650    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lrzrl" event={"ID":"fbb14eb0-b938-4d9a-a16e-9c9857c978ef","Type":"ContainerStarted","Data":"b2fb09dccc58361be74f855f609cddccb22a609ece9b8489f8503b1be5aa95cc"}
Sep 09 05:51:08.555207 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:08.555165    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lrzrl" podStartSLOduration=3.438966071 podStartE2EDuration="4.555151719s" totalImagesPullingTime="1.049320206s" totalInitContainerRuntime="66.865442ms" isStatefulPod=true podCreationTimestamp="2026-09-09 05:51:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:51:08.553949678 +0000 UTC m=+93.894437586" watchObservedRunningTime="2026-09-09 05:51:08.555151719 +0000 UTC m=+93.895639684"
Sep 09 05:51:08.962985 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:08.962958    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-649f6cfb6f-8c5s5"]
Sep 09 05:51:08.968567 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:08.968552    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-649f6cfb6f-8c5s5"
Sep 09 05:51:08.971449 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:08.971427    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 09 05:51:08.971550 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:08.971460    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 09 05:51:08.971550 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:08.971502    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-b6dgl\""
Sep 09 05:51:08.971655 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:08.971559    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-7kmeqlml0uaf7\""
Sep 09 05:51:08.971655 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:08.971585    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 09 05:51:08.971795 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:08.971782    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 09 05:51:08.981694 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:08.981674    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-649f6cfb6f-8c5s5"]
Sep 09 05:51:09.077479 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.077449    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/32c4b08b-c3e9-4c12-b35a-2b2ca5e59757-audit-log\") pod \"metrics-server-649f6cfb6f-8c5s5\" (UID: \"32c4b08b-c3e9-4c12-b35a-2b2ca5e59757\") " pod="openshift-monitoring/metrics-server-649f6cfb6f-8c5s5"
Sep 09 05:51:09.077649 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.077492    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxkvs\" (UniqueName: \"kubernetes.io/projected/32c4b08b-c3e9-4c12-b35a-2b2ca5e59757-kube-api-access-gxkvs\") pod \"metrics-server-649f6cfb6f-8c5s5\" (UID: \"32c4b08b-c3e9-4c12-b35a-2b2ca5e59757\") " pod="openshift-monitoring/metrics-server-649f6cfb6f-8c5s5"
Sep 09 05:51:09.077649 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.077537    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32c4b08b-c3e9-4c12-b35a-2b2ca5e59757-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-649f6cfb6f-8c5s5\" (UID: \"32c4b08b-c3e9-4c12-b35a-2b2ca5e59757\") " pod="openshift-monitoring/metrics-server-649f6cfb6f-8c5s5"
Sep 09 05:51:09.077649 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.077563    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/32c4b08b-c3e9-4c12-b35a-2b2ca5e59757-client-ca-bundle\") pod \"metrics-server-649f6cfb6f-8c5s5\" (UID: \"32c4b08b-c3e9-4c12-b35a-2b2ca5e59757\") " pod="openshift-monitoring/metrics-server-649f6cfb6f-8c5s5"
Sep 09 05:51:09.077649 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.077587    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/32c4b08b-c3e9-4c12-b35a-2b2ca5e59757-secret-metrics-server-tls\") pod \"metrics-server-649f6cfb6f-8c5s5\" (UID: \"32c4b08b-c3e9-4c12-b35a-2b2ca5e59757\") " pod="openshift-monitoring/metrics-server-649f6cfb6f-8c5s5"
Sep 09 05:51:09.077832 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.077668    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/32c4b08b-c3e9-4c12-b35a-2b2ca5e59757-metrics-server-audit-profiles\") pod \"metrics-server-649f6cfb6f-8c5s5\" (UID: \"32c4b08b-c3e9-4c12-b35a-2b2ca5e59757\") " pod="openshift-monitoring/metrics-server-649f6cfb6f-8c5s5"
Sep 09 05:51:09.077832 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.077708    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/32c4b08b-c3e9-4c12-b35a-2b2ca5e59757-secret-metrics-server-client-certs\") pod \"metrics-server-649f6cfb6f-8c5s5\" (UID: \"32c4b08b-c3e9-4c12-b35a-2b2ca5e59757\") " pod="openshift-monitoring/metrics-server-649f6cfb6f-8c5s5"
Sep 09 05:51:09.178242 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.178193    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32c4b08b-c3e9-4c12-b35a-2b2ca5e59757-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-649f6cfb6f-8c5s5\" (UID: \"32c4b08b-c3e9-4c12-b35a-2b2ca5e59757\") " pod="openshift-monitoring/metrics-server-649f6cfb6f-8c5s5"
Sep 09 05:51:09.178351 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.178253    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/32c4b08b-c3e9-4c12-b35a-2b2ca5e59757-client-ca-bundle\") pod \"metrics-server-649f6cfb6f-8c5s5\" (UID: \"32c4b08b-c3e9-4c12-b35a-2b2ca5e59757\") " pod="openshift-monitoring/metrics-server-649f6cfb6f-8c5s5"
Sep 09 05:51:09.178351 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.178280    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/32c4b08b-c3e9-4c12-b35a-2b2ca5e59757-secret-metrics-server-tls\") pod \"metrics-server-649f6cfb6f-8c5s5\" (UID: \"32c4b08b-c3e9-4c12-b35a-2b2ca5e59757\") " pod="openshift-monitoring/metrics-server-649f6cfb6f-8c5s5"
Sep 09 05:51:09.178351 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.178306    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/32c4b08b-c3e9-4c12-b35a-2b2ca5e59757-metrics-server-audit-profiles\") pod \"metrics-server-649f6cfb6f-8c5s5\" (UID: \"32c4b08b-c3e9-4c12-b35a-2b2ca5e59757\") " pod="openshift-monitoring/metrics-server-649f6cfb6f-8c5s5"
Sep 09 05:51:09.178351 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.178345    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/32c4b08b-c3e9-4c12-b35a-2b2ca5e59757-secret-metrics-server-client-certs\") pod \"metrics-server-649f6cfb6f-8c5s5\" (UID: \"32c4b08b-c3e9-4c12-b35a-2b2ca5e59757\") " pod="openshift-monitoring/metrics-server-649f6cfb6f-8c5s5"
Sep 09 05:51:09.178553 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.178371    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/32c4b08b-c3e9-4c12-b35a-2b2ca5e59757-audit-log\") pod \"metrics-server-649f6cfb6f-8c5s5\" (UID: \"32c4b08b-c3e9-4c12-b35a-2b2ca5e59757\") " pod="openshift-monitoring/metrics-server-649f6cfb6f-8c5s5"
Sep 09 05:51:09.178553 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.178407    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxkvs\" (UniqueName: \"kubernetes.io/projected/32c4b08b-c3e9-4c12-b35a-2b2ca5e59757-kube-api-access-gxkvs\") pod \"metrics-server-649f6cfb6f-8c5s5\" (UID: \"32c4b08b-c3e9-4c12-b35a-2b2ca5e59757\") " pod="openshift-monitoring/metrics-server-649f6cfb6f-8c5s5"
Sep 09 05:51:09.178862 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.178835    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/32c4b08b-c3e9-4c12-b35a-2b2ca5e59757-audit-log\") pod \"metrics-server-649f6cfb6f-8c5s5\" (UID: \"32c4b08b-c3e9-4c12-b35a-2b2ca5e59757\") " pod="openshift-monitoring/metrics-server-649f6cfb6f-8c5s5"
Sep 09 05:51:09.179013 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.178991    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32c4b08b-c3e9-4c12-b35a-2b2ca5e59757-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-649f6cfb6f-8c5s5\" (UID: \"32c4b08b-c3e9-4c12-b35a-2b2ca5e59757\") " pod="openshift-monitoring/metrics-server-649f6cfb6f-8c5s5"
Sep 09 05:51:09.179530 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.179506    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/32c4b08b-c3e9-4c12-b35a-2b2ca5e59757-metrics-server-audit-profiles\") pod \"metrics-server-649f6cfb6f-8c5s5\" (UID: \"32c4b08b-c3e9-4c12-b35a-2b2ca5e59757\") " pod="openshift-monitoring/metrics-server-649f6cfb6f-8c5s5"
Sep 09 05:51:09.180913 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.180884    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/32c4b08b-c3e9-4c12-b35a-2b2ca5e59757-secret-metrics-server-tls\") pod \"metrics-server-649f6cfb6f-8c5s5\" (UID: \"32c4b08b-c3e9-4c12-b35a-2b2ca5e59757\") " pod="openshift-monitoring/metrics-server-649f6cfb6f-8c5s5"
Sep 09 05:51:09.180995 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.180979    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/32c4b08b-c3e9-4c12-b35a-2b2ca5e59757-client-ca-bundle\") pod \"metrics-server-649f6cfb6f-8c5s5\" (UID: \"32c4b08b-c3e9-4c12-b35a-2b2ca5e59757\") " pod="openshift-monitoring/metrics-server-649f6cfb6f-8c5s5"
Sep 09 05:51:09.181141 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.181118    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/32c4b08b-c3e9-4c12-b35a-2b2ca5e59757-secret-metrics-server-client-certs\") pod \"metrics-server-649f6cfb6f-8c5s5\" (UID: \"32c4b08b-c3e9-4c12-b35a-2b2ca5e59757\") " pod="openshift-monitoring/metrics-server-649f6cfb6f-8c5s5"
Sep 09 05:51:09.192658 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.192618    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxkvs\" (UniqueName: \"kubernetes.io/projected/32c4b08b-c3e9-4c12-b35a-2b2ca5e59757-kube-api-access-gxkvs\") pod \"metrics-server-649f6cfb6f-8c5s5\" (UID: \"32c4b08b-c3e9-4c12-b35a-2b2ca5e59757\") " pod="openshift-monitoring/metrics-server-649f6cfb6f-8c5s5"
Sep 09 05:51:09.276935 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.276874    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-649f6cfb6f-8c5s5"
Sep 09 05:51:09.549004 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.548937    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-649f6cfb6f-8c5s5"]
Sep 09 05:51:09.551986 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:51:09.551957    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32c4b08b_c3e9_4c12_b35a_2b2ca5e59757.slice/crio-a9430ed8e45da62a69d76a2da76f7e02e0ce2c75d873efad68e2fd01a3caa755 WatchSource:0}: Error finding container a9430ed8e45da62a69d76a2da76f7e02e0ce2c75d873efad68e2fd01a3caa755: Status 404 returned error can't find the container with id a9430ed8e45da62a69d76a2da76f7e02e0ce2c75d873efad68e2fd01a3caa755
Sep 09 05:51:09.647771 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.647745    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7f6bf9cf8-zcbh7"]
Sep 09 05:51:09.653489 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.653473    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7f6bf9cf8-zcbh7"
Sep 09 05:51:09.655800 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.655773    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 09 05:51:09.655800 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.655776    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 09 05:51:09.655947 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.655815    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 09 05:51:09.656060 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.656046    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 09 05:51:09.656232 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.656220    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 09 05:51:09.656285 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.656271    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-2b5bq\""
Sep 09 05:51:09.661646 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.661611    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 09 05:51:09.668904 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.668883    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7f6bf9cf8-zcbh7"]
Sep 09 05:51:09.783597 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.783572    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ea34ee7-a545-437f-ab74-68f893f758ff-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f6bf9cf8-zcbh7\" (UID: \"1ea34ee7-a545-437f-ab74-68f893f758ff\") " pod="openshift-monitoring/telemeter-client-7f6bf9cf8-zcbh7"
Sep 09 05:51:09.783731 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.783604    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qwhp\" (UniqueName: \"kubernetes.io/projected/1ea34ee7-a545-437f-ab74-68f893f758ff-kube-api-access-7qwhp\") pod \"telemeter-client-7f6bf9cf8-zcbh7\" (UID: \"1ea34ee7-a545-437f-ab74-68f893f758ff\") " pod="openshift-monitoring/telemeter-client-7f6bf9cf8-zcbh7"
Sep 09 05:51:09.783731 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.783654    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ea34ee7-a545-437f-ab74-68f893f758ff-serving-certs-ca-bundle\") pod \"telemeter-client-7f6bf9cf8-zcbh7\" (UID: \"1ea34ee7-a545-437f-ab74-68f893f758ff\") " pod="openshift-monitoring/telemeter-client-7f6bf9cf8-zcbh7"
Sep 09 05:51:09.783731 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.783695    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1ea34ee7-a545-437f-ab74-68f893f758ff-telemeter-client-tls\") pod \"telemeter-client-7f6bf9cf8-zcbh7\" (UID: \"1ea34ee7-a545-437f-ab74-68f893f758ff\") " pod="openshift-monitoring/telemeter-client-7f6bf9cf8-zcbh7"
Sep 09 05:51:09.783731 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.783725    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ea34ee7-a545-437f-ab74-68f893f758ff-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f6bf9cf8-zcbh7\" (UID: \"1ea34ee7-a545-437f-ab74-68f893f758ff\") " pod="openshift-monitoring/telemeter-client-7f6bf9cf8-zcbh7"
Sep 09 05:51:09.783879 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.783752    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1ea34ee7-a545-437f-ab74-68f893f758ff-federate-client-tls\") pod \"telemeter-client-7f6bf9cf8-zcbh7\" (UID: \"1ea34ee7-a545-437f-ab74-68f893f758ff\") " pod="openshift-monitoring/telemeter-client-7f6bf9cf8-zcbh7"
Sep 09 05:51:09.783879 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.783776    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1ea34ee7-a545-437f-ab74-68f893f758ff-secret-telemeter-client\") pod \"telemeter-client-7f6bf9cf8-zcbh7\" (UID: \"1ea34ee7-a545-437f-ab74-68f893f758ff\") " pod="openshift-monitoring/telemeter-client-7f6bf9cf8-zcbh7"
Sep 09 05:51:09.783879 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.783798    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ea34ee7-a545-437f-ab74-68f893f758ff-metrics-client-ca\") pod \"telemeter-client-7f6bf9cf8-zcbh7\" (UID: \"1ea34ee7-a545-437f-ab74-68f893f758ff\") " pod="openshift-monitoring/telemeter-client-7f6bf9cf8-zcbh7"
Sep 09 05:51:09.884864 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.884840    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ea34ee7-a545-437f-ab74-68f893f758ff-metrics-client-ca\") pod \"telemeter-client-7f6bf9cf8-zcbh7\" (UID: \"1ea34ee7-a545-437f-ab74-68f893f758ff\") " pod="openshift-monitoring/telemeter-client-7f6bf9cf8-zcbh7"
Sep 09 05:51:09.884980 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.884872    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ea34ee7-a545-437f-ab74-68f893f758ff-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f6bf9cf8-zcbh7\" (UID: \"1ea34ee7-a545-437f-ab74-68f893f758ff\") " pod="openshift-monitoring/telemeter-client-7f6bf9cf8-zcbh7"
Sep 09 05:51:09.884980 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.884895    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qwhp\" (UniqueName: \"kubernetes.io/projected/1ea34ee7-a545-437f-ab74-68f893f758ff-kube-api-access-7qwhp\") pod \"telemeter-client-7f6bf9cf8-zcbh7\" (UID: \"1ea34ee7-a545-437f-ab74-68f893f758ff\") " pod="openshift-monitoring/telemeter-client-7f6bf9cf8-zcbh7"
Sep 09 05:51:09.885075 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.885025    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ea34ee7-a545-437f-ab74-68f893f758ff-serving-certs-ca-bundle\") pod \"telemeter-client-7f6bf9cf8-zcbh7\" (UID: \"1ea34ee7-a545-437f-ab74-68f893f758ff\") " pod="openshift-monitoring/telemeter-client-7f6bf9cf8-zcbh7"
Sep 09 05:51:09.885075 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.885062    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1ea34ee7-a545-437f-ab74-68f893f758ff-telemeter-client-tls\") pod \"telemeter-client-7f6bf9cf8-zcbh7\" (UID: \"1ea34ee7-a545-437f-ab74-68f893f758ff\") " pod="openshift-monitoring/telemeter-client-7f6bf9cf8-zcbh7"
Sep 09 05:51:09.885163 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.885099    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ea34ee7-a545-437f-ab74-68f893f758ff-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f6bf9cf8-zcbh7\" (UID: \"1ea34ee7-a545-437f-ab74-68f893f758ff\") " pod="openshift-monitoring/telemeter-client-7f6bf9cf8-zcbh7"
Sep 09 05:51:09.885163 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.885134    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1ea34ee7-a545-437f-ab74-68f893f758ff-federate-client-tls\") pod \"telemeter-client-7f6bf9cf8-zcbh7\" (UID: \"1ea34ee7-a545-437f-ab74-68f893f758ff\") " pod="openshift-monitoring/telemeter-client-7f6bf9cf8-zcbh7"
Sep 09 05:51:09.885163 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.885150    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1ea34ee7-a545-437f-ab74-68f893f758ff-secret-telemeter-client\") pod \"telemeter-client-7f6bf9cf8-zcbh7\" (UID: \"1ea34ee7-a545-437f-ab74-68f893f758ff\") " pod="openshift-monitoring/telemeter-client-7f6bf9cf8-zcbh7"
Sep 09 05:51:09.885701 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.885669    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ea34ee7-a545-437f-ab74-68f893f758ff-metrics-client-ca\") pod \"telemeter-client-7f6bf9cf8-zcbh7\" (UID: \"1ea34ee7-a545-437f-ab74-68f893f758ff\") " pod="openshift-monitoring/telemeter-client-7f6bf9cf8-zcbh7"
Sep 09 05:51:09.885978 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.885791    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ea34ee7-a545-437f-ab74-68f893f758ff-serving-certs-ca-bundle\") pod \"telemeter-client-7f6bf9cf8-zcbh7\" (UID: \"1ea34ee7-a545-437f-ab74-68f893f758ff\") " pod="openshift-monitoring/telemeter-client-7f6bf9cf8-zcbh7"
Sep 09 05:51:09.885978 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.885914    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ea34ee7-a545-437f-ab74-68f893f758ff-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f6bf9cf8-zcbh7\" (UID: \"1ea34ee7-a545-437f-ab74-68f893f758ff\") " pod="openshift-monitoring/telemeter-client-7f6bf9cf8-zcbh7"
Sep 09 05:51:09.887681 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.887655    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1ea34ee7-a545-437f-ab74-68f893f758ff-secret-telemeter-client\") pod \"telemeter-client-7f6bf9cf8-zcbh7\" (UID: \"1ea34ee7-a545-437f-ab74-68f893f758ff\") " pod="openshift-monitoring/telemeter-client-7f6bf9cf8-zcbh7"
Sep 09 05:51:09.887744 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.887684    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1ea34ee7-a545-437f-ab74-68f893f758ff-federate-client-tls\") pod \"telemeter-client-7f6bf9cf8-zcbh7\" (UID: \"1ea34ee7-a545-437f-ab74-68f893f758ff\") " pod="openshift-monitoring/telemeter-client-7f6bf9cf8-zcbh7"
Sep 09 05:51:09.888069 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.888049    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ea34ee7-a545-437f-ab74-68f893f758ff-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f6bf9cf8-zcbh7\" (UID: \"1ea34ee7-a545-437f-ab74-68f893f758ff\") " pod="openshift-monitoring/telemeter-client-7f6bf9cf8-zcbh7"
Sep 09 05:51:09.888105 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.888060    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1ea34ee7-a545-437f-ab74-68f893f758ff-telemeter-client-tls\") pod \"telemeter-client-7f6bf9cf8-zcbh7\" (UID: \"1ea34ee7-a545-437f-ab74-68f893f758ff\") " pod="openshift-monitoring/telemeter-client-7f6bf9cf8-zcbh7"
Sep 09 05:51:09.894806 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.894788    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qwhp\" (UniqueName: \"kubernetes.io/projected/1ea34ee7-a545-437f-ab74-68f893f758ff-kube-api-access-7qwhp\") pod \"telemeter-client-7f6bf9cf8-zcbh7\" (UID: \"1ea34ee7-a545-437f-ab74-68f893f758ff\") " pod="openshift-monitoring/telemeter-client-7f6bf9cf8-zcbh7"
Sep 09 05:51:09.962725 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:09.962704    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7f6bf9cf8-zcbh7"
Sep 09 05:51:10.243052 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:10.242990    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7f6bf9cf8-zcbh7"]
Sep 09 05:51:10.246056 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:51:10.246029    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ea34ee7_a545_437f_ab74_68f893f758ff.slice/crio-b40ff3ef5d010055d4a272a99e475b3fa5a0ed16271b4edbd91ea867afe24e0f WatchSource:0}: Error finding container b40ff3ef5d010055d4a272a99e475b3fa5a0ed16271b4edbd91ea867afe24e0f: Status 404 returned error can't find the container with id b40ff3ef5d010055d4a272a99e475b3fa5a0ed16271b4edbd91ea867afe24e0f
Sep 09 05:51:10.545966 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:10.545890    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f6bf9cf8-zcbh7" event={"ID":"1ea34ee7-a545-437f-ab74-68f893f758ff","Type":"ContainerStarted","Data":"b40ff3ef5d010055d4a272a99e475b3fa5a0ed16271b4edbd91ea867afe24e0f"}
Sep 09 05:51:10.547094 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:10.547072    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-649f6cfb6f-8c5s5" event={"ID":"32c4b08b-c3e9-4c12-b35a-2b2ca5e59757","Type":"ContainerStarted","Data":"a9430ed8e45da62a69d76a2da76f7e02e0ce2c75d873efad68e2fd01a3caa755"}
Sep 09 05:51:12.557097 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:12.557008    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f6bf9cf8-zcbh7" event={"ID":"1ea34ee7-a545-437f-ab74-68f893f758ff","Type":"ContainerStarted","Data":"8bf92b93afa3975b81248d4f7fe54edf84aa448b9c1559466813f7b3e8cb76f2"}
Sep 09 05:51:13.563984 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:13.563947    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f6bf9cf8-zcbh7" event={"ID":"1ea34ee7-a545-437f-ab74-68f893f758ff","Type":"ContainerStarted","Data":"7fcd02e66e8b0a2482ba6eceab85cd7ac05847625ab4655437075ecfbb365168"}
Sep 09 05:51:13.564388 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:13.564017    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f6bf9cf8-zcbh7" event={"ID":"1ea34ee7-a545-437f-ab74-68f893f758ff","Type":"ContainerStarted","Data":"2dd1123135b8e91dfef1747fe4d62118cdf61334920c2248e913ac503141100e"}
Sep 09 05:51:13.588832 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:13.588780    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7f6bf9cf8-zcbh7" podStartSLOduration=1.867366478 podStartE2EDuration="4.588761506s" totalImagesPullingTime="2.721395028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:51:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:51:13.586978363 +0000 UTC m=+98.927466273" watchObservedRunningTime="2026-09-09 05:51:13.588761506 +0000 UTC m=+98.929249417"
Sep 09 05:51:14.569870 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:14.569833    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-649f6cfb6f-8c5s5" event={"ID":"32c4b08b-c3e9-4c12-b35a-2b2ca5e59757","Type":"ContainerStarted","Data":"5905cd798bc5951b70d3af0cd548a859a58a0ac2a8ce6778ef5d00815277a442"}
Sep 09 05:51:14.585852 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:14.585802    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-649f6cfb6f-8c5s5" podStartSLOduration=2.141397643 podStartE2EDuration="6.585783679s" totalImagesPullingTime="4.444386036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:51:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:51:14.585198274 +0000 UTC m=+99.925686184" watchObservedRunningTime="2026-09-09 05:51:14.585783679 +0000 UTC m=+99.926271590"
Sep 09 05:51:21.482604 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:21.482577    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-b78pk"
Sep 09 05:51:29.277911 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:29.277875    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-649f6cfb6f-8c5s5"
Sep 09 05:51:29.278277 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:29.277932    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-649f6cfb6f-8c5s5"
Sep 09 05:51:49.283175 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:49.283144    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-649f6cfb6f-8c5s5"
Sep 09 05:51:49.287040 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:51:49.287013    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-649f6cfb6f-8c5s5"
Sep 09 05:54:16.012567 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:16.012528    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-nqlt2"]
Sep 09 05:54:16.015653 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:16.015612    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-nqlt2"
Sep 09 05:54:16.017549 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:16.017523    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 09 05:54:16.017691 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:16.017640    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 09 05:54:16.017691 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:16.017649    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-jn6zb\""
Sep 09 05:54:16.022553 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:16.022535    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-nqlt2"]
Sep 09 05:54:16.214993 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:16.214957    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/093ecefc-89d1-4f16-bcb5-a0941dc3622e-bound-sa-token\") pod \"cert-manager-55b4d954d5-nqlt2\" (UID: \"093ecefc-89d1-4f16-bcb5-a0941dc3622e\") " pod="cert-manager/cert-manager-55b4d954d5-nqlt2"
Sep 09 05:54:16.215171 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:16.215008    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djxqk\" (UniqueName: \"kubernetes.io/projected/093ecefc-89d1-4f16-bcb5-a0941dc3622e-kube-api-access-djxqk\") pod \"cert-manager-55b4d954d5-nqlt2\" (UID: \"093ecefc-89d1-4f16-bcb5-a0941dc3622e\") " pod="cert-manager/cert-manager-55b4d954d5-nqlt2"
Sep 09 05:54:16.315340 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:16.315316    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djxqk\" (UniqueName: \"kubernetes.io/projected/093ecefc-89d1-4f16-bcb5-a0941dc3622e-kube-api-access-djxqk\") pod \"cert-manager-55b4d954d5-nqlt2\" (UID: \"093ecefc-89d1-4f16-bcb5-a0941dc3622e\") " pod="cert-manager/cert-manager-55b4d954d5-nqlt2"
Sep 09 05:54:16.315471 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:16.315366    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/093ecefc-89d1-4f16-bcb5-a0941dc3622e-bound-sa-token\") pod \"cert-manager-55b4d954d5-nqlt2\" (UID: \"093ecefc-89d1-4f16-bcb5-a0941dc3622e\") " pod="cert-manager/cert-manager-55b4d954d5-nqlt2"
Sep 09 05:54:16.323309 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:16.323285    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/093ecefc-89d1-4f16-bcb5-a0941dc3622e-bound-sa-token\") pod \"cert-manager-55b4d954d5-nqlt2\" (UID: \"093ecefc-89d1-4f16-bcb5-a0941dc3622e\") " pod="cert-manager/cert-manager-55b4d954d5-nqlt2"
Sep 09 05:54:16.323309 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:16.323304    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djxqk\" (UniqueName: \"kubernetes.io/projected/093ecefc-89d1-4f16-bcb5-a0941dc3622e-kube-api-access-djxqk\") pod \"cert-manager-55b4d954d5-nqlt2\" (UID: \"093ecefc-89d1-4f16-bcb5-a0941dc3622e\") " pod="cert-manager/cert-manager-55b4d954d5-nqlt2"
Sep 09 05:54:16.324167 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:16.324150    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-nqlt2"
Sep 09 05:54:16.592990 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:54:16.592962    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod093ecefc_89d1_4f16_bcb5_a0941dc3622e.slice/crio-e32dbce07dfbea6770315bd55ef630b8d509a6ace30d8634e21fed7845cc2766 WatchSource:0}: Error finding container e32dbce07dfbea6770315bd55ef630b8d509a6ace30d8634e21fed7845cc2766: Status 404 returned error can't find the container with id e32dbce07dfbea6770315bd55ef630b8d509a6ace30d8634e21fed7845cc2766
Sep 09 05:54:16.596797 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:16.596774    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-nqlt2"]
Sep 09 05:54:17.156211 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:17.156163    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-nqlt2" event={"ID":"093ecefc-89d1-4f16-bcb5-a0941dc3622e","Type":"ContainerStarted","Data":"e32dbce07dfbea6770315bd55ef630b8d509a6ace30d8634e21fed7845cc2766"}
Sep 09 05:54:20.168063 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:20.168024    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-nqlt2" event={"ID":"093ecefc-89d1-4f16-bcb5-a0941dc3622e","Type":"ContainerStarted","Data":"9cdbf7248e6e2cb02cfdc06243621b40612a10228006671e43296a33ae6ada87"}
Sep 09 05:54:20.183946 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:20.183899    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-nqlt2" podStartSLOduration=2.366964986 podStartE2EDuration="5.183885124s" totalImagesPullingTime="2.816920138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:54:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:54:20.183224146 +0000 UTC m=+285.523712056" watchObservedRunningTime="2026-09-09 05:54:20.183885124 +0000 UTC m=+285.524373033"
Sep 09 05:54:35.089426 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:35.089385    2579 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 05:54:42.287662 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:42.287594    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76b47fccc7-f4lfj"]
Sep 09 05:54:42.290955 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:42.290938    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-f4lfj"
Sep 09 05:54:42.294456 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:42.294431    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 09 05:54:42.294583 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:42.294562    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 09 05:54:42.294677 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:42.294660    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 09 05:54:42.294744 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:42.294688    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 09 05:54:42.294926 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:42.294911    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-xfnz9\""
Sep 09 05:54:42.304514 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:42.304494    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76b47fccc7-f4lfj"]
Sep 09 05:54:42.386361 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:42.386334    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tmj2\" (UniqueName: \"kubernetes.io/projected/284a6006-17ba-4ed4-ab73-173bf23c0eae-kube-api-access-2tmj2\") pod \"opendatahub-operator-controller-manager-76b47fccc7-f4lfj\" (UID: \"284a6006-17ba-4ed4-ab73-173bf23c0eae\") " pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-f4lfj"
Sep 09 05:54:42.386481 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:42.386386    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/284a6006-17ba-4ed4-ab73-173bf23c0eae-tmp\") pod \"opendatahub-operator-controller-manager-76b47fccc7-f4lfj\" (UID: \"284a6006-17ba-4ed4-ab73-173bf23c0eae\") " pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-f4lfj"
Sep 09 05:54:42.386481 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:42.386409    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/284a6006-17ba-4ed4-ab73-173bf23c0eae-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76b47fccc7-f4lfj\" (UID: \"284a6006-17ba-4ed4-ab73-173bf23c0eae\") " pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-f4lfj"
Sep 09 05:54:42.386481 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:42.386447    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/284a6006-17ba-4ed4-ab73-173bf23c0eae-manifests\") pod \"opendatahub-operator-controller-manager-76b47fccc7-f4lfj\" (UID: \"284a6006-17ba-4ed4-ab73-173bf23c0eae\") " pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-f4lfj"
Sep 09 05:54:42.386481 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:42.386467    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/284a6006-17ba-4ed4-ab73-173bf23c0eae-webhook-cert\") pod \"opendatahub-operator-controller-manager-76b47fccc7-f4lfj\" (UID: \"284a6006-17ba-4ed4-ab73-173bf23c0eae\") " pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-f4lfj"
Sep 09 05:54:42.487607 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:42.487579    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/284a6006-17ba-4ed4-ab73-173bf23c0eae-manifests\") pod \"opendatahub-operator-controller-manager-76b47fccc7-f4lfj\" (UID: \"284a6006-17ba-4ed4-ab73-173bf23c0eae\") " pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-f4lfj"
Sep 09 05:54:42.487749 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:42.487615    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/284a6006-17ba-4ed4-ab73-173bf23c0eae-webhook-cert\") pod \"opendatahub-operator-controller-manager-76b47fccc7-f4lfj\" (UID: \"284a6006-17ba-4ed4-ab73-173bf23c0eae\") " pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-f4lfj"
Sep 09 05:54:42.487749 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:42.487657    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tmj2\" (UniqueName: \"kubernetes.io/projected/284a6006-17ba-4ed4-ab73-173bf23c0eae-kube-api-access-2tmj2\") pod \"opendatahub-operator-controller-manager-76b47fccc7-f4lfj\" (UID: \"284a6006-17ba-4ed4-ab73-173bf23c0eae\") " pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-f4lfj"
Sep 09 05:54:42.487749 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:42.487703    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/284a6006-17ba-4ed4-ab73-173bf23c0eae-tmp\") pod \"opendatahub-operator-controller-manager-76b47fccc7-f4lfj\" (UID: \"284a6006-17ba-4ed4-ab73-173bf23c0eae\") " pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-f4lfj"
Sep 09 05:54:42.487749 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:42.487723    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/284a6006-17ba-4ed4-ab73-173bf23c0eae-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76b47fccc7-f4lfj\" (UID: \"284a6006-17ba-4ed4-ab73-173bf23c0eae\") " pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-f4lfj"
Sep 09 05:54:42.488016 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:42.487994    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/284a6006-17ba-4ed4-ab73-173bf23c0eae-manifests\") pod \"opendatahub-operator-controller-manager-76b47fccc7-f4lfj\" (UID: \"284a6006-17ba-4ed4-ab73-173bf23c0eae\") " pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-f4lfj"
Sep 09 05:54:42.490021 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:42.489990    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/284a6006-17ba-4ed4-ab73-173bf23c0eae-tmp\") pod \"opendatahub-operator-controller-manager-76b47fccc7-f4lfj\" (UID: \"284a6006-17ba-4ed4-ab73-173bf23c0eae\") " pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-f4lfj"
Sep 09 05:54:42.490135 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:42.490030    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/284a6006-17ba-4ed4-ab73-173bf23c0eae-webhook-cert\") pod \"opendatahub-operator-controller-manager-76b47fccc7-f4lfj\" (UID: \"284a6006-17ba-4ed4-ab73-173bf23c0eae\") " pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-f4lfj"
Sep 09 05:54:42.490135 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:42.490115    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/284a6006-17ba-4ed4-ab73-173bf23c0eae-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76b47fccc7-f4lfj\" (UID: \"284a6006-17ba-4ed4-ab73-173bf23c0eae\") " pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-f4lfj"
Sep 09 05:54:42.494834 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:42.494816    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tmj2\" (UniqueName: \"kubernetes.io/projected/284a6006-17ba-4ed4-ab73-173bf23c0eae-kube-api-access-2tmj2\") pod \"opendatahub-operator-controller-manager-76b47fccc7-f4lfj\" (UID: \"284a6006-17ba-4ed4-ab73-173bf23c0eae\") " pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-f4lfj"
Sep 09 05:54:42.600231 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:42.600199    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-f4lfj"
Sep 09 05:54:42.881501 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:42.881429    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76b47fccc7-f4lfj"]
Sep 09 05:54:42.885000 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:54:42.884977    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod284a6006_17ba_4ed4_ab73_173bf23c0eae.slice/crio-eae9fdafcd508ef5c23b4c2a8700663dab562b02124cb8aac6f7b8c9becd3cbe WatchSource:0}: Error finding container eae9fdafcd508ef5c23b4c2a8700663dab562b02124cb8aac6f7b8c9becd3cbe: Status 404 returned error can't find the container with id eae9fdafcd508ef5c23b4c2a8700663dab562b02124cb8aac6f7b8c9becd3cbe
Sep 09 05:54:42.886794 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:42.886775    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 05:54:43.241808 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:43.241726    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-f4lfj" event={"ID":"284a6006-17ba-4ed4-ab73-173bf23c0eae","Type":"ContainerStarted","Data":"eae9fdafcd508ef5c23b4c2a8700663dab562b02124cb8aac6f7b8c9becd3cbe"}
Sep 09 05:54:46.254916 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:46.254881    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-f4lfj" event={"ID":"284a6006-17ba-4ed4-ab73-173bf23c0eae","Type":"ContainerStarted","Data":"e1b1974002378defc9cd2a3a2fc3a5b9602147a5682dee8cebbaaa8fd8efd9b0"}
Sep 09 05:54:47.259444 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:47.259355    2579 generic.go:356] "Generic (PLEG): container finished" podID="284a6006-17ba-4ed4-ab73-173bf23c0eae" containerID="e1b1974002378defc9cd2a3a2fc3a5b9602147a5682dee8cebbaaa8fd8efd9b0" exitCode=0
Sep 09 05:54:47.259444 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:47.259405    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-f4lfj" event={"ID":"284a6006-17ba-4ed4-ab73-173bf23c0eae","Type":"ContainerDied","Data":"e1b1974002378defc9cd2a3a2fc3a5b9602147a5682dee8cebbaaa8fd8efd9b0"}
Sep 09 05:54:48.264811 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:48.264768    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-f4lfj" event={"ID":"284a6006-17ba-4ed4-ab73-173bf23c0eae","Type":"ContainerStarted","Data":"7b2b00db8c902ba0db6c8c6ec10d6aae0986f2002d8e65467fdfcbfc3a473ce0"}
Sep 09 05:54:48.265202 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:48.264885    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-f4lfj"
Sep 09 05:54:48.285093 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:48.285031    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-f4lfj" podStartSLOduration=2.116393446 podStartE2EDuration="6.285011317s" totalImagesPullingTime="3.161079352s" totalInitContainerRuntime="1.007538519s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:54:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:54:48.283200598 +0000 UTC m=+313.623688507" watchObservedRunningTime="2026-09-09 05:54:48.285011317 +0000 UTC m=+313.625499226"
Sep 09 05:54:59.270739 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:54:59.270711    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-76b47fccc7-f4lfj"
Sep 09 05:55:01.735183 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:01.735147    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7f4d448d7f-58zzf"]
Sep 09 05:55:01.738303 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:01.738287    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7f4d448d7f-58zzf"
Sep 09 05:55:01.740666 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:01.740644    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 09 05:55:01.740878 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:01.740796    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 09 05:55:01.740878 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:01.740811    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 05:55:01.740878 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:01.740817    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 05:55:01.741060 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:01.740930    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-zv7bh\""
Sep 09 05:55:01.751359 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:01.751336    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7f4d448d7f-58zzf"]
Sep 09 05:55:01.835239 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:01.835210    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf900674-4feb-4a9d-aed9-ed685de58c8f-tmp\") pod \"kube-auth-proxy-7f4d448d7f-58zzf\" (UID: \"bf900674-4feb-4a9d-aed9-ed685de58c8f\") " pod="openshift-ingress/kube-auth-proxy-7f4d448d7f-58zzf"
Sep 09 05:55:01.835352 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:01.835269    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf900674-4feb-4a9d-aed9-ed685de58c8f-tls-certs\") pod \"kube-auth-proxy-7f4d448d7f-58zzf\" (UID: \"bf900674-4feb-4a9d-aed9-ed685de58c8f\") " pod="openshift-ingress/kube-auth-proxy-7f4d448d7f-58zzf"
Sep 09 05:55:01.835406 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:01.835356    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcsrk\" (UniqueName: \"kubernetes.io/projected/bf900674-4feb-4a9d-aed9-ed685de58c8f-kube-api-access-bcsrk\") pod \"kube-auth-proxy-7f4d448d7f-58zzf\" (UID: \"bf900674-4feb-4a9d-aed9-ed685de58c8f\") " pod="openshift-ingress/kube-auth-proxy-7f4d448d7f-58zzf"
Sep 09 05:55:01.935819 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:01.935790    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcsrk\" (UniqueName: \"kubernetes.io/projected/bf900674-4feb-4a9d-aed9-ed685de58c8f-kube-api-access-bcsrk\") pod \"kube-auth-proxy-7f4d448d7f-58zzf\" (UID: \"bf900674-4feb-4a9d-aed9-ed685de58c8f\") " pod="openshift-ingress/kube-auth-proxy-7f4d448d7f-58zzf"
Sep 09 05:55:01.935927 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:01.935834    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf900674-4feb-4a9d-aed9-ed685de58c8f-tmp\") pod \"kube-auth-proxy-7f4d448d7f-58zzf\" (UID: \"bf900674-4feb-4a9d-aed9-ed685de58c8f\") " pod="openshift-ingress/kube-auth-proxy-7f4d448d7f-58zzf"
Sep 09 05:55:01.935927 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:01.935871    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf900674-4feb-4a9d-aed9-ed685de58c8f-tls-certs\") pod \"kube-auth-proxy-7f4d448d7f-58zzf\" (UID: \"bf900674-4feb-4a9d-aed9-ed685de58c8f\") " pod="openshift-ingress/kube-auth-proxy-7f4d448d7f-58zzf"
Sep 09 05:55:01.938316 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:01.938294    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf900674-4feb-4a9d-aed9-ed685de58c8f-tmp\") pod \"kube-auth-proxy-7f4d448d7f-58zzf\" (UID: \"bf900674-4feb-4a9d-aed9-ed685de58c8f\") " pod="openshift-ingress/kube-auth-proxy-7f4d448d7f-58zzf"
Sep 09 05:55:01.938413 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:01.938325    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf900674-4feb-4a9d-aed9-ed685de58c8f-tls-certs\") pod \"kube-auth-proxy-7f4d448d7f-58zzf\" (UID: \"bf900674-4feb-4a9d-aed9-ed685de58c8f\") " pod="openshift-ingress/kube-auth-proxy-7f4d448d7f-58zzf"
Sep 09 05:55:01.946212 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:01.946192    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcsrk\" (UniqueName: \"kubernetes.io/projected/bf900674-4feb-4a9d-aed9-ed685de58c8f-kube-api-access-bcsrk\") pod \"kube-auth-proxy-7f4d448d7f-58zzf\" (UID: \"bf900674-4feb-4a9d-aed9-ed685de58c8f\") " pod="openshift-ingress/kube-auth-proxy-7f4d448d7f-58zzf"
Sep 09 05:55:02.048194 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:02.048130    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7f4d448d7f-58zzf"
Sep 09 05:55:02.324748 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:02.324605    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7f4d448d7f-58zzf"]
Sep 09 05:55:02.327119 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:55:02.327084    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf900674_4feb_4a9d_aed9_ed685de58c8f.slice/crio-b866ff8742f4528251c2ed060e0ae4261ccb80c95ac414aaee9bf2d6b6349e38 WatchSource:0}: Error finding container b866ff8742f4528251c2ed060e0ae4261ccb80c95ac414aaee9bf2d6b6349e38: Status 404 returned error can't find the container with id b866ff8742f4528251c2ed060e0ae4261ccb80c95ac414aaee9bf2d6b6349e38
Sep 09 05:55:03.137327 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:03.136681    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfl9w"]
Sep 09 05:55:03.140316 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:03.140279    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfl9w"
Sep 09 05:55:03.142560 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:03.142536    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 09 05:55:03.142763 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:03.142716    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 09 05:55:03.142893 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:03.142876    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-njpwm\""
Sep 09 05:55:03.142964 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:03.142923    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 09 05:55:03.143079 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:03.143063    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 09 05:55:03.151307 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:03.150611    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfl9w"]
Sep 09 05:55:03.243581 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:03.243533    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/45d00781-7997-48b5-97f1-c3434738a214-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-cfl9w\" (UID: \"45d00781-7997-48b5-97f1-c3434738a214\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfl9w"
Sep 09 05:55:03.243801 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:03.243605    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/45d00781-7997-48b5-97f1-c3434738a214-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-cfl9w\" (UID: \"45d00781-7997-48b5-97f1-c3434738a214\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfl9w"
Sep 09 05:55:03.243801 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:03.243655    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/45d00781-7997-48b5-97f1-c3434738a214-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-cfl9w\" (UID: \"45d00781-7997-48b5-97f1-c3434738a214\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfl9w"
Sep 09 05:55:03.243801 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:03.243691    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d00781-7997-48b5-97f1-c3434738a214-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-cfl9w\" (UID: \"45d00781-7997-48b5-97f1-c3434738a214\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfl9w"
Sep 09 05:55:03.243801 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:03.243710    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/45d00781-7997-48b5-97f1-c3434738a214-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-cfl9w\" (UID: \"45d00781-7997-48b5-97f1-c3434738a214\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfl9w"
Sep 09 05:55:03.243801 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:03.243751    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45d00781-7997-48b5-97f1-c3434738a214-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-cfl9w\" (UID: \"45d00781-7997-48b5-97f1-c3434738a214\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfl9w"
Sep 09 05:55:03.243801 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:03.243773    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c56kn\" (UniqueName: \"kubernetes.io/projected/45d00781-7997-48b5-97f1-c3434738a214-kube-api-access-c56kn\") pod \"istiod-openshift-gateway-7b7995d5cd-cfl9w\" (UID: \"45d00781-7997-48b5-97f1-c3434738a214\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfl9w"
Sep 09 05:55:03.314804 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:03.314766    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7f4d448d7f-58zzf" event={"ID":"bf900674-4feb-4a9d-aed9-ed685de58c8f","Type":"ContainerStarted","Data":"b866ff8742f4528251c2ed060e0ae4261ccb80c95ac414aaee9bf2d6b6349e38"}
Sep 09 05:55:03.344232 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:03.344192    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/45d00781-7997-48b5-97f1-c3434738a214-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-cfl9w\" (UID: \"45d00781-7997-48b5-97f1-c3434738a214\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfl9w"
Sep 09 05:55:03.344232 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:03.344234    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/45d00781-7997-48b5-97f1-c3434738a214-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-cfl9w\" (UID: \"45d00781-7997-48b5-97f1-c3434738a214\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfl9w"
Sep 09 05:55:03.344427 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:03.344268    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d00781-7997-48b5-97f1-c3434738a214-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-cfl9w\" (UID: \"45d00781-7997-48b5-97f1-c3434738a214\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfl9w"
Sep 09 05:55:03.344427 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:03.344289    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/45d00781-7997-48b5-97f1-c3434738a214-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-cfl9w\" (UID: \"45d00781-7997-48b5-97f1-c3434738a214\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfl9w"
Sep 09 05:55:03.344427 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:03.344379    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45d00781-7997-48b5-97f1-c3434738a214-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-cfl9w\" (UID: \"45d00781-7997-48b5-97f1-c3434738a214\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfl9w"
Sep 09 05:55:03.344427 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:03.344406    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c56kn\" (UniqueName: \"kubernetes.io/projected/45d00781-7997-48b5-97f1-c3434738a214-kube-api-access-c56kn\") pod \"istiod-openshift-gateway-7b7995d5cd-cfl9w\" (UID: \"45d00781-7997-48b5-97f1-c3434738a214\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfl9w"
Sep 09 05:55:03.344660 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:03.344522    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/45d00781-7997-48b5-97f1-c3434738a214-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-cfl9w\" (UID: \"45d00781-7997-48b5-97f1-c3434738a214\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfl9w"
Sep 09 05:55:03.345383 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:03.345357    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/45d00781-7997-48b5-97f1-c3434738a214-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-cfl9w\" (UID: \"45d00781-7997-48b5-97f1-c3434738a214\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfl9w"
Sep 09 05:55:03.348290 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:03.348106    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/45d00781-7997-48b5-97f1-c3434738a214-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-cfl9w\" (UID: \"45d00781-7997-48b5-97f1-c3434738a214\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfl9w"
Sep 09 05:55:03.348773 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:03.348748    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d00781-7997-48b5-97f1-c3434738a214-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-cfl9w\" (UID: \"45d00781-7997-48b5-97f1-c3434738a214\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfl9w"
Sep 09 05:55:03.349277 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:03.349226    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/45d00781-7997-48b5-97f1-c3434738a214-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-cfl9w\" (UID: \"45d00781-7997-48b5-97f1-c3434738a214\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfl9w"
Sep 09 05:55:03.349375 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:03.349301    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/45d00781-7997-48b5-97f1-c3434738a214-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-cfl9w\" (UID: \"45d00781-7997-48b5-97f1-c3434738a214\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfl9w"
Sep 09 05:55:03.353353 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:03.353328    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45d00781-7997-48b5-97f1-c3434738a214-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-cfl9w\" (UID: \"45d00781-7997-48b5-97f1-c3434738a214\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfl9w"
Sep 09 05:55:03.353450 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:03.353413    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c56kn\" (UniqueName: \"kubernetes.io/projected/45d00781-7997-48b5-97f1-c3434738a214-kube-api-access-c56kn\") pod \"istiod-openshift-gateway-7b7995d5cd-cfl9w\" (UID: \"45d00781-7997-48b5-97f1-c3434738a214\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfl9w"
Sep 09 05:55:03.454251 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:03.454179    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfl9w"
Sep 09 05:55:03.738493 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:03.738456    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfl9w"]
Sep 09 05:55:05.018304 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:55:05.018265    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45d00781_7997_48b5_97f1_c3434738a214.slice/crio-084b4e5227a466240c07cfb4398b5fccf6ad94bf32119f68a8dc651756abc6cb WatchSource:0}: Error finding container 084b4e5227a466240c07cfb4398b5fccf6ad94bf32119f68a8dc651756abc6cb: Status 404 returned error can't find the container with id 084b4e5227a466240c07cfb4398b5fccf6ad94bf32119f68a8dc651756abc6cb
Sep 09 05:55:05.324926 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:05.324886    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfl9w" event={"ID":"45d00781-7997-48b5-97f1-c3434738a214","Type":"ContainerStarted","Data":"084b4e5227a466240c07cfb4398b5fccf6ad94bf32119f68a8dc651756abc6cb"}
Sep 09 05:55:05.326343 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:05.326321    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7f4d448d7f-58zzf" event={"ID":"bf900674-4feb-4a9d-aed9-ed685de58c8f","Type":"ContainerStarted","Data":"85d617afe75aa380b686d8cbcae799245b939d26540f73d9d8fc95091c0c5ea9"}
Sep 09 05:55:05.344852 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:05.344805    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7f4d448d7f-58zzf" podStartSLOduration=1.6022411760000002 podStartE2EDuration="4.344791058s" totalImagesPullingTime="2.742549882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:55:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:55:05.343444657 +0000 UTC m=+330.683932566" watchObservedRunningTime="2026-09-09 05:55:05.344791058 +0000 UTC m=+330.685278967"
Sep 09 05:55:07.713824 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:07.713783    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Sep 09 05:55:07.714181 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:07.713866    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Sep 09 05:55:08.340276 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:08.340242    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfl9w" event={"ID":"45d00781-7997-48b5-97f1-c3434738a214","Type":"ContainerStarted","Data":"c34535fe7c44a2b843cc4904f247a2f7bdcc51458a11060bed80cc411039659d"}
Sep 09 05:55:08.340481 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:08.340458    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfl9w"
Sep 09 05:55:08.342028 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:08.342002    2579 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-cfl9w container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 05:55:08.342150 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:08.342052    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfl9w" podUID="45d00781-7997-48b5-97f1-c3434738a214" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 05:55:08.359677 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:08.359607    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfl9w" podStartSLOduration=2.666821385 podStartE2EDuration="5.35959429s" totalImagesPullingTime="2.692772905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:55:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:55:08.357580989 +0000 UTC m=+333.698068938" watchObservedRunningTime="2026-09-09 05:55:08.35959429 +0000 UTC m=+333.700082200"
Sep 09 05:55:09.343794 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:09.343769    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfl9w"
Sep 09 05:55:19.897912 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:19.897879    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-58b7d76c47-c8lp2"]
Sep 09 05:55:19.904342 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:19.904320    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-58b7d76c47-c8lp2"
Sep 09 05:55:19.911352 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:19.911333    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 09 05:55:19.911472 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:19.911393    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 09 05:55:19.911472 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:19.911426    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 09 05:55:19.912974 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:19.912949    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 09 05:55:19.913485 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:19.913468    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 09 05:55:19.914326 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:19.914311    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-zn2ts\""
Sep 09 05:55:19.915209 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:19.915190    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-58b7d76c47-c8lp2"]
Sep 09 05:55:19.977331 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:19.977310    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/26d3cd25-48db-43be-8b1e-b3ace484daf4-manager-config\") pod \"lws-controller-manager-58b7d76c47-c8lp2\" (UID: \"26d3cd25-48db-43be-8b1e-b3ace484daf4\") " pod="openshift-lws-operator/lws-controller-manager-58b7d76c47-c8lp2"
Sep 09 05:55:19.977431 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:19.977344    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26d3cd25-48db-43be-8b1e-b3ace484daf4-cert\") pod \"lws-controller-manager-58b7d76c47-c8lp2\" (UID: \"26d3cd25-48db-43be-8b1e-b3ace484daf4\") " pod="openshift-lws-operator/lws-controller-manager-58b7d76c47-c8lp2"
Sep 09 05:55:19.977431 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:19.977370    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clcpx\" (UniqueName: \"kubernetes.io/projected/26d3cd25-48db-43be-8b1e-b3ace484daf4-kube-api-access-clcpx\") pod \"lws-controller-manager-58b7d76c47-c8lp2\" (UID: \"26d3cd25-48db-43be-8b1e-b3ace484daf4\") " pod="openshift-lws-operator/lws-controller-manager-58b7d76c47-c8lp2"
Sep 09 05:55:19.977508 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:19.977452    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/26d3cd25-48db-43be-8b1e-b3ace484daf4-metrics-cert\") pod \"lws-controller-manager-58b7d76c47-c8lp2\" (UID: \"26d3cd25-48db-43be-8b1e-b3ace484daf4\") " pod="openshift-lws-operator/lws-controller-manager-58b7d76c47-c8lp2"
Sep 09 05:55:20.077914 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:20.077889    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/26d3cd25-48db-43be-8b1e-b3ace484daf4-metrics-cert\") pod \"lws-controller-manager-58b7d76c47-c8lp2\" (UID: \"26d3cd25-48db-43be-8b1e-b3ace484daf4\") " pod="openshift-lws-operator/lws-controller-manager-58b7d76c47-c8lp2"
Sep 09 05:55:20.078023 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:20.077928    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/26d3cd25-48db-43be-8b1e-b3ace484daf4-manager-config\") pod \"lws-controller-manager-58b7d76c47-c8lp2\" (UID: \"26d3cd25-48db-43be-8b1e-b3ace484daf4\") " pod="openshift-lws-operator/lws-controller-manager-58b7d76c47-c8lp2"
Sep 09 05:55:20.078023 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:20.077954    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26d3cd25-48db-43be-8b1e-b3ace484daf4-cert\") pod \"lws-controller-manager-58b7d76c47-c8lp2\" (UID: \"26d3cd25-48db-43be-8b1e-b3ace484daf4\") " pod="openshift-lws-operator/lws-controller-manager-58b7d76c47-c8lp2"
Sep 09 05:55:20.078023 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:20.077981    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clcpx\" (UniqueName: \"kubernetes.io/projected/26d3cd25-48db-43be-8b1e-b3ace484daf4-kube-api-access-clcpx\") pod \"lws-controller-manager-58b7d76c47-c8lp2\" (UID: \"26d3cd25-48db-43be-8b1e-b3ace484daf4\") " pod="openshift-lws-operator/lws-controller-manager-58b7d76c47-c8lp2"
Sep 09 05:55:20.078560 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:20.078540    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/26d3cd25-48db-43be-8b1e-b3ace484daf4-manager-config\") pod \"lws-controller-manager-58b7d76c47-c8lp2\" (UID: \"26d3cd25-48db-43be-8b1e-b3ace484daf4\") " pod="openshift-lws-operator/lws-controller-manager-58b7d76c47-c8lp2"
Sep 09 05:55:20.080528 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:20.080510    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/26d3cd25-48db-43be-8b1e-b3ace484daf4-metrics-cert\") pod \"lws-controller-manager-58b7d76c47-c8lp2\" (UID: \"26d3cd25-48db-43be-8b1e-b3ace484daf4\") " pod="openshift-lws-operator/lws-controller-manager-58b7d76c47-c8lp2"
Sep 09 05:55:20.080575 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:20.080556    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26d3cd25-48db-43be-8b1e-b3ace484daf4-cert\") pod \"lws-controller-manager-58b7d76c47-c8lp2\" (UID: \"26d3cd25-48db-43be-8b1e-b3ace484daf4\") " pod="openshift-lws-operator/lws-controller-manager-58b7d76c47-c8lp2"
Sep 09 05:55:20.097593 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:20.097570    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clcpx\" (UniqueName: \"kubernetes.io/projected/26d3cd25-48db-43be-8b1e-b3ace484daf4-kube-api-access-clcpx\") pod \"lws-controller-manager-58b7d76c47-c8lp2\" (UID: \"26d3cd25-48db-43be-8b1e-b3ace484daf4\") " pod="openshift-lws-operator/lws-controller-manager-58b7d76c47-c8lp2"
Sep 09 05:55:20.213569 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:20.213520    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-58b7d76c47-c8lp2"
Sep 09 05:55:20.498014 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:20.497992    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-58b7d76c47-c8lp2"]
Sep 09 05:55:20.499970 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:55:20.499946    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26d3cd25_48db_43be_8b1e_b3ace484daf4.slice/crio-2c1bfac42d2df511de274fe587e19e5e745fd0d20593c41d26ccc8f05ede16be WatchSource:0}: Error finding container 2c1bfac42d2df511de274fe587e19e5e745fd0d20593c41d26ccc8f05ede16be: Status 404 returned error can't find the container with id 2c1bfac42d2df511de274fe587e19e5e745fd0d20593c41d26ccc8f05ede16be
Sep 09 05:55:21.387570 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:21.387535    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-58b7d76c47-c8lp2" event={"ID":"26d3cd25-48db-43be-8b1e-b3ace484daf4","Type":"ContainerStarted","Data":"2c1bfac42d2df511de274fe587e19e5e745fd0d20593c41d26ccc8f05ede16be"}
Sep 09 05:55:23.126957 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:23.126923    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfl9w"]
Sep 09 05:55:23.127328 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:23.127102    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfl9w" podUID="45d00781-7997-48b5-97f1-c3434738a214" containerName="discovery" containerID="cri-o://c34535fe7c44a2b843cc4904f247a2f7bdcc51458a11060bed80cc411039659d" gracePeriod=30
Sep 09 05:55:23.397534 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:23.397454    2579 generic.go:356] "Generic (PLEG): container finished" podID="45d00781-7997-48b5-97f1-c3434738a214" containerID="c34535fe7c44a2b843cc4904f247a2f7bdcc51458a11060bed80cc411039659d" exitCode=0
Sep 09 05:55:23.397534 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:23.397513    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfl9w" event={"ID":"45d00781-7997-48b5-97f1-c3434738a214","Type":"ContainerDied","Data":"c34535fe7c44a2b843cc4904f247a2f7bdcc51458a11060bed80cc411039659d"}
Sep 09 05:55:23.631256 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:23.631235    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfl9w"
Sep 09 05:55:23.710213 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:23.710143    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/45d00781-7997-48b5-97f1-c3434738a214-istio-csr-dns-cert\") pod \"45d00781-7997-48b5-97f1-c3434738a214\" (UID: \"45d00781-7997-48b5-97f1-c3434738a214\") "
Sep 09 05:55:23.710213 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:23.710192    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/45d00781-7997-48b5-97f1-c3434738a214-istio-kubeconfig\") pod \"45d00781-7997-48b5-97f1-c3434738a214\" (UID: \"45d00781-7997-48b5-97f1-c3434738a214\") "
Sep 09 05:55:23.710213 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:23.710209    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d00781-7997-48b5-97f1-c3434738a214-local-certs\") pod \"45d00781-7997-48b5-97f1-c3434738a214\" (UID: \"45d00781-7997-48b5-97f1-c3434738a214\") "
Sep 09 05:55:23.710442 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:23.710228    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/45d00781-7997-48b5-97f1-c3434738a214-istio-csr-ca-configmap\") pod \"45d00781-7997-48b5-97f1-c3434738a214\" (UID: \"45d00781-7997-48b5-97f1-c3434738a214\") "
Sep 09 05:55:23.710442 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:23.710415    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45d00781-7997-48b5-97f1-c3434738a214-istio-token\") pod \"45d00781-7997-48b5-97f1-c3434738a214\" (UID: \"45d00781-7997-48b5-97f1-c3434738a214\") "
Sep 09 05:55:23.710538 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:23.710513    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/45d00781-7997-48b5-97f1-c3434738a214-cacerts\") pod \"45d00781-7997-48b5-97f1-c3434738a214\" (UID: \"45d00781-7997-48b5-97f1-c3434738a214\") "
Sep 09 05:55:23.710592 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:23.710540    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45d00781-7997-48b5-97f1-c3434738a214-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "45d00781-7997-48b5-97f1-c3434738a214" (UID: "45d00781-7997-48b5-97f1-c3434738a214"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 05:55:23.710592 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:23.710551    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c56kn\" (UniqueName: \"kubernetes.io/projected/45d00781-7997-48b5-97f1-c3434738a214-kube-api-access-c56kn\") pod \"45d00781-7997-48b5-97f1-c3434738a214\" (UID: \"45d00781-7997-48b5-97f1-c3434738a214\") "
Sep 09 05:55:23.710795 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:23.710778    2579 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/45d00781-7997-48b5-97f1-c3434738a214-istio-csr-ca-configmap\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 05:55:23.712790 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:23.712765    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45d00781-7997-48b5-97f1-c3434738a214-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "45d00781-7997-48b5-97f1-c3434738a214" (UID: "45d00781-7997-48b5-97f1-c3434738a214"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 05:55:23.712902 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:23.712825    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45d00781-7997-48b5-97f1-c3434738a214-istio-token" (OuterVolumeSpecName: "istio-token") pod "45d00781-7997-48b5-97f1-c3434738a214" (UID: "45d00781-7997-48b5-97f1-c3434738a214"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 05:55:23.712955 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:23.712908    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45d00781-7997-48b5-97f1-c3434738a214-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "45d00781-7997-48b5-97f1-c3434738a214" (UID: "45d00781-7997-48b5-97f1-c3434738a214"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 05:55:23.712955 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:23.712917    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d00781-7997-48b5-97f1-c3434738a214-local-certs" (OuterVolumeSpecName: "local-certs") pod "45d00781-7997-48b5-97f1-c3434738a214" (UID: "45d00781-7997-48b5-97f1-c3434738a214"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 05:55:23.713166 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:23.713139    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45d00781-7997-48b5-97f1-c3434738a214-kube-api-access-c56kn" (OuterVolumeSpecName: "kube-api-access-c56kn") pod "45d00781-7997-48b5-97f1-c3434738a214" (UID: "45d00781-7997-48b5-97f1-c3434738a214"). InnerVolumeSpecName "kube-api-access-c56kn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 05:55:23.713166 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:23.713156    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45d00781-7997-48b5-97f1-c3434738a214-cacerts" (OuterVolumeSpecName: "cacerts") pod "45d00781-7997-48b5-97f1-c3434738a214" (UID: "45d00781-7997-48b5-97f1-c3434738a214"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 05:55:23.811597 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:23.811572    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c56kn\" (UniqueName: \"kubernetes.io/projected/45d00781-7997-48b5-97f1-c3434738a214-kube-api-access-c56kn\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 05:55:23.811597 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:23.811599    2579 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/45d00781-7997-48b5-97f1-c3434738a214-istio-csr-dns-cert\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 05:55:23.811783 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:23.811613    2579 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/45d00781-7997-48b5-97f1-c3434738a214-istio-kubeconfig\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 05:55:23.811783 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:23.811650    2579 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d00781-7997-48b5-97f1-c3434738a214-local-certs\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 05:55:23.811783 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:23.811663    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45d00781-7997-48b5-97f1-c3434738a214-istio-token\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 05:55:23.811783 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:23.811675    2579 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/45d00781-7997-48b5-97f1-c3434738a214-cacerts\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 05:55:24.402505 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:24.402467    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfl9w" event={"ID":"45d00781-7997-48b5-97f1-c3434738a214","Type":"ContainerDied","Data":"084b4e5227a466240c07cfb4398b5fccf6ad94bf32119f68a8dc651756abc6cb"}
Sep 09 05:55:24.402999 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:24.402478    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfl9w"
Sep 09 05:55:24.402999 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:24.402525    2579 scope.go:117] "RemoveContainer" containerID="c34535fe7c44a2b843cc4904f247a2f7bdcc51458a11060bed80cc411039659d"
Sep 09 05:55:24.404439 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:24.404385    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-58b7d76c47-c8lp2" event={"ID":"26d3cd25-48db-43be-8b1e-b3ace484daf4","Type":"ContainerStarted","Data":"f95bb06a01f53fc20e92bd75f1c73f7a1ab59489bbf4fa49007c16620925aa50"}
Sep 09 05:55:24.404532 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:24.404465    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-58b7d76c47-c8lp2"
Sep 09 05:55:24.434427 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:24.434404    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfl9w"]
Sep 09 05:55:24.440467 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:24.440448    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cfl9w"]
Sep 09 05:55:24.461559 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:24.461518    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-58b7d76c47-c8lp2" podStartSLOduration=2.392226925 podStartE2EDuration="5.461504846s" totalImagesPullingTime="3.069277921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:55:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:55:24.46019567 +0000 UTC m=+349.800683590" watchObservedRunningTime="2026-09-09 05:55:24.461504846 +0000 UTC m=+349.801992754"
Sep 09 05:55:25.147249 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:25.147216    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45d00781-7997-48b5-97f1-c3434738a214" path="/var/lib/kubelet/pods/45d00781-7997-48b5-97f1-c3434738a214/volumes"
Sep 09 05:55:35.411197 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:55:35.411132    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-58b7d76c47-c8lp2"
Sep 09 05:56:47.768471 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:56:47.768440    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-54mm9"]
Sep 09 05:56:47.768951 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:56:47.768707    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45d00781-7997-48b5-97f1-c3434738a214" containerName="discovery"
Sep 09 05:56:47.768951 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:56:47.768720    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="45d00781-7997-48b5-97f1-c3434738a214" containerName="discovery"
Sep 09 05:56:47.768951 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:56:47.768773    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="45d00781-7997-48b5-97f1-c3434738a214" containerName="discovery"
Sep 09 05:56:47.771484 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:56:47.771465    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-54mm9"
Sep 09 05:56:47.773729 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:56:47.773707    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 09 05:56:47.773871 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:56:47.773854    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 05:56:47.773951 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:56:47.773935    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 09 05:56:47.775852 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:56:47.775835    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-58nxl\""
Sep 09 05:56:47.775944 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:56:47.775834    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 05:56:47.780852 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:56:47.780833    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-54mm9"]
Sep 09 05:56:47.889693 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:56:47.889669    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8bd638b-da4c-4e1d-bdaf-3b7978363e51-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-54mm9\" (UID: \"d8bd638b-da4c-4e1d-bdaf-3b7978363e51\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-54mm9"
Sep 09 05:56:47.889798 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:56:47.889696    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d8bd638b-da4c-4e1d-bdaf-3b7978363e51-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-54mm9\" (UID: \"d8bd638b-da4c-4e1d-bdaf-3b7978363e51\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-54mm9"
Sep 09 05:56:47.889798 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:56:47.889787    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-752fz\" (UniqueName: \"kubernetes.io/projected/d8bd638b-da4c-4e1d-bdaf-3b7978363e51-kube-api-access-752fz\") pod \"kuadrant-console-plugin-fbdd4f445-54mm9\" (UID: \"d8bd638b-da4c-4e1d-bdaf-3b7978363e51\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-54mm9"
Sep 09 05:56:47.990779 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:56:47.990757    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8bd638b-da4c-4e1d-bdaf-3b7978363e51-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-54mm9\" (UID: \"d8bd638b-da4c-4e1d-bdaf-3b7978363e51\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-54mm9"
Sep 09 05:56:47.990844 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:56:47.990784    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d8bd638b-da4c-4e1d-bdaf-3b7978363e51-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-54mm9\" (UID: \"d8bd638b-da4c-4e1d-bdaf-3b7978363e51\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-54mm9"
Sep 09 05:56:47.990844 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:56:47.990832    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-752fz\" (UniqueName: \"kubernetes.io/projected/d8bd638b-da4c-4e1d-bdaf-3b7978363e51-kube-api-access-752fz\") pod \"kuadrant-console-plugin-fbdd4f445-54mm9\" (UID: \"d8bd638b-da4c-4e1d-bdaf-3b7978363e51\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-54mm9"
Sep 09 05:56:47.990918 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:56:47.990904    2579 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 09 05:56:47.990990 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:56:47.990979    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8bd638b-da4c-4e1d-bdaf-3b7978363e51-plugin-serving-cert podName:d8bd638b-da4c-4e1d-bdaf-3b7978363e51 nodeName:}" failed. No retries permitted until 2026-09-09 05:56:48.490959158 +0000 UTC m=+433.831447059 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/d8bd638b-da4c-4e1d-bdaf-3b7978363e51-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-54mm9" (UID: "d8bd638b-da4c-4e1d-bdaf-3b7978363e51") : secret "plugin-serving-cert" not found
Sep 09 05:56:47.991352 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:56:47.991335    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d8bd638b-da4c-4e1d-bdaf-3b7978363e51-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-54mm9\" (UID: \"d8bd638b-da4c-4e1d-bdaf-3b7978363e51\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-54mm9"
Sep 09 05:56:47.999843 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:56:47.999824    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-752fz\" (UniqueName: \"kubernetes.io/projected/d8bd638b-da4c-4e1d-bdaf-3b7978363e51-kube-api-access-752fz\") pod \"kuadrant-console-plugin-fbdd4f445-54mm9\" (UID: \"d8bd638b-da4c-4e1d-bdaf-3b7978363e51\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-54mm9"
Sep 09 05:56:48.494280 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:56:48.494242    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8bd638b-da4c-4e1d-bdaf-3b7978363e51-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-54mm9\" (UID: \"d8bd638b-da4c-4e1d-bdaf-3b7978363e51\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-54mm9"
Sep 09 05:56:48.496768 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:56:48.496751    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8bd638b-da4c-4e1d-bdaf-3b7978363e51-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-54mm9\" (UID: \"d8bd638b-da4c-4e1d-bdaf-3b7978363e51\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-54mm9"
Sep 09 05:56:48.680833 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:56:48.680801    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-54mm9"
Sep 09 05:56:48.959883 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:56:48.959853    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-54mm9"]
Sep 09 05:56:48.962890 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:56:48.962860    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8bd638b_da4c_4e1d_bdaf_3b7978363e51.slice/crio-2ff98a6937f73738f9f9ed178dc4f730b7435beb0e67afc515199da55ce959c1 WatchSource:0}: Error finding container 2ff98a6937f73738f9f9ed178dc4f730b7435beb0e67afc515199da55ce959c1: Status 404 returned error can't find the container with id 2ff98a6937f73738f9f9ed178dc4f730b7435beb0e67afc515199da55ce959c1
Sep 09 05:56:49.706614 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:56:49.706580    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-54mm9" event={"ID":"d8bd638b-da4c-4e1d-bdaf-3b7978363e51","Type":"ContainerStarted","Data":"2ff98a6937f73738f9f9ed178dc4f730b7435beb0e67afc515199da55ce959c1"}
Sep 09 05:56:54.727539 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:56:54.727505    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-54mm9" event={"ID":"d8bd638b-da4c-4e1d-bdaf-3b7978363e51","Type":"ContainerStarted","Data":"dbf7c91f3c541329cacb28e052414ee753cd1fd5a93ea4f78e32c58a6be4f5db"}
Sep 09 05:56:54.743497 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:56:54.743324    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-54mm9" podStartSLOduration=2.938616302 podStartE2EDuration="7.743305544s" totalImagesPullingTime="4.804689242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:56:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:56:54.742816042 +0000 UTC m=+440.083303962" watchObservedRunningTime="2026-09-09 05:56:54.743305544 +0000 UTC m=+440.083793454"
Sep 09 05:56:58.591716 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:56:58.591642    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6pp2x"]
Sep 09 05:56:58.594634 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:56:58.594600    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6pp2x"
Sep 09 05:56:58.596785 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:56:58.596764    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-pm7rx\""
Sep 09 05:56:58.609115 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:56:58.609088    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6pp2x"]
Sep 09 05:56:58.671919 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:56:58.671891    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d7a7c450-c27f-4a5b-958e-aca8024aed9e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-6pp2x\" (UID: \"d7a7c450-c27f-4a5b-958e-aca8024aed9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6pp2x"
Sep 09 05:56:58.672059 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:56:58.671934    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldb89\" (UniqueName: \"kubernetes.io/projected/d7a7c450-c27f-4a5b-958e-aca8024aed9e-kube-api-access-ldb89\") pod \"kuadrant-operator-controller-manager-5f685c496f-6pp2x\" (UID: \"d7a7c450-c27f-4a5b-958e-aca8024aed9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6pp2x"
Sep 09 05:56:58.773167 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:56:58.773125    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d7a7c450-c27f-4a5b-958e-aca8024aed9e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-6pp2x\" (UID: \"d7a7c450-c27f-4a5b-958e-aca8024aed9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6pp2x"
Sep 09 05:56:58.773167 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:56:58.773174    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldb89\" (UniqueName: \"kubernetes.io/projected/d7a7c450-c27f-4a5b-958e-aca8024aed9e-kube-api-access-ldb89\") pod \"kuadrant-operator-controller-manager-5f685c496f-6pp2x\" (UID: \"d7a7c450-c27f-4a5b-958e-aca8024aed9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6pp2x"
Sep 09 05:56:58.773591 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:56:58.773566    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d7a7c450-c27f-4a5b-958e-aca8024aed9e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-6pp2x\" (UID: \"d7a7c450-c27f-4a5b-958e-aca8024aed9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6pp2x"
Sep 09 05:56:58.789561 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:56:58.789534    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldb89\" (UniqueName: \"kubernetes.io/projected/d7a7c450-c27f-4a5b-958e-aca8024aed9e-kube-api-access-ldb89\") pod \"kuadrant-operator-controller-manager-5f685c496f-6pp2x\" (UID: \"d7a7c450-c27f-4a5b-958e-aca8024aed9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6pp2x"
Sep 09 05:56:58.903998 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:56:58.903924    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6pp2x"
Sep 09 05:56:59.182156 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:56:59.182075    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6pp2x"]
Sep 09 05:56:59.185049 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:56:59.185017    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7a7c450_c27f_4a5b_958e_aca8024aed9e.slice/crio-fe28eed665f215e7197efe761882267b71efdd3f229a651505f02b7642494f90 WatchSource:0}: Error finding container fe28eed665f215e7197efe761882267b71efdd3f229a651505f02b7642494f90: Status 404 returned error can't find the container with id fe28eed665f215e7197efe761882267b71efdd3f229a651505f02b7642494f90
Sep 09 05:56:59.747606 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:56:59.747570    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6pp2x" event={"ID":"d7a7c450-c27f-4a5b-958e-aca8024aed9e","Type":"ContainerStarted","Data":"fe28eed665f215e7197efe761882267b71efdd3f229a651505f02b7642494f90"}
Sep 09 05:57:03.769303 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:03.769268    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6pp2x" event={"ID":"d7a7c450-c27f-4a5b-958e-aca8024aed9e","Type":"ContainerStarted","Data":"d785cedaa2807c3f2f7893570b0eceb52f344d42c148e5f58178e1c60db32479"}
Sep 09 05:57:03.769791 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:03.769404    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6pp2x"
Sep 09 05:57:03.790064 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:03.790008    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6pp2x" podStartSLOduration=1.838705645 podStartE2EDuration="5.789992905s" totalImagesPullingTime="3.95128726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:56:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:57:03.788034662 +0000 UTC m=+449.128522581" watchObservedRunningTime="2026-09-09 05:57:03.789992905 +0000 UTC m=+449.130480814"
Sep 09 05:57:14.775017 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:14.774990    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6pp2x"
Sep 09 05:57:27.741675 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:27.741643    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6pp2x"]
Sep 09 05:57:27.742150 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:27.741849    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6pp2x" podUID="d7a7c450-c27f-4a5b-958e-aca8024aed9e" containerName="manager" containerID="cri-o://d785cedaa2807c3f2f7893570b0eceb52f344d42c148e5f58178e1c60db32479" gracePeriod=10
Sep 09 05:57:27.864075 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:27.864047    2579 generic.go:356] "Generic (PLEG): container finished" podID="d7a7c450-c27f-4a5b-958e-aca8024aed9e" containerID="d785cedaa2807c3f2f7893570b0eceb52f344d42c148e5f58178e1c60db32479" exitCode=0
Sep 09 05:57:27.864181 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:27.864114    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6pp2x" event={"ID":"d7a7c450-c27f-4a5b-958e-aca8024aed9e","Type":"ContainerDied","Data":"d785cedaa2807c3f2f7893570b0eceb52f344d42c148e5f58178e1c60db32479"}
Sep 09 05:57:28.003373 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:28.003321    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6pp2x"
Sep 09 05:57:28.082484 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:28.082459    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldb89\" (UniqueName: \"kubernetes.io/projected/d7a7c450-c27f-4a5b-958e-aca8024aed9e-kube-api-access-ldb89\") pod \"d7a7c450-c27f-4a5b-958e-aca8024aed9e\" (UID: \"d7a7c450-c27f-4a5b-958e-aca8024aed9e\") "
Sep 09 05:57:28.082611 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:28.082519    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d7a7c450-c27f-4a5b-958e-aca8024aed9e-extensions-socket-volume\") pod \"d7a7c450-c27f-4a5b-958e-aca8024aed9e\" (UID: \"d7a7c450-c27f-4a5b-958e-aca8024aed9e\") "
Sep 09 05:57:28.083005 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:28.082984    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7a7c450-c27f-4a5b-958e-aca8024aed9e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d7a7c450-c27f-4a5b-958e-aca8024aed9e" (UID: "d7a7c450-c27f-4a5b-958e-aca8024aed9e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 05:57:28.084809 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:28.084783    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7a7c450-c27f-4a5b-958e-aca8024aed9e-kube-api-access-ldb89" (OuterVolumeSpecName: "kube-api-access-ldb89") pod "d7a7c450-c27f-4a5b-958e-aca8024aed9e" (UID: "d7a7c450-c27f-4a5b-958e-aca8024aed9e"). InnerVolumeSpecName "kube-api-access-ldb89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 05:57:28.183658 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:28.183613    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldb89\" (UniqueName: \"kubernetes.io/projected/d7a7c450-c27f-4a5b-958e-aca8024aed9e-kube-api-access-ldb89\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 05:57:28.183658 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:28.183656    2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d7a7c450-c27f-4a5b-958e-aca8024aed9e-extensions-socket-volume\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 05:57:28.869302 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:28.869276    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6pp2x"
Sep 09 05:57:28.869681 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:28.869272    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6pp2x" event={"ID":"d7a7c450-c27f-4a5b-958e-aca8024aed9e","Type":"ContainerDied","Data":"fe28eed665f215e7197efe761882267b71efdd3f229a651505f02b7642494f90"}
Sep 09 05:57:28.869681 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:28.869412    2579 scope.go:117] "RemoveContainer" containerID="d785cedaa2807c3f2f7893570b0eceb52f344d42c148e5f58178e1c60db32479"
Sep 09 05:57:28.893222 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:28.893197    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6pp2x"]
Sep 09 05:57:28.897338 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:28.897316    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6pp2x"]
Sep 09 05:57:29.147407 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:29.147351    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7a7c450-c27f-4a5b-958e-aca8024aed9e" path="/var/lib/kubelet/pods/d7a7c450-c27f-4a5b-958e-aca8024aed9e/volumes"
Sep 09 05:57:46.688975 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:46.688945    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 05:57:46.689464 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:46.689346    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7a7c450-c27f-4a5b-958e-aca8024aed9e" containerName="manager"
Sep 09 05:57:46.689464 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:46.689364    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7a7c450-c27f-4a5b-958e-aca8024aed9e" containerName="manager"
Sep 09 05:57:46.689464 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:46.689446    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7a7c450-c27f-4a5b-958e-aca8024aed9e" containerName="manager"
Sep 09 05:57:46.693668 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:46.693646    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"
Sep 09 05:57:46.695813 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:46.695793    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 09 05:57:46.702704 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:46.702683    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 05:57:46.736989 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:46.736959    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 05:57:46.799244 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:46.799222    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwtvj\" (UniqueName: \"kubernetes.io/projected/1919f47d-0880-4d98-ad97-1e8dd6d34408-kube-api-access-zwtvj\") pod \"limitador-limitador-84994f7ffc-f5j8g\" (UID: \"1919f47d-0880-4d98-ad97-1e8dd6d34408\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"
Sep 09 05:57:46.799343 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:46.799253    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1919f47d-0880-4d98-ad97-1e8dd6d34408-config-file\") pod \"limitador-limitador-84994f7ffc-f5j8g\" (UID: \"1919f47d-0880-4d98-ad97-1e8dd6d34408\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"
Sep 09 05:57:46.899793 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:46.899771    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwtvj\" (UniqueName: \"kubernetes.io/projected/1919f47d-0880-4d98-ad97-1e8dd6d34408-kube-api-access-zwtvj\") pod \"limitador-limitador-84994f7ffc-f5j8g\" (UID: \"1919f47d-0880-4d98-ad97-1e8dd6d34408\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"
Sep 09 05:57:46.899892 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:46.899805    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1919f47d-0880-4d98-ad97-1e8dd6d34408-config-file\") pod \"limitador-limitador-84994f7ffc-f5j8g\" (UID: \"1919f47d-0880-4d98-ad97-1e8dd6d34408\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"
Sep 09 05:57:46.900338 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:46.900322    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1919f47d-0880-4d98-ad97-1e8dd6d34408-config-file\") pod \"limitador-limitador-84994f7ffc-f5j8g\" (UID: \"1919f47d-0880-4d98-ad97-1e8dd6d34408\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"
Sep 09 05:57:46.908651 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:46.908614    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwtvj\" (UniqueName: \"kubernetes.io/projected/1919f47d-0880-4d98-ad97-1e8dd6d34408-kube-api-access-zwtvj\") pod \"limitador-limitador-84994f7ffc-f5j8g\" (UID: \"1919f47d-0880-4d98-ad97-1e8dd6d34408\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"
Sep 09 05:57:47.003568 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:47.003517    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"
Sep 09 05:57:47.096098 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:47.096071    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-9stg4"]
Sep 09 05:57:47.099323 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:47.099302    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-9stg4"
Sep 09 05:57:47.101963 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:47.101946    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ncznm\""
Sep 09 05:57:47.107167 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:47.107146    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-9stg4"]
Sep 09 05:57:47.202849 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:47.202824    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsv8t\" (UniqueName: \"kubernetes.io/projected/a336ec99-1ec3-4fac-a24c-e85e5af484de-kube-api-access-xsv8t\") pod \"authorino-7bcfd8f9cd-9stg4\" (UID: \"a336ec99-1ec3-4fac-a24c-e85e5af484de\") " pod="kuadrant-system/authorino-7bcfd8f9cd-9stg4"
Sep 09 05:57:47.271946 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:47.271925    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 05:57:47.274404 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:57:47.274379    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1919f47d_0880_4d98_ad97_1e8dd6d34408.slice/crio-7a8a6db131cf3434e178ce1bc6a94dd81f27a8f846074e54ff790d91b6e8c273 WatchSource:0}: Error finding container 7a8a6db131cf3434e178ce1bc6a94dd81f27a8f846074e54ff790d91b6e8c273: Status 404 returned error can't find the container with id 7a8a6db131cf3434e178ce1bc6a94dd81f27a8f846074e54ff790d91b6e8c273
Sep 09 05:57:47.303374 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:47.303340    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsv8t\" (UniqueName: \"kubernetes.io/projected/a336ec99-1ec3-4fac-a24c-e85e5af484de-kube-api-access-xsv8t\") pod \"authorino-7bcfd8f9cd-9stg4\" (UID: \"a336ec99-1ec3-4fac-a24c-e85e5af484de\") " pod="kuadrant-system/authorino-7bcfd8f9cd-9stg4"
Sep 09 05:57:47.311659 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:47.311641    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsv8t\" (UniqueName: \"kubernetes.io/projected/a336ec99-1ec3-4fac-a24c-e85e5af484de-kube-api-access-xsv8t\") pod \"authorino-7bcfd8f9cd-9stg4\" (UID: \"a336ec99-1ec3-4fac-a24c-e85e5af484de\") " pod="kuadrant-system/authorino-7bcfd8f9cd-9stg4"
Sep 09 05:57:47.347441 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:47.347417    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-8rlv9"]
Sep 09 05:57:47.350495 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:47.350481    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-8rlv9"
Sep 09 05:57:47.357322 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:47.357301    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-8rlv9"]
Sep 09 05:57:47.404432 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:47.404395    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwkcr\" (UniqueName: \"kubernetes.io/projected/9c735929-00ab-4837-b588-87dc5e1b5d87-kube-api-access-xwkcr\") pod \"authorino-59c974bd48-8rlv9\" (UID: \"9c735929-00ab-4837-b588-87dc5e1b5d87\") " pod="kuadrant-system/authorino-59c974bd48-8rlv9"
Sep 09 05:57:47.408372 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:47.408346    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-9stg4"
Sep 09 05:57:47.504883 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:47.504855    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwkcr\" (UniqueName: \"kubernetes.io/projected/9c735929-00ab-4837-b588-87dc5e1b5d87-kube-api-access-xwkcr\") pod \"authorino-59c974bd48-8rlv9\" (UID: \"9c735929-00ab-4837-b588-87dc5e1b5d87\") " pod="kuadrant-system/authorino-59c974bd48-8rlv9"
Sep 09 05:57:47.512367 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:47.512342    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwkcr\" (UniqueName: \"kubernetes.io/projected/9c735929-00ab-4837-b588-87dc5e1b5d87-kube-api-access-xwkcr\") pod \"authorino-59c974bd48-8rlv9\" (UID: \"9c735929-00ab-4837-b588-87dc5e1b5d87\") " pod="kuadrant-system/authorino-59c974bd48-8rlv9"
Sep 09 05:57:47.660162 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:47.660142    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-8rlv9"
Sep 09 05:57:47.673201 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:47.673173    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-9stg4"]
Sep 09 05:57:47.676742 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:57:47.676714    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda336ec99_1ec3_4fac_a24c_e85e5af484de.slice/crio-6e0740c2b17812265f291d3e67a763bc984c133ce50ca1bea4ec790239851761 WatchSource:0}: Error finding container 6e0740c2b17812265f291d3e67a763bc984c133ce50ca1bea4ec790239851761: Status 404 returned error can't find the container with id 6e0740c2b17812265f291d3e67a763bc984c133ce50ca1bea4ec790239851761
Sep 09 05:57:47.928325 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:47.928266    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-8rlv9"]
Sep 09 05:57:47.931722 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:57:47.931688    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c735929_00ab_4837_b588_87dc5e1b5d87.slice/crio-b0f06f66a1c6522b37cc5fb23b64942a792616e3add22fdb4d83f83eaae849a9 WatchSource:0}: Error finding container b0f06f66a1c6522b37cc5fb23b64942a792616e3add22fdb4d83f83eaae849a9: Status 404 returned error can't find the container with id b0f06f66a1c6522b37cc5fb23b64942a792616e3add22fdb4d83f83eaae849a9
Sep 09 05:57:47.944649 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:47.944607    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-f5j8g" event={"ID":"1919f47d-0880-4d98-ad97-1e8dd6d34408","Type":"ContainerStarted","Data":"7a8a6db131cf3434e178ce1bc6a94dd81f27a8f846074e54ff790d91b6e8c273"}
Sep 09 05:57:47.945762 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:47.945738    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-9stg4" event={"ID":"a336ec99-1ec3-4fac-a24c-e85e5af484de","Type":"ContainerStarted","Data":"6e0740c2b17812265f291d3e67a763bc984c133ce50ca1bea4ec790239851761"}
Sep 09 05:57:47.946919 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:47.946899    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-8rlv9" event={"ID":"9c735929-00ab-4837-b588-87dc5e1b5d87","Type":"ContainerStarted","Data":"b0f06f66a1c6522b37cc5fb23b64942a792616e3add22fdb4d83f83eaae849a9"}
Sep 09 05:57:48.952581 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:48.952540    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-f5j8g" event={"ID":"1919f47d-0880-4d98-ad97-1e8dd6d34408","Type":"ContainerStarted","Data":"b31c4ccc1dbda49e80239f8458e96b816a303447e5eb2cb078b645455846ee50"}
Sep 09 05:57:48.952950 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:48.952621    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"
Sep 09 05:57:48.969282 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:48.969241    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-f5j8g" podStartSLOduration=1.7681248489999999 podStartE2EDuration="2.969226631s" totalImagesPullingTime="1.201101782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:57:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:57:48.967879236 +0000 UTC m=+494.308367147" watchObservedRunningTime="2026-09-09 05:57:48.969226631 +0000 UTC m=+494.309714539"
Sep 09 05:57:50.963137 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:50.963102    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-9stg4" event={"ID":"a336ec99-1ec3-4fac-a24c-e85e5af484de","Type":"ContainerStarted","Data":"37300cac670d060e1567af4dc9921ce40a4dcac963074c7deb2f02f7d3c32296"}
Sep 09 05:57:50.964536 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:50.964509    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-8rlv9" event={"ID":"9c735929-00ab-4837-b588-87dc5e1b5d87","Type":"ContainerStarted","Data":"0f274289edeac9f20f2bbd05448a426fe3118c1be61ebd69d90b059821baae6b"}
Sep 09 05:57:50.977442 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:50.977401    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-9stg4" podStartSLOduration=1.219873142 podStartE2EDuration="3.977387741s" totalImagesPullingTime="2.757514599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:57:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:57:50.976165151 +0000 UTC m=+496.316653055" watchObservedRunningTime="2026-09-09 05:57:50.977387741 +0000 UTC m=+496.317875649"
Sep 09 05:57:50.989175 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:50.988995    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-8rlv9" podStartSLOduration=1.999247443 podStartE2EDuration="3.988977791s" totalImagesPullingTime="1.989730348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:57:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:57:50.988887892 +0000 UTC m=+496.329375802" watchObservedRunningTime="2026-09-09 05:57:50.988977791 +0000 UTC m=+496.329465702"
Sep 09 05:57:51.014181 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:51.014153    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-9stg4"]
Sep 09 05:57:52.972148 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:52.972111    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-9stg4" podUID="a336ec99-1ec3-4fac-a24c-e85e5af484de" containerName="authorino" containerID="cri-o://37300cac670d060e1567af4dc9921ce40a4dcac963074c7deb2f02f7d3c32296" gracePeriod=30
Sep 09 05:57:53.212408 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:53.212386    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-9stg4"
Sep 09 05:57:53.246706 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:53.246651    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsv8t\" (UniqueName: \"kubernetes.io/projected/a336ec99-1ec3-4fac-a24c-e85e5af484de-kube-api-access-xsv8t\") pod \"a336ec99-1ec3-4fac-a24c-e85e5af484de\" (UID: \"a336ec99-1ec3-4fac-a24c-e85e5af484de\") "
Sep 09 05:57:53.248777 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:53.248747    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a336ec99-1ec3-4fac-a24c-e85e5af484de-kube-api-access-xsv8t" (OuterVolumeSpecName: "kube-api-access-xsv8t") pod "a336ec99-1ec3-4fac-a24c-e85e5af484de" (UID: "a336ec99-1ec3-4fac-a24c-e85e5af484de"). InnerVolumeSpecName "kube-api-access-xsv8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 05:57:53.347964 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:53.347929    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsv8t\" (UniqueName: \"kubernetes.io/projected/a336ec99-1ec3-4fac-a24c-e85e5af484de-kube-api-access-xsv8t\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 05:57:53.977521 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:53.977487    2579 generic.go:356] "Generic (PLEG): container finished" podID="a336ec99-1ec3-4fac-a24c-e85e5af484de" containerID="37300cac670d060e1567af4dc9921ce40a4dcac963074c7deb2f02f7d3c32296" exitCode=0
Sep 09 05:57:53.977940 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:53.977530    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-9stg4"
Sep 09 05:57:53.977940 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:53.977555    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-9stg4" event={"ID":"a336ec99-1ec3-4fac-a24c-e85e5af484de","Type":"ContainerDied","Data":"37300cac670d060e1567af4dc9921ce40a4dcac963074c7deb2f02f7d3c32296"}
Sep 09 05:57:53.977940 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:53.977585    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-9stg4" event={"ID":"a336ec99-1ec3-4fac-a24c-e85e5af484de","Type":"ContainerDied","Data":"6e0740c2b17812265f291d3e67a763bc984c133ce50ca1bea4ec790239851761"}
Sep 09 05:57:53.977940 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:53.977600    2579 scope.go:117] "RemoveContainer" containerID="37300cac670d060e1567af4dc9921ce40a4dcac963074c7deb2f02f7d3c32296"
Sep 09 05:57:53.985878 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:53.985858    2579 scope.go:117] "RemoveContainer" containerID="37300cac670d060e1567af4dc9921ce40a4dcac963074c7deb2f02f7d3c32296"
Sep 09 05:57:53.986132 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:57:53.986115    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37300cac670d060e1567af4dc9921ce40a4dcac963074c7deb2f02f7d3c32296\": container with ID starting with 37300cac670d060e1567af4dc9921ce40a4dcac963074c7deb2f02f7d3c32296 not found: ID does not exist" containerID="37300cac670d060e1567af4dc9921ce40a4dcac963074c7deb2f02f7d3c32296"
Sep 09 05:57:53.986181 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:53.986138    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37300cac670d060e1567af4dc9921ce40a4dcac963074c7deb2f02f7d3c32296"} err="failed to get container status \"37300cac670d060e1567af4dc9921ce40a4dcac963074c7deb2f02f7d3c32296\": rpc error: code = NotFound desc = could not find container \"37300cac670d060e1567af4dc9921ce40a4dcac963074c7deb2f02f7d3c32296\": container with ID starting with 37300cac670d060e1567af4dc9921ce40a4dcac963074c7deb2f02f7d3c32296 not found: ID does not exist"
Sep 09 05:57:53.996221 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:53.996201    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-9stg4"]
Sep 09 05:57:53.998435 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:53.998418    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-9stg4"]
Sep 09 05:57:55.147461 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:55.147421    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a336ec99-1ec3-4fac-a24c-e85e5af484de" path="/var/lib/kubelet/pods/a336ec99-1ec3-4fac-a24c-e85e5af484de/volumes"
Sep 09 05:57:59.959589 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:57:59.959559    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"
Sep 09 05:58:16.333701 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:16.333662    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-gbtkz"]
Sep 09 05:58:16.334213 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:16.334038    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a336ec99-1ec3-4fac-a24c-e85e5af484de" containerName="authorino"
Sep 09 05:58:16.334213 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:16.334056    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a336ec99-1ec3-4fac-a24c-e85e5af484de" containerName="authorino"
Sep 09 05:58:16.334213 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:16.334150    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a336ec99-1ec3-4fac-a24c-e85e5af484de" containerName="authorino"
Sep 09 05:58:16.336614 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:16.336593    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-gbtkz"
Sep 09 05:58:16.343376 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:16.343351    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-gbtkz"]
Sep 09 05:58:16.418469 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:16.418437    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qd7n\" (UniqueName: \"kubernetes.io/projected/926a11e8-8ddc-4736-8300-6a5b55ccb73c-kube-api-access-7qd7n\") pod \"authorino-557f876f69-gbtkz\" (UID: \"926a11e8-8ddc-4736-8300-6a5b55ccb73c\") " pod="kuadrant-system/authorino-557f876f69-gbtkz"
Sep 09 05:58:16.519002 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:16.518972    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qd7n\" (UniqueName: \"kubernetes.io/projected/926a11e8-8ddc-4736-8300-6a5b55ccb73c-kube-api-access-7qd7n\") pod \"authorino-557f876f69-gbtkz\" (UID: \"926a11e8-8ddc-4736-8300-6a5b55ccb73c\") " pod="kuadrant-system/authorino-557f876f69-gbtkz"
Sep 09 05:58:16.526899 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:16.526869    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qd7n\" (UniqueName: \"kubernetes.io/projected/926a11e8-8ddc-4736-8300-6a5b55ccb73c-kube-api-access-7qd7n\") pod \"authorino-557f876f69-gbtkz\" (UID: \"926a11e8-8ddc-4736-8300-6a5b55ccb73c\") " pod="kuadrant-system/authorino-557f876f69-gbtkz"
Sep 09 05:58:16.584759 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:16.584681    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-gbtkz"]
Sep 09 05:58:16.584935 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:16.584920    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-gbtkz"
Sep 09 05:58:16.598170 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:16.598149    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-9479df6d9-r5mlh"]
Sep 09 05:58:16.601483 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:16.601465    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9479df6d9-r5mlh"
Sep 09 05:58:16.604292 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:16.604265    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 09 05:58:16.613196 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:16.613177    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-9479df6d9-r5mlh"]
Sep 09 05:58:16.720360 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:16.720328    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4s7n\" (UniqueName: \"kubernetes.io/projected/4786ff3c-db02-4087-8b02-79babede8757-kube-api-access-g4s7n\") pod \"authorino-9479df6d9-r5mlh\" (UID: \"4786ff3c-db02-4087-8b02-79babede8757\") " pod="kuadrant-system/authorino-9479df6d9-r5mlh"
Sep 09 05:58:16.720516 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:16.720393    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4786ff3c-db02-4087-8b02-79babede8757-tls-cert\") pod \"authorino-9479df6d9-r5mlh\" (UID: \"4786ff3c-db02-4087-8b02-79babede8757\") " pod="kuadrant-system/authorino-9479df6d9-r5mlh"
Sep 09 05:58:16.821233 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:16.821193    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4786ff3c-db02-4087-8b02-79babede8757-tls-cert\") pod \"authorino-9479df6d9-r5mlh\" (UID: \"4786ff3c-db02-4087-8b02-79babede8757\") " pod="kuadrant-system/authorino-9479df6d9-r5mlh"
Sep 09 05:58:16.821400 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:16.821253    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4s7n\" (UniqueName: \"kubernetes.io/projected/4786ff3c-db02-4087-8b02-79babede8757-kube-api-access-g4s7n\") pod \"authorino-9479df6d9-r5mlh\" (UID: \"4786ff3c-db02-4087-8b02-79babede8757\") " pod="kuadrant-system/authorino-9479df6d9-r5mlh"
Sep 09 05:58:16.823761 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:16.823737    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4786ff3c-db02-4087-8b02-79babede8757-tls-cert\") pod \"authorino-9479df6d9-r5mlh\" (UID: \"4786ff3c-db02-4087-8b02-79babede8757\") " pod="kuadrant-system/authorino-9479df6d9-r5mlh"
Sep 09 05:58:16.829278 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:16.829252    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4s7n\" (UniqueName: \"kubernetes.io/projected/4786ff3c-db02-4087-8b02-79babede8757-kube-api-access-g4s7n\") pod \"authorino-9479df6d9-r5mlh\" (UID: \"4786ff3c-db02-4087-8b02-79babede8757\") " pod="kuadrant-system/authorino-9479df6d9-r5mlh"
Sep 09 05:58:16.853562 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:16.853538    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-gbtkz"]
Sep 09 05:58:16.855873 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:58:16.855851    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod926a11e8_8ddc_4736_8300_6a5b55ccb73c.slice/crio-0d423498e18251ba97fa4edc6c8d5cca228ba918a8586022afeb3c3ecafbc3e9 WatchSource:0}: Error finding container 0d423498e18251ba97fa4edc6c8d5cca228ba918a8586022afeb3c3ecafbc3e9: Status 404 returned error can't find the container with id 0d423498e18251ba97fa4edc6c8d5cca228ba918a8586022afeb3c3ecafbc3e9
Sep 09 05:58:16.913399 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:16.913369    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9479df6d9-r5mlh"
Sep 09 05:58:17.073641 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:17.073594    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-gbtkz" event={"ID":"926a11e8-8ddc-4736-8300-6a5b55ccb73c","Type":"ContainerStarted","Data":"0d423498e18251ba97fa4edc6c8d5cca228ba918a8586022afeb3c3ecafbc3e9"}
Sep 09 05:58:17.181018 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:17.180995    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-9479df6d9-r5mlh"]
Sep 09 05:58:17.183308 ip-10-0-131-142 kubenswrapper[2579]: W0909 05:58:17.183276    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4786ff3c_db02_4087_8b02_79babede8757.slice/crio-cffe26231698322719d5f192c49f0e066fa4ab7f13553b17b8288070617ed791 WatchSource:0}: Error finding container cffe26231698322719d5f192c49f0e066fa4ab7f13553b17b8288070617ed791: Status 404 returned error can't find the container with id cffe26231698322719d5f192c49f0e066fa4ab7f13553b17b8288070617ed791
Sep 09 05:58:18.080620 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:18.080578    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9479df6d9-r5mlh" event={"ID":"4786ff3c-db02-4087-8b02-79babede8757","Type":"ContainerStarted","Data":"552fcd1452ad5eb071666f92af4ba8a929a83ad9ca431c69edbe42f41fde368c"}
Sep 09 05:58:18.081077 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:18.080643    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9479df6d9-r5mlh" event={"ID":"4786ff3c-db02-4087-8b02-79babede8757","Type":"ContainerStarted","Data":"cffe26231698322719d5f192c49f0e066fa4ab7f13553b17b8288070617ed791"}
Sep 09 05:58:18.082272 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:18.082243    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-gbtkz" event={"ID":"926a11e8-8ddc-4736-8300-6a5b55ccb73c","Type":"ContainerStarted","Data":"64d14653fbc7fcf49b2ac5a7e05454da3fb4d7696ce1f024fe029519d3fb5991"}
Sep 09 05:58:18.082398 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:18.082307    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-557f876f69-gbtkz" podUID="926a11e8-8ddc-4736-8300-6a5b55ccb73c" containerName="authorino" containerID="cri-o://64d14653fbc7fcf49b2ac5a7e05454da3fb4d7696ce1f024fe029519d3fb5991" gracePeriod=30
Sep 09 05:58:18.095553 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:18.095511    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-9479df6d9-r5mlh" podStartSLOduration=1.619217607 podStartE2EDuration="2.095495764s" totalImagesPullingTime="476.278157ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:58:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:58:18.094061306 +0000 UTC m=+523.434549217" watchObservedRunningTime="2026-09-09 05:58:18.095495764 +0000 UTC m=+523.435983676"
Sep 09 05:58:18.108425 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:18.108381    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-557f876f69-gbtkz" podStartSLOduration=1.447659233 podStartE2EDuration="2.108366347s" totalImagesPullingTime="660.707114ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 05:58:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 05:58:18.107951083 +0000 UTC m=+523.448438993" watchObservedRunningTime="2026-09-09 05:58:18.108366347 +0000 UTC m=+523.448854289"
Sep 09 05:58:18.121600 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:18.121567    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-8rlv9"]
Sep 09 05:58:18.121769 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:18.121750    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-8rlv9" podUID="9c735929-00ab-4837-b588-87dc5e1b5d87" containerName="authorino" containerID="cri-o://0f274289edeac9f20f2bbd05448a426fe3118c1be61ebd69d90b059821baae6b" gracePeriod=30
Sep 09 05:58:18.360539 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:18.360391    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-gbtkz"
Sep 09 05:58:18.367015 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:18.366995    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-8rlv9"
Sep 09 05:58:18.436767 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:18.436742    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwkcr\" (UniqueName: \"kubernetes.io/projected/9c735929-00ab-4837-b588-87dc5e1b5d87-kube-api-access-xwkcr\") pod \"9c735929-00ab-4837-b588-87dc5e1b5d87\" (UID: \"9c735929-00ab-4837-b588-87dc5e1b5d87\") "
Sep 09 05:58:18.436921 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:18.436774    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qd7n\" (UniqueName: \"kubernetes.io/projected/926a11e8-8ddc-4736-8300-6a5b55ccb73c-kube-api-access-7qd7n\") pod \"926a11e8-8ddc-4736-8300-6a5b55ccb73c\" (UID: \"926a11e8-8ddc-4736-8300-6a5b55ccb73c\") "
Sep 09 05:58:18.438827 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:18.438802    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c735929-00ab-4837-b588-87dc5e1b5d87-kube-api-access-xwkcr" (OuterVolumeSpecName: "kube-api-access-xwkcr") pod "9c735929-00ab-4837-b588-87dc5e1b5d87" (UID: "9c735929-00ab-4837-b588-87dc5e1b5d87"). InnerVolumeSpecName "kube-api-access-xwkcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 05:58:18.438926 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:18.438863    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/926a11e8-8ddc-4736-8300-6a5b55ccb73c-kube-api-access-7qd7n" (OuterVolumeSpecName: "kube-api-access-7qd7n") pod "926a11e8-8ddc-4736-8300-6a5b55ccb73c" (UID: "926a11e8-8ddc-4736-8300-6a5b55ccb73c"). InnerVolumeSpecName "kube-api-access-7qd7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 05:58:18.538214 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:18.538190    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwkcr\" (UniqueName: \"kubernetes.io/projected/9c735929-00ab-4837-b588-87dc5e1b5d87-kube-api-access-xwkcr\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 05:58:18.538214 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:18.538215    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qd7n\" (UniqueName: \"kubernetes.io/projected/926a11e8-8ddc-4736-8300-6a5b55ccb73c-kube-api-access-7qd7n\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 05:58:19.088272 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:19.088240    2579 generic.go:356] "Generic (PLEG): container finished" podID="926a11e8-8ddc-4736-8300-6a5b55ccb73c" containerID="64d14653fbc7fcf49b2ac5a7e05454da3fb4d7696ce1f024fe029519d3fb5991" exitCode=0
Sep 09 05:58:19.088764 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:19.088282    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-gbtkz"
Sep 09 05:58:19.088764 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:19.088326    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-gbtkz" event={"ID":"926a11e8-8ddc-4736-8300-6a5b55ccb73c","Type":"ContainerDied","Data":"64d14653fbc7fcf49b2ac5a7e05454da3fb4d7696ce1f024fe029519d3fb5991"}
Sep 09 05:58:19.088764 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:19.088363    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-gbtkz" event={"ID":"926a11e8-8ddc-4736-8300-6a5b55ccb73c","Type":"ContainerDied","Data":"0d423498e18251ba97fa4edc6c8d5cca228ba918a8586022afeb3c3ecafbc3e9"}
Sep 09 05:58:19.088764 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:19.088381    2579 scope.go:117] "RemoveContainer" containerID="64d14653fbc7fcf49b2ac5a7e05454da3fb4d7696ce1f024fe029519d3fb5991"
Sep 09 05:58:19.089722 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:19.089701    2579 generic.go:356] "Generic (PLEG): container finished" podID="9c735929-00ab-4837-b588-87dc5e1b5d87" containerID="0f274289edeac9f20f2bbd05448a426fe3118c1be61ebd69d90b059821baae6b" exitCode=0
Sep 09 05:58:19.089773 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:19.089741    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-8rlv9"
Sep 09 05:58:19.089773 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:19.089739    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-8rlv9" event={"ID":"9c735929-00ab-4837-b588-87dc5e1b5d87","Type":"ContainerDied","Data":"0f274289edeac9f20f2bbd05448a426fe3118c1be61ebd69d90b059821baae6b"}
Sep 09 05:58:19.089854 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:19.089793    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-8rlv9" event={"ID":"9c735929-00ab-4837-b588-87dc5e1b5d87","Type":"ContainerDied","Data":"b0f06f66a1c6522b37cc5fb23b64942a792616e3add22fdb4d83f83eaae849a9"}
Sep 09 05:58:19.097750 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:19.097728    2579 scope.go:117] "RemoveContainer" containerID="64d14653fbc7fcf49b2ac5a7e05454da3fb4d7696ce1f024fe029519d3fb5991"
Sep 09 05:58:19.098030 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:58:19.098004    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64d14653fbc7fcf49b2ac5a7e05454da3fb4d7696ce1f024fe029519d3fb5991\": container with ID starting with 64d14653fbc7fcf49b2ac5a7e05454da3fb4d7696ce1f024fe029519d3fb5991 not found: ID does not exist" containerID="64d14653fbc7fcf49b2ac5a7e05454da3fb4d7696ce1f024fe029519d3fb5991"
Sep 09 05:58:19.098106 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:19.098044    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64d14653fbc7fcf49b2ac5a7e05454da3fb4d7696ce1f024fe029519d3fb5991"} err="failed to get container status \"64d14653fbc7fcf49b2ac5a7e05454da3fb4d7696ce1f024fe029519d3fb5991\": rpc error: code = NotFound desc = could not find container \"64d14653fbc7fcf49b2ac5a7e05454da3fb4d7696ce1f024fe029519d3fb5991\": container with ID starting with 64d14653fbc7fcf49b2ac5a7e05454da3fb4d7696ce1f024fe029519d3fb5991 not found: ID does not exist"
Sep 09 05:58:19.098106 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:19.098068    2579 scope.go:117] "RemoveContainer" containerID="0f274289edeac9f20f2bbd05448a426fe3118c1be61ebd69d90b059821baae6b"
Sep 09 05:58:19.105386 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:19.105370    2579 scope.go:117] "RemoveContainer" containerID="0f274289edeac9f20f2bbd05448a426fe3118c1be61ebd69d90b059821baae6b"
Sep 09 05:58:19.105655 ip-10-0-131-142 kubenswrapper[2579]: E0909 05:58:19.105636    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f274289edeac9f20f2bbd05448a426fe3118c1be61ebd69d90b059821baae6b\": container with ID starting with 0f274289edeac9f20f2bbd05448a426fe3118c1be61ebd69d90b059821baae6b not found: ID does not exist" containerID="0f274289edeac9f20f2bbd05448a426fe3118c1be61ebd69d90b059821baae6b"
Sep 09 05:58:19.105711 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:19.105661    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f274289edeac9f20f2bbd05448a426fe3118c1be61ebd69d90b059821baae6b"} err="failed to get container status \"0f274289edeac9f20f2bbd05448a426fe3118c1be61ebd69d90b059821baae6b\": rpc error: code = NotFound desc = could not find container \"0f274289edeac9f20f2bbd05448a426fe3118c1be61ebd69d90b059821baae6b\": container with ID starting with 0f274289edeac9f20f2bbd05448a426fe3118c1be61ebd69d90b059821baae6b not found: ID does not exist"
Sep 09 05:58:19.107703 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:19.107685    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-gbtkz"]
Sep 09 05:58:19.109747 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:19.109730    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-gbtkz"]
Sep 09 05:58:19.118829 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:19.118810    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-8rlv9"]
Sep 09 05:58:19.122894 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:19.122879    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-8rlv9"]
Sep 09 05:58:19.147754 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:19.147736    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="926a11e8-8ddc-4736-8300-6a5b55ccb73c" path="/var/lib/kubelet/pods/926a11e8-8ddc-4736-8300-6a5b55ccb73c/volumes"
Sep 09 05:58:19.148056 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:58:19.148043    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c735929-00ab-4837-b588-87dc5e1b5d87" path="/var/lib/kubelet/pods/9c735929-00ab-4837-b588-87dc5e1b5d87/volumes"
Sep 09 05:59:00.545913 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:59:00.545878    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 05:59:21.899587 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:59:21.899550    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 05:59:33.692480 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:59:33.692438    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 05:59:44.700783 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:59:44.700749    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 05:59:54.195995 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:59:54.195959    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 05:59:59.896767 ip-10-0-131-142 kubenswrapper[2579]: I0909 05:59:59.896679    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:00:00.135496 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:00.135467    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29815560-pmw8d"]
Sep 09 06:00:00.135761 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:00.135748    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="926a11e8-8ddc-4736-8300-6a5b55ccb73c" containerName="authorino"
Sep 09 06:00:00.135810 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:00.135763    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="926a11e8-8ddc-4736-8300-6a5b55ccb73c" containerName="authorino"
Sep 09 06:00:00.135810 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:00.135771    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c735929-00ab-4837-b588-87dc5e1b5d87" containerName="authorino"
Sep 09 06:00:00.135810 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:00.135777    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c735929-00ab-4837-b588-87dc5e1b5d87" containerName="authorino"
Sep 09 06:00:00.135902 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:00.135842    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c735929-00ab-4837-b588-87dc5e1b5d87" containerName="authorino"
Sep 09 06:00:00.135902 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:00.135851    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="926a11e8-8ddc-4736-8300-6a5b55ccb73c" containerName="authorino"
Sep 09 06:00:00.138571 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:00.138555    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815560-pmw8d"
Sep 09 06:00:00.140588 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:00.140565    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ksjk6\""
Sep 09 06:00:00.144680 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:00.144654    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29815560-pmw8d"]
Sep 09 06:00:00.149523 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:00.149470    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815560-pmw8d"
Sep 09 06:00:00.422272 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:00.422200    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29815560-pmw8d"]
Sep 09 06:00:00.425204 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:00:00.425176    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8bc8589_e0c9_4be7_a38d_e3ec3f9bbbf6.slice/crio-2383950667ed9422fbbb616486353f6c56ed37b90d5fda7fa3638a4e292788a0 WatchSource:0}: Error finding container 2383950667ed9422fbbb616486353f6c56ed37b90d5fda7fa3638a4e292788a0: Status 404 returned error can't find the container with id 2383950667ed9422fbbb616486353f6c56ed37b90d5fda7fa3638a4e292788a0
Sep 09 06:00:00.427358 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:00.427343    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 06:00:00.479393 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:00.479363    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815560-pmw8d" event={"ID":"a8bc8589-e0c9-4be7-a38d-e3ec3f9bbbf6","Type":"ContainerStarted","Data":"2383950667ed9422fbbb616486353f6c56ed37b90d5fda7fa3638a4e292788a0"}
Sep 09 06:00:04.497313 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:04.497274    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815560-pmw8d" event={"ID":"a8bc8589-e0c9-4be7-a38d-e3ec3f9bbbf6","Type":"ContainerStarted","Data":"a4680a14e8188a530d2dd3c154223afcb8ec070fe1f7523900fd3a28b25fc08a"}
Sep 09 06:00:04.512123 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:04.512083    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815560-pmw8d" podStartSLOduration=0.966023468 podStartE2EDuration="4.512070231s" totalImagesPullingTime="3.546046763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:00:04.511461656 +0000 UTC m=+629.851949567" watchObservedRunningTime="2026-09-09 06:00:04.512070231 +0000 UTC m=+629.852558151"
Sep 09 06:00:05.502646 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:05.502593    2579 generic.go:356] "Generic (PLEG): container finished" podID="a8bc8589-e0c9-4be7-a38d-e3ec3f9bbbf6" containerID="a4680a14e8188a530d2dd3c154223afcb8ec070fe1f7523900fd3a28b25fc08a" exitCode=0
Sep 09 06:00:05.503045 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:05.502652    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815560-pmw8d" event={"ID":"a8bc8589-e0c9-4be7-a38d-e3ec3f9bbbf6","Type":"ContainerDied","Data":"a4680a14e8188a530d2dd3c154223afcb8ec070fe1f7523900fd3a28b25fc08a"}
Sep 09 06:00:06.625298 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:06.625275    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815560-pmw8d"
Sep 09 06:00:07.511829 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:07.511761    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815560-pmw8d"
Sep 09 06:00:07.511953 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:07.511756    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815560-pmw8d" event={"ID":"a8bc8589-e0c9-4be7-a38d-e3ec3f9bbbf6","Type":"ContainerDied","Data":"2383950667ed9422fbbb616486353f6c56ed37b90d5fda7fa3638a4e292788a0"}
Sep 09 06:00:07.511953 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:07.511893    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2383950667ed9422fbbb616486353f6c56ed37b90d5fda7fa3638a4e292788a0"
Sep 09 06:00:09.231960 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:09.231930    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-vzp9x"]
Sep 09 06:00:09.232303 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:09.232203    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8bc8589-e0c9-4be7-a38d-e3ec3f9bbbf6" containerName="cleanup"
Sep 09 06:00:09.232303 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:09.232215    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8bc8589-e0c9-4be7-a38d-e3ec3f9bbbf6" containerName="cleanup"
Sep 09 06:00:09.232303 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:09.232265    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8bc8589-e0c9-4be7-a38d-e3ec3f9bbbf6" containerName="cleanup"
Sep 09 06:00:09.234988 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:09.234972    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vzp9x"
Sep 09 06:00:09.238315 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:09.238289    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-w84m4\""
Sep 09 06:00:09.238423 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:09.238323    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 09 06:00:09.238423 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:09.238293    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 09 06:00:09.238423 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:09.238418    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 09 06:00:09.242779 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:09.242759    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-vzp9x"]
Sep 09 06:00:09.321079 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:09.321048    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68d348a3-d2ba-4526-ac49-64c657279cd0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vzp9x\" (UID: \"68d348a3-d2ba-4526-ac49-64c657279cd0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vzp9x"
Sep 09 06:00:09.321079 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:09.321078    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbb6v\" (UniqueName: \"kubernetes.io/projected/68d348a3-d2ba-4526-ac49-64c657279cd0-kube-api-access-fbb6v\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vzp9x\" (UID: \"68d348a3-d2ba-4526-ac49-64c657279cd0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vzp9x"
Sep 09 06:00:09.321240 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:09.321104    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68d348a3-d2ba-4526-ac49-64c657279cd0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vzp9x\" (UID: \"68d348a3-d2ba-4526-ac49-64c657279cd0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vzp9x"
Sep 09 06:00:09.321240 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:09.321216    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68d348a3-d2ba-4526-ac49-64c657279cd0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vzp9x\" (UID: \"68d348a3-d2ba-4526-ac49-64c657279cd0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vzp9x"
Sep 09 06:00:09.321304 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:09.321252    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68d348a3-d2ba-4526-ac49-64c657279cd0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vzp9x\" (UID: \"68d348a3-d2ba-4526-ac49-64c657279cd0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vzp9x"
Sep 09 06:00:09.321304 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:09.321289    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68d348a3-d2ba-4526-ac49-64c657279cd0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vzp9x\" (UID: \"68d348a3-d2ba-4526-ac49-64c657279cd0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vzp9x"
Sep 09 06:00:09.422202 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:09.422173    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68d348a3-d2ba-4526-ac49-64c657279cd0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vzp9x\" (UID: \"68d348a3-d2ba-4526-ac49-64c657279cd0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vzp9x"
Sep 09 06:00:09.422377 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:09.422211    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68d348a3-d2ba-4526-ac49-64c657279cd0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vzp9x\" (UID: \"68d348a3-d2ba-4526-ac49-64c657279cd0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vzp9x"
Sep 09 06:00:09.422377 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:09.422237    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68d348a3-d2ba-4526-ac49-64c657279cd0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vzp9x\" (UID: \"68d348a3-d2ba-4526-ac49-64c657279cd0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vzp9x"
Sep 09 06:00:09.422377 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:09.422272    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68d348a3-d2ba-4526-ac49-64c657279cd0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vzp9x\" (UID: \"68d348a3-d2ba-4526-ac49-64c657279cd0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vzp9x"
Sep 09 06:00:09.422377 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:09.422296    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbb6v\" (UniqueName: \"kubernetes.io/projected/68d348a3-d2ba-4526-ac49-64c657279cd0-kube-api-access-fbb6v\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vzp9x\" (UID: \"68d348a3-d2ba-4526-ac49-64c657279cd0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vzp9x"
Sep 09 06:00:09.422377 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:09.422322    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68d348a3-d2ba-4526-ac49-64c657279cd0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vzp9x\" (UID: \"68d348a3-d2ba-4526-ac49-64c657279cd0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vzp9x"
Sep 09 06:00:09.422649 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:09.422608    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68d348a3-d2ba-4526-ac49-64c657279cd0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vzp9x\" (UID: \"68d348a3-d2ba-4526-ac49-64c657279cd0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vzp9x"
Sep 09 06:00:09.422820 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:09.422709    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68d348a3-d2ba-4526-ac49-64c657279cd0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vzp9x\" (UID: \"68d348a3-d2ba-4526-ac49-64c657279cd0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vzp9x"
Sep 09 06:00:09.422906 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:09.422749    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68d348a3-d2ba-4526-ac49-64c657279cd0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vzp9x\" (UID: \"68d348a3-d2ba-4526-ac49-64c657279cd0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vzp9x"
Sep 09 06:00:09.424753 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:09.424735    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68d348a3-d2ba-4526-ac49-64c657279cd0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vzp9x\" (UID: \"68d348a3-d2ba-4526-ac49-64c657279cd0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vzp9x"
Sep 09 06:00:09.425112 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:09.425092    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68d348a3-d2ba-4526-ac49-64c657279cd0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vzp9x\" (UID: \"68d348a3-d2ba-4526-ac49-64c657279cd0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vzp9x"
Sep 09 06:00:09.435760 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:09.435741    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbb6v\" (UniqueName: \"kubernetes.io/projected/68d348a3-d2ba-4526-ac49-64c657279cd0-kube-api-access-fbb6v\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vzp9x\" (UID: \"68d348a3-d2ba-4526-ac49-64c657279cd0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vzp9x"
Sep 09 06:00:09.545586 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:09.545512    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vzp9x"
Sep 09 06:00:09.815774 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:09.815737    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-vzp9x"]
Sep 09 06:00:09.818447 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:00:09.818420    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68d348a3_d2ba_4526_ac49_64c657279cd0.slice/crio-a6556b78922dd470f2ff1751788c9dbee0cf92156caeef06f24334dcf64fc38b WatchSource:0}: Error finding container a6556b78922dd470f2ff1751788c9dbee0cf92156caeef06f24334dcf64fc38b: Status 404 returned error can't find the container with id a6556b78922dd470f2ff1751788c9dbee0cf92156caeef06f24334dcf64fc38b
Sep 09 06:00:10.524471 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:10.524428    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vzp9x" event={"ID":"68d348a3-d2ba-4526-ac49-64c657279cd0","Type":"ContainerStarted","Data":"a6556b78922dd470f2ff1751788c9dbee0cf92156caeef06f24334dcf64fc38b"}
Sep 09 06:00:11.199615 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:11.199538    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:00:11.531937 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:11.531860    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vzp9x" event={"ID":"68d348a3-d2ba-4526-ac49-64c657279cd0","Type":"ContainerStarted","Data":"df7a576edac8a11c15e7972430e4a012042f15751a348349a5675a308dbc04ab"}
Sep 09 06:00:11.549393 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:11.549347    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vzp9x" podStartSLOduration=1.470001189 podStartE2EDuration="2.54933331s" totalImagesPullingTime="1.079332121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:00:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:00:11.547714507 +0000 UTC m=+636.888202418" watchObservedRunningTime="2026-09-09 06:00:11.54933331 +0000 UTC m=+636.889821218"
Sep 09 06:00:19.546019 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:19.545978    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vzp9x"
Sep 09 06:00:19.546525 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:19.546040    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vzp9x"
Sep 09 06:00:19.558699 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:19.558676    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vzp9x"
Sep 09 06:00:19.573164 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:19.573135    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vzp9x"
Sep 09 06:00:21.118492 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:21.118452    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw"]
Sep 09 06:00:21.125484 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:21.125454    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw"
Sep 09 06:00:21.126988 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:21.126966    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw"]
Sep 09 06:00:21.127534 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:21.127516    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 09 06:00:21.210062 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:21.210029    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqg7q\" (UniqueName: \"kubernetes.io/projected/277bc490-a853-416c-a048-f06558f49f85-kube-api-access-jqg7q\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw\" (UID: \"277bc490-a853-416c-a048-f06558f49f85\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw"
Sep 09 06:00:21.210217 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:21.210076    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/277bc490-a853-416c-a048-f06558f49f85-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw\" (UID: \"277bc490-a853-416c-a048-f06558f49f85\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw"
Sep 09 06:00:21.210217 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:21.210100    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/277bc490-a853-416c-a048-f06558f49f85-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw\" (UID: \"277bc490-a853-416c-a048-f06558f49f85\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw"
Sep 09 06:00:21.210217 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:21.210193    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/277bc490-a853-416c-a048-f06558f49f85-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw\" (UID: \"277bc490-a853-416c-a048-f06558f49f85\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw"
Sep 09 06:00:21.210325 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:21.210225    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/277bc490-a853-416c-a048-f06558f49f85-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw\" (UID: \"277bc490-a853-416c-a048-f06558f49f85\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw"
Sep 09 06:00:21.210325 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:21.210257    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/277bc490-a853-416c-a048-f06558f49f85-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw\" (UID: \"277bc490-a853-416c-a048-f06558f49f85\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw"
Sep 09 06:00:21.310866 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:21.310834    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/277bc490-a853-416c-a048-f06558f49f85-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw\" (UID: \"277bc490-a853-416c-a048-f06558f49f85\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw"
Sep 09 06:00:21.310866 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:21.310871    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/277bc490-a853-416c-a048-f06558f49f85-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw\" (UID: \"277bc490-a853-416c-a048-f06558f49f85\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw"
Sep 09 06:00:21.311064 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:21.310901    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/277bc490-a853-416c-a048-f06558f49f85-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw\" (UID: \"277bc490-a853-416c-a048-f06558f49f85\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw"
Sep 09 06:00:21.311064 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:21.310982    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqg7q\" (UniqueName: \"kubernetes.io/projected/277bc490-a853-416c-a048-f06558f49f85-kube-api-access-jqg7q\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw\" (UID: \"277bc490-a853-416c-a048-f06558f49f85\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw"
Sep 09 06:00:21.311273 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:21.311247    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/277bc490-a853-416c-a048-f06558f49f85-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw\" (UID: \"277bc490-a853-416c-a048-f06558f49f85\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw"
Sep 09 06:00:21.311423 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:21.311310    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/277bc490-a853-416c-a048-f06558f49f85-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw\" (UID: \"277bc490-a853-416c-a048-f06558f49f85\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw"
Sep 09 06:00:21.311423 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:21.311347    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/277bc490-a853-416c-a048-f06558f49f85-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw\" (UID: \"277bc490-a853-416c-a048-f06558f49f85\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw"
Sep 09 06:00:21.311718 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:21.311609    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/277bc490-a853-416c-a048-f06558f49f85-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw\" (UID: \"277bc490-a853-416c-a048-f06558f49f85\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw"
Sep 09 06:00:21.312056 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:21.312030    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/277bc490-a853-416c-a048-f06558f49f85-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw\" (UID: \"277bc490-a853-416c-a048-f06558f49f85\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw"
Sep 09 06:00:21.313334 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:21.313313    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/277bc490-a853-416c-a048-f06558f49f85-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw\" (UID: \"277bc490-a853-416c-a048-f06558f49f85\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw"
Sep 09 06:00:21.313617 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:21.313599    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/277bc490-a853-416c-a048-f06558f49f85-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw\" (UID: \"277bc490-a853-416c-a048-f06558f49f85\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw"
Sep 09 06:00:21.317906 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:21.317885    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqg7q\" (UniqueName: \"kubernetes.io/projected/277bc490-a853-416c-a048-f06558f49f85-kube-api-access-jqg7q\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw\" (UID: \"277bc490-a853-416c-a048-f06558f49f85\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw"
Sep 09 06:00:21.435236 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:21.435166    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw"
Sep 09 06:00:21.705591 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:21.705521    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw"]
Sep 09 06:00:21.708512 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:00:21.708481    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod277bc490_a853_416c_a048_f06558f49f85.slice/crio-1690429eb76668324891b81bed719e1cfc46e50906f3e6ee284b27fe65e624bd WatchSource:0}: Error finding container 1690429eb76668324891b81bed719e1cfc46e50906f3e6ee284b27fe65e624bd: Status 404 returned error can't find the container with id 1690429eb76668324891b81bed719e1cfc46e50906f3e6ee284b27fe65e624bd
Sep 09 06:00:22.574545 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:22.574517    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw" event={"ID":"277bc490-a853-416c-a048-f06558f49f85","Type":"ContainerStarted","Data":"be6a0c737d0682e315c1388d739c068c4df82efaa4033e5d9db444549d7aaf4c"}
Sep 09 06:00:22.574942 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:22.574553    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw" event={"ID":"277bc490-a853-416c-a048-f06558f49f85","Type":"ContainerStarted","Data":"1690429eb76668324891b81bed719e1cfc46e50906f3e6ee284b27fe65e624bd"}
Sep 09 06:00:22.592431 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:22.592384    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw" podStartSLOduration=1.395009296 podStartE2EDuration="1.592369624s" totalImagesPullingTime="197.360328ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:00:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:00:22.590798989 +0000 UTC m=+647.931286898" watchObservedRunningTime="2026-09-09 06:00:22.592369624 +0000 UTC m=+647.932857532"
Sep 09 06:00:23.398607 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:23.398575    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:00:31.435706 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:31.435677    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw"
Sep 09 06:00:31.436083 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:31.435932    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw"
Sep 09 06:00:31.448168 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:31.448143    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw"
Sep 09 06:00:31.621911 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:00:31.621883    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw"
Sep 09 06:01:26.812662 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:26.812565    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-788587d9b8-42kfs"]
Sep 09 06:01:26.815771 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:26.815755    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-788587d9b8-42kfs"
Sep 09 06:01:26.823552 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:26.823527    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-788587d9b8-42kfs"]
Sep 09 06:01:26.881324 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:26.881296    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7456999c-8987-428f-ac8d-4065a6fb5f76-tls-cert\") pod \"authorino-788587d9b8-42kfs\" (UID: \"7456999c-8987-428f-ac8d-4065a6fb5f76\") " pod="kuadrant-system/authorino-788587d9b8-42kfs"
Sep 09 06:01:26.881429 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:26.881337    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c847h\" (UniqueName: \"kubernetes.io/projected/7456999c-8987-428f-ac8d-4065a6fb5f76-kube-api-access-c847h\") pod \"authorino-788587d9b8-42kfs\" (UID: \"7456999c-8987-428f-ac8d-4065a6fb5f76\") " pod="kuadrant-system/authorino-788587d9b8-42kfs"
Sep 09 06:01:26.981754 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:26.981728    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c847h\" (UniqueName: \"kubernetes.io/projected/7456999c-8987-428f-ac8d-4065a6fb5f76-kube-api-access-c847h\") pod \"authorino-788587d9b8-42kfs\" (UID: \"7456999c-8987-428f-ac8d-4065a6fb5f76\") " pod="kuadrant-system/authorino-788587d9b8-42kfs"
Sep 09 06:01:26.981853 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:26.981788    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7456999c-8987-428f-ac8d-4065a6fb5f76-tls-cert\") pod \"authorino-788587d9b8-42kfs\" (UID: \"7456999c-8987-428f-ac8d-4065a6fb5f76\") " pod="kuadrant-system/authorino-788587d9b8-42kfs"
Sep 09 06:01:26.984166 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:26.984147    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7456999c-8987-428f-ac8d-4065a6fb5f76-tls-cert\") pod \"authorino-788587d9b8-42kfs\" (UID: \"7456999c-8987-428f-ac8d-4065a6fb5f76\") " pod="kuadrant-system/authorino-788587d9b8-42kfs"
Sep 09 06:01:26.989349 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:26.989319    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c847h\" (UniqueName: \"kubernetes.io/projected/7456999c-8987-428f-ac8d-4065a6fb5f76-kube-api-access-c847h\") pod \"authorino-788587d9b8-42kfs\" (UID: \"7456999c-8987-428f-ac8d-4065a6fb5f76\") " pod="kuadrant-system/authorino-788587d9b8-42kfs"
Sep 09 06:01:27.125281 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:27.125229    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-788587d9b8-42kfs"
Sep 09 06:01:27.392898 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:27.392819    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-788587d9b8-42kfs"]
Sep 09 06:01:27.397366 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:01:27.397336    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7456999c_8987_428f_ac8d_4065a6fb5f76.slice/crio-fce189c54a0a23a332d10438440dbd5f5bfe68fd89d4c86e2c485e52b2b1a4ed WatchSource:0}: Error finding container fce189c54a0a23a332d10438440dbd5f5bfe68fd89d4c86e2c485e52b2b1a4ed: Status 404 returned error can't find the container with id fce189c54a0a23a332d10438440dbd5f5bfe68fd89d4c86e2c485e52b2b1a4ed
Sep 09 06:01:27.835013 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:27.834980    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-788587d9b8-42kfs" event={"ID":"7456999c-8987-428f-ac8d-4065a6fb5f76","Type":"ContainerStarted","Data":"fce189c54a0a23a332d10438440dbd5f5bfe68fd89d4c86e2c485e52b2b1a4ed"}
Sep 09 06:01:28.843341 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:28.843295    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-788587d9b8-42kfs" event={"ID":"7456999c-8987-428f-ac8d-4065a6fb5f76","Type":"ContainerStarted","Data":"06bc9d76de265360042932dec75e6b14b0e265d8eae4c7218719d87bab24e963"}
Sep 09 06:01:28.860827 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:28.860770    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-788587d9b8-42kfs" podStartSLOduration=2.030725567 podStartE2EDuration="2.860753585s" totalImagesPullingTime="830.028018ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:01:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:01:28.859709529 +0000 UTC m=+714.200197439" watchObservedRunningTime="2026-09-09 06:01:28.860753585 +0000 UTC m=+714.201241507"
Sep 09 06:01:28.887780 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:28.887742    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-9479df6d9-r5mlh"]
Sep 09 06:01:28.887955 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:28.887921    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-9479df6d9-r5mlh" podUID="4786ff3c-db02-4087-8b02-79babede8757" containerName="authorino" containerID="cri-o://552fcd1452ad5eb071666f92af4ba8a929a83ad9ca431c69edbe42f41fde368c" gracePeriod=30
Sep 09 06:01:29.148477 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:29.148452    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9479df6d9-r5mlh"
Sep 09 06:01:29.198836 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:29.198801    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4s7n\" (UniqueName: \"kubernetes.io/projected/4786ff3c-db02-4087-8b02-79babede8757-kube-api-access-g4s7n\") pod \"4786ff3c-db02-4087-8b02-79babede8757\" (UID: \"4786ff3c-db02-4087-8b02-79babede8757\") "
Sep 09 06:01:29.199026 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:29.198847    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4786ff3c-db02-4087-8b02-79babede8757-tls-cert\") pod \"4786ff3c-db02-4087-8b02-79babede8757\" (UID: \"4786ff3c-db02-4087-8b02-79babede8757\") "
Sep 09 06:01:29.201263 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:29.201231    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4786ff3c-db02-4087-8b02-79babede8757-kube-api-access-g4s7n" (OuterVolumeSpecName: "kube-api-access-g4s7n") pod "4786ff3c-db02-4087-8b02-79babede8757" (UID: "4786ff3c-db02-4087-8b02-79babede8757"). InnerVolumeSpecName "kube-api-access-g4s7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:01:29.210436 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:29.210411    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4786ff3c-db02-4087-8b02-79babede8757-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "4786ff3c-db02-4087-8b02-79babede8757" (UID: "4786ff3c-db02-4087-8b02-79babede8757"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:01:29.300052 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:29.300022    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4s7n\" (UniqueName: \"kubernetes.io/projected/4786ff3c-db02-4087-8b02-79babede8757-kube-api-access-g4s7n\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:01:29.300052 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:29.300049    2579 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4786ff3c-db02-4087-8b02-79babede8757-tls-cert\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:01:29.848637 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:29.848600    2579 generic.go:356] "Generic (PLEG): container finished" podID="4786ff3c-db02-4087-8b02-79babede8757" containerID="552fcd1452ad5eb071666f92af4ba8a929a83ad9ca431c69edbe42f41fde368c" exitCode=0
Sep 09 06:01:29.849113 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:29.848658    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9479df6d9-r5mlh" event={"ID":"4786ff3c-db02-4087-8b02-79babede8757","Type":"ContainerDied","Data":"552fcd1452ad5eb071666f92af4ba8a929a83ad9ca431c69edbe42f41fde368c"}
Sep 09 06:01:29.849113 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:29.848699    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9479df6d9-r5mlh" event={"ID":"4786ff3c-db02-4087-8b02-79babede8757","Type":"ContainerDied","Data":"cffe26231698322719d5f192c49f0e066fa4ab7f13553b17b8288070617ed791"}
Sep 09 06:01:29.849113 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:29.848717    2579 scope.go:117] "RemoveContainer" containerID="552fcd1452ad5eb071666f92af4ba8a929a83ad9ca431c69edbe42f41fde368c"
Sep 09 06:01:29.849113 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:29.848666    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9479df6d9-r5mlh"
Sep 09 06:01:29.859037 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:29.859013    2579 scope.go:117] "RemoveContainer" containerID="552fcd1452ad5eb071666f92af4ba8a929a83ad9ca431c69edbe42f41fde368c"
Sep 09 06:01:29.859316 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:01:29.859297    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"552fcd1452ad5eb071666f92af4ba8a929a83ad9ca431c69edbe42f41fde368c\": container with ID starting with 552fcd1452ad5eb071666f92af4ba8a929a83ad9ca431c69edbe42f41fde368c not found: ID does not exist" containerID="552fcd1452ad5eb071666f92af4ba8a929a83ad9ca431c69edbe42f41fde368c"
Sep 09 06:01:29.859382 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:29.859325    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"552fcd1452ad5eb071666f92af4ba8a929a83ad9ca431c69edbe42f41fde368c"} err="failed to get container status \"552fcd1452ad5eb071666f92af4ba8a929a83ad9ca431c69edbe42f41fde368c\": rpc error: code = NotFound desc = could not find container \"552fcd1452ad5eb071666f92af4ba8a929a83ad9ca431c69edbe42f41fde368c\": container with ID starting with 552fcd1452ad5eb071666f92af4ba8a929a83ad9ca431c69edbe42f41fde368c not found: ID does not exist"
Sep 09 06:01:29.869027 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:29.869000    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-9479df6d9-r5mlh"]
Sep 09 06:01:29.872718 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:29.872694    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-9479df6d9-r5mlh"]
Sep 09 06:01:31.147694 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:31.147657    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4786ff3c-db02-4087-8b02-79babede8757" path="/var/lib/kubelet/pods/4786ff3c-db02-4087-8b02-79babede8757/volumes"
Sep 09 06:01:49.827472 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:49.827428    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7991-139747575834288"]
Sep 09 06:01:49.827982 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:49.827733    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4786ff3c-db02-4087-8b02-79babede8757" containerName="authorino"
Sep 09 06:01:49.827982 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:49.827746    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4786ff3c-db02-4087-8b02-79babede8757" containerName="authorino"
Sep 09 06:01:49.827982 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:49.827801    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4786ff3c-db02-4087-8b02-79babede8757" containerName="authorino"
Sep 09 06:01:49.838173 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:49.838133    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7991-139747575834288"
Sep 09 06:01:49.839008 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:49.838985    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7991-139747575834288"]
Sep 09 06:01:49.842285 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:49.842268    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vljc5\""
Sep 09 06:01:49.953709 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:49.953681    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plr9m\" (UniqueName: \"kubernetes.io/projected/b0214c9b-ca1d-47f1-adf0-a60f2eda77b0-kube-api-access-plr9m\") pod \"test-curl-7991-139747575834288\" (UID: \"b0214c9b-ca1d-47f1-adf0-a60f2eda77b0\") " pod="opendatahub/test-curl-7991-139747575834288"
Sep 09 06:01:50.055051 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:50.055013    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plr9m\" (UniqueName: \"kubernetes.io/projected/b0214c9b-ca1d-47f1-adf0-a60f2eda77b0-kube-api-access-plr9m\") pod \"test-curl-7991-139747575834288\" (UID: \"b0214c9b-ca1d-47f1-adf0-a60f2eda77b0\") " pod="opendatahub/test-curl-7991-139747575834288"
Sep 09 06:01:50.064511 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:50.064477    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plr9m\" (UniqueName: \"kubernetes.io/projected/b0214c9b-ca1d-47f1-adf0-a60f2eda77b0-kube-api-access-plr9m\") pod \"test-curl-7991-139747575834288\" (UID: \"b0214c9b-ca1d-47f1-adf0-a60f2eda77b0\") " pod="opendatahub/test-curl-7991-139747575834288"
Sep 09 06:01:50.147745 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:50.147687    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7991-139747575834288"
Sep 09 06:01:50.413398 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:50.413374    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7991-139747575834288"]
Sep 09 06:01:50.416043 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:01:50.416015    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0214c9b_ca1d_47f1_adf0_a60f2eda77b0.slice/crio-daf37d0fb4641d8d229c0b4e7ff6931b0e0934c983662137d9d407ea1d072ed0 WatchSource:0}: Error finding container daf37d0fb4641d8d229c0b4e7ff6931b0e0934c983662137d9d407ea1d072ed0: Status 404 returned error can't find the container with id daf37d0fb4641d8d229c0b4e7ff6931b0e0934c983662137d9d407ea1d072ed0
Sep 09 06:01:50.944531 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:50.944497    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7991-139747575834288" event={"ID":"b0214c9b-ca1d-47f1-adf0-a60f2eda77b0","Type":"ContainerStarted","Data":"daf37d0fb4641d8d229c0b4e7ff6931b0e0934c983662137d9d407ea1d072ed0"}
Sep 09 06:01:51.600319 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:51.600276    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:01:52.954901 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:52.954865    2579 generic.go:356] "Generic (PLEG): container finished" podID="b0214c9b-ca1d-47f1-adf0-a60f2eda77b0" containerID="032f390afb3d0de7b71fc02d950367b44e8b48dcdd6241b3e0b3b46e2df28a90" exitCode=0
Sep 09 06:01:52.955277 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:52.954912    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7991-139747575834288" event={"ID":"b0214c9b-ca1d-47f1-adf0-a60f2eda77b0","Type":"ContainerDied","Data":"032f390afb3d0de7b71fc02d950367b44e8b48dcdd6241b3e0b3b46e2df28a90"}
Sep 09 06:01:54.086883 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:54.086862    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7991-139747575834288"
Sep 09 06:01:54.119031 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:54.119003    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7991-139747575834288_b0214c9b-ca1d-47f1-adf0-a60f2eda77b0/test-curl-7991-139747575834288/0.log"
Sep 09 06:01:54.123011 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:54.122987    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ecd13e-6bc8846b59-vq8gl"]
Sep 09 06:01:54.123306 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:54.123289    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0214c9b-ca1d-47f1-adf0-a60f2eda77b0" containerName="test-curl-7991-139747575834288"
Sep 09 06:01:54.123375 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:54.123309    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0214c9b-ca1d-47f1-adf0-a60f2eda77b0" containerName="test-curl-7991-139747575834288"
Sep 09 06:01:54.123423 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:54.123394    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0214c9b-ca1d-47f1-adf0-a60f2eda77b0" containerName="test-curl-7991-139747575834288"
Sep 09 06:01:54.126364 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:54.126346    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ecd13e-6bc8846b59-vq8gl"
Sep 09 06:01:54.128866 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:54.128841    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-ecd13e-dockercfg-rvtmv\""
Sep 09 06:01:54.129288 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:54.129271    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ecd13e\""
Sep 09 06:01:54.139660 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:54.139639    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ecd13e-6bc8846b59-vq8gl"]
Sep 09 06:01:54.149852 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:54.149832    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7991-139747575834288"]
Sep 09 06:01:54.154035 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:54.154015    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7991-139747575834288"]
Sep 09 06:01:54.192643 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:54.192604    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plr9m\" (UniqueName: \"kubernetes.io/projected/b0214c9b-ca1d-47f1-adf0-a60f2eda77b0-kube-api-access-plr9m\") pod \"b0214c9b-ca1d-47f1-adf0-a60f2eda77b0\" (UID: \"b0214c9b-ca1d-47f1-adf0-a60f2eda77b0\") "
Sep 09 06:01:54.194865 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:54.194842    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0214c9b-ca1d-47f1-adf0-a60f2eda77b0-kube-api-access-plr9m" (OuterVolumeSpecName: "kube-api-access-plr9m") pod "b0214c9b-ca1d-47f1-adf0-a60f2eda77b0" (UID: "b0214c9b-ca1d-47f1-adf0-a60f2eda77b0"). InnerVolumeSpecName "kube-api-access-plr9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:01:54.275710 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:54.275661    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7991-139747575834288"]
Sep 09 06:01:54.278987 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:54.278970    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7991-139747575834288"
Sep 09 06:01:54.285869 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:54.285838    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7991-139747575834288"]
Sep 09 06:01:54.293363 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:54.293344    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rv56\" (UniqueName: \"kubernetes.io/projected/c727c74f-dc8b-4efd-9bf2-10bdc66c2c08-kube-api-access-8rv56\") pod \"payload-processing-e2e-inf-a-ecd13e-6bc8846b59-vq8gl\" (UID: \"c727c74f-dc8b-4efd-9bf2-10bdc66c2c08\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ecd13e-6bc8846b59-vq8gl"
Sep 09 06:01:54.293493 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:54.293405    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c727c74f-dc8b-4efd-9bf2-10bdc66c2c08-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ecd13e-6bc8846b59-vq8gl\" (UID: \"c727c74f-dc8b-4efd-9bf2-10bdc66c2c08\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ecd13e-6bc8846b59-vq8gl"
Sep 09 06:01:54.293569 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:54.293520    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plr9m\" (UniqueName: \"kubernetes.io/projected/b0214c9b-ca1d-47f1-adf0-a60f2eda77b0-kube-api-access-plr9m\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:01:54.394425 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:54.394392    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c727c74f-dc8b-4efd-9bf2-10bdc66c2c08-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ecd13e-6bc8846b59-vq8gl\" (UID: \"c727c74f-dc8b-4efd-9bf2-10bdc66c2c08\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ecd13e-6bc8846b59-vq8gl"
Sep 09 06:01:54.394536 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:54.394447    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psn27\" (UniqueName: \"kubernetes.io/projected/1ef81e3c-4419-4d33-a93e-a605d2be712e-kube-api-access-psn27\") pod \"test-curl-7991-139747575834288\" (UID: \"1ef81e3c-4419-4d33-a93e-a605d2be712e\") " pod="opendatahub/test-curl-7991-139747575834288"
Sep 09 06:01:54.394536 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:54.394496    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rv56\" (UniqueName: \"kubernetes.io/projected/c727c74f-dc8b-4efd-9bf2-10bdc66c2c08-kube-api-access-8rv56\") pod \"payload-processing-e2e-inf-a-ecd13e-6bc8846b59-vq8gl\" (UID: \"c727c74f-dc8b-4efd-9bf2-10bdc66c2c08\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ecd13e-6bc8846b59-vq8gl"
Sep 09 06:01:54.395102 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:54.395083    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c727c74f-dc8b-4efd-9bf2-10bdc66c2c08-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ecd13e-6bc8846b59-vq8gl\" (UID: \"c727c74f-dc8b-4efd-9bf2-10bdc66c2c08\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ecd13e-6bc8846b59-vq8gl"
Sep 09 06:01:54.403455 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:54.403437    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rv56\" (UniqueName: \"kubernetes.io/projected/c727c74f-dc8b-4efd-9bf2-10bdc66c2c08-kube-api-access-8rv56\") pod \"payload-processing-e2e-inf-a-ecd13e-6bc8846b59-vq8gl\" (UID: \"c727c74f-dc8b-4efd-9bf2-10bdc66c2c08\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ecd13e-6bc8846b59-vq8gl"
Sep 09 06:01:54.436409 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:54.436391    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ecd13e-6bc8846b59-vq8gl"
Sep 09 06:01:54.496100 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:54.496017    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psn27\" (UniqueName: \"kubernetes.io/projected/1ef81e3c-4419-4d33-a93e-a605d2be712e-kube-api-access-psn27\") pod \"test-curl-7991-139747575834288\" (UID: \"1ef81e3c-4419-4d33-a93e-a605d2be712e\") " pod="opendatahub/test-curl-7991-139747575834288"
Sep 09 06:01:54.504495 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:54.504463    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psn27\" (UniqueName: \"kubernetes.io/projected/1ef81e3c-4419-4d33-a93e-a605d2be712e-kube-api-access-psn27\") pod \"test-curl-7991-139747575834288\" (UID: \"1ef81e3c-4419-4d33-a93e-a605d2be712e\") " pod="opendatahub/test-curl-7991-139747575834288"
Sep 09 06:01:54.587896 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:54.587874    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7991-139747575834288"
Sep 09 06:01:54.711758 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:54.711732    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ecd13e-6bc8846b59-vq8gl"]
Sep 09 06:01:54.712730 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:01:54.712702    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc727c74f_dc8b_4efd_9bf2_10bdc66c2c08.slice/crio-d02d577c3f7798042fce4ef92ac16b14f0c74a85bdb31669703f0554a5889ab1 WatchSource:0}: Error finding container d02d577c3f7798042fce4ef92ac16b14f0c74a85bdb31669703f0554a5889ab1: Status 404 returned error can't find the container with id d02d577c3f7798042fce4ef92ac16b14f0c74a85bdb31669703f0554a5889ab1
Sep 09 06:01:54.858824 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:54.858758    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7991-139747575834288"]
Sep 09 06:01:54.861433 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:01:54.861412    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ef81e3c_4419_4d33_a93e_a605d2be712e.slice/crio-b097bf58f0edbdf68e7d353a50eb6456aafd7c6219a27e08933ebfead79b0674 WatchSource:0}: Error finding container b097bf58f0edbdf68e7d353a50eb6456aafd7c6219a27e08933ebfead79b0674: Status 404 returned error can't find the container with id b097bf58f0edbdf68e7d353a50eb6456aafd7c6219a27e08933ebfead79b0674
Sep 09 06:01:54.964138 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:54.964116    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="daf37d0fb4641d8d229c0b4e7ff6931b0e0934c983662137d9d407ea1d072ed0"
Sep 09 06:01:54.964138 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:54.964119    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7991-139747575834288"
Sep 09 06:01:54.965357 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:54.965326    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7991-139747575834288" event={"ID":"1ef81e3c-4419-4d33-a93e-a605d2be712e","Type":"ContainerStarted","Data":"b097bf58f0edbdf68e7d353a50eb6456aafd7c6219a27e08933ebfead79b0674"}
Sep 09 06:01:54.966532 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:54.966508    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ecd13e-6bc8846b59-vq8gl" event={"ID":"c727c74f-dc8b-4efd-9bf2-10bdc66c2c08","Type":"ContainerStarted","Data":"d02d577c3f7798042fce4ef92ac16b14f0c74a85bdb31669703f0554a5889ab1"}
Sep 09 06:01:54.977247 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:54.977217    2579 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7991-139747575834288" oldPodUID="b0214c9b-ca1d-47f1-adf0-a60f2eda77b0" podUID="1ef81e3c-4419-4d33-a93e-a605d2be712e"
Sep 09 06:01:55.147368 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:55.147293    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0214c9b-ca1d-47f1-adf0-a60f2eda77b0" path="/var/lib/kubelet/pods/b0214c9b-ca1d-47f1-adf0-a60f2eda77b0/volumes"
Sep 09 06:01:56.976719 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:56.976682    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ecd13e-6bc8846b59-vq8gl" event={"ID":"c727c74f-dc8b-4efd-9bf2-10bdc66c2c08","Type":"ContainerStarted","Data":"8ed563b0d004084bbc95f0854c0f84ee0cc554b70af56b33a5e4d2888ae0d518"}
Sep 09 06:01:56.977120 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:56.976773    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ecd13e-6bc8846b59-vq8gl"
Sep 09 06:01:56.994764 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:56.994707    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-ecd13e-6bc8846b59-vq8gl" podStartSLOduration=1.377525968 podStartE2EDuration="2.994687311s" totalImagesPullingTime="1.617161343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:01:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:01:56.993451692 +0000 UTC m=+742.333939600" watchObservedRunningTime="2026-09-09 06:01:56.994687311 +0000 UTC m=+742.335175224"
Sep 09 06:01:58.988091 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:58.988053    2579 generic.go:356] "Generic (PLEG): container finished" podID="1ef81e3c-4419-4d33-a93e-a605d2be712e" containerID="9ea92371a34b93a05e4c07c7678b4e7d020d64c85504bf65469a595ba1817fff" exitCode=0
Sep 09 06:01:58.988433 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:01:58.988134    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7991-139747575834288" event={"ID":"1ef81e3c-4419-4d33-a93e-a605d2be712e","Type":"ContainerDied","Data":"9ea92371a34b93a05e4c07c7678b4e7d020d64c85504bf65469a595ba1817fff"}
Sep 09 06:02:00.115511 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:00.115490    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7991-139747575834288"
Sep 09 06:02:00.136048 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:00.134262    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7991-139747575834288_1ef81e3c-4419-4d33-a93e-a605d2be712e/test-curl-7991-139747575834288/0.log"
Sep 09 06:02:00.155466 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:00.155436    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7991-139747575834288"]
Sep 09 06:02:00.161832 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:00.161811    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7991-139747575834288"]
Sep 09 06:02:00.242009 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:00.241986    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-psn27\" (UniqueName: \"kubernetes.io/projected/1ef81e3c-4419-4d33-a93e-a605d2be712e-kube-api-access-psn27\") pod \"1ef81e3c-4419-4d33-a93e-a605d2be712e\" (UID: \"1ef81e3c-4419-4d33-a93e-a605d2be712e\") "
Sep 09 06:02:00.244194 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:00.244142    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ef81e3c-4419-4d33-a93e-a605d2be712e-kube-api-access-psn27" (OuterVolumeSpecName: "kube-api-access-psn27") pod "1ef81e3c-4419-4d33-a93e-a605d2be712e" (UID: "1ef81e3c-4419-4d33-a93e-a605d2be712e"). InnerVolumeSpecName "kube-api-access-psn27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:02:00.342676 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:00.342646    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psn27\" (UniqueName: \"kubernetes.io/projected/1ef81e3c-4419-4d33-a93e-a605d2be712e-kube-api-access-psn27\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:02:00.386675 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:00.386651    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb"]
Sep 09 06:02:00.387016 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:00.386999    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ef81e3c-4419-4d33-a93e-a605d2be712e" containerName="test-curl-7991-139747575834288"
Sep 09 06:02:00.387095 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:00.387019    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ef81e3c-4419-4d33-a93e-a605d2be712e" containerName="test-curl-7991-139747575834288"
Sep 09 06:02:00.387095 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:00.387091    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ef81e3c-4419-4d33-a93e-a605d2be712e" containerName="test-curl-7991-139747575834288"
Sep 09 06:02:00.389954 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:00.389936    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb"
Sep 09 06:02:00.392859 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:00.392831    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 06:02:00.392963 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:00.392898    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c53eb05a-serving-cert\""
Sep 09 06:02:00.393025 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:00.392960    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-927tl\""
Sep 09 06:02:00.393662 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:00.393617    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 06:02:00.393776 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:00.393715    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 09 06:02:00.401295 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:00.401274    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb"]
Sep 09 06:02:00.544881 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:00.544788    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf4c9dfa-482e-4a77-9c98-7bde5b854fbe-metrics-certs\") pod \"maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb\" (UID: \"cf4c9dfa-482e-4a77-9c98-7bde5b854fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb"
Sep 09 06:02:00.544881 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:00.544858    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24g8v\" (UniqueName: \"kubernetes.io/projected/cf4c9dfa-482e-4a77-9c98-7bde5b854fbe-kube-api-access-24g8v\") pod \"maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb\" (UID: \"cf4c9dfa-482e-4a77-9c98-7bde5b854fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb"
Sep 09 06:02:00.545063 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:00.544894    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf4c9dfa-482e-4a77-9c98-7bde5b854fbe-openshift-service-ca\") pod \"maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb\" (UID: \"cf4c9dfa-482e-4a77-9c98-7bde5b854fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb"
Sep 09 06:02:00.545063 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:00.544910    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf4c9dfa-482e-4a77-9c98-7bde5b854fbe-maas-api-tls\") pod \"maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb\" (UID: \"cf4c9dfa-482e-4a77-9c98-7bde5b854fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb"
Sep 09 06:02:00.645428 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:00.645401    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf4c9dfa-482e-4a77-9c98-7bde5b854fbe-openshift-service-ca\") pod \"maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb\" (UID: \"cf4c9dfa-482e-4a77-9c98-7bde5b854fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb"
Sep 09 06:02:00.645428 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:00.645431    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf4c9dfa-482e-4a77-9c98-7bde5b854fbe-maas-api-tls\") pod \"maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb\" (UID: \"cf4c9dfa-482e-4a77-9c98-7bde5b854fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb"
Sep 09 06:02:00.645651 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:00.645463    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf4c9dfa-482e-4a77-9c98-7bde5b854fbe-metrics-certs\") pod \"maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb\" (UID: \"cf4c9dfa-482e-4a77-9c98-7bde5b854fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb"
Sep 09 06:02:00.645651 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:00.645520    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24g8v\" (UniqueName: \"kubernetes.io/projected/cf4c9dfa-482e-4a77-9c98-7bde5b854fbe-kube-api-access-24g8v\") pod \"maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb\" (UID: \"cf4c9dfa-482e-4a77-9c98-7bde5b854fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb"
Sep 09 06:02:00.645651 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:02:00.645568    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-c53eb05a-serving-cert: secret "maas-api-e2e-ait-c53eb05a-serving-cert" not found
Sep 09 06:02:00.645651 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:02:00.645650    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf4c9dfa-482e-4a77-9c98-7bde5b854fbe-maas-api-tls podName:cf4c9dfa-482e-4a77-9c98-7bde5b854fbe nodeName:}" failed. No retries permitted until 2026-09-09 06:02:01.145612725 +0000 UTC m=+746.486100615 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cf4c9dfa-482e-4a77-9c98-7bde5b854fbe-maas-api-tls") pod "maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb" (UID: "cf4c9dfa-482e-4a77-9c98-7bde5b854fbe") : secret "maas-api-e2e-ait-c53eb05a-serving-cert" not found
Sep 09 06:02:00.646105 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:00.646087    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf4c9dfa-482e-4a77-9c98-7bde5b854fbe-openshift-service-ca\") pod \"maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb\" (UID: \"cf4c9dfa-482e-4a77-9c98-7bde5b854fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb"
Sep 09 06:02:00.647995 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:00.647975    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf4c9dfa-482e-4a77-9c98-7bde5b854fbe-metrics-certs\") pod \"maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb\" (UID: \"cf4c9dfa-482e-4a77-9c98-7bde5b854fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb"
Sep 09 06:02:00.654658 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:00.654617    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24g8v\" (UniqueName: \"kubernetes.io/projected/cf4c9dfa-482e-4a77-9c98-7bde5b854fbe-kube-api-access-24g8v\") pod \"maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb\" (UID: \"cf4c9dfa-482e-4a77-9c98-7bde5b854fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb"
Sep 09 06:02:00.998437 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:00.998410    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7991-139747575834288"
Sep 09 06:02:00.998437 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:00.998426    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b097bf58f0edbdf68e7d353a50eb6456aafd7c6219a27e08933ebfead79b0674"
Sep 09 06:02:01.008346 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:02:01.008292    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7991-139747575834288\" is forbidden: User \"system:node:ip-10-0-131-142.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-142.ec2.internal' and this object" podUID="1ef81e3c-4419-4d33-a93e-a605d2be712e" pod="opendatahub/test-curl-7991-139747575834288"
Sep 09 06:02:01.148169 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:01.148145    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ef81e3c-4419-4d33-a93e-a605d2be712e" path="/var/lib/kubelet/pods/1ef81e3c-4419-4d33-a93e-a605d2be712e/volumes"
Sep 09 06:02:01.149658 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:01.149620    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf4c9dfa-482e-4a77-9c98-7bde5b854fbe-maas-api-tls\") pod \"maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb\" (UID: \"cf4c9dfa-482e-4a77-9c98-7bde5b854fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb"
Sep 09 06:02:01.152031 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:01.152013    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf4c9dfa-482e-4a77-9c98-7bde5b854fbe-maas-api-tls\") pod \"maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb\" (UID: \"cf4c9dfa-482e-4a77-9c98-7bde5b854fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb"
Sep 09 06:02:01.301918 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:01.301847    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb"
Sep 09 06:02:01.589669 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:01.589615    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb"]
Sep 09 06:02:01.593707 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:02:01.593676    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf4c9dfa_482e_4a77_9c98_7bde5b854fbe.slice/crio-693f47b7f449c2217a476e3a2e641bfb35bb595008ed2dc117e7b4dd0e7a76b8 WatchSource:0}: Error finding container 693f47b7f449c2217a476e3a2e641bfb35bb595008ed2dc117e7b4dd0e7a76b8: Status 404 returned error can't find the container with id 693f47b7f449c2217a476e3a2e641bfb35bb595008ed2dc117e7b4dd0e7a76b8
Sep 09 06:02:02.004191 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:02.004106    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb" event={"ID":"cf4c9dfa-482e-4a77-9c98-7bde5b854fbe","Type":"ContainerStarted","Data":"693f47b7f449c2217a476e3a2e641bfb35bb595008ed2dc117e7b4dd0e7a76b8"}
Sep 09 06:02:03.602610 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:03.602580    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:02:04.014691 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:04.014594    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb" event={"ID":"cf4c9dfa-482e-4a77-9c98-7bde5b854fbe","Type":"ContainerStarted","Data":"a5c0bc0371b0ba1606b317422d188fab9ea190649e37eecf50589b25bd8ae17c"}
Sep 09 06:02:04.014835 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:04.014707    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb"
Sep 09 06:02:04.992320 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:04.992254    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb" podStartSLOduration=3.527846626 podStartE2EDuration="4.992230545s" totalImagesPullingTime="1.464383919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:02:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:02:04.031217305 +0000 UTC m=+749.371705216" watchObservedRunningTime="2026-09-09 06:02:04.992230545 +0000 UTC m=+750.332718457"
Sep 09 06:02:04.993972 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:04.993950    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7a357a-546f6db798-gf8rb"]
Sep 09 06:02:04.997308 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:04.997290    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a357a-546f6db798-gf8rb"
Sep 09 06:02:04.999801 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:04.999778    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-7a357a-serving-cert\""
Sep 09 06:02:05.010262 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:05.010237    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7a357a-546f6db798-gf8rb"]
Sep 09 06:02:05.186362 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:05.186331    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c45ba77-e983-43dc-97b4-48a96037b1a1-openshift-service-ca\") pod \"maas-api-e2e-autores-7a357a-546f6db798-gf8rb\" (UID: \"0c45ba77-e983-43dc-97b4-48a96037b1a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a357a-546f6db798-gf8rb"
Sep 09 06:02:05.186524 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:05.186413    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csdw8\" (UniqueName: \"kubernetes.io/projected/0c45ba77-e983-43dc-97b4-48a96037b1a1-kube-api-access-csdw8\") pod \"maas-api-e2e-autores-7a357a-546f6db798-gf8rb\" (UID: \"0c45ba77-e983-43dc-97b4-48a96037b1a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a357a-546f6db798-gf8rb"
Sep 09 06:02:05.186524 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:05.186447    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c45ba77-e983-43dc-97b4-48a96037b1a1-metrics-certs\") pod \"maas-api-e2e-autores-7a357a-546f6db798-gf8rb\" (UID: \"0c45ba77-e983-43dc-97b4-48a96037b1a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a357a-546f6db798-gf8rb"
Sep 09 06:02:05.186524 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:05.186510    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c45ba77-e983-43dc-97b4-48a96037b1a1-maas-api-tls\") pod \"maas-api-e2e-autores-7a357a-546f6db798-gf8rb\" (UID: \"0c45ba77-e983-43dc-97b4-48a96037b1a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a357a-546f6db798-gf8rb"
Sep 09 06:02:05.287060 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:05.286977    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c45ba77-e983-43dc-97b4-48a96037b1a1-openshift-service-ca\") pod \"maas-api-e2e-autores-7a357a-546f6db798-gf8rb\" (UID: \"0c45ba77-e983-43dc-97b4-48a96037b1a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a357a-546f6db798-gf8rb"
Sep 09 06:02:05.287228 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:05.287070    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csdw8\" (UniqueName: \"kubernetes.io/projected/0c45ba77-e983-43dc-97b4-48a96037b1a1-kube-api-access-csdw8\") pod \"maas-api-e2e-autores-7a357a-546f6db798-gf8rb\" (UID: \"0c45ba77-e983-43dc-97b4-48a96037b1a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a357a-546f6db798-gf8rb"
Sep 09 06:02:05.287228 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:05.287104    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c45ba77-e983-43dc-97b4-48a96037b1a1-metrics-certs\") pod \"maas-api-e2e-autores-7a357a-546f6db798-gf8rb\" (UID: \"0c45ba77-e983-43dc-97b4-48a96037b1a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a357a-546f6db798-gf8rb"
Sep 09 06:02:05.287228 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:05.287155    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c45ba77-e983-43dc-97b4-48a96037b1a1-maas-api-tls\") pod \"maas-api-e2e-autores-7a357a-546f6db798-gf8rb\" (UID: \"0c45ba77-e983-43dc-97b4-48a96037b1a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a357a-546f6db798-gf8rb"
Sep 09 06:02:05.287732 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:05.287706    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c45ba77-e983-43dc-97b4-48a96037b1a1-openshift-service-ca\") pod \"maas-api-e2e-autores-7a357a-546f6db798-gf8rb\" (UID: \"0c45ba77-e983-43dc-97b4-48a96037b1a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a357a-546f6db798-gf8rb"
Sep 09 06:02:05.289960 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:05.289936    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c45ba77-e983-43dc-97b4-48a96037b1a1-metrics-certs\") pod \"maas-api-e2e-autores-7a357a-546f6db798-gf8rb\" (UID: \"0c45ba77-e983-43dc-97b4-48a96037b1a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a357a-546f6db798-gf8rb"
Sep 09 06:02:05.290058 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:05.290042    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c45ba77-e983-43dc-97b4-48a96037b1a1-maas-api-tls\") pod \"maas-api-e2e-autores-7a357a-546f6db798-gf8rb\" (UID: \"0c45ba77-e983-43dc-97b4-48a96037b1a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a357a-546f6db798-gf8rb"
Sep 09 06:02:05.295589 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:05.295567    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csdw8\" (UniqueName: \"kubernetes.io/projected/0c45ba77-e983-43dc-97b4-48a96037b1a1-kube-api-access-csdw8\") pod \"maas-api-e2e-autores-7a357a-546f6db798-gf8rb\" (UID: \"0c45ba77-e983-43dc-97b4-48a96037b1a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a357a-546f6db798-gf8rb"
Sep 09 06:02:05.309017 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:05.308995    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a357a-546f6db798-gf8rb"
Sep 09 06:02:05.586494 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:05.586465    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7a357a-546f6db798-gf8rb"]
Sep 09 06:02:05.589047 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:02:05.589018    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c45ba77_e983_43dc_97b4_48a96037b1a1.slice/crio-ed8fc72868aba6a594118da99c62f21d0f799485be0e174f5acbcf61b73a3970 WatchSource:0}: Error finding container ed8fc72868aba6a594118da99c62f21d0f799485be0e174f5acbcf61b73a3970: Status 404 returned error can't find the container with id ed8fc72868aba6a594118da99c62f21d0f799485be0e174f5acbcf61b73a3970
Sep 09 06:02:06.027794 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:06.027764    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a357a-546f6db798-gf8rb" event={"ID":"0c45ba77-e983-43dc-97b4-48a96037b1a1","Type":"ContainerStarted","Data":"99465667c6f89ec02c34f4a5fc7a3ed21e95f6f64f1f395aea50f27c12a7397f"}
Sep 09 06:02:06.028191 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:06.027815    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a357a-546f6db798-gf8rb" event={"ID":"0c45ba77-e983-43dc-97b4-48a96037b1a1","Type":"ContainerStarted","Data":"ed8fc72868aba6a594118da99c62f21d0f799485be0e174f5acbcf61b73a3970"}
Sep 09 06:02:06.028191 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:06.027844    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a357a-546f6db798-gf8rb"
Sep 09 06:02:06.044193 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:06.044142    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a357a-546f6db798-gf8rb" podStartSLOduration=1.661568066 podStartE2EDuration="2.044126456s" totalImagesPullingTime="382.55839ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:02:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:02:06.043063874 +0000 UTC m=+751.383551784" watchObservedRunningTime="2026-09-09 06:02:06.044126456 +0000 UTC m=+751.384614365"
Sep 09 06:02:07.984255 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:07.984226    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ecd13e-6bc8846b59-vq8gl"
Sep 09 06:02:10.026810 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:10.026783    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb"
Sep 09 06:02:12.038248 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:12.038218    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a357a-546f6db798-gf8rb"
Sep 09 06:02:21.217996 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:21.217965    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:02:22.510058 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:22.510026    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh"]
Sep 09 06:02:22.513122 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:22.513107    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh"
Sep 09 06:02:22.515119 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:22.515097    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-fe8eaa-serving-cert\""
Sep 09 06:02:22.522325 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:22.522298    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh"]
Sep 09 06:02:22.613193 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:22.613160    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4-maas-api-tls\") pod \"maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh\" (UID: \"0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh"
Sep 09 06:02:22.613359 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:22.613197    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4-metrics-certs\") pod \"maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh\" (UID: \"0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh"
Sep 09 06:02:22.613359 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:22.613290    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh\" (UID: \"0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh"
Sep 09 06:02:22.613359 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:22.613327    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f47qp\" (UniqueName: \"kubernetes.io/projected/0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4-kube-api-access-f47qp\") pod \"maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh\" (UID: \"0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh"
Sep 09 06:02:22.713815 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:22.713783    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f47qp\" (UniqueName: \"kubernetes.io/projected/0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4-kube-api-access-f47qp\") pod \"maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh\" (UID: \"0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh"
Sep 09 06:02:22.713996 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:22.713843    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4-maas-api-tls\") pod \"maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh\" (UID: \"0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh"
Sep 09 06:02:22.713996 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:22.713862    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4-metrics-certs\") pod \"maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh\" (UID: \"0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh"
Sep 09 06:02:22.713996 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:22.713902    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh\" (UID: \"0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh"
Sep 09 06:02:22.714162 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:02:22.714004    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-fe8eaa-serving-cert: secret "maas-api-e2e-inf-b-fe8eaa-serving-cert" not found
Sep 09 06:02:22.714162 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:02:22.714077    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4-maas-api-tls podName:0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4 nodeName:}" failed. No retries permitted until 2026-09-09 06:02:23.214052881 +0000 UTC m=+768.554540768 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4-maas-api-tls") pod "maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh" (UID: "0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4") : secret "maas-api-e2e-inf-b-fe8eaa-serving-cert" not found
Sep 09 06:02:22.714480 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:22.714464    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh\" (UID: \"0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh"
Sep 09 06:02:22.716583 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:22.716551    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4-metrics-certs\") pod \"maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh\" (UID: \"0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh"
Sep 09 06:02:22.724343 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:22.724321    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f47qp\" (UniqueName: \"kubernetes.io/projected/0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4-kube-api-access-f47qp\") pod \"maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh\" (UID: \"0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh"
Sep 09 06:02:23.218140 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:23.218088    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4-maas-api-tls\") pod \"maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh\" (UID: \"0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh"
Sep 09 06:02:23.220583 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:23.220544    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4-maas-api-tls\") pod \"maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh\" (UID: \"0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh"
Sep 09 06:02:23.363325 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:23.363292    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fe8eaa-76889d76c4-mphkg"]
Sep 09 06:02:23.366721 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:23.366705    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fe8eaa-76889d76c4-mphkg"
Sep 09 06:02:23.369002 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:23.368981    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-fe8eaa\""
Sep 09 06:02:23.369248 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:23.369232    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-fe8eaa-dockercfg-xmsd4\""
Sep 09 06:02:23.376814 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:23.376786    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fe8eaa-76889d76c4-mphkg"]
Sep 09 06:02:23.419802 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:23.419776    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ada3927-7884-47a4-8827-3cdb556fb999-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-fe8eaa-76889d76c4-mphkg\" (UID: \"6ada3927-7884-47a4-8827-3cdb556fb999\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fe8eaa-76889d76c4-mphkg"
Sep 09 06:02:23.419908 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:23.419824    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jclp2\" (UniqueName: \"kubernetes.io/projected/6ada3927-7884-47a4-8827-3cdb556fb999-kube-api-access-jclp2\") pod \"payload-pre-processing-e2e-inf-b-fe8eaa-76889d76c4-mphkg\" (UID: \"6ada3927-7884-47a4-8827-3cdb556fb999\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fe8eaa-76889d76c4-mphkg"
Sep 09 06:02:23.423605 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:23.423585    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh"
Sep 09 06:02:23.431409 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:23.431389    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fe8eaa-568f87ccc5-5z4qp"]
Sep 09 06:02:23.434788 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:23.434774    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fe8eaa-568f87ccc5-5z4qp"
Sep 09 06:02:23.450292 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:23.450268    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fe8eaa-568f87ccc5-5z4qp"]
Sep 09 06:02:23.520866 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:23.520837    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jclp2\" (UniqueName: \"kubernetes.io/projected/6ada3927-7884-47a4-8827-3cdb556fb999-kube-api-access-jclp2\") pod \"payload-pre-processing-e2e-inf-b-fe8eaa-76889d76c4-mphkg\" (UID: \"6ada3927-7884-47a4-8827-3cdb556fb999\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fe8eaa-76889d76c4-mphkg"
Sep 09 06:02:23.521180 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:23.520903    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7txw\" (UniqueName: \"kubernetes.io/projected/a4d64e57-8756-43c5-a798-1690572ade99-kube-api-access-f7txw\") pod \"payload-processing-e2e-inf-b-fe8eaa-568f87ccc5-5z4qp\" (UID: \"a4d64e57-8756-43c5-a798-1690572ade99\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fe8eaa-568f87ccc5-5z4qp"
Sep 09 06:02:23.521180 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:23.520975    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ada3927-7884-47a4-8827-3cdb556fb999-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-fe8eaa-76889d76c4-mphkg\" (UID: \"6ada3927-7884-47a4-8827-3cdb556fb999\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fe8eaa-76889d76c4-mphkg"
Sep 09 06:02:23.521180 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:23.521013    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4d64e57-8756-43c5-a798-1690572ade99-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-fe8eaa-568f87ccc5-5z4qp\" (UID: \"a4d64e57-8756-43c5-a798-1690572ade99\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fe8eaa-568f87ccc5-5z4qp"
Sep 09 06:02:23.521677 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:23.521659    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ada3927-7884-47a4-8827-3cdb556fb999-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-fe8eaa-76889d76c4-mphkg\" (UID: \"6ada3927-7884-47a4-8827-3cdb556fb999\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fe8eaa-76889d76c4-mphkg"
Sep 09 06:02:23.530133 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:23.530110    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jclp2\" (UniqueName: \"kubernetes.io/projected/6ada3927-7884-47a4-8827-3cdb556fb999-kube-api-access-jclp2\") pod \"payload-pre-processing-e2e-inf-b-fe8eaa-76889d76c4-mphkg\" (UID: \"6ada3927-7884-47a4-8827-3cdb556fb999\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fe8eaa-76889d76c4-mphkg"
Sep 09 06:02:23.621513 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:23.621481    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7txw\" (UniqueName: \"kubernetes.io/projected/a4d64e57-8756-43c5-a798-1690572ade99-kube-api-access-f7txw\") pod \"payload-processing-e2e-inf-b-fe8eaa-568f87ccc5-5z4qp\" (UID: \"a4d64e57-8756-43c5-a798-1690572ade99\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fe8eaa-568f87ccc5-5z4qp"
Sep 09 06:02:23.621706 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:23.621560    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4d64e57-8756-43c5-a798-1690572ade99-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-fe8eaa-568f87ccc5-5z4qp\" (UID: \"a4d64e57-8756-43c5-a798-1690572ade99\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fe8eaa-568f87ccc5-5z4qp"
Sep 09 06:02:23.622240 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:23.622218    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4d64e57-8756-43c5-a798-1690572ade99-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-fe8eaa-568f87ccc5-5z4qp\" (UID: \"a4d64e57-8756-43c5-a798-1690572ade99\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fe8eaa-568f87ccc5-5z4qp"
Sep 09 06:02:23.629153 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:23.629133    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7txw\" (UniqueName: \"kubernetes.io/projected/a4d64e57-8756-43c5-a798-1690572ade99-kube-api-access-f7txw\") pod \"payload-processing-e2e-inf-b-fe8eaa-568f87ccc5-5z4qp\" (UID: \"a4d64e57-8756-43c5-a798-1690572ade99\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fe8eaa-568f87ccc5-5z4qp"
Sep 09 06:02:23.678136 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:23.678114    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fe8eaa-76889d76c4-mphkg"
Sep 09 06:02:23.702793 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:23.702770    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh"]
Sep 09 06:02:23.705283 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:02:23.705258    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0762cdd5_6db6_4e44_b2be_ddbd5ff6a0c4.slice/crio-67e6734faae53296bbc942b1b938811641d4b33f1874516cfb2a9d12a65be6b3 WatchSource:0}: Error finding container 67e6734faae53296bbc942b1b938811641d4b33f1874516cfb2a9d12a65be6b3: Status 404 returned error can't find the container with id 67e6734faae53296bbc942b1b938811641d4b33f1874516cfb2a9d12a65be6b3
Sep 09 06:02:23.759150 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:23.759121    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fe8eaa-568f87ccc5-5z4qp"
Sep 09 06:02:23.959660 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:23.959620    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fe8eaa-76889d76c4-mphkg"]
Sep 09 06:02:23.961025 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:02:23.960997    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ada3927_7884_47a4_8827_3cdb556fb999.slice/crio-4cf7e0f3cde7b4da68b1c63fb26a7aceea85bce6c85803ae0932c48eec064caf WatchSource:0}: Error finding container 4cf7e0f3cde7b4da68b1c63fb26a7aceea85bce6c85803ae0932c48eec064caf: Status 404 returned error can't find the container with id 4cf7e0f3cde7b4da68b1c63fb26a7aceea85bce6c85803ae0932c48eec064caf
Sep 09 06:02:24.047219 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:24.047047    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fe8eaa-568f87ccc5-5z4qp"]
Sep 09 06:02:24.051136 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:02:24.051111    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4d64e57_8756_43c5_a798_1690572ade99.slice/crio-bc8549e2b1005155e71b767defe22a0e4d2ae3d9ecc75a356b57674ca0558e25 WatchSource:0}: Error finding container bc8549e2b1005155e71b767defe22a0e4d2ae3d9ecc75a356b57674ca0558e25: Status 404 returned error can't find the container with id bc8549e2b1005155e71b767defe22a0e4d2ae3d9ecc75a356b57674ca0558e25
Sep 09 06:02:24.111483 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:24.111456    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fe8eaa-568f87ccc5-5z4qp" event={"ID":"a4d64e57-8756-43c5-a798-1690572ade99","Type":"ContainerStarted","Data":"bc8549e2b1005155e71b767defe22a0e4d2ae3d9ecc75a356b57674ca0558e25"}
Sep 09 06:02:24.113581 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:24.113558    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh" event={"ID":"0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4","Type":"ContainerStarted","Data":"67e6734faae53296bbc942b1b938811641d4b33f1874516cfb2a9d12a65be6b3"}
Sep 09 06:02:24.115393 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:24.115367    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fe8eaa-76889d76c4-mphkg" event={"ID":"6ada3927-7884-47a4-8827-3cdb556fb999","Type":"ContainerStarted","Data":"dfcdb5ea07c5f66040abd77c9eac21f70c7ac10f508b4f7af7f2cabc5afe04db"}
Sep 09 06:02:24.115492 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:24.115418    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fe8eaa-76889d76c4-mphkg" event={"ID":"6ada3927-7884-47a4-8827-3cdb556fb999","Type":"ContainerStarted","Data":"4cf7e0f3cde7b4da68b1c63fb26a7aceea85bce6c85803ae0932c48eec064caf"}
Sep 09 06:02:24.115647 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:24.115595    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fe8eaa-76889d76c4-mphkg"
Sep 09 06:02:24.154853 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:24.154795    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fe8eaa-76889d76c4-mphkg" podStartSLOduration=1.154773117 podStartE2EDuration="1.154773117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:02:24.154473083 +0000 UTC m=+769.494960995" watchObservedRunningTime="2026-09-09 06:02:24.154773117 +0000 UTC m=+769.495261027"
Sep 09 06:02:25.122208 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:25.122164    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh" event={"ID":"0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4","Type":"ContainerStarted","Data":"e64578a3fda3f0d41bc8faaf3998a95b2e3f8a0aba9a504079acf7c1fa495fb7"}
Sep 09 06:02:25.122669 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:25.122317    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh"
Sep 09 06:02:25.123720 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:25.123697    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fe8eaa-568f87ccc5-5z4qp" event={"ID":"a4d64e57-8756-43c5-a798-1690572ade99","Type":"ContainerStarted","Data":"a2004bc32f90cebc920db828454f54223da1030f061716844fed97b24c3934dd"}
Sep 09 06:02:25.143935 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:25.143894    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh" podStartSLOduration=2.796972347 podStartE2EDuration="3.143878822s" totalImagesPullingTime="346.906475ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:02:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:02:25.142424014 +0000 UTC m=+770.482911924" watchObservedRunningTime="2026-09-09 06:02:25.143878822 +0000 UTC m=+770.484366731"
Sep 09 06:02:25.165853 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:25.165809    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-fe8eaa-568f87ccc5-5z4qp" podStartSLOduration=2.1657953819999998 podStartE2EDuration="2.165795382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:02:25.1648145 +0000 UTC m=+770.505302408" watchObservedRunningTime="2026-09-09 06:02:25.165795382 +0000 UTC m=+770.506283291"
Sep 09 06:02:26.129716 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:26.129683    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-fe8eaa-568f87ccc5-5z4qp"
Sep 09 06:02:31.135563 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:31.135537    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh"
Sep 09 06:02:35.124950 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:35.124915    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fe8eaa-76889d76c4-mphkg"
Sep 09 06:02:35.503482 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:35.503410    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:02:36.613465 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:36.613430    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-43eb2d/simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk"]
Sep 09 06:02:36.619755 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:36.619738    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-43eb2d/simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk"
Sep 09 06:02:36.622133 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:36.622112    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-43eb2d\"/\"openshift-service-ca.crt\""
Sep 09 06:02:36.622250 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:36.622165    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-43eb2d\"/\"kube-root-ca.crt\""
Sep 09 06:02:36.622975 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:36.622958    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-43eb2d\"/\"default-dockercfg-j4z2r\""
Sep 09 06:02:36.623053 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:36.622977    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-43eb2d\"/\"simulator-w4-43eb2d-kserve-self-signed-certs\""
Sep 09 06:02:36.628826 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:36.628804    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-43eb2d/simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk"]
Sep 09 06:02:36.719403 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:36.719374    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82a478a9-69bb-4e81-b5bd-e30a56f2733d-tls-certs\") pod \"simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk\" (UID: \"82a478a9-69bb-4e81-b5bd-e30a56f2733d\") " pod="e2e-models-e2e-worker-w4-43eb2d/simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk"
Sep 09 06:02:36.719511 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:36.719428    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82a478a9-69bb-4e81-b5bd-e30a56f2733d-home\") pod \"simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk\" (UID: \"82a478a9-69bb-4e81-b5bd-e30a56f2733d\") " pod="e2e-models-e2e-worker-w4-43eb2d/simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk"
Sep 09 06:02:36.719511 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:36.719497    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9nkj\" (UniqueName: \"kubernetes.io/projected/82a478a9-69bb-4e81-b5bd-e30a56f2733d-kube-api-access-v9nkj\") pod \"simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk\" (UID: \"82a478a9-69bb-4e81-b5bd-e30a56f2733d\") " pod="e2e-models-e2e-worker-w4-43eb2d/simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk"
Sep 09 06:02:36.719609 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:36.719531    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82a478a9-69bb-4e81-b5bd-e30a56f2733d-dshm\") pod \"simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk\" (UID: \"82a478a9-69bb-4e81-b5bd-e30a56f2733d\") " pod="e2e-models-e2e-worker-w4-43eb2d/simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk"
Sep 09 06:02:36.719609 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:36.719550    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82a478a9-69bb-4e81-b5bd-e30a56f2733d-model-cache\") pod \"simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk\" (UID: \"82a478a9-69bb-4e81-b5bd-e30a56f2733d\") " pod="e2e-models-e2e-worker-w4-43eb2d/simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk"
Sep 09 06:02:36.719734 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:36.719673    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82a478a9-69bb-4e81-b5bd-e30a56f2733d-tmp-dir\") pod \"simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk\" (UID: \"82a478a9-69bb-4e81-b5bd-e30a56f2733d\") " pod="e2e-models-e2e-worker-w4-43eb2d/simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk"
Sep 09 06:02:36.820237 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:36.820204    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82a478a9-69bb-4e81-b5bd-e30a56f2733d-model-cache\") pod \"simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk\" (UID: \"82a478a9-69bb-4e81-b5bd-e30a56f2733d\") " pod="e2e-models-e2e-worker-w4-43eb2d/simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk"
Sep 09 06:02:36.820386 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:36.820255    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82a478a9-69bb-4e81-b5bd-e30a56f2733d-tmp-dir\") pod \"simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk\" (UID: \"82a478a9-69bb-4e81-b5bd-e30a56f2733d\") " pod="e2e-models-e2e-worker-w4-43eb2d/simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk"
Sep 09 06:02:36.820429 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:36.820378    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82a478a9-69bb-4e81-b5bd-e30a56f2733d-tls-certs\") pod \"simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk\" (UID: \"82a478a9-69bb-4e81-b5bd-e30a56f2733d\") " pod="e2e-models-e2e-worker-w4-43eb2d/simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk"
Sep 09 06:02:36.820481 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:36.820441    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82a478a9-69bb-4e81-b5bd-e30a56f2733d-home\") pod \"simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk\" (UID: \"82a478a9-69bb-4e81-b5bd-e30a56f2733d\") " pod="e2e-models-e2e-worker-w4-43eb2d/simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk"
Sep 09 06:02:36.820535 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:36.820514    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9nkj\" (UniqueName: \"kubernetes.io/projected/82a478a9-69bb-4e81-b5bd-e30a56f2733d-kube-api-access-v9nkj\") pod \"simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk\" (UID: \"82a478a9-69bb-4e81-b5bd-e30a56f2733d\") " pod="e2e-models-e2e-worker-w4-43eb2d/simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk"
Sep 09 06:02:36.820610 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:36.820566    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82a478a9-69bb-4e81-b5bd-e30a56f2733d-dshm\") pod \"simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk\" (UID: \"82a478a9-69bb-4e81-b5bd-e30a56f2733d\") " pod="e2e-models-e2e-worker-w4-43eb2d/simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk"
Sep 09 06:02:36.820610 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:36.820581    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82a478a9-69bb-4e81-b5bd-e30a56f2733d-tmp-dir\") pod \"simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk\" (UID: \"82a478a9-69bb-4e81-b5bd-e30a56f2733d\") " pod="e2e-models-e2e-worker-w4-43eb2d/simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk"
Sep 09 06:02:36.820753 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:36.820664    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82a478a9-69bb-4e81-b5bd-e30a56f2733d-model-cache\") pod \"simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk\" (UID: \"82a478a9-69bb-4e81-b5bd-e30a56f2733d\") " pod="e2e-models-e2e-worker-w4-43eb2d/simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk"
Sep 09 06:02:36.820825 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:36.820801    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82a478a9-69bb-4e81-b5bd-e30a56f2733d-home\") pod \"simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk\" (UID: \"82a478a9-69bb-4e81-b5bd-e30a56f2733d\") " pod="e2e-models-e2e-worker-w4-43eb2d/simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk"
Sep 09 06:02:36.822837 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:36.822817    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82a478a9-69bb-4e81-b5bd-e30a56f2733d-dshm\") pod \"simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk\" (UID: \"82a478a9-69bb-4e81-b5bd-e30a56f2733d\") " pod="e2e-models-e2e-worker-w4-43eb2d/simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk"
Sep 09 06:02:36.823004 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:36.822988    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82a478a9-69bb-4e81-b5bd-e30a56f2733d-tls-certs\") pod \"simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk\" (UID: \"82a478a9-69bb-4e81-b5bd-e30a56f2733d\") " pod="e2e-models-e2e-worker-w4-43eb2d/simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk"
Sep 09 06:02:36.829073 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:36.829054    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9nkj\" (UniqueName: \"kubernetes.io/projected/82a478a9-69bb-4e81-b5bd-e30a56f2733d-kube-api-access-v9nkj\") pod \"simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk\" (UID: \"82a478a9-69bb-4e81-b5bd-e30a56f2733d\") " pod="e2e-models-e2e-worker-w4-43eb2d/simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk"
Sep 09 06:02:36.930108 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:36.930035    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-43eb2d/simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk"
Sep 09 06:02:37.137799 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:37.137772    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-fe8eaa-568f87ccc5-5z4qp"
Sep 09 06:02:37.200724 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:37.200503    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-43eb2d/simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk"]
Sep 09 06:02:37.203314 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:02:37.203289    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82a478a9_69bb_4e81_b5bd_e30a56f2733d.slice/crio-51bbc891f96d89ebb40ae07d5d61f0e4cbfdb97c97d7e8b04ea6f73dc101cf74 WatchSource:0}: Error finding container 51bbc891f96d89ebb40ae07d5d61f0e4cbfdb97c97d7e8b04ea6f73dc101cf74: Status 404 returned error can't find the container with id 51bbc891f96d89ebb40ae07d5d61f0e4cbfdb97c97d7e8b04ea6f73dc101cf74
Sep 09 06:02:38.184902 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:38.184868    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-43eb2d/simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk" event={"ID":"82a478a9-69bb-4e81-b5bd-e30a56f2733d","Type":"ContainerStarted","Data":"d87861771eb71c73f10abadc4d2ea767eec7506378f0d4fad15c14c911edb235"}
Sep 09 06:02:38.185248 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:38.184912    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-43eb2d/simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk" event={"ID":"82a478a9-69bb-4e81-b5bd-e30a56f2733d","Type":"ContainerStarted","Data":"51bbc891f96d89ebb40ae07d5d61f0e4cbfdb97c97d7e8b04ea6f73dc101cf74"}
Sep 09 06:02:38.202115 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:38.202071    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-43eb2d/simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk" podStartSLOduration=2.20205545 podStartE2EDuration="2.20205545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:02:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:02:38.200792992 +0000 UTC m=+783.541280902" watchObservedRunningTime="2026-09-09 06:02:38.20205545 +0000 UTC m=+783.542543360"
Sep 09 06:02:41.152165 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:41.152130    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb"]
Sep 09 06:02:41.152647 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:41.152320    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb" podUID="cf4c9dfa-482e-4a77-9c98-7bde5b854fbe" containerName="maas-api" containerID="cri-o://a5c0bc0371b0ba1606b317422d188fab9ea190649e37eecf50589b25bd8ae17c" gracePeriod=30
Sep 09 06:02:41.393740 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:41.393719    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb"
Sep 09 06:02:41.454249 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:41.454188    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf4c9dfa-482e-4a77-9c98-7bde5b854fbe-openshift-service-ca\") pod \"cf4c9dfa-482e-4a77-9c98-7bde5b854fbe\" (UID: \"cf4c9dfa-482e-4a77-9c98-7bde5b854fbe\") "
Sep 09 06:02:41.454249 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:41.454238    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24g8v\" (UniqueName: \"kubernetes.io/projected/cf4c9dfa-482e-4a77-9c98-7bde5b854fbe-kube-api-access-24g8v\") pod \"cf4c9dfa-482e-4a77-9c98-7bde5b854fbe\" (UID: \"cf4c9dfa-482e-4a77-9c98-7bde5b854fbe\") "
Sep 09 06:02:41.454448 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:41.454262    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf4c9dfa-482e-4a77-9c98-7bde5b854fbe-metrics-certs\") pod \"cf4c9dfa-482e-4a77-9c98-7bde5b854fbe\" (UID: \"cf4c9dfa-482e-4a77-9c98-7bde5b854fbe\") "
Sep 09 06:02:41.454448 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:41.454311    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf4c9dfa-482e-4a77-9c98-7bde5b854fbe-maas-api-tls\") pod \"cf4c9dfa-482e-4a77-9c98-7bde5b854fbe\" (UID: \"cf4c9dfa-482e-4a77-9c98-7bde5b854fbe\") "
Sep 09 06:02:41.454600 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:41.454578    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf4c9dfa-482e-4a77-9c98-7bde5b854fbe-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cf4c9dfa-482e-4a77-9c98-7bde5b854fbe" (UID: "cf4c9dfa-482e-4a77-9c98-7bde5b854fbe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:02:41.456545 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:41.456518    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf4c9dfa-482e-4a77-9c98-7bde5b854fbe-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cf4c9dfa-482e-4a77-9c98-7bde5b854fbe" (UID: "cf4c9dfa-482e-4a77-9c98-7bde5b854fbe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:02:41.456673 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:41.456545    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf4c9dfa-482e-4a77-9c98-7bde5b854fbe-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cf4c9dfa-482e-4a77-9c98-7bde5b854fbe" (UID: "cf4c9dfa-482e-4a77-9c98-7bde5b854fbe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:02:41.456673 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:41.456575    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf4c9dfa-482e-4a77-9c98-7bde5b854fbe-kube-api-access-24g8v" (OuterVolumeSpecName: "kube-api-access-24g8v") pod "cf4c9dfa-482e-4a77-9c98-7bde5b854fbe" (UID: "cf4c9dfa-482e-4a77-9c98-7bde5b854fbe"). InnerVolumeSpecName "kube-api-access-24g8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:02:41.555918 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:41.555887    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24g8v\" (UniqueName: \"kubernetes.io/projected/cf4c9dfa-482e-4a77-9c98-7bde5b854fbe-kube-api-access-24g8v\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:02:41.555918 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:41.555912    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf4c9dfa-482e-4a77-9c98-7bde5b854fbe-metrics-certs\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:02:41.555918 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:41.555922    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf4c9dfa-482e-4a77-9c98-7bde5b854fbe-maas-api-tls\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:02:41.556120 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:41.555929    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf4c9dfa-482e-4a77-9c98-7bde5b854fbe-openshift-service-ca\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:02:42.205193 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:42.205158    2579 generic.go:356] "Generic (PLEG): container finished" podID="cf4c9dfa-482e-4a77-9c98-7bde5b854fbe" containerID="a5c0bc0371b0ba1606b317422d188fab9ea190649e37eecf50589b25bd8ae17c" exitCode=0
Sep 09 06:02:42.205598 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:42.205204    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb"
Sep 09 06:02:42.205598 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:42.205248    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb" event={"ID":"cf4c9dfa-482e-4a77-9c98-7bde5b854fbe","Type":"ContainerDied","Data":"a5c0bc0371b0ba1606b317422d188fab9ea190649e37eecf50589b25bd8ae17c"}
Sep 09 06:02:42.205598 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:42.205301    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb" event={"ID":"cf4c9dfa-482e-4a77-9c98-7bde5b854fbe","Type":"ContainerDied","Data":"693f47b7f449c2217a476e3a2e641bfb35bb595008ed2dc117e7b4dd0e7a76b8"}
Sep 09 06:02:42.205598 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:42.205328    2579 scope.go:117] "RemoveContainer" containerID="a5c0bc0371b0ba1606b317422d188fab9ea190649e37eecf50589b25bd8ae17c"
Sep 09 06:02:42.214402 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:42.214382    2579 scope.go:117] "RemoveContainer" containerID="a5c0bc0371b0ba1606b317422d188fab9ea190649e37eecf50589b25bd8ae17c"
Sep 09 06:02:42.214671 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:02:42.214650    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5c0bc0371b0ba1606b317422d188fab9ea190649e37eecf50589b25bd8ae17c\": container with ID starting with a5c0bc0371b0ba1606b317422d188fab9ea190649e37eecf50589b25bd8ae17c not found: ID does not exist" containerID="a5c0bc0371b0ba1606b317422d188fab9ea190649e37eecf50589b25bd8ae17c"
Sep 09 06:02:42.214765 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:42.214677    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5c0bc0371b0ba1606b317422d188fab9ea190649e37eecf50589b25bd8ae17c"} err="failed to get container status \"a5c0bc0371b0ba1606b317422d188fab9ea190649e37eecf50589b25bd8ae17c\": rpc error: code = NotFound desc = could not find container \"a5c0bc0371b0ba1606b317422d188fab9ea190649e37eecf50589b25bd8ae17c\": container with ID starting with a5c0bc0371b0ba1606b317422d188fab9ea190649e37eecf50589b25bd8ae17c not found: ID does not exist"
Sep 09 06:02:42.228447 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:42.228427    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb"]
Sep 09 06:02:42.234363 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:42.234340    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c53eb05a-f6b89cdb7-mwwhb"]
Sep 09 06:02:43.147713 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:43.147680    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf4c9dfa-482e-4a77-9c98-7bde5b854fbe" path="/var/lib/kubelet/pods/cf4c9dfa-482e-4a77-9c98-7bde5b854fbe/volumes"
Sep 09 06:02:46.203213 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:46.203179    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e7b233dd-b7df67b9d-m5szr"]
Sep 09 06:02:46.203556 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:46.203489    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf4c9dfa-482e-4a77-9c98-7bde5b854fbe" containerName="maas-api"
Sep 09 06:02:46.203556 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:46.203500    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf4c9dfa-482e-4a77-9c98-7bde5b854fbe" containerName="maas-api"
Sep 09 06:02:46.203556 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:46.203553    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf4c9dfa-482e-4a77-9c98-7bde5b854fbe" containerName="maas-api"
Sep 09 06:02:46.207892 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:46.207876    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e7b233dd-b7df67b9d-m5szr"
Sep 09 06:02:46.209865 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:46.209848    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-e7b233dd\""
Sep 09 06:02:46.218343 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:46.218321    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e7b233dd-b7df67b9d-m5szr"]
Sep 09 06:02:46.239024 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:46.239002    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e7b233dd-65b9ff4f46-5t56v"]
Sep 09 06:02:46.242222 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:46.242208    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e7b233dd-65b9ff4f46-5t56v"
Sep 09 06:02:46.244359 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:46.244340    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-e7b233dd-dockercfg-9vf54\""
Sep 09 06:02:46.252773 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:46.252754    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e7b233dd-65b9ff4f46-5t56v"]
Sep 09 06:02:46.300048 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:46.300023    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvcqc\" (UniqueName: \"kubernetes.io/projected/90af83c5-b738-4369-bfa3-ac6f760b1c88-kube-api-access-kvcqc\") pod \"payload-pre-processing-e2e-migrate-e7b233dd-b7df67b9d-m5szr\" (UID: \"90af83c5-b738-4369-bfa3-ac6f760b1c88\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e7b233dd-b7df67b9d-m5szr"
Sep 09 06:02:46.300143 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:46.300054    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgvcc\" (UniqueName: \"kubernetes.io/projected/ce74270c-59f9-4485-b322-e379a2232662-kube-api-access-lgvcc\") pod \"payload-processing-e2e-migrate-e7b233dd-65b9ff4f46-5t56v\" (UID: \"ce74270c-59f9-4485-b322-e379a2232662\") " pod="openshift-ingress/payload-processing-e2e-migrate-e7b233dd-65b9ff4f46-5t56v"
Sep 09 06:02:46.300143 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:46.300124    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90af83c5-b738-4369-bfa3-ac6f760b1c88-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e7b233dd-b7df67b9d-m5szr\" (UID: \"90af83c5-b738-4369-bfa3-ac6f760b1c88\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e7b233dd-b7df67b9d-m5szr"
Sep 09 06:02:46.300212 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:46.300149    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce74270c-59f9-4485-b322-e379a2232662-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e7b233dd-65b9ff4f46-5t56v\" (UID: \"ce74270c-59f9-4485-b322-e379a2232662\") " pod="openshift-ingress/payload-processing-e2e-migrate-e7b233dd-65b9ff4f46-5t56v"
Sep 09 06:02:46.401391 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:46.401359    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvcqc\" (UniqueName: \"kubernetes.io/projected/90af83c5-b738-4369-bfa3-ac6f760b1c88-kube-api-access-kvcqc\") pod \"payload-pre-processing-e2e-migrate-e7b233dd-b7df67b9d-m5szr\" (UID: \"90af83c5-b738-4369-bfa3-ac6f760b1c88\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e7b233dd-b7df67b9d-m5szr"
Sep 09 06:02:46.401529 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:46.401409    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgvcc\" (UniqueName: \"kubernetes.io/projected/ce74270c-59f9-4485-b322-e379a2232662-kube-api-access-lgvcc\") pod \"payload-processing-e2e-migrate-e7b233dd-65b9ff4f46-5t56v\" (UID: \"ce74270c-59f9-4485-b322-e379a2232662\") " pod="openshift-ingress/payload-processing-e2e-migrate-e7b233dd-65b9ff4f46-5t56v"
Sep 09 06:02:46.401529 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:46.401444    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90af83c5-b738-4369-bfa3-ac6f760b1c88-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e7b233dd-b7df67b9d-m5szr\" (UID: \"90af83c5-b738-4369-bfa3-ac6f760b1c88\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e7b233dd-b7df67b9d-m5szr"
Sep 09 06:02:46.401529 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:46.401458    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce74270c-59f9-4485-b322-e379a2232662-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e7b233dd-65b9ff4f46-5t56v\" (UID: \"ce74270c-59f9-4485-b322-e379a2232662\") " pod="openshift-ingress/payload-processing-e2e-migrate-e7b233dd-65b9ff4f46-5t56v"
Sep 09 06:02:46.402346 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:46.402322    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce74270c-59f9-4485-b322-e379a2232662-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e7b233dd-65b9ff4f46-5t56v\" (UID: \"ce74270c-59f9-4485-b322-e379a2232662\") " pod="openshift-ingress/payload-processing-e2e-migrate-e7b233dd-65b9ff4f46-5t56v"
Sep 09 06:02:46.402404 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:46.402322    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90af83c5-b738-4369-bfa3-ac6f760b1c88-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e7b233dd-b7df67b9d-m5szr\" (UID: \"90af83c5-b738-4369-bfa3-ac6f760b1c88\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e7b233dd-b7df67b9d-m5szr"
Sep 09 06:02:46.409699 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:46.409679    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvcqc\" (UniqueName: \"kubernetes.io/projected/90af83c5-b738-4369-bfa3-ac6f760b1c88-kube-api-access-kvcqc\") pod \"payload-pre-processing-e2e-migrate-e7b233dd-b7df67b9d-m5szr\" (UID: \"90af83c5-b738-4369-bfa3-ac6f760b1c88\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e7b233dd-b7df67b9d-m5szr"
Sep 09 06:02:46.409778 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:46.409743    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgvcc\" (UniqueName: \"kubernetes.io/projected/ce74270c-59f9-4485-b322-e379a2232662-kube-api-access-lgvcc\") pod \"payload-processing-e2e-migrate-e7b233dd-65b9ff4f46-5t56v\" (UID: \"ce74270c-59f9-4485-b322-e379a2232662\") " pod="openshift-ingress/payload-processing-e2e-migrate-e7b233dd-65b9ff4f46-5t56v"
Sep 09 06:02:46.517898 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:46.517842    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e7b233dd-b7df67b9d-m5szr"
Sep 09 06:02:46.552266 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:46.552232    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e7b233dd-65b9ff4f46-5t56v"
Sep 09 06:02:46.803566 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:46.803540    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e7b233dd-b7df67b9d-m5szr"]
Sep 09 06:02:46.806038 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:02:46.806004    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90af83c5_b738_4369_bfa3_ac6f760b1c88.slice/crio-80210ad1619e40e5549e8bea28b8e976da2448ca58476d8b26cce9fb8a4ef585 WatchSource:0}: Error finding container 80210ad1619e40e5549e8bea28b8e976da2448ca58476d8b26cce9fb8a4ef585: Status 404 returned error can't find the container with id 80210ad1619e40e5549e8bea28b8e976da2448ca58476d8b26cce9fb8a4ef585
Sep 09 06:02:46.831790 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:46.831772    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e7b233dd-65b9ff4f46-5t56v"]
Sep 09 06:02:46.840603 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:02:46.840574    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce74270c_59f9_4485_b322_e379a2232662.slice/crio-c54094e24168b9ab05daba199102ad2daff6b8e18739b498de79d51ab878c263 WatchSource:0}: Error finding container c54094e24168b9ab05daba199102ad2daff6b8e18739b498de79d51ab878c263: Status 404 returned error can't find the container with id c54094e24168b9ab05daba199102ad2daff6b8e18739b498de79d51ab878c263
Sep 09 06:02:46.930611 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:46.930577    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-43eb2d/simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk"
Sep 09 06:02:46.930771 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:46.930663    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-43eb2d/simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk"
Sep 09 06:02:46.942774 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:46.942754    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-43eb2d/simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk"
Sep 09 06:02:47.232578 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:47.232535    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e7b233dd-b7df67b9d-m5szr" event={"ID":"90af83c5-b738-4369-bfa3-ac6f760b1c88","Type":"ContainerStarted","Data":"9e272f28574f777ec06cf78fd9e2ea02588f99582aa746efa848a4e47f44771b"}
Sep 09 06:02:47.233066 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:47.232601    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e7b233dd-b7df67b9d-m5szr" event={"ID":"90af83c5-b738-4369-bfa3-ac6f760b1c88","Type":"ContainerStarted","Data":"80210ad1619e40e5549e8bea28b8e976da2448ca58476d8b26cce9fb8a4ef585"}
Sep 09 06:02:47.233066 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:47.232647    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e7b233dd-b7df67b9d-m5szr"
Sep 09 06:02:47.234301 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:47.234270    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e7b233dd-65b9ff4f46-5t56v" event={"ID":"ce74270c-59f9-4485-b322-e379a2232662","Type":"ContainerStarted","Data":"bcc808e6c069d2b397e7d71ef94f6884e988c9dc1ede1500575a129e4f22c6f3"}
Sep 09 06:02:47.234407 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:47.234320    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e7b233dd-65b9ff4f46-5t56v" event={"ID":"ce74270c-59f9-4485-b322-e379a2232662","Type":"ContainerStarted","Data":"c54094e24168b9ab05daba199102ad2daff6b8e18739b498de79d51ab878c263"}
Sep 09 06:02:47.234475 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:47.234424    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-e7b233dd-65b9ff4f46-5t56v"
Sep 09 06:02:47.245766 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:47.245744    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-43eb2d/simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk"
Sep 09 06:02:47.247984 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:47.247930    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e7b233dd-b7df67b9d-m5szr" podStartSLOduration=1.247913136 podStartE2EDuration="1.247913136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:02:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:02:47.246685158 +0000 UTC m=+792.587173068" watchObservedRunningTime="2026-09-09 06:02:47.247913136 +0000 UTC m=+792.588401049"
Sep 09 06:02:47.262551 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:47.262514    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-e7b233dd-65b9ff4f46-5t56v" podStartSLOduration=1.262504971 podStartE2EDuration="1.262504971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:02:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:02:47.261496105 +0000 UTC m=+792.601984013" watchObservedRunningTime="2026-09-09 06:02:47.262504971 +0000 UTC m=+792.602992880"
Sep 09 06:02:53.597308 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:53.597269    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:02:58.241869 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:58.241789    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-e7b233dd-65b9ff4f46-5t56v"
Sep 09 06:02:58.241869 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:02:58.241870    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e7b233dd-b7df67b9d-m5szr"
Sep 09 06:03:13.311284 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:13.311250    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-7a357a/autores-model-7a357a-kserve-5cd69c7b59-frt28"]
Sep 09 06:03:13.314609 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:13.314591    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-7a357a/autores-model-7a357a-kserve-5cd69c7b59-frt28"
Sep 09 06:03:13.317688 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:13.317667    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-7a357a\"/\"openshift-service-ca.crt\""
Sep 09 06:03:13.317782 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:13.317667    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-7a357a\"/\"autores-model-7a357a-kserve-self-signed-certs\""
Sep 09 06:03:13.317782 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:13.317668    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-7a357a\"/\"default-dockercfg-87ftx\""
Sep 09 06:03:13.317782 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:13.317722    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-7a357a\"/\"kube-root-ca.crt\""
Sep 09 06:03:13.322105 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:13.322087    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-7a357a/autores-model-7a357a-kserve-5cd69c7b59-frt28"]
Sep 09 06:03:13.427276 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:13.427250    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b0e85c8-14fc-4cea-a780-b9db00e0d874-home\") pod \"autores-model-7a357a-kserve-5cd69c7b59-frt28\" (UID: \"7b0e85c8-14fc-4cea-a780-b9db00e0d874\") " pod="ai-tenant-e2e-autores-7a357a/autores-model-7a357a-kserve-5cd69c7b59-frt28"
Sep 09 06:03:13.427436 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:13.427283    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b0e85c8-14fc-4cea-a780-b9db00e0d874-tls-certs\") pod \"autores-model-7a357a-kserve-5cd69c7b59-frt28\" (UID: \"7b0e85c8-14fc-4cea-a780-b9db00e0d874\") " pod="ai-tenant-e2e-autores-7a357a/autores-model-7a357a-kserve-5cd69c7b59-frt28"
Sep 09 06:03:13.427436 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:13.427308    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzcrd\" (UniqueName: \"kubernetes.io/projected/7b0e85c8-14fc-4cea-a780-b9db00e0d874-kube-api-access-qzcrd\") pod \"autores-model-7a357a-kserve-5cd69c7b59-frt28\" (UID: \"7b0e85c8-14fc-4cea-a780-b9db00e0d874\") " pod="ai-tenant-e2e-autores-7a357a/autores-model-7a357a-kserve-5cd69c7b59-frt28"
Sep 09 06:03:13.427436 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:13.427397    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b0e85c8-14fc-4cea-a780-b9db00e0d874-model-cache\") pod \"autores-model-7a357a-kserve-5cd69c7b59-frt28\" (UID: \"7b0e85c8-14fc-4cea-a780-b9db00e0d874\") " pod="ai-tenant-e2e-autores-7a357a/autores-model-7a357a-kserve-5cd69c7b59-frt28"
Sep 09 06:03:13.427436 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:13.427430    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b0e85c8-14fc-4cea-a780-b9db00e0d874-tmp-dir\") pod \"autores-model-7a357a-kserve-5cd69c7b59-frt28\" (UID: \"7b0e85c8-14fc-4cea-a780-b9db00e0d874\") " pod="ai-tenant-e2e-autores-7a357a/autores-model-7a357a-kserve-5cd69c7b59-frt28"
Sep 09 06:03:13.427661 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:13.427470    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b0e85c8-14fc-4cea-a780-b9db00e0d874-dshm\") pod \"autores-model-7a357a-kserve-5cd69c7b59-frt28\" (UID: \"7b0e85c8-14fc-4cea-a780-b9db00e0d874\") " pod="ai-tenant-e2e-autores-7a357a/autores-model-7a357a-kserve-5cd69c7b59-frt28"
Sep 09 06:03:13.528286 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:13.528264    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzcrd\" (UniqueName: \"kubernetes.io/projected/7b0e85c8-14fc-4cea-a780-b9db00e0d874-kube-api-access-qzcrd\") pod \"autores-model-7a357a-kserve-5cd69c7b59-frt28\" (UID: \"7b0e85c8-14fc-4cea-a780-b9db00e0d874\") " pod="ai-tenant-e2e-autores-7a357a/autores-model-7a357a-kserve-5cd69c7b59-frt28"
Sep 09 06:03:13.528398 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:13.528297    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b0e85c8-14fc-4cea-a780-b9db00e0d874-model-cache\") pod \"autores-model-7a357a-kserve-5cd69c7b59-frt28\" (UID: \"7b0e85c8-14fc-4cea-a780-b9db00e0d874\") " pod="ai-tenant-e2e-autores-7a357a/autores-model-7a357a-kserve-5cd69c7b59-frt28"
Sep 09 06:03:13.528398 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:13.528315    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b0e85c8-14fc-4cea-a780-b9db00e0d874-tmp-dir\") pod \"autores-model-7a357a-kserve-5cd69c7b59-frt28\" (UID: \"7b0e85c8-14fc-4cea-a780-b9db00e0d874\") " pod="ai-tenant-e2e-autores-7a357a/autores-model-7a357a-kserve-5cd69c7b59-frt28"
Sep 09 06:03:13.528502 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:13.528446    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b0e85c8-14fc-4cea-a780-b9db00e0d874-dshm\") pod \"autores-model-7a357a-kserve-5cd69c7b59-frt28\" (UID: \"7b0e85c8-14fc-4cea-a780-b9db00e0d874\") " pod="ai-tenant-e2e-autores-7a357a/autores-model-7a357a-kserve-5cd69c7b59-frt28"
Sep 09 06:03:13.528569 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:13.528511    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b0e85c8-14fc-4cea-a780-b9db00e0d874-home\") pod \"autores-model-7a357a-kserve-5cd69c7b59-frt28\" (UID: \"7b0e85c8-14fc-4cea-a780-b9db00e0d874\") " pod="ai-tenant-e2e-autores-7a357a/autores-model-7a357a-kserve-5cd69c7b59-frt28"
Sep 09 06:03:13.528569 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:13.528542    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b0e85c8-14fc-4cea-a780-b9db00e0d874-tls-certs\") pod \"autores-model-7a357a-kserve-5cd69c7b59-frt28\" (UID: \"7b0e85c8-14fc-4cea-a780-b9db00e0d874\") " pod="ai-tenant-e2e-autores-7a357a/autores-model-7a357a-kserve-5cd69c7b59-frt28"
Sep 09 06:03:13.528751 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:13.528731    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b0e85c8-14fc-4cea-a780-b9db00e0d874-tmp-dir\") pod \"autores-model-7a357a-kserve-5cd69c7b59-frt28\" (UID: \"7b0e85c8-14fc-4cea-a780-b9db00e0d874\") " pod="ai-tenant-e2e-autores-7a357a/autores-model-7a357a-kserve-5cd69c7b59-frt28"
Sep 09 06:03:13.528813 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:13.528756    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b0e85c8-14fc-4cea-a780-b9db00e0d874-model-cache\") pod \"autores-model-7a357a-kserve-5cd69c7b59-frt28\" (UID: \"7b0e85c8-14fc-4cea-a780-b9db00e0d874\") " pod="ai-tenant-e2e-autores-7a357a/autores-model-7a357a-kserve-5cd69c7b59-frt28"
Sep 09 06:03:13.528939 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:13.528921    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b0e85c8-14fc-4cea-a780-b9db00e0d874-home\") pod \"autores-model-7a357a-kserve-5cd69c7b59-frt28\" (UID: \"7b0e85c8-14fc-4cea-a780-b9db00e0d874\") " pod="ai-tenant-e2e-autores-7a357a/autores-model-7a357a-kserve-5cd69c7b59-frt28"
Sep 09 06:03:13.530874 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:13.530853    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b0e85c8-14fc-4cea-a780-b9db00e0d874-dshm\") pod \"autores-model-7a357a-kserve-5cd69c7b59-frt28\" (UID: \"7b0e85c8-14fc-4cea-a780-b9db00e0d874\") " pod="ai-tenant-e2e-autores-7a357a/autores-model-7a357a-kserve-5cd69c7b59-frt28"
Sep 09 06:03:13.531087 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:13.531073    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b0e85c8-14fc-4cea-a780-b9db00e0d874-tls-certs\") pod \"autores-model-7a357a-kserve-5cd69c7b59-frt28\" (UID: \"7b0e85c8-14fc-4cea-a780-b9db00e0d874\") " pod="ai-tenant-e2e-autores-7a357a/autores-model-7a357a-kserve-5cd69c7b59-frt28"
Sep 09 06:03:13.535585 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:13.535561    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzcrd\" (UniqueName: \"kubernetes.io/projected/7b0e85c8-14fc-4cea-a780-b9db00e0d874-kube-api-access-qzcrd\") pod \"autores-model-7a357a-kserve-5cd69c7b59-frt28\" (UID: \"7b0e85c8-14fc-4cea-a780-b9db00e0d874\") " pod="ai-tenant-e2e-autores-7a357a/autores-model-7a357a-kserve-5cd69c7b59-frt28"
Sep 09 06:03:13.625199 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:13.625140    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-7a357a/autores-model-7a357a-kserve-5cd69c7b59-frt28"
Sep 09 06:03:13.796068 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:13.796029    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:03:13.895518 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:13.895496    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-7a357a/autores-model-7a357a-kserve-5cd69c7b59-frt28"]
Sep 09 06:03:13.897667 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:03:13.897618    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b0e85c8_14fc_4cea_a780_b9db00e0d874.slice/crio-883fcb772bd3b18bdeb0dc56febe9ac1d0e2f29b49c96e5fcc44b0b742ae4c43 WatchSource:0}: Error finding container 883fcb772bd3b18bdeb0dc56febe9ac1d0e2f29b49c96e5fcc44b0b742ae4c43: Status 404 returned error can't find the container with id 883fcb772bd3b18bdeb0dc56febe9ac1d0e2f29b49c96e5fcc44b0b742ae4c43
Sep 09 06:03:14.354963 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:14.354933    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-7a357a/autores-model-7a357a-kserve-5cd69c7b59-frt28" event={"ID":"7b0e85c8-14fc-4cea-a780-b9db00e0d874","Type":"ContainerStarted","Data":"7a3e917134a671c7e2a09846c06c9fc4e2fd4cc724a643edf56f057db4f0893d"}
Sep 09 06:03:14.355328 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:14.354975    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-7a357a/autores-model-7a357a-kserve-5cd69c7b59-frt28" event={"ID":"7b0e85c8-14fc-4cea-a780-b9db00e0d874","Type":"ContainerStarted","Data":"883fcb772bd3b18bdeb0dc56febe9ac1d0e2f29b49c96e5fcc44b0b742ae4c43"}
Sep 09 06:03:14.371684 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:14.371616    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-7a357a/autores-model-7a357a-kserve-5cd69c7b59-frt28" podStartSLOduration=1.3715995460000001 podStartE2EDuration="1.371599546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:03:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:03:14.36998329 +0000 UTC m=+819.710471202" watchObservedRunningTime="2026-09-09 06:03:14.371599546 +0000 UTC m=+819.712087488"
Sep 09 06:03:15.995658 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:15.995606    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e7b233dd-65b9ff4f46-5t56v"]
Sep 09 06:03:15.996102 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:15.995891    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-e7b233dd-65b9ff4f46-5t56v" podUID="ce74270c-59f9-4485-b322-e379a2232662" containerName="payload-processing" containerID="cri-o://bcc808e6c069d2b397e7d71ef94f6884e988c9dc1ede1500575a129e4f22c6f3" gracePeriod=30
Sep 09 06:03:16.003776 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:16.003722    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e7b233dd-b7df67b9d-m5szr"]
Sep 09 06:03:16.004056 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:16.003969    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e7b233dd-b7df67b9d-m5szr" podUID="90af83c5-b738-4369-bfa3-ac6f760b1c88" containerName="payload-pre-processing" containerID="cri-o://9e272f28574f777ec06cf78fd9e2ea02588f99582aa746efa848a4e47f44771b" gracePeriod=30
Sep 09 06:03:16.251048 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:16.250996    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e7b233dd-65b9ff4f46-5t56v"
Sep 09 06:03:16.277077 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:16.277038    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e7b233dd-b7df67b9d-m5szr"
Sep 09 06:03:16.351244 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:16.351210    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce74270c-59f9-4485-b322-e379a2232662-plugins-config-volume\") pod \"ce74270c-59f9-4485-b322-e379a2232662\" (UID: \"ce74270c-59f9-4485-b322-e379a2232662\") "
Sep 09 06:03:16.351394 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:16.351316    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgvcc\" (UniqueName: \"kubernetes.io/projected/ce74270c-59f9-4485-b322-e379a2232662-kube-api-access-lgvcc\") pod \"ce74270c-59f9-4485-b322-e379a2232662\" (UID: \"ce74270c-59f9-4485-b322-e379a2232662\") "
Sep 09 06:03:16.351659 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:16.351611    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce74270c-59f9-4485-b322-e379a2232662-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ce74270c-59f9-4485-b322-e379a2232662" (UID: "ce74270c-59f9-4485-b322-e379a2232662"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:03:16.353430 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:16.353404    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce74270c-59f9-4485-b322-e379a2232662-kube-api-access-lgvcc" (OuterVolumeSpecName: "kube-api-access-lgvcc") pod "ce74270c-59f9-4485-b322-e379a2232662" (UID: "ce74270c-59f9-4485-b322-e379a2232662"). InnerVolumeSpecName "kube-api-access-lgvcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:03:16.364603 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:16.364578    2579 generic.go:356] "Generic (PLEG): container finished" podID="90af83c5-b738-4369-bfa3-ac6f760b1c88" containerID="9e272f28574f777ec06cf78fd9e2ea02588f99582aa746efa848a4e47f44771b" exitCode=0
Sep 09 06:03:16.364758 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:16.364653    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e7b233dd-b7df67b9d-m5szr" event={"ID":"90af83c5-b738-4369-bfa3-ac6f760b1c88","Type":"ContainerDied","Data":"9e272f28574f777ec06cf78fd9e2ea02588f99582aa746efa848a4e47f44771b"}
Sep 09 06:03:16.364758 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:16.364659    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e7b233dd-b7df67b9d-m5szr"
Sep 09 06:03:16.364758 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:16.364716    2579 scope.go:117] "RemoveContainer" containerID="9e272f28574f777ec06cf78fd9e2ea02588f99582aa746efa848a4e47f44771b"
Sep 09 06:03:16.364897 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:16.364704    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e7b233dd-b7df67b9d-m5szr" event={"ID":"90af83c5-b738-4369-bfa3-ac6f760b1c88","Type":"ContainerDied","Data":"80210ad1619e40e5549e8bea28b8e976da2448ca58476d8b26cce9fb8a4ef585"}
Sep 09 06:03:16.366040 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:16.366018    2579 generic.go:356] "Generic (PLEG): container finished" podID="ce74270c-59f9-4485-b322-e379a2232662" containerID="bcc808e6c069d2b397e7d71ef94f6884e988c9dc1ede1500575a129e4f22c6f3" exitCode=0
Sep 09 06:03:16.366101 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:16.366052    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e7b233dd-65b9ff4f46-5t56v" event={"ID":"ce74270c-59f9-4485-b322-e379a2232662","Type":"ContainerDied","Data":"bcc808e6c069d2b397e7d71ef94f6884e988c9dc1ede1500575a129e4f22c6f3"}
Sep 09 06:03:16.366101 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:16.366097    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e7b233dd-65b9ff4f46-5t56v" event={"ID":"ce74270c-59f9-4485-b322-e379a2232662","Type":"ContainerDied","Data":"c54094e24168b9ab05daba199102ad2daff6b8e18739b498de79d51ab878c263"}
Sep 09 06:03:16.366245 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:16.366229    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e7b233dd-65b9ff4f46-5t56v"
Sep 09 06:03:16.374356 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:16.374334    2579 scope.go:117] "RemoveContainer" containerID="9e272f28574f777ec06cf78fd9e2ea02588f99582aa746efa848a4e47f44771b"
Sep 09 06:03:16.374678 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:03:16.374620    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e272f28574f777ec06cf78fd9e2ea02588f99582aa746efa848a4e47f44771b\": container with ID starting with 9e272f28574f777ec06cf78fd9e2ea02588f99582aa746efa848a4e47f44771b not found: ID does not exist" containerID="9e272f28574f777ec06cf78fd9e2ea02588f99582aa746efa848a4e47f44771b"
Sep 09 06:03:16.374742 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:16.374673    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e272f28574f777ec06cf78fd9e2ea02588f99582aa746efa848a4e47f44771b"} err="failed to get container status \"9e272f28574f777ec06cf78fd9e2ea02588f99582aa746efa848a4e47f44771b\": rpc error: code = NotFound desc = could not find container \"9e272f28574f777ec06cf78fd9e2ea02588f99582aa746efa848a4e47f44771b\": container with ID starting with 9e272f28574f777ec06cf78fd9e2ea02588f99582aa746efa848a4e47f44771b not found: ID does not exist"
Sep 09 06:03:16.374742 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:16.374696    2579 scope.go:117] "RemoveContainer" containerID="bcc808e6c069d2b397e7d71ef94f6884e988c9dc1ede1500575a129e4f22c6f3"
Sep 09 06:03:16.382611 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:16.382592    2579 scope.go:117] "RemoveContainer" containerID="bcc808e6c069d2b397e7d71ef94f6884e988c9dc1ede1500575a129e4f22c6f3"
Sep 09 06:03:16.382877 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:03:16.382858    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcc808e6c069d2b397e7d71ef94f6884e988c9dc1ede1500575a129e4f22c6f3\": container with ID starting with bcc808e6c069d2b397e7d71ef94f6884e988c9dc1ede1500575a129e4f22c6f3 not found: ID does not exist" containerID="bcc808e6c069d2b397e7d71ef94f6884e988c9dc1ede1500575a129e4f22c6f3"
Sep 09 06:03:16.382922 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:16.382885    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcc808e6c069d2b397e7d71ef94f6884e988c9dc1ede1500575a129e4f22c6f3"} err="failed to get container status \"bcc808e6c069d2b397e7d71ef94f6884e988c9dc1ede1500575a129e4f22c6f3\": rpc error: code = NotFound desc = could not find container \"bcc808e6c069d2b397e7d71ef94f6884e988c9dc1ede1500575a129e4f22c6f3\": container with ID starting with bcc808e6c069d2b397e7d71ef94f6884e988c9dc1ede1500575a129e4f22c6f3 not found: ID does not exist"
Sep 09 06:03:16.388047 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:16.388026    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e7b233dd-65b9ff4f46-5t56v"]
Sep 09 06:03:16.391332 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:16.391306    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e7b233dd-65b9ff4f46-5t56v"]
Sep 09 06:03:16.452446 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:16.452420    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90af83c5-b738-4369-bfa3-ac6f760b1c88-plugins-config-volume\") pod \"90af83c5-b738-4369-bfa3-ac6f760b1c88\" (UID: \"90af83c5-b738-4369-bfa3-ac6f760b1c88\") "
Sep 09 06:03:16.452550 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:16.452467    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvcqc\" (UniqueName: \"kubernetes.io/projected/90af83c5-b738-4369-bfa3-ac6f760b1c88-kube-api-access-kvcqc\") pod \"90af83c5-b738-4369-bfa3-ac6f760b1c88\" (UID: \"90af83c5-b738-4369-bfa3-ac6f760b1c88\") "
Sep 09 06:03:16.452715 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:16.452701    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgvcc\" (UniqueName: \"kubernetes.io/projected/ce74270c-59f9-4485-b322-e379a2232662-kube-api-access-lgvcc\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:03:16.452793 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:16.452716    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce74270c-59f9-4485-b322-e379a2232662-plugins-config-volume\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:03:16.452889 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:16.452839    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90af83c5-b738-4369-bfa3-ac6f760b1c88-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "90af83c5-b738-4369-bfa3-ac6f760b1c88" (UID: "90af83c5-b738-4369-bfa3-ac6f760b1c88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:03:16.454553 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:16.454536    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90af83c5-b738-4369-bfa3-ac6f760b1c88-kube-api-access-kvcqc" (OuterVolumeSpecName: "kube-api-access-kvcqc") pod "90af83c5-b738-4369-bfa3-ac6f760b1c88" (UID: "90af83c5-b738-4369-bfa3-ac6f760b1c88"). InnerVolumeSpecName "kube-api-access-kvcqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:03:16.554040 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:16.553987    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90af83c5-b738-4369-bfa3-ac6f760b1c88-plugins-config-volume\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:03:16.554040 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:16.554007    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvcqc\" (UniqueName: \"kubernetes.io/projected/90af83c5-b738-4369-bfa3-ac6f760b1c88-kube-api-access-kvcqc\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:03:16.684311 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:16.684290    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e7b233dd-b7df67b9d-m5szr"]
Sep 09 06:03:16.687270 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:16.687251    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e7b233dd-b7df67b9d-m5szr"]
Sep 09 06:03:17.148865 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:17.148832    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90af83c5-b738-4369-bfa3-ac6f760b1c88" path="/var/lib/kubelet/pods/90af83c5-b738-4369-bfa3-ac6f760b1c88/volumes"
Sep 09 06:03:17.149210 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:17.149170    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce74270c-59f9-4485-b322-e379a2232662" path="/var/lib/kubelet/pods/ce74270c-59f9-4485-b322-e379a2232662/volumes"
Sep 09 06:03:21.719192 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:21.719146    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8b70f171-7858689f58-4f4t8"]
Sep 09 06:03:21.719815 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:21.719792    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90af83c5-b738-4369-bfa3-ac6f760b1c88" containerName="payload-pre-processing"
Sep 09 06:03:21.719871 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:21.719823    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="90af83c5-b738-4369-bfa3-ac6f760b1c88" containerName="payload-pre-processing"
Sep 09 06:03:21.719871 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:21.719848    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce74270c-59f9-4485-b322-e379a2232662" containerName="payload-processing"
Sep 09 06:03:21.719871 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:21.719859    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce74270c-59f9-4485-b322-e379a2232662" containerName="payload-processing"
Sep 09 06:03:21.719961 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:21.719946    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce74270c-59f9-4485-b322-e379a2232662" containerName="payload-processing"
Sep 09 06:03:21.719961 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:21.719958    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="90af83c5-b738-4369-bfa3-ac6f760b1c88" containerName="payload-pre-processing"
Sep 09 06:03:21.724984 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:21.724963    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8b70f171-7858689f58-4f4t8"
Sep 09 06:03:21.727369 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:21.727341    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8b70f171-dockercfg-6wpt4\""
Sep 09 06:03:21.727369 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:21.727364    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8b70f171\""
Sep 09 06:03:21.734317 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:21.733998    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8b70f171-7858689f58-4f4t8"]
Sep 09 06:03:21.740757 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:21.740734    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8b70f171-7b95cb769f-nxfcf"]
Sep 09 06:03:21.743922 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:21.743901    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8b70f171-7b95cb769f-nxfcf"
Sep 09 06:03:21.754733 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:21.754585    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8b70f171-7b95cb769f-nxfcf"]
Sep 09 06:03:21.901204 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:21.900939    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb4zs\" (UniqueName: \"kubernetes.io/projected/d61639ca-d4d1-44c0-90e8-c98e54d939cf-kube-api-access-pb4zs\") pod \"payload-pre-processing-e2e-ait-8b70f171-7858689f58-4f4t8\" (UID: \"d61639ca-d4d1-44c0-90e8-c98e54d939cf\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8b70f171-7858689f58-4f4t8"
Sep 09 06:03:21.901204 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:21.901041    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/261a4acc-243a-4a4e-ab00-f02529ff0d60-plugins-config-volume\") pod \"payload-processing-e2e-ait-8b70f171-7b95cb769f-nxfcf\" (UID: \"261a4acc-243a-4a4e-ab00-f02529ff0d60\") " pod="openshift-ingress/payload-processing-e2e-ait-8b70f171-7b95cb769f-nxfcf"
Sep 09 06:03:21.901204 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:21.901095    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d61639ca-d4d1-44c0-90e8-c98e54d939cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8b70f171-7858689f58-4f4t8\" (UID: \"d61639ca-d4d1-44c0-90e8-c98e54d939cf\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8b70f171-7858689f58-4f4t8"
Sep 09 06:03:21.901204 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:21.901127    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76gw9\" (UniqueName: \"kubernetes.io/projected/261a4acc-243a-4a4e-ab00-f02529ff0d60-kube-api-access-76gw9\") pod \"payload-processing-e2e-ait-8b70f171-7b95cb769f-nxfcf\" (UID: \"261a4acc-243a-4a4e-ab00-f02529ff0d60\") " pod="openshift-ingress/payload-processing-e2e-ait-8b70f171-7b95cb769f-nxfcf"
Sep 09 06:03:22.002410 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:22.002310    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/261a4acc-243a-4a4e-ab00-f02529ff0d60-plugins-config-volume\") pod \"payload-processing-e2e-ait-8b70f171-7b95cb769f-nxfcf\" (UID: \"261a4acc-243a-4a4e-ab00-f02529ff0d60\") " pod="openshift-ingress/payload-processing-e2e-ait-8b70f171-7b95cb769f-nxfcf"
Sep 09 06:03:22.002410 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:22.002401    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d61639ca-d4d1-44c0-90e8-c98e54d939cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8b70f171-7858689f58-4f4t8\" (UID: \"d61639ca-d4d1-44c0-90e8-c98e54d939cf\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8b70f171-7858689f58-4f4t8"
Sep 09 06:03:22.002650 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:22.002437    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76gw9\" (UniqueName: \"kubernetes.io/projected/261a4acc-243a-4a4e-ab00-f02529ff0d60-kube-api-access-76gw9\") pod \"payload-processing-e2e-ait-8b70f171-7b95cb769f-nxfcf\" (UID: \"261a4acc-243a-4a4e-ab00-f02529ff0d60\") " pod="openshift-ingress/payload-processing-e2e-ait-8b70f171-7b95cb769f-nxfcf"
Sep 09 06:03:22.002650 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:22.002499    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb4zs\" (UniqueName: \"kubernetes.io/projected/d61639ca-d4d1-44c0-90e8-c98e54d939cf-kube-api-access-pb4zs\") pod \"payload-pre-processing-e2e-ait-8b70f171-7858689f58-4f4t8\" (UID: \"d61639ca-d4d1-44c0-90e8-c98e54d939cf\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8b70f171-7858689f58-4f4t8"
Sep 09 06:03:22.003142 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:22.003119    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/261a4acc-243a-4a4e-ab00-f02529ff0d60-plugins-config-volume\") pod \"payload-processing-e2e-ait-8b70f171-7b95cb769f-nxfcf\" (UID: \"261a4acc-243a-4a4e-ab00-f02529ff0d60\") " pod="openshift-ingress/payload-processing-e2e-ait-8b70f171-7b95cb769f-nxfcf"
Sep 09 06:03:22.003232 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:22.003206    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d61639ca-d4d1-44c0-90e8-c98e54d939cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8b70f171-7858689f58-4f4t8\" (UID: \"d61639ca-d4d1-44c0-90e8-c98e54d939cf\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8b70f171-7858689f58-4f4t8"
Sep 09 06:03:22.011909 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:22.011885    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76gw9\" (UniqueName: \"kubernetes.io/projected/261a4acc-243a-4a4e-ab00-f02529ff0d60-kube-api-access-76gw9\") pod \"payload-processing-e2e-ait-8b70f171-7b95cb769f-nxfcf\" (UID: \"261a4acc-243a-4a4e-ab00-f02529ff0d60\") " pod="openshift-ingress/payload-processing-e2e-ait-8b70f171-7b95cb769f-nxfcf"
Sep 09 06:03:22.012039 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:22.011959    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb4zs\" (UniqueName: \"kubernetes.io/projected/d61639ca-d4d1-44c0-90e8-c98e54d939cf-kube-api-access-pb4zs\") pod \"payload-pre-processing-e2e-ait-8b70f171-7858689f58-4f4t8\" (UID: \"d61639ca-d4d1-44c0-90e8-c98e54d939cf\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8b70f171-7858689f58-4f4t8"
Sep 09 06:03:22.038506 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:22.038484    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8b70f171-7858689f58-4f4t8"
Sep 09 06:03:22.055787 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:22.055759    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8b70f171-7b95cb769f-nxfcf"
Sep 09 06:03:22.316536 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:22.316505    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8b70f171-7858689f58-4f4t8"]
Sep 09 06:03:22.319656 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:03:22.319605    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd61639ca_d4d1_44c0_90e8_c98e54d939cf.slice/crio-a5779047ca474deca2bc1cf0b397dfcc0a839f6600b1229187d39782c89caf14 WatchSource:0}: Error finding container a5779047ca474deca2bc1cf0b397dfcc0a839f6600b1229187d39782c89caf14: Status 404 returned error can't find the container with id a5779047ca474deca2bc1cf0b397dfcc0a839f6600b1229187d39782c89caf14
Sep 09 06:03:22.340396 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:22.340371    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8b70f171-7b95cb769f-nxfcf"]
Sep 09 06:03:22.342081 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:03:22.342055    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod261a4acc_243a_4a4e_ab00_f02529ff0d60.slice/crio-e9193d675de9cab043eb3a22c5bf0c22aec29b9a1c3065003e969661fc7eb2d9 WatchSource:0}: Error finding container e9193d675de9cab043eb3a22c5bf0c22aec29b9a1c3065003e969661fc7eb2d9: Status 404 returned error can't find the container with id e9193d675de9cab043eb3a22c5bf0c22aec29b9a1c3065003e969661fc7eb2d9
Sep 09 06:03:22.398692 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:22.398657    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8b70f171-7858689f58-4f4t8" event={"ID":"d61639ca-d4d1-44c0-90e8-c98e54d939cf","Type":"ContainerStarted","Data":"8ea1b4b1ec4b80b9cda1540e08f73f177d94b78ccce27f9b0815275370aec3b5"}
Sep 09 06:03:22.398808 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:22.398730    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8b70f171-7858689f58-4f4t8" event={"ID":"d61639ca-d4d1-44c0-90e8-c98e54d939cf","Type":"ContainerStarted","Data":"a5779047ca474deca2bc1cf0b397dfcc0a839f6600b1229187d39782c89caf14"}
Sep 09 06:03:22.398808 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:22.398763    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8b70f171-7858689f58-4f4t8"
Sep 09 06:03:22.399989 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:22.399961    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8b70f171-7b95cb769f-nxfcf" event={"ID":"261a4acc-243a-4a4e-ab00-f02529ff0d60","Type":"ContainerStarted","Data":"e9193d675de9cab043eb3a22c5bf0c22aec29b9a1c3065003e969661fc7eb2d9"}
Sep 09 06:03:22.414851 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:22.414803    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-8b70f171-7858689f58-4f4t8" podStartSLOduration=1.414789044 podStartE2EDuration="1.414789044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:03:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:03:22.413423948 +0000 UTC m=+827.753911858" watchObservedRunningTime="2026-09-09 06:03:22.414789044 +0000 UTC m=+827.755276954"
Sep 09 06:03:23.406868 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:23.406837    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8b70f171-7b95cb769f-nxfcf" event={"ID":"261a4acc-243a-4a4e-ab00-f02529ff0d60","Type":"ContainerStarted","Data":"fd48b00fae1ea686220256a8f9a5e61b07c14571eba427908335064b9e1c14db"}
Sep 09 06:03:23.407311 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:23.406919    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-8b70f171-7b95cb769f-nxfcf"
Sep 09 06:03:23.423966 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:23.423915    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-8b70f171-7b95cb769f-nxfcf" podStartSLOduration=2.423896447 podStartE2EDuration="2.423896447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:03:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:03:23.422691925 +0000 UTC m=+828.763179837" watchObservedRunningTime="2026-09-09 06:03:23.423896447 +0000 UTC m=+828.764384363"
Sep 09 06:03:23.625294 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:23.625272    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-7a357a/autores-model-7a357a-kserve-5cd69c7b59-frt28"
Sep 09 06:03:23.625425 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:23.625346    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-7a357a/autores-model-7a357a-kserve-5cd69c7b59-frt28"
Sep 09 06:03:23.638040 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:23.638012    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-7a357a/autores-model-7a357a-kserve-5cd69c7b59-frt28"
Sep 09 06:03:24.422291 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:24.422266    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-7a357a/autores-model-7a357a-kserve-5cd69c7b59-frt28"
Sep 09 06:03:31.001866 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:31.001823    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:03:33.409449 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:33.409415    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8b70f171-7858689f58-4f4t8"
Sep 09 06:03:34.413666 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:34.413619    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-8b70f171-7b95cb769f-nxfcf"
Sep 09 06:03:44.689213 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:44.689180    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-7a357a/autores-model-7a357a-kserve-5cd69c7b59-frt28"]
Sep 09 06:03:45.609955 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:45.609923    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8b70f171-xmxkz"]
Sep 09 06:03:45.614438 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:45.614423    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8b70f171-xmxkz"
Sep 09 06:03:45.616715 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:45.616693    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ksjk6\""
Sep 09 06:03:45.620216 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:45.620192    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8b70f171-xmxkz"]
Sep 09 06:03:45.692413 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:45.692369    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/314c05dd-1040-4cc1-b50e-9ca03cfd2de6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8b70f171-xmxkz\" (UID: \"314c05dd-1040-4cc1-b50e-9ca03cfd2de6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8b70f171-xmxkz"
Sep 09 06:03:45.793207 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:45.793187    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/314c05dd-1040-4cc1-b50e-9ca03cfd2de6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8b70f171-xmxkz\" (UID: \"314c05dd-1040-4cc1-b50e-9ca03cfd2de6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8b70f171-xmxkz"
Sep 09 06:03:45.793771 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:45.793754    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/314c05dd-1040-4cc1-b50e-9ca03cfd2de6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8b70f171-xmxkz\" (UID: \"314c05dd-1040-4cc1-b50e-9ca03cfd2de6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8b70f171-xmxkz"
Sep 09 06:03:45.925297 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:45.925241    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8b70f171-xmxkz"
Sep 09 06:03:46.195355 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:46.195282    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8b70f171-xmxkz"]
Sep 09 06:03:46.198157 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:03:46.198129    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod314c05dd_1040_4cc1_b50e_9ca03cfd2de6.slice/crio-e5261cd8c95ef0776e1d983dfd806bdfb0f3bed1ed4c49dd79dc80d7c8e6a238 WatchSource:0}: Error finding container e5261cd8c95ef0776e1d983dfd806bdfb0f3bed1ed4c49dd79dc80d7c8e6a238: Status 404 returned error can't find the container with id e5261cd8c95ef0776e1d983dfd806bdfb0f3bed1ed4c49dd79dc80d7c8e6a238
Sep 09 06:03:46.521021 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:46.520995    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8b70f171-xmxkz" event={"ID":"314c05dd-1040-4cc1-b50e-9ca03cfd2de6","Type":"ContainerStarted","Data":"e5261cd8c95ef0776e1d983dfd806bdfb0f3bed1ed4c49dd79dc80d7c8e6a238"}
Sep 09 06:03:47.526768 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:47.526733    2579 generic.go:356] "Generic (PLEG): container finished" podID="314c05dd-1040-4cc1-b50e-9ca03cfd2de6" containerID="26cc04b3f97062ebfd51689fbb1f776b16153a07dbd326f0e84ef390cb24576b" exitCode=0
Sep 09 06:03:47.527273 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:47.526825    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8b70f171-xmxkz" event={"ID":"314c05dd-1040-4cc1-b50e-9ca03cfd2de6","Type":"ContainerDied","Data":"26cc04b3f97062ebfd51689fbb1f776b16153a07dbd326f0e84ef390cb24576b"}
Sep 09 06:03:48.606946 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:48.606913    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:03:48.661430 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:48.661407    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8b70f171-xmxkz"
Sep 09 06:03:48.717614 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:48.717584    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/314c05dd-1040-4cc1-b50e-9ca03cfd2de6-maas-api-service-ca\") pod \"314c05dd-1040-4cc1-b50e-9ca03cfd2de6\" (UID: \"314c05dd-1040-4cc1-b50e-9ca03cfd2de6\") "
Sep 09 06:03:48.717922 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:48.717893    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/314c05dd-1040-4cc1-b50e-9ca03cfd2de6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "314c05dd-1040-4cc1-b50e-9ca03cfd2de6" (UID: "314c05dd-1040-4cc1-b50e-9ca03cfd2de6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:03:48.818990 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:48.818961    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/314c05dd-1040-4cc1-b50e-9ca03cfd2de6-maas-api-service-ca\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:03:49.538715 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:49.538616    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8b70f171-xmxkz"
Sep 09 06:03:49.538715 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:49.538652    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8b70f171-xmxkz" event={"ID":"314c05dd-1040-4cc1-b50e-9ca03cfd2de6","Type":"ContainerDied","Data":"e5261cd8c95ef0776e1d983dfd806bdfb0f3bed1ed4c49dd79dc80d7c8e6a238"}
Sep 09 06:03:49.538898 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:49.538742    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5261cd8c95ef0776e1d983dfd806bdfb0f3bed1ed4c49dd79dc80d7c8e6a238"
Sep 09 06:03:51.777727 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:51.777687    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8b70f171-7b95cb769f-nxfcf"]
Sep 09 06:03:51.778172 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:51.777953    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-8b70f171-7b95cb769f-nxfcf" podUID="261a4acc-243a-4a4e-ab00-f02529ff0d60" containerName="payload-processing" containerID="cri-o://fd48b00fae1ea686220256a8f9a5e61b07c14571eba427908335064b9e1c14db" gracePeriod=30
Sep 09 06:03:51.786057 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:51.786033    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8b70f171-7858689f58-4f4t8"]
Sep 09 06:03:51.786287 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:51.786250    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-8b70f171-7858689f58-4f4t8" podUID="d61639ca-d4d1-44c0-90e8-c98e54d939cf" containerName="payload-pre-processing" containerID="cri-o://8ea1b4b1ec4b80b9cda1540e08f73f177d94b78ccce27f9b0815275370aec3b5" gracePeriod=30
Sep 09 06:03:51.943934 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:51.943853    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8b70f171-xmxkz"]
Sep 09 06:03:51.946042 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:51.945589    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8b70f171-xmxkz"]
Sep 09 06:03:52.052671 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:52.052599    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8b70f171-7b95cb769f-nxfcf"
Sep 09 06:03:52.055736 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:52.055718    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8b70f171-7858689f58-4f4t8"
Sep 09 06:03:52.149291 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:52.149269    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pb4zs\" (UniqueName: \"kubernetes.io/projected/d61639ca-d4d1-44c0-90e8-c98e54d939cf-kube-api-access-pb4zs\") pod \"d61639ca-d4d1-44c0-90e8-c98e54d939cf\" (UID: \"d61639ca-d4d1-44c0-90e8-c98e54d939cf\") "
Sep 09 06:03:52.149452 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:52.149303    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/261a4acc-243a-4a4e-ab00-f02529ff0d60-plugins-config-volume\") pod \"261a4acc-243a-4a4e-ab00-f02529ff0d60\" (UID: \"261a4acc-243a-4a4e-ab00-f02529ff0d60\") "
Sep 09 06:03:52.149452 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:52.149368    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76gw9\" (UniqueName: \"kubernetes.io/projected/261a4acc-243a-4a4e-ab00-f02529ff0d60-kube-api-access-76gw9\") pod \"261a4acc-243a-4a4e-ab00-f02529ff0d60\" (UID: \"261a4acc-243a-4a4e-ab00-f02529ff0d60\") "
Sep 09 06:03:52.149452 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:52.149391    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d61639ca-d4d1-44c0-90e8-c98e54d939cf-plugins-config-volume\") pod \"d61639ca-d4d1-44c0-90e8-c98e54d939cf\" (UID: \"d61639ca-d4d1-44c0-90e8-c98e54d939cf\") "
Sep 09 06:03:52.149741 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:52.149715    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/261a4acc-243a-4a4e-ab00-f02529ff0d60-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "261a4acc-243a-4a4e-ab00-f02529ff0d60" (UID: "261a4acc-243a-4a4e-ab00-f02529ff0d60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:03:52.149842 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:52.149788    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d61639ca-d4d1-44c0-90e8-c98e54d939cf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d61639ca-d4d1-44c0-90e8-c98e54d939cf" (UID: "d61639ca-d4d1-44c0-90e8-c98e54d939cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:03:52.151500 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:52.151478    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d61639ca-d4d1-44c0-90e8-c98e54d939cf-kube-api-access-pb4zs" (OuterVolumeSpecName: "kube-api-access-pb4zs") pod "d61639ca-d4d1-44c0-90e8-c98e54d939cf" (UID: "d61639ca-d4d1-44c0-90e8-c98e54d939cf"). InnerVolumeSpecName "kube-api-access-pb4zs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:03:52.151603 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:52.151506    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/261a4acc-243a-4a4e-ab00-f02529ff0d60-kube-api-access-76gw9" (OuterVolumeSpecName: "kube-api-access-76gw9") pod "261a4acc-243a-4a4e-ab00-f02529ff0d60" (UID: "261a4acc-243a-4a4e-ab00-f02529ff0d60"). InnerVolumeSpecName "kube-api-access-76gw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:03:52.250291 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:52.250259    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb4zs\" (UniqueName: \"kubernetes.io/projected/d61639ca-d4d1-44c0-90e8-c98e54d939cf-kube-api-access-pb4zs\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:03:52.250291 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:52.250289    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/261a4acc-243a-4a4e-ab00-f02529ff0d60-plugins-config-volume\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:03:52.250416 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:52.250305    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76gw9\" (UniqueName: \"kubernetes.io/projected/261a4acc-243a-4a4e-ab00-f02529ff0d60-kube-api-access-76gw9\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:03:52.250416 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:52.250315    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d61639ca-d4d1-44c0-90e8-c98e54d939cf-plugins-config-volume\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:03:52.553612 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:52.553578    2579 generic.go:356] "Generic (PLEG): container finished" podID="261a4acc-243a-4a4e-ab00-f02529ff0d60" containerID="fd48b00fae1ea686220256a8f9a5e61b07c14571eba427908335064b9e1c14db" exitCode=0
Sep 09 06:03:52.553795 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:52.553645    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8b70f171-7b95cb769f-nxfcf"
Sep 09 06:03:52.553795 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:52.553674    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8b70f171-7b95cb769f-nxfcf" event={"ID":"261a4acc-243a-4a4e-ab00-f02529ff0d60","Type":"ContainerDied","Data":"fd48b00fae1ea686220256a8f9a5e61b07c14571eba427908335064b9e1c14db"}
Sep 09 06:03:52.553795 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:52.553733    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8b70f171-7b95cb769f-nxfcf" event={"ID":"261a4acc-243a-4a4e-ab00-f02529ff0d60","Type":"ContainerDied","Data":"e9193d675de9cab043eb3a22c5bf0c22aec29b9a1c3065003e969661fc7eb2d9"}
Sep 09 06:03:52.553795 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:52.553763    2579 scope.go:117] "RemoveContainer" containerID="fd48b00fae1ea686220256a8f9a5e61b07c14571eba427908335064b9e1c14db"
Sep 09 06:03:52.554989 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:52.554968    2579 generic.go:356] "Generic (PLEG): container finished" podID="d61639ca-d4d1-44c0-90e8-c98e54d939cf" containerID="8ea1b4b1ec4b80b9cda1540e08f73f177d94b78ccce27f9b0815275370aec3b5" exitCode=0
Sep 09 06:03:52.555062 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:52.555008    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8b70f171-7858689f58-4f4t8"
Sep 09 06:03:52.555062 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:52.555044    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8b70f171-7858689f58-4f4t8" event={"ID":"d61639ca-d4d1-44c0-90e8-c98e54d939cf","Type":"ContainerDied","Data":"8ea1b4b1ec4b80b9cda1540e08f73f177d94b78ccce27f9b0815275370aec3b5"}
Sep 09 06:03:52.555153 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:52.555083    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8b70f171-7858689f58-4f4t8" event={"ID":"d61639ca-d4d1-44c0-90e8-c98e54d939cf","Type":"ContainerDied","Data":"a5779047ca474deca2bc1cf0b397dfcc0a839f6600b1229187d39782c89caf14"}
Sep 09 06:03:52.562981 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:52.562958    2579 scope.go:117] "RemoveContainer" containerID="fd48b00fae1ea686220256a8f9a5e61b07c14571eba427908335064b9e1c14db"
Sep 09 06:03:52.563336 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:03:52.563267    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd48b00fae1ea686220256a8f9a5e61b07c14571eba427908335064b9e1c14db\": container with ID starting with fd48b00fae1ea686220256a8f9a5e61b07c14571eba427908335064b9e1c14db not found: ID does not exist" containerID="fd48b00fae1ea686220256a8f9a5e61b07c14571eba427908335064b9e1c14db"
Sep 09 06:03:52.563336 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:52.563297    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd48b00fae1ea686220256a8f9a5e61b07c14571eba427908335064b9e1c14db"} err="failed to get container status \"fd48b00fae1ea686220256a8f9a5e61b07c14571eba427908335064b9e1c14db\": rpc error: code = NotFound desc = could not find container \"fd48b00fae1ea686220256a8f9a5e61b07c14571eba427908335064b9e1c14db\": container with ID starting with fd48b00fae1ea686220256a8f9a5e61b07c14571eba427908335064b9e1c14db not found: ID does not exist"
Sep 09 06:03:52.563336 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:52.563322    2579 scope.go:117] "RemoveContainer" containerID="8ea1b4b1ec4b80b9cda1540e08f73f177d94b78ccce27f9b0815275370aec3b5"
Sep 09 06:03:52.571041 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:52.571024    2579 scope.go:117] "RemoveContainer" containerID="8ea1b4b1ec4b80b9cda1540e08f73f177d94b78ccce27f9b0815275370aec3b5"
Sep 09 06:03:52.571290 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:03:52.571271    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ea1b4b1ec4b80b9cda1540e08f73f177d94b78ccce27f9b0815275370aec3b5\": container with ID starting with 8ea1b4b1ec4b80b9cda1540e08f73f177d94b78ccce27f9b0815275370aec3b5 not found: ID does not exist" containerID="8ea1b4b1ec4b80b9cda1540e08f73f177d94b78ccce27f9b0815275370aec3b5"
Sep 09 06:03:52.571349 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:52.571300    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ea1b4b1ec4b80b9cda1540e08f73f177d94b78ccce27f9b0815275370aec3b5"} err="failed to get container status \"8ea1b4b1ec4b80b9cda1540e08f73f177d94b78ccce27f9b0815275370aec3b5\": rpc error: code = NotFound desc = could not find container \"8ea1b4b1ec4b80b9cda1540e08f73f177d94b78ccce27f9b0815275370aec3b5\": container with ID starting with 8ea1b4b1ec4b80b9cda1540e08f73f177d94b78ccce27f9b0815275370aec3b5 not found: ID does not exist"
Sep 09 06:03:52.578194 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:52.578172    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8b70f171-7b95cb769f-nxfcf"]
Sep 09 06:03:52.582046 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:52.582028    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8b70f171-7b95cb769f-nxfcf"]
Sep 09 06:03:52.591533 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:52.591513    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8b70f171-7858689f58-4f4t8"]
Sep 09 06:03:52.593813 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:52.593795    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8b70f171-7858689f58-4f4t8"]
Sep 09 06:03:53.148511 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:53.148482    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="261a4acc-243a-4a4e-ab00-f02529ff0d60" path="/var/lib/kubelet/pods/261a4acc-243a-4a4e-ab00-f02529ff0d60/volumes"
Sep 09 06:03:53.148875 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:53.148842    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="314c05dd-1040-4cc1-b50e-9ca03cfd2de6" path="/var/lib/kubelet/pods/314c05dd-1040-4cc1-b50e-9ca03cfd2de6/volumes"
Sep 09 06:03:53.149176 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:53.149162    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d61639ca-d4d1-44c0-90e8-c98e54d939cf" path="/var/lib/kubelet/pods/d61639ca-d4d1-44c0-90e8-c98e54d939cf/volumes"
Sep 09 06:03:55.657482 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:55.657451    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7a357a-546f6db798-gf8rb"]
Sep 09 06:03:55.657965 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:55.657655    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a357a-546f6db798-gf8rb" podUID="0c45ba77-e983-43dc-97b4-48a96037b1a1" containerName="maas-api" containerID="cri-o://99465667c6f89ec02c34f4a5fc7a3ed21e95f6f64f1f395aea50f27c12a7397f" gracePeriod=30
Sep 09 06:03:55.907457 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:55.907436    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a357a-546f6db798-gf8rb"
Sep 09 06:03:55.982307 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:55.982216    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c45ba77-e983-43dc-97b4-48a96037b1a1-openshift-service-ca\") pod \"0c45ba77-e983-43dc-97b4-48a96037b1a1\" (UID: \"0c45ba77-e983-43dc-97b4-48a96037b1a1\") "
Sep 09 06:03:55.982307 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:55.982287    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c45ba77-e983-43dc-97b4-48a96037b1a1-metrics-certs\") pod \"0c45ba77-e983-43dc-97b4-48a96037b1a1\" (UID: \"0c45ba77-e983-43dc-97b4-48a96037b1a1\") "
Sep 09 06:03:55.982566 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:55.982362    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csdw8\" (UniqueName: \"kubernetes.io/projected/0c45ba77-e983-43dc-97b4-48a96037b1a1-kube-api-access-csdw8\") pod \"0c45ba77-e983-43dc-97b4-48a96037b1a1\" (UID: \"0c45ba77-e983-43dc-97b4-48a96037b1a1\") "
Sep 09 06:03:55.982566 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:55.982548    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c45ba77-e983-43dc-97b4-48a96037b1a1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0c45ba77-e983-43dc-97b4-48a96037b1a1" (UID: "0c45ba77-e983-43dc-97b4-48a96037b1a1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:03:55.982747 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:55.982686    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c45ba77-e983-43dc-97b4-48a96037b1a1-maas-api-tls\") pod \"0c45ba77-e983-43dc-97b4-48a96037b1a1\" (UID: \"0c45ba77-e983-43dc-97b4-48a96037b1a1\") "
Sep 09 06:03:55.982951 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:55.982935    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c45ba77-e983-43dc-97b4-48a96037b1a1-openshift-service-ca\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:03:55.984931 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:55.984907    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c45ba77-e983-43dc-97b4-48a96037b1a1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0c45ba77-e983-43dc-97b4-48a96037b1a1" (UID: "0c45ba77-e983-43dc-97b4-48a96037b1a1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:03:55.985047 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:55.985028    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c45ba77-e983-43dc-97b4-48a96037b1a1-kube-api-access-csdw8" (OuterVolumeSpecName: "kube-api-access-csdw8") pod "0c45ba77-e983-43dc-97b4-48a96037b1a1" (UID: "0c45ba77-e983-43dc-97b4-48a96037b1a1"). InnerVolumeSpecName "kube-api-access-csdw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:03:55.985263 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:55.985244    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c45ba77-e983-43dc-97b4-48a96037b1a1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0c45ba77-e983-43dc-97b4-48a96037b1a1" (UID: "0c45ba77-e983-43dc-97b4-48a96037b1a1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:03:56.083914 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:56.083881    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c45ba77-e983-43dc-97b4-48a96037b1a1-metrics-certs\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:03:56.083914 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:56.083912    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csdw8\" (UniqueName: \"kubernetes.io/projected/0c45ba77-e983-43dc-97b4-48a96037b1a1-kube-api-access-csdw8\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:03:56.083914 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:56.083924    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c45ba77-e983-43dc-97b4-48a96037b1a1-maas-api-tls\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:03:56.575119 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:56.575083    2579 generic.go:356] "Generic (PLEG): container finished" podID="0c45ba77-e983-43dc-97b4-48a96037b1a1" containerID="99465667c6f89ec02c34f4a5fc7a3ed21e95f6f64f1f395aea50f27c12a7397f" exitCode=0
Sep 09 06:03:56.575270 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:56.575129    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a357a-546f6db798-gf8rb"
Sep 09 06:03:56.575270 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:56.575170    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a357a-546f6db798-gf8rb" event={"ID":"0c45ba77-e983-43dc-97b4-48a96037b1a1","Type":"ContainerDied","Data":"99465667c6f89ec02c34f4a5fc7a3ed21e95f6f64f1f395aea50f27c12a7397f"}
Sep 09 06:03:56.575270 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:56.575223    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a357a-546f6db798-gf8rb" event={"ID":"0c45ba77-e983-43dc-97b4-48a96037b1a1","Type":"ContainerDied","Data":"ed8fc72868aba6a594118da99c62f21d0f799485be0e174f5acbcf61b73a3970"}
Sep 09 06:03:56.575270 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:56.575266    2579 scope.go:117] "RemoveContainer" containerID="99465667c6f89ec02c34f4a5fc7a3ed21e95f6f64f1f395aea50f27c12a7397f"
Sep 09 06:03:56.584750 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:56.584731    2579 scope.go:117] "RemoveContainer" containerID="99465667c6f89ec02c34f4a5fc7a3ed21e95f6f64f1f395aea50f27c12a7397f"
Sep 09 06:03:56.585010 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:03:56.584992    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99465667c6f89ec02c34f4a5fc7a3ed21e95f6f64f1f395aea50f27c12a7397f\": container with ID starting with 99465667c6f89ec02c34f4a5fc7a3ed21e95f6f64f1f395aea50f27c12a7397f not found: ID does not exist" containerID="99465667c6f89ec02c34f4a5fc7a3ed21e95f6f64f1f395aea50f27c12a7397f"
Sep 09 06:03:56.585058 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:56.585018    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99465667c6f89ec02c34f4a5fc7a3ed21e95f6f64f1f395aea50f27c12a7397f"} err="failed to get container status \"99465667c6f89ec02c34f4a5fc7a3ed21e95f6f64f1f395aea50f27c12a7397f\": rpc error: code = NotFound desc = could not find container \"99465667c6f89ec02c34f4a5fc7a3ed21e95f6f64f1f395aea50f27c12a7397f\": container with ID starting with 99465667c6f89ec02c34f4a5fc7a3ed21e95f6f64f1f395aea50f27c12a7397f not found: ID does not exist"
Sep 09 06:03:56.621518 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:56.621496    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7a357a-546f6db798-gf8rb"]
Sep 09 06:03:56.634763 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:56.634737    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7a357a-546f6db798-gf8rb"]
Sep 09 06:03:57.148257 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:57.148227    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c45ba77-e983-43dc-97b4-48a96037b1a1" path="/var/lib/kubelet/pods/0c45ba77-e983-43dc-97b4-48a96037b1a1/volumes"
Sep 09 06:03:58.444996 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:58.444962    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d1d9fd16-8667c49dc8-ttvbt"]
Sep 09 06:03:58.445339 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:58.445295    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c45ba77-e983-43dc-97b4-48a96037b1a1" containerName="maas-api"
Sep 09 06:03:58.445339 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:58.445310    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c45ba77-e983-43dc-97b4-48a96037b1a1" containerName="maas-api"
Sep 09 06:03:58.445339 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:58.445321    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d61639ca-d4d1-44c0-90e8-c98e54d939cf" containerName="payload-pre-processing"
Sep 09 06:03:58.445339 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:58.445327    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d61639ca-d4d1-44c0-90e8-c98e54d939cf" containerName="payload-pre-processing"
Sep 09 06:03:58.445471 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:58.445344    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="314c05dd-1040-4cc1-b50e-9ca03cfd2de6" containerName="revoke-keys"
Sep 09 06:03:58.445471 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:58.445349    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="314c05dd-1040-4cc1-b50e-9ca03cfd2de6" containerName="revoke-keys"
Sep 09 06:03:58.445471 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:58.445356    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="261a4acc-243a-4a4e-ab00-f02529ff0d60" containerName="payload-processing"
Sep 09 06:03:58.445471 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:58.445361    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="261a4acc-243a-4a4e-ab00-f02529ff0d60" containerName="payload-processing"
Sep 09 06:03:58.445471 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:58.445408    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="314c05dd-1040-4cc1-b50e-9ca03cfd2de6" containerName="revoke-keys"
Sep 09 06:03:58.445471 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:58.445416    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="261a4acc-243a-4a4e-ab00-f02529ff0d60" containerName="payload-processing"
Sep 09 06:03:58.445471 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:58.445423    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c45ba77-e983-43dc-97b4-48a96037b1a1" containerName="maas-api"
Sep 09 06:03:58.445471 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:58.445429    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d61639ca-d4d1-44c0-90e8-c98e54d939cf" containerName="payload-pre-processing"
Sep 09 06:03:58.449600 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:58.449582    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-d1d9fd16-8667c49dc8-ttvbt"
Sep 09 06:03:58.451869 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:58.451853    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-d1d9fd16\""
Sep 09 06:03:58.452033 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:58.452006    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-d1d9fd16-dockercfg-w92fh\""
Sep 09 06:03:58.456327 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:58.456307    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d1d9fd16-8667c49dc8-ttvbt"]
Sep 09 06:03:58.603824 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:58.603795    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/064d14a1-f5dd-4f01-96b8-f16dbd8b7ae8-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-d1d9fd16-8667c49dc8-ttvbt\" (UID: \"064d14a1-f5dd-4f01-96b8-f16dbd8b7ae8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d1d9fd16-8667c49dc8-ttvbt"
Sep 09 06:03:58.603975 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:58.603870    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wssps\" (UniqueName: \"kubernetes.io/projected/064d14a1-f5dd-4f01-96b8-f16dbd8b7ae8-kube-api-access-wssps\") pod \"payload-pre-processing-e2e-derive-d1d9fd16-8667c49dc8-ttvbt\" (UID: \"064d14a1-f5dd-4f01-96b8-f16dbd8b7ae8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d1d9fd16-8667c49dc8-ttvbt"
Sep 09 06:03:58.704440 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:58.704367    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/064d14a1-f5dd-4f01-96b8-f16dbd8b7ae8-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-d1d9fd16-8667c49dc8-ttvbt\" (UID: \"064d14a1-f5dd-4f01-96b8-f16dbd8b7ae8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d1d9fd16-8667c49dc8-ttvbt"
Sep 09 06:03:58.704561 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:58.704460    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wssps\" (UniqueName: \"kubernetes.io/projected/064d14a1-f5dd-4f01-96b8-f16dbd8b7ae8-kube-api-access-wssps\") pod \"payload-pre-processing-e2e-derive-d1d9fd16-8667c49dc8-ttvbt\" (UID: \"064d14a1-f5dd-4f01-96b8-f16dbd8b7ae8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d1d9fd16-8667c49dc8-ttvbt"
Sep 09 06:03:58.705055 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:58.705036    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/064d14a1-f5dd-4f01-96b8-f16dbd8b7ae8-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-d1d9fd16-8667c49dc8-ttvbt\" (UID: \"064d14a1-f5dd-4f01-96b8-f16dbd8b7ae8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d1d9fd16-8667c49dc8-ttvbt"
Sep 09 06:03:58.711987 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:58.711970    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wssps\" (UniqueName: \"kubernetes.io/projected/064d14a1-f5dd-4f01-96b8-f16dbd8b7ae8-kube-api-access-wssps\") pod \"payload-pre-processing-e2e-derive-d1d9fd16-8667c49dc8-ttvbt\" (UID: \"064d14a1-f5dd-4f01-96b8-f16dbd8b7ae8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d1d9fd16-8667c49dc8-ttvbt"
Sep 09 06:03:58.760572 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:58.760545    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-d1d9fd16-8667c49dc8-ttvbt"
Sep 09 06:03:59.028567 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:59.028536    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d1d9fd16-8667c49dc8-ttvbt"]
Sep 09 06:03:59.031262 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:03:59.031240    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod064d14a1_f5dd_4f01_96b8_f16dbd8b7ae8.slice/crio-e67589ab72b9031f855fb7ffba342cddfbc1e67a2c66b2fe19bb53e4b6a33c23 WatchSource:0}: Error finding container e67589ab72b9031f855fb7ffba342cddfbc1e67a2c66b2fe19bb53e4b6a33c23: Status 404 returned error can't find the container with id e67589ab72b9031f855fb7ffba342cddfbc1e67a2c66b2fe19bb53e4b6a33c23
Sep 09 06:03:59.596299 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:59.596264    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-d1d9fd16-8667c49dc8-ttvbt" event={"ID":"064d14a1-f5dd-4f01-96b8-f16dbd8b7ae8","Type":"ContainerStarted","Data":"eb5889211be11b9e73366d0d5100e3358a5ac651b5bc9b4ffdcb46d47cb21396"}
Sep 09 06:03:59.596748 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:59.596324    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-d1d9fd16-8667c49dc8-ttvbt" event={"ID":"064d14a1-f5dd-4f01-96b8-f16dbd8b7ae8","Type":"ContainerStarted","Data":"e67589ab72b9031f855fb7ffba342cddfbc1e67a2c66b2fe19bb53e4b6a33c23"}
Sep 09 06:03:59.596748 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:59.596363    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-d1d9fd16-8667c49dc8-ttvbt"
Sep 09 06:03:59.611709 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:59.611666    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-d1d9fd16-8667c49dc8-ttvbt" podStartSLOduration=1.611651206 podStartE2EDuration="1.611651206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:03:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:03:59.610399029 +0000 UTC m=+864.950886939" watchObservedRunningTime="2026-09-09 06:03:59.611651206 +0000 UTC m=+864.952139117"
Sep 09 06:03:59.708397 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:59.708358    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-7a357a/autores-model-7a357a-kserve-5cd69c7b59-frt28" podUID="7b0e85c8-14fc-4cea-a780-b9db00e0d874" containerName="main" containerID="cri-o://7a3e917134a671c7e2a09846c06c9fc4e2fd4cc724a643edf56f057db4f0893d" gracePeriod=45
Sep 09 06:03:59.951397 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:03:59.951376    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-7a357a/autores-model-7a357a-kserve-5cd69c7b59-frt28"
Sep 09 06:04:00.015411 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:00.015382    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b0e85c8-14fc-4cea-a780-b9db00e0d874-home\") pod \"7b0e85c8-14fc-4cea-a780-b9db00e0d874\" (UID: \"7b0e85c8-14fc-4cea-a780-b9db00e0d874\") "
Sep 09 06:04:00.015577 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:00.015443    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b0e85c8-14fc-4cea-a780-b9db00e0d874-model-cache\") pod \"7b0e85c8-14fc-4cea-a780-b9db00e0d874\" (UID: \"7b0e85c8-14fc-4cea-a780-b9db00e0d874\") "
Sep 09 06:04:00.015577 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:00.015489    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b0e85c8-14fc-4cea-a780-b9db00e0d874-tls-certs\") pod \"7b0e85c8-14fc-4cea-a780-b9db00e0d874\" (UID: \"7b0e85c8-14fc-4cea-a780-b9db00e0d874\") "
Sep 09 06:04:00.015577 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:00.015543    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qzcrd\" (UniqueName: \"kubernetes.io/projected/7b0e85c8-14fc-4cea-a780-b9db00e0d874-kube-api-access-qzcrd\") pod \"7b0e85c8-14fc-4cea-a780-b9db00e0d874\" (UID: \"7b0e85c8-14fc-4cea-a780-b9db00e0d874\") "
Sep 09 06:04:00.015577 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:00.015565    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b0e85c8-14fc-4cea-a780-b9db00e0d874-dshm\") pod \"7b0e85c8-14fc-4cea-a780-b9db00e0d874\" (UID: \"7b0e85c8-14fc-4cea-a780-b9db00e0d874\") "
Sep 09 06:04:00.015815 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:00.015602    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b0e85c8-14fc-4cea-a780-b9db00e0d874-tmp-dir\") pod \"7b0e85c8-14fc-4cea-a780-b9db00e0d874\" (UID: \"7b0e85c8-14fc-4cea-a780-b9db00e0d874\") "
Sep 09 06:04:00.015815 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:00.015651    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b0e85c8-14fc-4cea-a780-b9db00e0d874-home" (OuterVolumeSpecName: "home") pod "7b0e85c8-14fc-4cea-a780-b9db00e0d874" (UID: "7b0e85c8-14fc-4cea-a780-b9db00e0d874"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:04:00.015815 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:00.015716    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b0e85c8-14fc-4cea-a780-b9db00e0d874-model-cache" (OuterVolumeSpecName: "model-cache") pod "7b0e85c8-14fc-4cea-a780-b9db00e0d874" (UID: "7b0e85c8-14fc-4cea-a780-b9db00e0d874"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:04:00.015965 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:00.015924    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b0e85c8-14fc-4cea-a780-b9db00e0d874-home\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:04:00.015965 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:00.015941    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b0e85c8-14fc-4cea-a780-b9db00e0d874-model-cache\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:04:00.015965 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:00.015940    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b0e85c8-14fc-4cea-a780-b9db00e0d874-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7b0e85c8-14fc-4cea-a780-b9db00e0d874" (UID: "7b0e85c8-14fc-4cea-a780-b9db00e0d874"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:04:00.017857 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:00.017833    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b0e85c8-14fc-4cea-a780-b9db00e0d874-kube-api-access-qzcrd" (OuterVolumeSpecName: "kube-api-access-qzcrd") pod "7b0e85c8-14fc-4cea-a780-b9db00e0d874" (UID: "7b0e85c8-14fc-4cea-a780-b9db00e0d874"). InnerVolumeSpecName "kube-api-access-qzcrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:04:00.017939 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:00.017884    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b0e85c8-14fc-4cea-a780-b9db00e0d874-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7b0e85c8-14fc-4cea-a780-b9db00e0d874" (UID: "7b0e85c8-14fc-4cea-a780-b9db00e0d874"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:04:00.017975 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:00.017928    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b0e85c8-14fc-4cea-a780-b9db00e0d874-dshm" (OuterVolumeSpecName: "dshm") pod "7b0e85c8-14fc-4cea-a780-b9db00e0d874" (UID: "7b0e85c8-14fc-4cea-a780-b9db00e0d874"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:04:00.116972 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:00.116947    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzcrd\" (UniqueName: \"kubernetes.io/projected/7b0e85c8-14fc-4cea-a780-b9db00e0d874-kube-api-access-qzcrd\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:04:00.116972 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:00.116970    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b0e85c8-14fc-4cea-a780-b9db00e0d874-dshm\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:04:00.117093 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:00.116980    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b0e85c8-14fc-4cea-a780-b9db00e0d874-tmp-dir\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:04:00.117093 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:00.116988    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b0e85c8-14fc-4cea-a780-b9db00e0d874-tls-certs\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:04:00.602317 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:00.602284    2579 generic.go:356] "Generic (PLEG): container finished" podID="7b0e85c8-14fc-4cea-a780-b9db00e0d874" containerID="7a3e917134a671c7e2a09846c06c9fc4e2fd4cc724a643edf56f057db4f0893d" exitCode=0
Sep 09 06:04:00.602780 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:00.602325    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-7a357a/autores-model-7a357a-kserve-5cd69c7b59-frt28"
Sep 09 06:04:00.602780 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:00.602366    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-7a357a/autores-model-7a357a-kserve-5cd69c7b59-frt28" event={"ID":"7b0e85c8-14fc-4cea-a780-b9db00e0d874","Type":"ContainerDied","Data":"7a3e917134a671c7e2a09846c06c9fc4e2fd4cc724a643edf56f057db4f0893d"}
Sep 09 06:04:00.602780 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:00.602418    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-7a357a/autores-model-7a357a-kserve-5cd69c7b59-frt28" event={"ID":"7b0e85c8-14fc-4cea-a780-b9db00e0d874","Type":"ContainerDied","Data":"883fcb772bd3b18bdeb0dc56febe9ac1d0e2f29b49c96e5fcc44b0b742ae4c43"}
Sep 09 06:04:00.602780 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:00.602452    2579 scope.go:117] "RemoveContainer" containerID="7a3e917134a671c7e2a09846c06c9fc4e2fd4cc724a643edf56f057db4f0893d"
Sep 09 06:04:00.611452 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:00.611437    2579 scope.go:117] "RemoveContainer" containerID="7a3e917134a671c7e2a09846c06c9fc4e2fd4cc724a643edf56f057db4f0893d"
Sep 09 06:04:00.611723 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:04:00.611703    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a3e917134a671c7e2a09846c06c9fc4e2fd4cc724a643edf56f057db4f0893d\": container with ID starting with 7a3e917134a671c7e2a09846c06c9fc4e2fd4cc724a643edf56f057db4f0893d not found: ID does not exist" containerID="7a3e917134a671c7e2a09846c06c9fc4e2fd4cc724a643edf56f057db4f0893d"
Sep 09 06:04:00.611795 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:00.611730    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a3e917134a671c7e2a09846c06c9fc4e2fd4cc724a643edf56f057db4f0893d"} err="failed to get container status \"7a3e917134a671c7e2a09846c06c9fc4e2fd4cc724a643edf56f057db4f0893d\": rpc error: code = NotFound desc = could not find container \"7a3e917134a671c7e2a09846c06c9fc4e2fd4cc724a643edf56f057db4f0893d\": container with ID starting with 7a3e917134a671c7e2a09846c06c9fc4e2fd4cc724a643edf56f057db4f0893d not found: ID does not exist"
Sep 09 06:04:00.622579 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:00.622559    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-7a357a/autores-model-7a357a-kserve-5cd69c7b59-frt28"]
Sep 09 06:04:00.628934 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:00.628914    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-7a357a/autores-model-7a357a-kserve-5cd69c7b59-frt28"]
Sep 09 06:04:01.148582 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:01.148541    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b0e85c8-14fc-4cea-a780-b9db00e0d874" path="/var/lib/kubelet/pods/7b0e85c8-14fc-4cea-a780-b9db00e0d874/volumes"
Sep 09 06:04:07.676863 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:07.676824    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f631ee-74796bf5d6-rsgxg"]
Sep 09 06:04:07.677417 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:07.677318    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b0e85c8-14fc-4cea-a780-b9db00e0d874" containerName="main"
Sep 09 06:04:07.677417 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:07.677338    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b0e85c8-14fc-4cea-a780-b9db00e0d874" containerName="main"
Sep 09 06:04:07.677534 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:07.677431    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b0e85c8-14fc-4cea-a780-b9db00e0d874" containerName="main"
Sep 09 06:04:07.681740 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:07.681720    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f631ee-74796bf5d6-rsgxg"
Sep 09 06:04:07.684011 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:07.683961    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-f631ee-dockercfg-vrfvq\""
Sep 09 06:04:07.684011 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:07.683964    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-f631ee\""
Sep 09 06:04:07.690165 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:07.690142    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f631ee-74796bf5d6-rsgxg"]
Sep 09 06:04:07.720166 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:07.720134    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f631ee-6559b7c7-zmggw"]
Sep 09 06:04:07.723374 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:07.723354    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-f631ee-6559b7c7-zmggw"
Sep 09 06:04:07.738039 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:07.738010    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f631ee-6559b7c7-zmggw"]
Sep 09 06:04:07.780486 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:07.780457    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qggrw\" (UniqueName: \"kubernetes.io/projected/7f8bdb0d-c657-4ff9-a5f0-756e0642aa28-kube-api-access-qggrw\") pod \"payload-pre-processing-e2e-explicit-f631ee-74796bf5d6-rsgxg\" (UID: \"7f8bdb0d-c657-4ff9-a5f0-756e0642aa28\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f631ee-74796bf5d6-rsgxg"
Sep 09 06:04:07.780663 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:07.780506    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f8bdb0d-c657-4ff9-a5f0-756e0642aa28-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-f631ee-74796bf5d6-rsgxg\" (UID: \"7f8bdb0d-c657-4ff9-a5f0-756e0642aa28\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f631ee-74796bf5d6-rsgxg"
Sep 09 06:04:07.881801 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:07.881765    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/754a6601-d71f-486e-9b99-5fb50d498feb-plugins-config-volume\") pod \"payload-processing-e2e-explicit-f631ee-6559b7c7-zmggw\" (UID: \"754a6601-d71f-486e-9b99-5fb50d498feb\") " pod="openshift-ingress/payload-processing-e2e-explicit-f631ee-6559b7c7-zmggw"
Sep 09 06:04:07.882016 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:07.881814    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qggrw\" (UniqueName: \"kubernetes.io/projected/7f8bdb0d-c657-4ff9-a5f0-756e0642aa28-kube-api-access-qggrw\") pod \"payload-pre-processing-e2e-explicit-f631ee-74796bf5d6-rsgxg\" (UID: \"7f8bdb0d-c657-4ff9-a5f0-756e0642aa28\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f631ee-74796bf5d6-rsgxg"
Sep 09 06:04:07.882016 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:07.881854    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f8bdb0d-c657-4ff9-a5f0-756e0642aa28-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-f631ee-74796bf5d6-rsgxg\" (UID: \"7f8bdb0d-c657-4ff9-a5f0-756e0642aa28\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f631ee-74796bf5d6-rsgxg"
Sep 09 06:04:07.882016 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:07.881897    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlvts\" (UniqueName: \"kubernetes.io/projected/754a6601-d71f-486e-9b99-5fb50d498feb-kube-api-access-tlvts\") pod \"payload-processing-e2e-explicit-f631ee-6559b7c7-zmggw\" (UID: \"754a6601-d71f-486e-9b99-5fb50d498feb\") " pod="openshift-ingress/payload-processing-e2e-explicit-f631ee-6559b7c7-zmggw"
Sep 09 06:04:07.883084 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:07.883057    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f8bdb0d-c657-4ff9-a5f0-756e0642aa28-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-f631ee-74796bf5d6-rsgxg\" (UID: \"7f8bdb0d-c657-4ff9-a5f0-756e0642aa28\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f631ee-74796bf5d6-rsgxg"
Sep 09 06:04:07.889516 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:07.889493    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qggrw\" (UniqueName: \"kubernetes.io/projected/7f8bdb0d-c657-4ff9-a5f0-756e0642aa28-kube-api-access-qggrw\") pod \"payload-pre-processing-e2e-explicit-f631ee-74796bf5d6-rsgxg\" (UID: \"7f8bdb0d-c657-4ff9-a5f0-756e0642aa28\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f631ee-74796bf5d6-rsgxg"
Sep 09 06:04:07.982820 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:07.982751    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlvts\" (UniqueName: \"kubernetes.io/projected/754a6601-d71f-486e-9b99-5fb50d498feb-kube-api-access-tlvts\") pod \"payload-processing-e2e-explicit-f631ee-6559b7c7-zmggw\" (UID: \"754a6601-d71f-486e-9b99-5fb50d498feb\") " pod="openshift-ingress/payload-processing-e2e-explicit-f631ee-6559b7c7-zmggw"
Sep 09 06:04:07.982944 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:07.982830    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/754a6601-d71f-486e-9b99-5fb50d498feb-plugins-config-volume\") pod \"payload-processing-e2e-explicit-f631ee-6559b7c7-zmggw\" (UID: \"754a6601-d71f-486e-9b99-5fb50d498feb\") " pod="openshift-ingress/payload-processing-e2e-explicit-f631ee-6559b7c7-zmggw"
Sep 09 06:04:07.983421 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:07.983404    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/754a6601-d71f-486e-9b99-5fb50d498feb-plugins-config-volume\") pod \"payload-processing-e2e-explicit-f631ee-6559b7c7-zmggw\" (UID: \"754a6601-d71f-486e-9b99-5fb50d498feb\") " pod="openshift-ingress/payload-processing-e2e-explicit-f631ee-6559b7c7-zmggw"
Sep 09 06:04:07.990386 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:07.990367    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlvts\" (UniqueName: \"kubernetes.io/projected/754a6601-d71f-486e-9b99-5fb50d498feb-kube-api-access-tlvts\") pod \"payload-processing-e2e-explicit-f631ee-6559b7c7-zmggw\" (UID: \"754a6601-d71f-486e-9b99-5fb50d498feb\") " pod="openshift-ingress/payload-processing-e2e-explicit-f631ee-6559b7c7-zmggw"
Sep 09 06:04:07.993242 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:07.993224    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f631ee-74796bf5d6-rsgxg"
Sep 09 06:04:08.032891 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:08.032861    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-f631ee-6559b7c7-zmggw"
Sep 09 06:04:08.269678 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:08.268614    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f631ee-74796bf5d6-rsgxg"]
Sep 09 06:04:08.317300 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:08.317278    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f631ee-6559b7c7-zmggw"]
Sep 09 06:04:08.319461 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:04:08.319421    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod754a6601_d71f_486e_9b99_5fb50d498feb.slice/crio-53be8174bcfc0ea66d0a37f5bbe9695050215e607170188a0ef4274659c54165 WatchSource:0}: Error finding container 53be8174bcfc0ea66d0a37f5bbe9695050215e607170188a0ef4274659c54165: Status 404 returned error can't find the container with id 53be8174bcfc0ea66d0a37f5bbe9695050215e607170188a0ef4274659c54165
Sep 09 06:04:08.643524 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:08.643491    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f631ee-74796bf5d6-rsgxg" event={"ID":"7f8bdb0d-c657-4ff9-a5f0-756e0642aa28","Type":"ContainerStarted","Data":"581ab85cf535774e1429c4e01f8d49060ae1c95aa609299fef3bf292a80c6f3d"}
Sep 09 06:04:08.643734 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:08.643544    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f631ee-74796bf5d6-rsgxg" event={"ID":"7f8bdb0d-c657-4ff9-a5f0-756e0642aa28","Type":"ContainerStarted","Data":"f66e1ec1c20cbddec5dbc38611e754023efdbb43a1e47426f748eb5d080bf376"}
Sep 09 06:04:08.643734 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:08.643611    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f631ee-74796bf5d6-rsgxg"
Sep 09 06:04:08.645337 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:08.645312    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-f631ee-6559b7c7-zmggw" event={"ID":"754a6601-d71f-486e-9b99-5fb50d498feb","Type":"ContainerStarted","Data":"83c8ae5b13e3103878edfcaebfd83d02e2b3745d2e7f1cdfcd8e61d37712b2f9"}
Sep 09 06:04:08.645430 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:08.645358    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-f631ee-6559b7c7-zmggw" event={"ID":"754a6601-d71f-486e-9b99-5fb50d498feb","Type":"ContainerStarted","Data":"53be8174bcfc0ea66d0a37f5bbe9695050215e607170188a0ef4274659c54165"}
Sep 09 06:04:08.645430 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:08.645386    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-f631ee-6559b7c7-zmggw"
Sep 09 06:04:08.660183 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:08.660121    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f631ee-74796bf5d6-rsgxg" podStartSLOduration=1.660104901 podStartE2EDuration="1.660104901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:04:08.659182052 +0000 UTC m=+873.999669964" watchObservedRunningTime="2026-09-09 06:04:08.660104901 +0000 UTC m=+874.000592812"
Sep 09 06:04:08.676711 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:08.676673    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-f631ee-6559b7c7-zmggw" podStartSLOduration=1.676655728 podStartE2EDuration="1.676655728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:04:08.67455534 +0000 UTC m=+874.015043249" watchObservedRunningTime="2026-09-09 06:04:08.676655728 +0000 UTC m=+874.017143832"
Sep 09 06:04:10.605203 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:10.605171    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-d1d9fd16-8667c49dc8-ttvbt"
Sep 09 06:04:11.197500 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:11.197464    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:04:19.652076 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:19.652050    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f631ee-74796bf5d6-rsgxg"
Sep 09 06:04:19.652442 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:19.652118    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-f631ee-6559b7c7-zmggw"
Sep 09 06:04:21.394003 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:21.393973    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1d9fd16-wqhbw"]
Sep 09 06:04:21.397737 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:21.397721    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1d9fd16-wqhbw"
Sep 09 06:04:21.400189 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:21.400169    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ksjk6\""
Sep 09 06:04:21.406065 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:21.406041    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1d9fd16-wqhbw"]
Sep 09 06:04:21.505228 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:21.505197    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99308cf7-033b-4782-ad70-cab473cfb0d7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1d9fd16-wqhbw\" (UID: \"99308cf7-033b-4782-ad70-cab473cfb0d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1d9fd16-wqhbw"
Sep 09 06:04:21.606226 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:21.606198    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99308cf7-033b-4782-ad70-cab473cfb0d7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1d9fd16-wqhbw\" (UID: \"99308cf7-033b-4782-ad70-cab473cfb0d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1d9fd16-wqhbw"
Sep 09 06:04:21.606825 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:21.606805    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99308cf7-033b-4782-ad70-cab473cfb0d7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1d9fd16-wqhbw\" (UID: \"99308cf7-033b-4782-ad70-cab473cfb0d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1d9fd16-wqhbw"
Sep 09 06:04:21.726754 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:21.726671    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1d9fd16-wqhbw"
Sep 09 06:04:21.994478 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:21.994404    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1d9fd16-wqhbw"]
Sep 09 06:04:21.997541 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:04:21.997513    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99308cf7_033b_4782_ad70_cab473cfb0d7.slice/crio-465e55e44c6d89d00572c5609b7968489445603c3cb4ed7591257e7afc211f79 WatchSource:0}: Error finding container 465e55e44c6d89d00572c5609b7968489445603c3cb4ed7591257e7afc211f79: Status 404 returned error can't find the container with id 465e55e44c6d89d00572c5609b7968489445603c3cb4ed7591257e7afc211f79
Sep 09 06:04:22.713502 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:22.713404    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1d9fd16-wqhbw" event={"ID":"99308cf7-033b-4782-ad70-cab473cfb0d7","Type":"ContainerStarted","Data":"fc0c80c60bbd1ca9e7c18d27e8e43acc3a21a0cd4346d320ad5cd75d85ad7459"}
Sep 09 06:04:22.713502 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:22.713477    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1d9fd16-wqhbw" event={"ID":"99308cf7-033b-4782-ad70-cab473cfb0d7","Type":"ContainerStarted","Data":"465e55e44c6d89d00572c5609b7968489445603c3cb4ed7591257e7afc211f79"}
Sep 09 06:04:22.727769 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:22.727719    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1d9fd16-wqhbw" podStartSLOduration=1.727704033 podStartE2EDuration="1.727704033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:04:22.726064734 +0000 UTC m=+888.066552642" watchObservedRunningTime="2026-09-09 06:04:22.727704033 +0000 UTC m=+888.068191942"
Sep 09 06:04:23.719063 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:23.719023    2579 generic.go:356] "Generic (PLEG): container finished" podID="99308cf7-033b-4782-ad70-cab473cfb0d7" containerID="fc0c80c60bbd1ca9e7c18d27e8e43acc3a21a0cd4346d320ad5cd75d85ad7459" exitCode=0
Sep 09 06:04:23.719429 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:23.719109    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1d9fd16-wqhbw" event={"ID":"99308cf7-033b-4782-ad70-cab473cfb0d7","Type":"ContainerDied","Data":"fc0c80c60bbd1ca9e7c18d27e8e43acc3a21a0cd4346d320ad5cd75d85ad7459"}
Sep 09 06:04:24.853821 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:24.853797    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1d9fd16-wqhbw"
Sep 09 06:04:24.933995 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:24.933968    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99308cf7-033b-4782-ad70-cab473cfb0d7-maas-api-service-ca\") pod \"99308cf7-033b-4782-ad70-cab473cfb0d7\" (UID: \"99308cf7-033b-4782-ad70-cab473cfb0d7\") "
Sep 09 06:04:24.934317 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:24.934293    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99308cf7-033b-4782-ad70-cab473cfb0d7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "99308cf7-033b-4782-ad70-cab473cfb0d7" (UID: "99308cf7-033b-4782-ad70-cab473cfb0d7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:04:25.035379 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:25.035280    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99308cf7-033b-4782-ad70-cab473cfb0d7-maas-api-service-ca\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:04:25.728843 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:25.728816    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1d9fd16-wqhbw"
Sep 09 06:04:25.729041 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:25.728819    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1d9fd16-wqhbw" event={"ID":"99308cf7-033b-4782-ad70-cab473cfb0d7","Type":"ContainerDied","Data":"465e55e44c6d89d00572c5609b7968489445603c3cb4ed7591257e7afc211f79"}
Sep 09 06:04:25.729041 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:25.728926    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="465e55e44c6d89d00572c5609b7968489445603c3cb4ed7591257e7afc211f79"
Sep 09 06:04:28.488979 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:28.488904    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d1d9fd16-8667c49dc8-ttvbt"]
Sep 09 06:04:28.489355 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:28.489207    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-d1d9fd16-8667c49dc8-ttvbt" podUID="064d14a1-f5dd-4f01-96b8-f16dbd8b7ae8" containerName="payload-pre-processing" containerID="cri-o://eb5889211be11b9e73366d0d5100e3358a5ac651b5bc9b4ffdcb46d47cb21396" gracePeriod=30
Sep 09 06:04:28.632308 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:28.632280    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1d9fd16-wqhbw"]
Sep 09 06:04:28.635390 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:28.635368    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1d9fd16-wqhbw"]
Sep 09 06:04:28.738544 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:28.738521    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-d1d9fd16-8667c49dc8-ttvbt"
Sep 09 06:04:28.743444 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:28.743389    2579 generic.go:356] "Generic (PLEG): container finished" podID="064d14a1-f5dd-4f01-96b8-f16dbd8b7ae8" containerID="eb5889211be11b9e73366d0d5100e3358a5ac651b5bc9b4ffdcb46d47cb21396" exitCode=0
Sep 09 06:04:28.743444 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:28.743422    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-d1d9fd16-8667c49dc8-ttvbt"
Sep 09 06:04:28.743559 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:28.743436    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-d1d9fd16-8667c49dc8-ttvbt" event={"ID":"064d14a1-f5dd-4f01-96b8-f16dbd8b7ae8","Type":"ContainerDied","Data":"eb5889211be11b9e73366d0d5100e3358a5ac651b5bc9b4ffdcb46d47cb21396"}
Sep 09 06:04:28.743559 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:28.743490    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-d1d9fd16-8667c49dc8-ttvbt" event={"ID":"064d14a1-f5dd-4f01-96b8-f16dbd8b7ae8","Type":"ContainerDied","Data":"e67589ab72b9031f855fb7ffba342cddfbc1e67a2c66b2fe19bb53e4b6a33c23"}
Sep 09 06:04:28.743559 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:28.743513    2579 scope.go:117] "RemoveContainer" containerID="eb5889211be11b9e73366d0d5100e3358a5ac651b5bc9b4ffdcb46d47cb21396"
Sep 09 06:04:28.752703 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:28.752683    2579 scope.go:117] "RemoveContainer" containerID="eb5889211be11b9e73366d0d5100e3358a5ac651b5bc9b4ffdcb46d47cb21396"
Sep 09 06:04:28.753031 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:04:28.753008    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb5889211be11b9e73366d0d5100e3358a5ac651b5bc9b4ffdcb46d47cb21396\": container with ID starting with eb5889211be11b9e73366d0d5100e3358a5ac651b5bc9b4ffdcb46d47cb21396 not found: ID does not exist" containerID="eb5889211be11b9e73366d0d5100e3358a5ac651b5bc9b4ffdcb46d47cb21396"
Sep 09 06:04:28.753118 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:28.753050    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb5889211be11b9e73366d0d5100e3358a5ac651b5bc9b4ffdcb46d47cb21396"} err="failed to get container status \"eb5889211be11b9e73366d0d5100e3358a5ac651b5bc9b4ffdcb46d47cb21396\": rpc error: code = NotFound desc = could not find container \"eb5889211be11b9e73366d0d5100e3358a5ac651b5bc9b4ffdcb46d47cb21396\": container with ID starting with eb5889211be11b9e73366d0d5100e3358a5ac651b5bc9b4ffdcb46d47cb21396 not found: ID does not exist"
Sep 09 06:04:28.770444 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:28.770414    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wssps\" (UniqueName: \"kubernetes.io/projected/064d14a1-f5dd-4f01-96b8-f16dbd8b7ae8-kube-api-access-wssps\") pod \"064d14a1-f5dd-4f01-96b8-f16dbd8b7ae8\" (UID: \"064d14a1-f5dd-4f01-96b8-f16dbd8b7ae8\") "
Sep 09 06:04:28.770660 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:28.770524    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/064d14a1-f5dd-4f01-96b8-f16dbd8b7ae8-plugins-config-volume\") pod \"064d14a1-f5dd-4f01-96b8-f16dbd8b7ae8\" (UID: \"064d14a1-f5dd-4f01-96b8-f16dbd8b7ae8\") "
Sep 09 06:04:28.770916 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:28.770894    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/064d14a1-f5dd-4f01-96b8-f16dbd8b7ae8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "064d14a1-f5dd-4f01-96b8-f16dbd8b7ae8" (UID: "064d14a1-f5dd-4f01-96b8-f16dbd8b7ae8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:04:28.774253 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:28.774226    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/064d14a1-f5dd-4f01-96b8-f16dbd8b7ae8-kube-api-access-wssps" (OuterVolumeSpecName: "kube-api-access-wssps") pod "064d14a1-f5dd-4f01-96b8-f16dbd8b7ae8" (UID: "064d14a1-f5dd-4f01-96b8-f16dbd8b7ae8"). InnerVolumeSpecName "kube-api-access-wssps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:04:28.871946 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:28.871920    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wssps\" (UniqueName: \"kubernetes.io/projected/064d14a1-f5dd-4f01-96b8-f16dbd8b7ae8-kube-api-access-wssps\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:04:28.871946 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:28.871945    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/064d14a1-f5dd-4f01-96b8-f16dbd8b7ae8-plugins-config-volume\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:04:29.063716 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:29.063685    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d1d9fd16-8667c49dc8-ttvbt"]
Sep 09 06:04:29.067148 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:29.067127    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d1d9fd16-8667c49dc8-ttvbt"]
Sep 09 06:04:29.147866 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:29.147841    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="064d14a1-f5dd-4f01-96b8-f16dbd8b7ae8" path="/var/lib/kubelet/pods/064d14a1-f5dd-4f01-96b8-f16dbd8b7ae8/volumes"
Sep 09 06:04:29.148151 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:29.148139    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99308cf7-033b-4782-ad70-cab473cfb0d7" path="/var/lib/kubelet/pods/99308cf7-033b-4782-ad70-cab473cfb0d7/volumes"
Sep 09 06:04:32.795757 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:32.792435    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:04:39.334617 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:39.334584    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-43eb2d/premium-w4-43eb2d-kserve-84b57f44-tlrbl"]
Sep 09 06:04:39.335106 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:39.335089    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="064d14a1-f5dd-4f01-96b8-f16dbd8b7ae8" containerName="payload-pre-processing"
Sep 09 06:04:39.335150 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:39.335110    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="064d14a1-f5dd-4f01-96b8-f16dbd8b7ae8" containerName="payload-pre-processing"
Sep 09 06:04:39.335150 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:39.335130    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99308cf7-033b-4782-ad70-cab473cfb0d7" containerName="revoke-keys"
Sep 09 06:04:39.335150 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:39.335139    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="99308cf7-033b-4782-ad70-cab473cfb0d7" containerName="revoke-keys"
Sep 09 06:04:39.335247 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:39.335231    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="064d14a1-f5dd-4f01-96b8-f16dbd8b7ae8" containerName="payload-pre-processing"
Sep 09 06:04:39.335280 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:39.335245    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="99308cf7-033b-4782-ad70-cab473cfb0d7" containerName="revoke-keys"
Sep 09 06:04:39.339961 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:39.339944    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-43eb2d/premium-w4-43eb2d-kserve-84b57f44-tlrbl"
Sep 09 06:04:39.342219 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:39.342195    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-43eb2d\"/\"premium-w4-43eb2d-kserve-self-signed-certs\""
Sep 09 06:04:39.347341 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:39.347099    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-43eb2d/premium-w4-43eb2d-kserve-84b57f44-tlrbl"]
Sep 09 06:04:39.456160 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:39.456134    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de67233d-58eb-43dc-8473-f5dd1ef69b05-tmp-dir\") pod \"premium-w4-43eb2d-kserve-84b57f44-tlrbl\" (UID: \"de67233d-58eb-43dc-8473-f5dd1ef69b05\") " pod="e2e-models-e2e-worker-w4-43eb2d/premium-w4-43eb2d-kserve-84b57f44-tlrbl"
Sep 09 06:04:39.456272 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:39.456172    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de67233d-58eb-43dc-8473-f5dd1ef69b05-tls-certs\") pod \"premium-w4-43eb2d-kserve-84b57f44-tlrbl\" (UID: \"de67233d-58eb-43dc-8473-f5dd1ef69b05\") " pod="e2e-models-e2e-worker-w4-43eb2d/premium-w4-43eb2d-kserve-84b57f44-tlrbl"
Sep 09 06:04:39.456272 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:39.456189    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5kkk\" (UniqueName: \"kubernetes.io/projected/de67233d-58eb-43dc-8473-f5dd1ef69b05-kube-api-access-v5kkk\") pod \"premium-w4-43eb2d-kserve-84b57f44-tlrbl\" (UID: \"de67233d-58eb-43dc-8473-f5dd1ef69b05\") " pod="e2e-models-e2e-worker-w4-43eb2d/premium-w4-43eb2d-kserve-84b57f44-tlrbl"
Sep 09 06:04:39.456344 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:39.456268    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de67233d-58eb-43dc-8473-f5dd1ef69b05-dshm\") pod \"premium-w4-43eb2d-kserve-84b57f44-tlrbl\" (UID: \"de67233d-58eb-43dc-8473-f5dd1ef69b05\") " pod="e2e-models-e2e-worker-w4-43eb2d/premium-w4-43eb2d-kserve-84b57f44-tlrbl"
Sep 09 06:04:39.456344 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:39.456291    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de67233d-58eb-43dc-8473-f5dd1ef69b05-model-cache\") pod \"premium-w4-43eb2d-kserve-84b57f44-tlrbl\" (UID: \"de67233d-58eb-43dc-8473-f5dd1ef69b05\") " pod="e2e-models-e2e-worker-w4-43eb2d/premium-w4-43eb2d-kserve-84b57f44-tlrbl"
Sep 09 06:04:39.456344 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:39.456335    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de67233d-58eb-43dc-8473-f5dd1ef69b05-home\") pod \"premium-w4-43eb2d-kserve-84b57f44-tlrbl\" (UID: \"de67233d-58eb-43dc-8473-f5dd1ef69b05\") " pod="e2e-models-e2e-worker-w4-43eb2d/premium-w4-43eb2d-kserve-84b57f44-tlrbl"
Sep 09 06:04:39.557348 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:39.557315    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de67233d-58eb-43dc-8473-f5dd1ef69b05-home\") pod \"premium-w4-43eb2d-kserve-84b57f44-tlrbl\" (UID: \"de67233d-58eb-43dc-8473-f5dd1ef69b05\") " pod="e2e-models-e2e-worker-w4-43eb2d/premium-w4-43eb2d-kserve-84b57f44-tlrbl"
Sep 09 06:04:39.557463 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:39.557358    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de67233d-58eb-43dc-8473-f5dd1ef69b05-tmp-dir\") pod \"premium-w4-43eb2d-kserve-84b57f44-tlrbl\" (UID: \"de67233d-58eb-43dc-8473-f5dd1ef69b05\") " pod="e2e-models-e2e-worker-w4-43eb2d/premium-w4-43eb2d-kserve-84b57f44-tlrbl"
Sep 09 06:04:39.557463 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:39.557386    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de67233d-58eb-43dc-8473-f5dd1ef69b05-tls-certs\") pod \"premium-w4-43eb2d-kserve-84b57f44-tlrbl\" (UID: \"de67233d-58eb-43dc-8473-f5dd1ef69b05\") " pod="e2e-models-e2e-worker-w4-43eb2d/premium-w4-43eb2d-kserve-84b57f44-tlrbl"
Sep 09 06:04:39.557463 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:39.557402    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5kkk\" (UniqueName: \"kubernetes.io/projected/de67233d-58eb-43dc-8473-f5dd1ef69b05-kube-api-access-v5kkk\") pod \"premium-w4-43eb2d-kserve-84b57f44-tlrbl\" (UID: \"de67233d-58eb-43dc-8473-f5dd1ef69b05\") " pod="e2e-models-e2e-worker-w4-43eb2d/premium-w4-43eb2d-kserve-84b57f44-tlrbl"
Sep 09 06:04:39.557463 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:39.557430    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de67233d-58eb-43dc-8473-f5dd1ef69b05-dshm\") pod \"premium-w4-43eb2d-kserve-84b57f44-tlrbl\" (UID: \"de67233d-58eb-43dc-8473-f5dd1ef69b05\") " pod="e2e-models-e2e-worker-w4-43eb2d/premium-w4-43eb2d-kserve-84b57f44-tlrbl"
Sep 09 06:04:39.557683 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:39.557515    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de67233d-58eb-43dc-8473-f5dd1ef69b05-model-cache\") pod \"premium-w4-43eb2d-kserve-84b57f44-tlrbl\" (UID: \"de67233d-58eb-43dc-8473-f5dd1ef69b05\") " pod="e2e-models-e2e-worker-w4-43eb2d/premium-w4-43eb2d-kserve-84b57f44-tlrbl"
Sep 09 06:04:39.557738 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:39.557707    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de67233d-58eb-43dc-8473-f5dd1ef69b05-home\") pod \"premium-w4-43eb2d-kserve-84b57f44-tlrbl\" (UID: \"de67233d-58eb-43dc-8473-f5dd1ef69b05\") " pod="e2e-models-e2e-worker-w4-43eb2d/premium-w4-43eb2d-kserve-84b57f44-tlrbl"
Sep 09 06:04:39.557796 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:39.557734    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de67233d-58eb-43dc-8473-f5dd1ef69b05-tmp-dir\") pod \"premium-w4-43eb2d-kserve-84b57f44-tlrbl\" (UID: \"de67233d-58eb-43dc-8473-f5dd1ef69b05\") " pod="e2e-models-e2e-worker-w4-43eb2d/premium-w4-43eb2d-kserve-84b57f44-tlrbl"
Sep 09 06:04:39.557917 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:39.557897    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de67233d-58eb-43dc-8473-f5dd1ef69b05-model-cache\") pod \"premium-w4-43eb2d-kserve-84b57f44-tlrbl\" (UID: \"de67233d-58eb-43dc-8473-f5dd1ef69b05\") " pod="e2e-models-e2e-worker-w4-43eb2d/premium-w4-43eb2d-kserve-84b57f44-tlrbl"
Sep 09 06:04:39.559792 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:39.559773    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de67233d-58eb-43dc-8473-f5dd1ef69b05-dshm\") pod \"premium-w4-43eb2d-kserve-84b57f44-tlrbl\" (UID: \"de67233d-58eb-43dc-8473-f5dd1ef69b05\") " pod="e2e-models-e2e-worker-w4-43eb2d/premium-w4-43eb2d-kserve-84b57f44-tlrbl"
Sep 09 06:04:39.559889 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:39.559871    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de67233d-58eb-43dc-8473-f5dd1ef69b05-tls-certs\") pod \"premium-w4-43eb2d-kserve-84b57f44-tlrbl\" (UID: \"de67233d-58eb-43dc-8473-f5dd1ef69b05\") " pod="e2e-models-e2e-worker-w4-43eb2d/premium-w4-43eb2d-kserve-84b57f44-tlrbl"
Sep 09 06:04:39.565360 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:39.565340    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5kkk\" (UniqueName: \"kubernetes.io/projected/de67233d-58eb-43dc-8473-f5dd1ef69b05-kube-api-access-v5kkk\") pod \"premium-w4-43eb2d-kserve-84b57f44-tlrbl\" (UID: \"de67233d-58eb-43dc-8473-f5dd1ef69b05\") " pod="e2e-models-e2e-worker-w4-43eb2d/premium-w4-43eb2d-kserve-84b57f44-tlrbl"
Sep 09 06:04:39.650308 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:39.650236    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-43eb2d/premium-w4-43eb2d-kserve-84b57f44-tlrbl"
Sep 09 06:04:39.920225 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:39.920199    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-43eb2d/premium-w4-43eb2d-kserve-84b57f44-tlrbl"]
Sep 09 06:04:39.922213 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:04:39.922180    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde67233d_58eb_43dc_8473_f5dd1ef69b05.slice/crio-a214f41992a48c06cee61b21851fb31448c0e515d080710773a6b12a3ff946bd WatchSource:0}: Error finding container a214f41992a48c06cee61b21851fb31448c0e515d080710773a6b12a3ff946bd: Status 404 returned error can't find the container with id a214f41992a48c06cee61b21851fb31448c0e515d080710773a6b12a3ff946bd
Sep 09 06:04:40.801903 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:40.801866    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-43eb2d/premium-w4-43eb2d-kserve-84b57f44-tlrbl" event={"ID":"de67233d-58eb-43dc-8473-f5dd1ef69b05","Type":"ContainerStarted","Data":"0f399fc050cc3a452af9697155c6d859046d4f4e8b3b805d6632bd6df1237614"}
Sep 09 06:04:40.801903 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:40.801907    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-43eb2d/premium-w4-43eb2d-kserve-84b57f44-tlrbl" event={"ID":"de67233d-58eb-43dc-8473-f5dd1ef69b05","Type":"ContainerStarted","Data":"a214f41992a48c06cee61b21851fb31448c0e515d080710773a6b12a3ff946bd"}
Sep 09 06:04:40.820662 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:40.820600    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-43eb2d/premium-w4-43eb2d-kserve-84b57f44-tlrbl" podStartSLOduration=1.820584653 podStartE2EDuration="1.820584653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:04:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:04:40.818979739 +0000 UTC m=+906.159467649" watchObservedRunningTime="2026-09-09 06:04:40.820584653 +0000 UTC m=+906.161072561"
Sep 09 06:04:49.650382 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:49.650350    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-43eb2d/premium-w4-43eb2d-kserve-84b57f44-tlrbl"
Sep 09 06:04:49.652910 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:49.650593    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-43eb2d/premium-w4-43eb2d-kserve-84b57f44-tlrbl"
Sep 09 06:04:49.663214 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:49.663196    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-43eb2d/premium-w4-43eb2d-kserve-84b57f44-tlrbl"
Sep 09 06:04:49.857812 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:49.857789    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-43eb2d/premium-w4-43eb2d-kserve-84b57f44-tlrbl"
Sep 09 06:04:53.091455 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:04:53.091419    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:05:16.406721 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:16.406683    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:05:43.437082 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:43.437049    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f631ee-f7q6b"]
Sep 09 06:05:43.440284 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:43.440267    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f631ee-f7q6b"
Sep 09 06:05:43.442516 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:43.442495    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ksjk6\""
Sep 09 06:05:43.446569 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:43.446551    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f631ee-f7q6b"]
Sep 09 06:05:43.462070 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:43.462050    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0aa173a1-d6a4-4728-83b6-fc387aa7c6b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-f631ee-f7q6b\" (UID: \"0aa173a1-d6a4-4728-83b6-fc387aa7c6b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f631ee-f7q6b"
Sep 09 06:05:43.562375 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:43.562350    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0aa173a1-d6a4-4728-83b6-fc387aa7c6b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-f631ee-f7q6b\" (UID: \"0aa173a1-d6a4-4728-83b6-fc387aa7c6b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f631ee-f7q6b"
Sep 09 06:05:43.562911 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:43.562894    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0aa173a1-d6a4-4728-83b6-fc387aa7c6b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-f631ee-f7q6b\" (UID: \"0aa173a1-d6a4-4728-83b6-fc387aa7c6b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f631ee-f7q6b"
Sep 09 06:05:43.751987 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:43.751932    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f631ee-f7q6b"
Sep 09 06:05:44.021153 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:44.021085    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f631ee-f7q6b"]
Sep 09 06:05:44.024184 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:05:44.024158    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aa173a1_d6a4_4728_83b6_fc387aa7c6b2.slice/crio-658b3f8e68e20ba7071a05c0bf1a548d844c1c7206068183f7f4f9d235ebda3c WatchSource:0}: Error finding container 658b3f8e68e20ba7071a05c0bf1a548d844c1c7206068183f7f4f9d235ebda3c: Status 404 returned error can't find the container with id 658b3f8e68e20ba7071a05c0bf1a548d844c1c7206068183f7f4f9d235ebda3c
Sep 09 06:05:44.092066 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:44.092039    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f631ee-f7q6b" event={"ID":"0aa173a1-d6a4-4728-83b6-fc387aa7c6b2","Type":"ContainerStarted","Data":"658b3f8e68e20ba7071a05c0bf1a548d844c1c7206068183f7f4f9d235ebda3c"}
Sep 09 06:05:44.742964 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:05:44.742936    2579 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aa173a1_d6a4_4728_83b6_fc387aa7c6b2.slice/crio-eaee0ca4be1a6861898f6fe2e38c37ce959549043e76d54a46a1a94e4229cb20.scope\": RecentStats: unable to find data in memory cache]"
Sep 09 06:05:45.098687 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:45.098657    2579 generic.go:356] "Generic (PLEG): container finished" podID="0aa173a1-d6a4-4728-83b6-fc387aa7c6b2" containerID="eaee0ca4be1a6861898f6fe2e38c37ce959549043e76d54a46a1a94e4229cb20" exitCode=0
Sep 09 06:05:45.098827 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:45.098716    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f631ee-f7q6b" event={"ID":"0aa173a1-d6a4-4728-83b6-fc387aa7c6b2","Type":"ContainerDied","Data":"eaee0ca4be1a6861898f6fe2e38c37ce959549043e76d54a46a1a94e4229cb20"}
Sep 09 06:05:46.228665 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:46.228643    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f631ee-f7q6b"
Sep 09 06:05:46.286348 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:46.286326    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0aa173a1-d6a4-4728-83b6-fc387aa7c6b2-maas-api-service-ca\") pod \"0aa173a1-d6a4-4728-83b6-fc387aa7c6b2\" (UID: \"0aa173a1-d6a4-4728-83b6-fc387aa7c6b2\") "
Sep 09 06:05:46.286707 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:46.286682    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aa173a1-d6a4-4728-83b6-fc387aa7c6b2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0aa173a1-d6a4-4728-83b6-fc387aa7c6b2" (UID: "0aa173a1-d6a4-4728-83b6-fc387aa7c6b2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:05:46.386988 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:46.386932    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0aa173a1-d6a4-4728-83b6-fc387aa7c6b2-maas-api-service-ca\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:05:47.108660 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:47.108603    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f631ee-f7q6b" event={"ID":"0aa173a1-d6a4-4728-83b6-fc387aa7c6b2","Type":"ContainerDied","Data":"658b3f8e68e20ba7071a05c0bf1a548d844c1c7206068183f7f4f9d235ebda3c"}
Sep 09 06:05:47.108660 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:47.108647    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f631ee-f7q6b"
Sep 09 06:05:47.108867 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:47.108683    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="658b3f8e68e20ba7071a05c0bf1a548d844c1c7206068183f7f4f9d235ebda3c"
Sep 09 06:05:52.700900 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:52.700870    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:05:53.940497 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:53.940458    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f631ee-6559b7c7-zmggw"]
Sep 09 06:05:53.940970 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:53.940777    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-f631ee-6559b7c7-zmggw" podUID="754a6601-d71f-486e-9b99-5fb50d498feb" containerName="payload-processing" containerID="cri-o://83c8ae5b13e3103878edfcaebfd83d02e2b3745d2e7f1cdfcd8e61d37712b2f9" gracePeriod=30
Sep 09 06:05:53.945825 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:53.945243    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f631ee-74796bf5d6-rsgxg"]
Sep 09 06:05:53.945825 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:53.945463    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f631ee-74796bf5d6-rsgxg" podUID="7f8bdb0d-c657-4ff9-a5f0-756e0642aa28" containerName="payload-pre-processing" containerID="cri-o://581ab85cf535774e1429c4e01f8d49060ae1c95aa609299fef3bf292a80c6f3d" gracePeriod=30
Sep 09 06:05:54.097525 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:54.097466    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f631ee-f7q6b"]
Sep 09 06:05:54.099954 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:54.099926    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f631ee-f7q6b"]
Sep 09 06:05:54.143825 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:54.143799    2579 generic.go:356] "Generic (PLEG): container finished" podID="7f8bdb0d-c657-4ff9-a5f0-756e0642aa28" containerID="581ab85cf535774e1429c4e01f8d49060ae1c95aa609299fef3bf292a80c6f3d" exitCode=0
Sep 09 06:05:54.143925 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:54.143906    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f631ee-74796bf5d6-rsgxg" event={"ID":"7f8bdb0d-c657-4ff9-a5f0-756e0642aa28","Type":"ContainerDied","Data":"581ab85cf535774e1429c4e01f8d49060ae1c95aa609299fef3bf292a80c6f3d"}
Sep 09 06:05:54.146058 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:54.146036    2579 generic.go:356] "Generic (PLEG): container finished" podID="754a6601-d71f-486e-9b99-5fb50d498feb" containerID="83c8ae5b13e3103878edfcaebfd83d02e2b3745d2e7f1cdfcd8e61d37712b2f9" exitCode=0
Sep 09 06:05:54.146158 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:54.146096    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-f631ee-6559b7c7-zmggw" event={"ID":"754a6601-d71f-486e-9b99-5fb50d498feb","Type":"ContainerDied","Data":"83c8ae5b13e3103878edfcaebfd83d02e2b3745d2e7f1cdfcd8e61d37712b2f9"}
Sep 09 06:05:54.219255 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:54.219227    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-f631ee-6559b7c7-zmggw"
Sep 09 06:05:54.221974 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:54.221958    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f631ee-74796bf5d6-rsgxg"
Sep 09 06:05:54.353849 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:54.353820    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/754a6601-d71f-486e-9b99-5fb50d498feb-plugins-config-volume\") pod \"754a6601-d71f-486e-9b99-5fb50d498feb\" (UID: \"754a6601-d71f-486e-9b99-5fb50d498feb\") "
Sep 09 06:05:54.353994 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:54.353858    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlvts\" (UniqueName: \"kubernetes.io/projected/754a6601-d71f-486e-9b99-5fb50d498feb-kube-api-access-tlvts\") pod \"754a6601-d71f-486e-9b99-5fb50d498feb\" (UID: \"754a6601-d71f-486e-9b99-5fb50d498feb\") "
Sep 09 06:05:54.353994 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:54.353893    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f8bdb0d-c657-4ff9-a5f0-756e0642aa28-plugins-config-volume\") pod \"7f8bdb0d-c657-4ff9-a5f0-756e0642aa28\" (UID: \"7f8bdb0d-c657-4ff9-a5f0-756e0642aa28\") "
Sep 09 06:05:54.353994 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:54.353922    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qggrw\" (UniqueName: \"kubernetes.io/projected/7f8bdb0d-c657-4ff9-a5f0-756e0642aa28-kube-api-access-qggrw\") pod \"7f8bdb0d-c657-4ff9-a5f0-756e0642aa28\" (UID: \"7f8bdb0d-c657-4ff9-a5f0-756e0642aa28\") "
Sep 09 06:05:54.354174 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:54.354152    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/754a6601-d71f-486e-9b99-5fb50d498feb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "754a6601-d71f-486e-9b99-5fb50d498feb" (UID: "754a6601-d71f-486e-9b99-5fb50d498feb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:05:54.354267 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:54.354247    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f8bdb0d-c657-4ff9-a5f0-756e0642aa28-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7f8bdb0d-c657-4ff9-a5f0-756e0642aa28" (UID: "7f8bdb0d-c657-4ff9-a5f0-756e0642aa28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:05:54.356151 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:54.356132    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f8bdb0d-c657-4ff9-a5f0-756e0642aa28-kube-api-access-qggrw" (OuterVolumeSpecName: "kube-api-access-qggrw") pod "7f8bdb0d-c657-4ff9-a5f0-756e0642aa28" (UID: "7f8bdb0d-c657-4ff9-a5f0-756e0642aa28"). InnerVolumeSpecName "kube-api-access-qggrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:05:54.356278 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:54.356258    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/754a6601-d71f-486e-9b99-5fb50d498feb-kube-api-access-tlvts" (OuterVolumeSpecName: "kube-api-access-tlvts") pod "754a6601-d71f-486e-9b99-5fb50d498feb" (UID: "754a6601-d71f-486e-9b99-5fb50d498feb"). InnerVolumeSpecName "kube-api-access-tlvts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:05:54.454504 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:54.454449    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/754a6601-d71f-486e-9b99-5fb50d498feb-plugins-config-volume\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:05:54.454504 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:54.454470    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlvts\" (UniqueName: \"kubernetes.io/projected/754a6601-d71f-486e-9b99-5fb50d498feb-kube-api-access-tlvts\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:05:54.454504 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:54.454480    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f8bdb0d-c657-4ff9-a5f0-756e0642aa28-plugins-config-volume\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:05:54.454504 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:54.454489    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qggrw\" (UniqueName: \"kubernetes.io/projected/7f8bdb0d-c657-4ff9-a5f0-756e0642aa28-kube-api-access-qggrw\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:05:55.147759 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:55.147720    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0aa173a1-d6a4-4728-83b6-fc387aa7c6b2" path="/var/lib/kubelet/pods/0aa173a1-d6a4-4728-83b6-fc387aa7c6b2/volumes"
Sep 09 06:05:55.153407 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:55.153378    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f631ee-74796bf5d6-rsgxg" event={"ID":"7f8bdb0d-c657-4ff9-a5f0-756e0642aa28","Type":"ContainerDied","Data":"f66e1ec1c20cbddec5dbc38611e754023efdbb43a1e47426f748eb5d080bf376"}
Sep 09 06:05:55.153407 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:55.153398    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f631ee-74796bf5d6-rsgxg"
Sep 09 06:05:55.153714 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:55.153442    2579 scope.go:117] "RemoveContainer" containerID="581ab85cf535774e1429c4e01f8d49060ae1c95aa609299fef3bf292a80c6f3d"
Sep 09 06:05:55.155212 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:55.155193    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-f631ee-6559b7c7-zmggw"
Sep 09 06:05:55.155212 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:55.155203    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-f631ee-6559b7c7-zmggw" event={"ID":"754a6601-d71f-486e-9b99-5fb50d498feb","Type":"ContainerDied","Data":"53be8174bcfc0ea66d0a37f5bbe9695050215e607170188a0ef4274659c54165"}
Sep 09 06:05:55.165408 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:55.165393    2579 scope.go:117] "RemoveContainer" containerID="83c8ae5b13e3103878edfcaebfd83d02e2b3745d2e7f1cdfcd8e61d37712b2f9"
Sep 09 06:05:55.175167 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:55.175150    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f631ee-6559b7c7-zmggw"]
Sep 09 06:05:55.177766 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:55.177746    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f631ee-6559b7c7-zmggw"]
Sep 09 06:05:55.186896 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:55.186874    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f631ee-74796bf5d6-rsgxg"]
Sep 09 06:05:55.188831 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:55.188814    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f631ee-74796bf5d6-rsgxg"]
Sep 09 06:05:57.148584 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:57.148507    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="754a6601-d71f-486e-9b99-5fb50d498feb" path="/var/lib/kubelet/pods/754a6601-d71f-486e-9b99-5fb50d498feb/volumes"
Sep 09 06:05:57.148952 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:05:57.148911    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f8bdb0d-c657-4ff9-a5f0-756e0642aa28" path="/var/lib/kubelet/pods/7f8bdb0d-c657-4ff9-a5f0-756e0642aa28/volumes"
Sep 09 06:06:08.660922 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:08.660887    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-464e41e7-6f4bb77674-dvx65"]
Sep 09 06:06:08.661255 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:08.661182    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="754a6601-d71f-486e-9b99-5fb50d498feb" containerName="payload-processing"
Sep 09 06:06:08.661255 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:08.661193    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="754a6601-d71f-486e-9b99-5fb50d498feb" containerName="payload-processing"
Sep 09 06:06:08.661255 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:08.661209    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f8bdb0d-c657-4ff9-a5f0-756e0642aa28" containerName="payload-pre-processing"
Sep 09 06:06:08.661255 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:08.661214    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f8bdb0d-c657-4ff9-a5f0-756e0642aa28" containerName="payload-pre-processing"
Sep 09 06:06:08.661255 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:08.661225    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0aa173a1-d6a4-4728-83b6-fc387aa7c6b2" containerName="revoke-keys"
Sep 09 06:06:08.661255 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:08.661231    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0aa173a1-d6a4-4728-83b6-fc387aa7c6b2" containerName="revoke-keys"
Sep 09 06:06:08.661456 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:08.661284    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f8bdb0d-c657-4ff9-a5f0-756e0642aa28" containerName="payload-pre-processing"
Sep 09 06:06:08.661456 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:08.661293    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0aa173a1-d6a4-4728-83b6-fc387aa7c6b2" containerName="revoke-keys"
Sep 09 06:06:08.661456 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:08.661301    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="754a6601-d71f-486e-9b99-5fb50d498feb" containerName="payload-processing"
Sep 09 06:06:08.664237 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:08.664219    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-464e41e7-6f4bb77674-dvx65"
Sep 09 06:06:08.666540 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:08.666518    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-464e41e7\""
Sep 09 06:06:08.666755 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:08.666682    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-464e41e7-dockercfg-ztd5h\""
Sep 09 06:06:08.675293 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:08.675274    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-464e41e7-6f4bb77674-dvx65"]
Sep 09 06:06:08.677159 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:08.677129    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-464e41e7-6f6d45f77-dm8t2"]
Sep 09 06:06:08.680246 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:08.680229    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-464e41e7-6f6d45f77-dm8t2"
Sep 09 06:06:08.691266 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:08.691244    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-464e41e7-6f6d45f77-dm8t2"]
Sep 09 06:06:08.765501 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:08.765470    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l2dw\" (UniqueName: \"kubernetes.io/projected/002ea6f3-16b9-4b6a-8ebf-ef83aa9c7e0c-kube-api-access-8l2dw\") pod \"payload-pre-processing-e2e-mt-464e41e7-6f4bb77674-dvx65\" (UID: \"002ea6f3-16b9-4b6a-8ebf-ef83aa9c7e0c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-464e41e7-6f4bb77674-dvx65"
Sep 09 06:06:08.765675 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:08.765519    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eec01a7-f17d-45b1-8c6f-036eff1bd9e5-plugins-config-volume\") pod \"payload-processing-e2e-mt-464e41e7-6f6d45f77-dm8t2\" (UID: \"5eec01a7-f17d-45b1-8c6f-036eff1bd9e5\") " pod="openshift-ingress/payload-processing-e2e-mt-464e41e7-6f6d45f77-dm8t2"
Sep 09 06:06:08.765675 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:08.765543    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/002ea6f3-16b9-4b6a-8ebf-ef83aa9c7e0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-464e41e7-6f4bb77674-dvx65\" (UID: \"002ea6f3-16b9-4b6a-8ebf-ef83aa9c7e0c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-464e41e7-6f4bb77674-dvx65"
Sep 09 06:06:08.765675 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:08.765604    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd98h\" (UniqueName: \"kubernetes.io/projected/5eec01a7-f17d-45b1-8c6f-036eff1bd9e5-kube-api-access-vd98h\") pod \"payload-processing-e2e-mt-464e41e7-6f6d45f77-dm8t2\" (UID: \"5eec01a7-f17d-45b1-8c6f-036eff1bd9e5\") " pod="openshift-ingress/payload-processing-e2e-mt-464e41e7-6f6d45f77-dm8t2"
Sep 09 06:06:08.866982 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:08.866946    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eec01a7-f17d-45b1-8c6f-036eff1bd9e5-plugins-config-volume\") pod \"payload-processing-e2e-mt-464e41e7-6f6d45f77-dm8t2\" (UID: \"5eec01a7-f17d-45b1-8c6f-036eff1bd9e5\") " pod="openshift-ingress/payload-processing-e2e-mt-464e41e7-6f6d45f77-dm8t2"
Sep 09 06:06:08.867144 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:08.866990    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/002ea6f3-16b9-4b6a-8ebf-ef83aa9c7e0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-464e41e7-6f4bb77674-dvx65\" (UID: \"002ea6f3-16b9-4b6a-8ebf-ef83aa9c7e0c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-464e41e7-6f4bb77674-dvx65"
Sep 09 06:06:08.867144 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:08.867041    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd98h\" (UniqueName: \"kubernetes.io/projected/5eec01a7-f17d-45b1-8c6f-036eff1bd9e5-kube-api-access-vd98h\") pod \"payload-processing-e2e-mt-464e41e7-6f6d45f77-dm8t2\" (UID: \"5eec01a7-f17d-45b1-8c6f-036eff1bd9e5\") " pod="openshift-ingress/payload-processing-e2e-mt-464e41e7-6f6d45f77-dm8t2"
Sep 09 06:06:08.867144 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:08.867094    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l2dw\" (UniqueName: \"kubernetes.io/projected/002ea6f3-16b9-4b6a-8ebf-ef83aa9c7e0c-kube-api-access-8l2dw\") pod \"payload-pre-processing-e2e-mt-464e41e7-6f4bb77674-dvx65\" (UID: \"002ea6f3-16b9-4b6a-8ebf-ef83aa9c7e0c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-464e41e7-6f4bb77674-dvx65"
Sep 09 06:06:08.867745 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:08.867726    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/002ea6f3-16b9-4b6a-8ebf-ef83aa9c7e0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-464e41e7-6f4bb77674-dvx65\" (UID: \"002ea6f3-16b9-4b6a-8ebf-ef83aa9c7e0c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-464e41e7-6f4bb77674-dvx65"
Sep 09 06:06:08.867818 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:08.867730    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eec01a7-f17d-45b1-8c6f-036eff1bd9e5-plugins-config-volume\") pod \"payload-processing-e2e-mt-464e41e7-6f6d45f77-dm8t2\" (UID: \"5eec01a7-f17d-45b1-8c6f-036eff1bd9e5\") " pod="openshift-ingress/payload-processing-e2e-mt-464e41e7-6f6d45f77-dm8t2"
Sep 09 06:06:08.876832 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:08.876812    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd98h\" (UniqueName: \"kubernetes.io/projected/5eec01a7-f17d-45b1-8c6f-036eff1bd9e5-kube-api-access-vd98h\") pod \"payload-processing-e2e-mt-464e41e7-6f6d45f77-dm8t2\" (UID: \"5eec01a7-f17d-45b1-8c6f-036eff1bd9e5\") " pod="openshift-ingress/payload-processing-e2e-mt-464e41e7-6f6d45f77-dm8t2"
Sep 09 06:06:08.876920 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:08.876828    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l2dw\" (UniqueName: \"kubernetes.io/projected/002ea6f3-16b9-4b6a-8ebf-ef83aa9c7e0c-kube-api-access-8l2dw\") pod \"payload-pre-processing-e2e-mt-464e41e7-6f4bb77674-dvx65\" (UID: \"002ea6f3-16b9-4b6a-8ebf-ef83aa9c7e0c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-464e41e7-6f4bb77674-dvx65"
Sep 09 06:06:08.975081 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:08.975001    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-464e41e7-6f4bb77674-dvx65"
Sep 09 06:06:08.991254 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:08.991224    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-464e41e7-6f6d45f77-dm8t2"
Sep 09 06:06:09.252560 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:09.252529    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-464e41e7-6f4bb77674-dvx65"]
Sep 09 06:06:09.254425 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:06:09.254393    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod002ea6f3_16b9_4b6a_8ebf_ef83aa9c7e0c.slice/crio-5ced5769b503d5df127c62ed0c1bb0c60bcca330f8309b12e5242d1cce6bf5c3 WatchSource:0}: Error finding container 5ced5769b503d5df127c62ed0c1bb0c60bcca330f8309b12e5242d1cce6bf5c3: Status 404 returned error can't find the container with id 5ced5769b503d5df127c62ed0c1bb0c60bcca330f8309b12e5242d1cce6bf5c3
Sep 09 06:06:09.276870 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:09.276847    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-464e41e7-6f6d45f77-dm8t2"]
Sep 09 06:06:09.282719 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:06:09.282692    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eec01a7_f17d_45b1_8c6f_036eff1bd9e5.slice/crio-80dfececb0b3259580c58897e49b1b27d8a98803548497478eda351fa428050b WatchSource:0}: Error finding container 80dfececb0b3259580c58897e49b1b27d8a98803548497478eda351fa428050b: Status 404 returned error can't find the container with id 80dfececb0b3259580c58897e49b1b27d8a98803548497478eda351fa428050b
Sep 09 06:06:10.235608 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:10.235574    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-464e41e7-6f6d45f77-dm8t2" event={"ID":"5eec01a7-f17d-45b1-8c6f-036eff1bd9e5","Type":"ContainerStarted","Data":"f834644924218298a65cbfef324d2c3422f2b16284f56b8c78d82d4ba091554c"}
Sep 09 06:06:10.236082 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:10.235680    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-464e41e7-6f6d45f77-dm8t2" event={"ID":"5eec01a7-f17d-45b1-8c6f-036eff1bd9e5","Type":"ContainerStarted","Data":"80dfececb0b3259580c58897e49b1b27d8a98803548497478eda351fa428050b"}
Sep 09 06:06:10.236082 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:10.235714    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-464e41e7-6f6d45f77-dm8t2"
Sep 09 06:06:10.237424 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:10.237394    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-464e41e7-6f4bb77674-dvx65" event={"ID":"002ea6f3-16b9-4b6a-8ebf-ef83aa9c7e0c","Type":"ContainerStarted","Data":"1a6da23a74675619bbe12b9e674f683d675ce2c3750595ff2bb9966a5777fa10"}
Sep 09 06:06:10.237535 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:10.237439    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-464e41e7-6f4bb77674-dvx65" event={"ID":"002ea6f3-16b9-4b6a-8ebf-ef83aa9c7e0c","Type":"ContainerStarted","Data":"5ced5769b503d5df127c62ed0c1bb0c60bcca330f8309b12e5242d1cce6bf5c3"}
Sep 09 06:06:10.237535 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:10.237493    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-464e41e7-6f4bb77674-dvx65"
Sep 09 06:06:10.252293 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:10.252255    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-464e41e7-6f6d45f77-dm8t2" podStartSLOduration=2.252242778 podStartE2EDuration="2.252242778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:06:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:06:10.25071522 +0000 UTC m=+995.591203130" watchObservedRunningTime="2026-09-09 06:06:10.252242778 +0000 UTC m=+995.592730688"
Sep 09 06:06:10.271720 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:10.271675    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-464e41e7-6f4bb77674-dvx65" podStartSLOduration=2.271661712 podStartE2EDuration="2.271661712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:06:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:06:10.269663735 +0000 UTC m=+995.610151648" watchObservedRunningTime="2026-09-09 06:06:10.271661712 +0000 UTC m=+995.612149621"
Sep 09 06:06:12.393339 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:12.393308    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:06:17.139015 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:17.138983    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-43eb2d/distinct-w4-43eb2d-kserve-596559c7f-8x4kz"]
Sep 09 06:06:17.142747 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:17.142719    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-43eb2d/distinct-w4-43eb2d-kserve-596559c7f-8x4kz"
Sep 09 06:06:17.145161 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:17.145135    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-43eb2d\"/\"distinct-w4-43eb2d-kserve-self-signed-certs\""
Sep 09 06:06:17.152950 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:17.152929    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-43eb2d/distinct-w4-43eb2d-kserve-596559c7f-8x4kz"]
Sep 09 06:06:17.231719 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:17.231689    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c4be116-1d2a-4d8a-8bf2-9aa980a954c4-tls-certs\") pod \"distinct-w4-43eb2d-kserve-596559c7f-8x4kz\" (UID: \"6c4be116-1d2a-4d8a-8bf2-9aa980a954c4\") " pod="e2e-models-e2e-worker-w4-43eb2d/distinct-w4-43eb2d-kserve-596559c7f-8x4kz"
Sep 09 06:06:17.231899 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:17.231724    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c4be116-1d2a-4d8a-8bf2-9aa980a954c4-tmp-dir\") pod \"distinct-w4-43eb2d-kserve-596559c7f-8x4kz\" (UID: \"6c4be116-1d2a-4d8a-8bf2-9aa980a954c4\") " pod="e2e-models-e2e-worker-w4-43eb2d/distinct-w4-43eb2d-kserve-596559c7f-8x4kz"
Sep 09 06:06:17.231899 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:17.231751    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c4be116-1d2a-4d8a-8bf2-9aa980a954c4-home\") pod \"distinct-w4-43eb2d-kserve-596559c7f-8x4kz\" (UID: \"6c4be116-1d2a-4d8a-8bf2-9aa980a954c4\") " pod="e2e-models-e2e-worker-w4-43eb2d/distinct-w4-43eb2d-kserve-596559c7f-8x4kz"
Sep 09 06:06:17.231899 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:17.231773    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c4be116-1d2a-4d8a-8bf2-9aa980a954c4-dshm\") pod \"distinct-w4-43eb2d-kserve-596559c7f-8x4kz\" (UID: \"6c4be116-1d2a-4d8a-8bf2-9aa980a954c4\") " pod="e2e-models-e2e-worker-w4-43eb2d/distinct-w4-43eb2d-kserve-596559c7f-8x4kz"
Sep 09 06:06:17.231899 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:17.231849    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c4be116-1d2a-4d8a-8bf2-9aa980a954c4-model-cache\") pod \"distinct-w4-43eb2d-kserve-596559c7f-8x4kz\" (UID: \"6c4be116-1d2a-4d8a-8bf2-9aa980a954c4\") " pod="e2e-models-e2e-worker-w4-43eb2d/distinct-w4-43eb2d-kserve-596559c7f-8x4kz"
Sep 09 06:06:17.231899 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:17.231875    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6bzf\" (UniqueName: \"kubernetes.io/projected/6c4be116-1d2a-4d8a-8bf2-9aa980a954c4-kube-api-access-w6bzf\") pod \"distinct-w4-43eb2d-kserve-596559c7f-8x4kz\" (UID: \"6c4be116-1d2a-4d8a-8bf2-9aa980a954c4\") " pod="e2e-models-e2e-worker-w4-43eb2d/distinct-w4-43eb2d-kserve-596559c7f-8x4kz"
Sep 09 06:06:17.332987 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:17.332959    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c4be116-1d2a-4d8a-8bf2-9aa980a954c4-tls-certs\") pod \"distinct-w4-43eb2d-kserve-596559c7f-8x4kz\" (UID: \"6c4be116-1d2a-4d8a-8bf2-9aa980a954c4\") " pod="e2e-models-e2e-worker-w4-43eb2d/distinct-w4-43eb2d-kserve-596559c7f-8x4kz"
Sep 09 06:06:17.332987 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:17.332990    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c4be116-1d2a-4d8a-8bf2-9aa980a954c4-tmp-dir\") pod \"distinct-w4-43eb2d-kserve-596559c7f-8x4kz\" (UID: \"6c4be116-1d2a-4d8a-8bf2-9aa980a954c4\") " pod="e2e-models-e2e-worker-w4-43eb2d/distinct-w4-43eb2d-kserve-596559c7f-8x4kz"
Sep 09 06:06:17.333218 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:17.333011    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c4be116-1d2a-4d8a-8bf2-9aa980a954c4-home\") pod \"distinct-w4-43eb2d-kserve-596559c7f-8x4kz\" (UID: \"6c4be116-1d2a-4d8a-8bf2-9aa980a954c4\") " pod="e2e-models-e2e-worker-w4-43eb2d/distinct-w4-43eb2d-kserve-596559c7f-8x4kz"
Sep 09 06:06:17.333218 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:17.333028    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c4be116-1d2a-4d8a-8bf2-9aa980a954c4-dshm\") pod \"distinct-w4-43eb2d-kserve-596559c7f-8x4kz\" (UID: \"6c4be116-1d2a-4d8a-8bf2-9aa980a954c4\") " pod="e2e-models-e2e-worker-w4-43eb2d/distinct-w4-43eb2d-kserve-596559c7f-8x4kz"
Sep 09 06:06:17.333218 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:17.333049    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c4be116-1d2a-4d8a-8bf2-9aa980a954c4-model-cache\") pod \"distinct-w4-43eb2d-kserve-596559c7f-8x4kz\" (UID: \"6c4be116-1d2a-4d8a-8bf2-9aa980a954c4\") " pod="e2e-models-e2e-worker-w4-43eb2d/distinct-w4-43eb2d-kserve-596559c7f-8x4kz"
Sep 09 06:06:17.333218 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:17.333162    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6bzf\" (UniqueName: \"kubernetes.io/projected/6c4be116-1d2a-4d8a-8bf2-9aa980a954c4-kube-api-access-w6bzf\") pod \"distinct-w4-43eb2d-kserve-596559c7f-8x4kz\" (UID: \"6c4be116-1d2a-4d8a-8bf2-9aa980a954c4\") " pod="e2e-models-e2e-worker-w4-43eb2d/distinct-w4-43eb2d-kserve-596559c7f-8x4kz"
Sep 09 06:06:17.333425 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:17.333403    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c4be116-1d2a-4d8a-8bf2-9aa980a954c4-home\") pod \"distinct-w4-43eb2d-kserve-596559c7f-8x4kz\" (UID: \"6c4be116-1d2a-4d8a-8bf2-9aa980a954c4\") " pod="e2e-models-e2e-worker-w4-43eb2d/distinct-w4-43eb2d-kserve-596559c7f-8x4kz"
Sep 09 06:06:17.333481 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:17.333448    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c4be116-1d2a-4d8a-8bf2-9aa980a954c4-tmp-dir\") pod \"distinct-w4-43eb2d-kserve-596559c7f-8x4kz\" (UID: \"6c4be116-1d2a-4d8a-8bf2-9aa980a954c4\") " pod="e2e-models-e2e-worker-w4-43eb2d/distinct-w4-43eb2d-kserve-596559c7f-8x4kz"
Sep 09 06:06:17.333545 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:17.333527    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c4be116-1d2a-4d8a-8bf2-9aa980a954c4-model-cache\") pod \"distinct-w4-43eb2d-kserve-596559c7f-8x4kz\" (UID: \"6c4be116-1d2a-4d8a-8bf2-9aa980a954c4\") " pod="e2e-models-e2e-worker-w4-43eb2d/distinct-w4-43eb2d-kserve-596559c7f-8x4kz"
Sep 09 06:06:17.335535 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:17.335513    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c4be116-1d2a-4d8a-8bf2-9aa980a954c4-dshm\") pod \"distinct-w4-43eb2d-kserve-596559c7f-8x4kz\" (UID: \"6c4be116-1d2a-4d8a-8bf2-9aa980a954c4\") " pod="e2e-models-e2e-worker-w4-43eb2d/distinct-w4-43eb2d-kserve-596559c7f-8x4kz"
Sep 09 06:06:17.335789 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:17.335772    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c4be116-1d2a-4d8a-8bf2-9aa980a954c4-tls-certs\") pod \"distinct-w4-43eb2d-kserve-596559c7f-8x4kz\" (UID: \"6c4be116-1d2a-4d8a-8bf2-9aa980a954c4\") " pod="e2e-models-e2e-worker-w4-43eb2d/distinct-w4-43eb2d-kserve-596559c7f-8x4kz"
Sep 09 06:06:17.341599 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:17.341578    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6bzf\" (UniqueName: \"kubernetes.io/projected/6c4be116-1d2a-4d8a-8bf2-9aa980a954c4-kube-api-access-w6bzf\") pod \"distinct-w4-43eb2d-kserve-596559c7f-8x4kz\" (UID: \"6c4be116-1d2a-4d8a-8bf2-9aa980a954c4\") " pod="e2e-models-e2e-worker-w4-43eb2d/distinct-w4-43eb2d-kserve-596559c7f-8x4kz"
Sep 09 06:06:17.453059 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:17.452987    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-43eb2d/distinct-w4-43eb2d-kserve-596559c7f-8x4kz"
Sep 09 06:06:17.723868 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:17.723845    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-43eb2d/distinct-w4-43eb2d-kserve-596559c7f-8x4kz"]
Sep 09 06:06:17.725545 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:06:17.725514    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c4be116_1d2a_4d8a_8bf2_9aa980a954c4.slice/crio-041bc02e4e0f69ac2e98a3560b13ae4fbf57ce525d55d12b5ff4ecdde1421c50 WatchSource:0}: Error finding container 041bc02e4e0f69ac2e98a3560b13ae4fbf57ce525d55d12b5ff4ecdde1421c50: Status 404 returned error can't find the container with id 041bc02e4e0f69ac2e98a3560b13ae4fbf57ce525d55d12b5ff4ecdde1421c50
Sep 09 06:06:18.279230 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:18.279194    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-43eb2d/distinct-w4-43eb2d-kserve-596559c7f-8x4kz" event={"ID":"6c4be116-1d2a-4d8a-8bf2-9aa980a954c4","Type":"ContainerStarted","Data":"3598743b6532f9d688daf0c608edb923edf4769496d12d7dfab4f0739ffab48a"}
Sep 09 06:06:18.279230 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:18.279235    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-43eb2d/distinct-w4-43eb2d-kserve-596559c7f-8x4kz" event={"ID":"6c4be116-1d2a-4d8a-8bf2-9aa980a954c4","Type":"ContainerStarted","Data":"041bc02e4e0f69ac2e98a3560b13ae4fbf57ce525d55d12b5ff4ecdde1421c50"}
Sep 09 06:06:18.295282 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:18.295235    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-43eb2d/distinct-w4-43eb2d-kserve-596559c7f-8x4kz" podStartSLOduration=1.295222262 podStartE2EDuration="1.295222262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:06:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:06:18.293908693 +0000 UTC m=+1003.634396602" watchObservedRunningTime="2026-09-09 06:06:18.295222262 +0000 UTC m=+1003.635710171"
Sep 09 06:06:21.244786 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:21.244758    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-464e41e7-6f6d45f77-dm8t2"
Sep 09 06:06:21.245148 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:21.244828    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-464e41e7-6f4bb77674-dvx65"
Sep 09 06:06:27.454180 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:27.454130    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-43eb2d/distinct-w4-43eb2d-kserve-596559c7f-8x4kz"
Sep 09 06:06:27.454180 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:27.454192    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-43eb2d/distinct-w4-43eb2d-kserve-596559c7f-8x4kz"
Sep 09 06:06:27.466846 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:27.466825    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-43eb2d/distinct-w4-43eb2d-kserve-596559c7f-8x4kz"
Sep 09 06:06:28.337913 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:28.337887    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-43eb2d/distinct-w4-43eb2d-kserve-596559c7f-8x4kz"
Sep 09 06:06:31.795437 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:31.795402    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:06:38.720888 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:38.720858    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464e41e7-bcrrc"]
Sep 09 06:06:38.725386 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:38.725370    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464e41e7-bcrrc"
Sep 09 06:06:38.728719 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:38.728297    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ksjk6\""
Sep 09 06:06:38.732774 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:38.732751    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464e41e7-bcrrc"]
Sep 09 06:06:38.812866 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:38.812838    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8128d424-7945-4bb4-bbdb-02be561febb6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-464e41e7-bcrrc\" (UID: \"8128d424-7945-4bb4-bbdb-02be561febb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464e41e7-bcrrc"
Sep 09 06:06:38.913898 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:38.913865    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8128d424-7945-4bb4-bbdb-02be561febb6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-464e41e7-bcrrc\" (UID: \"8128d424-7945-4bb4-bbdb-02be561febb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464e41e7-bcrrc"
Sep 09 06:06:38.914426 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:38.914408    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8128d424-7945-4bb4-bbdb-02be561febb6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-464e41e7-bcrrc\" (UID: \"8128d424-7945-4bb4-bbdb-02be561febb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464e41e7-bcrrc"
Sep 09 06:06:39.044193 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:39.044127    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464e41e7-bcrrc"
Sep 09 06:06:39.319680 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:39.319654    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464e41e7-bcrrc"]
Sep 09 06:06:39.321300 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:06:39.321275    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8128d424_7945_4bb4_bbdb_02be561febb6.slice/crio-39219cf6cb80f7000347684e488166f8fa512e4eab1036d1d1e58c55c5b0d7f5 WatchSource:0}: Error finding container 39219cf6cb80f7000347684e488166f8fa512e4eab1036d1d1e58c55c5b0d7f5: Status 404 returned error can't find the container with id 39219cf6cb80f7000347684e488166f8fa512e4eab1036d1d1e58c55c5b0d7f5
Sep 09 06:06:39.379341 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:39.379316    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464e41e7-bcrrc" event={"ID":"8128d424-7945-4bb4-bbdb-02be561febb6","Type":"ContainerStarted","Data":"39219cf6cb80f7000347684e488166f8fa512e4eab1036d1d1e58c55c5b0d7f5"}
Sep 09 06:06:40.387129 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:40.387095    2579 generic.go:356] "Generic (PLEG): container finished" podID="8128d424-7945-4bb4-bbdb-02be561febb6" containerID="4fdbf581da9e6efc61a921b0bd07f66edaeacd4dc21e29fdb251d927913f4b06" exitCode=0
Sep 09 06:06:40.387508 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:40.387184    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464e41e7-bcrrc" event={"ID":"8128d424-7945-4bb4-bbdb-02be561febb6","Type":"ContainerDied","Data":"4fdbf581da9e6efc61a921b0bd07f66edaeacd4dc21e29fdb251d927913f4b06"}
Sep 09 06:06:41.520511 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:41.520487    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464e41e7-bcrrc"
Sep 09 06:06:41.537029 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:41.537004    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8128d424-7945-4bb4-bbdb-02be561febb6-maas-api-service-ca\") pod \"8128d424-7945-4bb4-bbdb-02be561febb6\" (UID: \"8128d424-7945-4bb4-bbdb-02be561febb6\") "
Sep 09 06:06:41.537380 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:41.537354    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8128d424-7945-4bb4-bbdb-02be561febb6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8128d424-7945-4bb4-bbdb-02be561febb6" (UID: "8128d424-7945-4bb4-bbdb-02be561febb6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:06:41.638275 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:41.638251    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8128d424-7945-4bb4-bbdb-02be561febb6-maas-api-service-ca\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:06:42.399226 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:42.399202    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464e41e7-bcrrc"
Sep 09 06:06:42.399226 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:42.399213    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464e41e7-bcrrc" event={"ID":"8128d424-7945-4bb4-bbdb-02be561febb6","Type":"ContainerDied","Data":"39219cf6cb80f7000347684e488166f8fa512e4eab1036d1d1e58c55c5b0d7f5"}
Sep 09 06:06:42.399419 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:42.399256    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39219cf6cb80f7000347684e488166f8fa512e4eab1036d1d1e58c55c5b0d7f5"
Sep 09 06:06:49.266150 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:49.266115    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-464e41e7-6f6d45f77-dm8t2"]
Sep 09 06:06:49.266757 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:49.266364    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-464e41e7-6f6d45f77-dm8t2" podUID="5eec01a7-f17d-45b1-8c6f-036eff1bd9e5" containerName="payload-processing" containerID="cri-o://f834644924218298a65cbfef324d2c3422f2b16284f56b8c78d82d4ba091554c" gracePeriod=30
Sep 09 06:06:49.274150 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:49.274118    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-464e41e7-6f4bb77674-dvx65"]
Sep 09 06:06:49.274390 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:49.274358    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-464e41e7-6f4bb77674-dvx65" podUID="002ea6f3-16b9-4b6a-8ebf-ef83aa9c7e0c" containerName="payload-pre-processing" containerID="cri-o://1a6da23a74675619bbe12b9e674f683d675ce2c3750595ff2bb9966a5777fa10" gracePeriod=30
Sep 09 06:06:49.429803 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:49.429743    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464e41e7-bcrrc"]
Sep 09 06:06:49.432325 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:49.432295    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464e41e7-bcrrc"]
Sep 09 06:06:49.433711 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:49.433683    2579 generic.go:356] "Generic (PLEG): container finished" podID="002ea6f3-16b9-4b6a-8ebf-ef83aa9c7e0c" containerID="1a6da23a74675619bbe12b9e674f683d675ce2c3750595ff2bb9966a5777fa10" exitCode=0
Sep 09 06:06:49.433879 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:49.433820    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-464e41e7-6f4bb77674-dvx65" event={"ID":"002ea6f3-16b9-4b6a-8ebf-ef83aa9c7e0c","Type":"ContainerDied","Data":"1a6da23a74675619bbe12b9e674f683d675ce2c3750595ff2bb9966a5777fa10"}
Sep 09 06:06:49.435741 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:49.435619    2579 generic.go:356] "Generic (PLEG): container finished" podID="5eec01a7-f17d-45b1-8c6f-036eff1bd9e5" containerID="f834644924218298a65cbfef324d2c3422f2b16284f56b8c78d82d4ba091554c" exitCode=0
Sep 09 06:06:49.435741 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:49.435708    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-464e41e7-6f6d45f77-dm8t2" event={"ID":"5eec01a7-f17d-45b1-8c6f-036eff1bd9e5","Type":"ContainerDied","Data":"f834644924218298a65cbfef324d2c3422f2b16284f56b8c78d82d4ba091554c"}
Sep 09 06:06:49.546669 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:49.546646    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-464e41e7-6f6d45f77-dm8t2"
Sep 09 06:06:49.549382 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:49.549361    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-464e41e7-6f4bb77674-dvx65"
Sep 09 06:06:49.598411 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:49.598383    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8l2dw\" (UniqueName: \"kubernetes.io/projected/002ea6f3-16b9-4b6a-8ebf-ef83aa9c7e0c-kube-api-access-8l2dw\") pod \"002ea6f3-16b9-4b6a-8ebf-ef83aa9c7e0c\" (UID: \"002ea6f3-16b9-4b6a-8ebf-ef83aa9c7e0c\") "
Sep 09 06:06:49.598411 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:49.598412    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eec01a7-f17d-45b1-8c6f-036eff1bd9e5-plugins-config-volume\") pod \"5eec01a7-f17d-45b1-8c6f-036eff1bd9e5\" (UID: \"5eec01a7-f17d-45b1-8c6f-036eff1bd9e5\") "
Sep 09 06:06:49.598573 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:49.598507    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/002ea6f3-16b9-4b6a-8ebf-ef83aa9c7e0c-plugins-config-volume\") pod \"002ea6f3-16b9-4b6a-8ebf-ef83aa9c7e0c\" (UID: \"002ea6f3-16b9-4b6a-8ebf-ef83aa9c7e0c\") "
Sep 09 06:06:49.598573 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:49.598531    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vd98h\" (UniqueName: \"kubernetes.io/projected/5eec01a7-f17d-45b1-8c6f-036eff1bd9e5-kube-api-access-vd98h\") pod \"5eec01a7-f17d-45b1-8c6f-036eff1bd9e5\" (UID: \"5eec01a7-f17d-45b1-8c6f-036eff1bd9e5\") "
Sep 09 06:06:49.598898 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:49.598871    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eec01a7-f17d-45b1-8c6f-036eff1bd9e5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5eec01a7-f17d-45b1-8c6f-036eff1bd9e5" (UID: "5eec01a7-f17d-45b1-8c6f-036eff1bd9e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:06:49.599004 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:49.598905    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/002ea6f3-16b9-4b6a-8ebf-ef83aa9c7e0c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "002ea6f3-16b9-4b6a-8ebf-ef83aa9c7e0c" (UID: "002ea6f3-16b9-4b6a-8ebf-ef83aa9c7e0c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:06:49.600567 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:49.600540    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/002ea6f3-16b9-4b6a-8ebf-ef83aa9c7e0c-kube-api-access-8l2dw" (OuterVolumeSpecName: "kube-api-access-8l2dw") pod "002ea6f3-16b9-4b6a-8ebf-ef83aa9c7e0c" (UID: "002ea6f3-16b9-4b6a-8ebf-ef83aa9c7e0c"). InnerVolumeSpecName "kube-api-access-8l2dw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:06:49.600709 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:49.600690    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eec01a7-f17d-45b1-8c6f-036eff1bd9e5-kube-api-access-vd98h" (OuterVolumeSpecName: "kube-api-access-vd98h") pod "5eec01a7-f17d-45b1-8c6f-036eff1bd9e5" (UID: "5eec01a7-f17d-45b1-8c6f-036eff1bd9e5"). InnerVolumeSpecName "kube-api-access-vd98h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:06:49.699603 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:49.699577    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/002ea6f3-16b9-4b6a-8ebf-ef83aa9c7e0c-plugins-config-volume\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:06:49.699603 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:49.699601    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd98h\" (UniqueName: \"kubernetes.io/projected/5eec01a7-f17d-45b1-8c6f-036eff1bd9e5-kube-api-access-vd98h\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:06:49.699603 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:49.699610    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l2dw\" (UniqueName: \"kubernetes.io/projected/002ea6f3-16b9-4b6a-8ebf-ef83aa9c7e0c-kube-api-access-8l2dw\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:06:49.699813 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:49.699620    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eec01a7-f17d-45b1-8c6f-036eff1bd9e5-plugins-config-volume\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:06:50.442146 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:50.442119    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-464e41e7-6f6d45f77-dm8t2"
Sep 09 06:06:50.442540 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:50.442154    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-464e41e7-6f6d45f77-dm8t2" event={"ID":"5eec01a7-f17d-45b1-8c6f-036eff1bd9e5","Type":"ContainerDied","Data":"80dfececb0b3259580c58897e49b1b27d8a98803548497478eda351fa428050b"}
Sep 09 06:06:50.442540 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:50.442226    2579 scope.go:117] "RemoveContainer" containerID="f834644924218298a65cbfef324d2c3422f2b16284f56b8c78d82d4ba091554c"
Sep 09 06:06:50.443610 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:50.443591    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-464e41e7-6f4bb77674-dvx65"
Sep 09 06:06:50.443711 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:50.443596    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-464e41e7-6f4bb77674-dvx65" event={"ID":"002ea6f3-16b9-4b6a-8ebf-ef83aa9c7e0c","Type":"ContainerDied","Data":"5ced5769b503d5df127c62ed0c1bb0c60bcca330f8309b12e5242d1cce6bf5c3"}
Sep 09 06:06:50.451305 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:50.451281    2579 scope.go:117] "RemoveContainer" containerID="1a6da23a74675619bbe12b9e674f683d675ce2c3750595ff2bb9966a5777fa10"
Sep 09 06:06:50.466619 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:50.466602    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-464e41e7-6f4bb77674-dvx65"]
Sep 09 06:06:50.468592 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:50.468573    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-464e41e7-6f4bb77674-dvx65"]
Sep 09 06:06:50.478142 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:50.478124    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-464e41e7-6f6d45f77-dm8t2"]
Sep 09 06:06:50.481575 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:50.481553    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-464e41e7-6f6d45f77-dm8t2"]
Sep 09 06:06:51.147497 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:51.147464    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="002ea6f3-16b9-4b6a-8ebf-ef83aa9c7e0c" path="/var/lib/kubelet/pods/002ea6f3-16b9-4b6a-8ebf-ef83aa9c7e0c/volumes"
Sep 09 06:06:51.147827 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:51.147814    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5eec01a7-f17d-45b1-8c6f-036eff1bd9e5" path="/var/lib/kubelet/pods/5eec01a7-f17d-45b1-8c6f-036eff1bd9e5/volumes"
Sep 09 06:06:51.148087 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:51.148076    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8128d424-7945-4bb4-bbdb-02be561febb6" path="/var/lib/kubelet/pods/8128d424-7945-4bb4-bbdb-02be561febb6/volumes"
Sep 09 06:06:52.269659 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:52.269617    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-f8bdbb/noten-model-f8bdbb-kserve-69b9b4987f-6fpwp"]
Sep 09 06:06:52.270004 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:52.269922    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="002ea6f3-16b9-4b6a-8ebf-ef83aa9c7e0c" containerName="payload-pre-processing"
Sep 09 06:06:52.270004 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:52.269932    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="002ea6f3-16b9-4b6a-8ebf-ef83aa9c7e0c" containerName="payload-pre-processing"
Sep 09 06:06:52.270004 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:52.269947    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5eec01a7-f17d-45b1-8c6f-036eff1bd9e5" containerName="payload-processing"
Sep 09 06:06:52.270004 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:52.269952    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5eec01a7-f17d-45b1-8c6f-036eff1bd9e5" containerName="payload-processing"
Sep 09 06:06:52.270004 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:52.269959    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8128d424-7945-4bb4-bbdb-02be561febb6" containerName="revoke-keys"
Sep 09 06:06:52.270004 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:52.269965    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8128d424-7945-4bb4-bbdb-02be561febb6" containerName="revoke-keys"
Sep 09 06:06:52.270205 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:52.270021    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8128d424-7945-4bb4-bbdb-02be561febb6" containerName="revoke-keys"
Sep 09 06:06:52.270205 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:52.270030    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="002ea6f3-16b9-4b6a-8ebf-ef83aa9c7e0c" containerName="payload-pre-processing"
Sep 09 06:06:52.270205 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:52.270037    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5eec01a7-f17d-45b1-8c6f-036eff1bd9e5" containerName="payload-processing"
Sep 09 06:06:52.274245 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:52.274229    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-f8bdbb/noten-model-f8bdbb-kserve-69b9b4987f-6fpwp"
Sep 09 06:06:52.276407 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:52.276390    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-f8bdbb\"/\"openshift-service-ca.crt\""
Sep 09 06:06:52.277458 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:52.277441    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-f8bdbb\"/\"default-dockercfg-zlcmg\""
Sep 09 06:06:52.277564 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:52.277473    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-f8bdbb\"/\"noten-model-f8bdbb-kserve-self-signed-certs\""
Sep 09 06:06:52.277564 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:52.277498    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-f8bdbb\"/\"kube-root-ca.crt\""
Sep 09 06:06:52.283254 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:52.283233    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-f8bdbb/noten-model-f8bdbb-kserve-69b9b4987f-6fpwp"]
Sep 09 06:06:52.321473 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:52.321444    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fccee0a-43ae-4f08-9c7a-013128180c11-tls-certs\") pod \"noten-model-f8bdbb-kserve-69b9b4987f-6fpwp\" (UID: \"2fccee0a-43ae-4f08-9c7a-013128180c11\") " pod="ai-tenant-e2e-noten-f8bdbb/noten-model-f8bdbb-kserve-69b9b4987f-6fpwp"
Sep 09 06:06:52.321584 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:52.321477    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5bfg\" (UniqueName: \"kubernetes.io/projected/2fccee0a-43ae-4f08-9c7a-013128180c11-kube-api-access-z5bfg\") pod \"noten-model-f8bdbb-kserve-69b9b4987f-6fpwp\" (UID: \"2fccee0a-43ae-4f08-9c7a-013128180c11\") " pod="ai-tenant-e2e-noten-f8bdbb/noten-model-f8bdbb-kserve-69b9b4987f-6fpwp"
Sep 09 06:06:52.321584 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:52.321502    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fccee0a-43ae-4f08-9c7a-013128180c11-dshm\") pod \"noten-model-f8bdbb-kserve-69b9b4987f-6fpwp\" (UID: \"2fccee0a-43ae-4f08-9c7a-013128180c11\") " pod="ai-tenant-e2e-noten-f8bdbb/noten-model-f8bdbb-kserve-69b9b4987f-6fpwp"
Sep 09 06:06:52.321584 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:52.321516    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fccee0a-43ae-4f08-9c7a-013128180c11-model-cache\") pod \"noten-model-f8bdbb-kserve-69b9b4987f-6fpwp\" (UID: \"2fccee0a-43ae-4f08-9c7a-013128180c11\") " pod="ai-tenant-e2e-noten-f8bdbb/noten-model-f8bdbb-kserve-69b9b4987f-6fpwp"
Sep 09 06:06:52.321584 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:52.321564    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fccee0a-43ae-4f08-9c7a-013128180c11-tmp-dir\") pod \"noten-model-f8bdbb-kserve-69b9b4987f-6fpwp\" (UID: \"2fccee0a-43ae-4f08-9c7a-013128180c11\") " pod="ai-tenant-e2e-noten-f8bdbb/noten-model-f8bdbb-kserve-69b9b4987f-6fpwp"
Sep 09 06:06:52.321769 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:52.321670    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fccee0a-43ae-4f08-9c7a-013128180c11-home\") pod \"noten-model-f8bdbb-kserve-69b9b4987f-6fpwp\" (UID: \"2fccee0a-43ae-4f08-9c7a-013128180c11\") " pod="ai-tenant-e2e-noten-f8bdbb/noten-model-f8bdbb-kserve-69b9b4987f-6fpwp"
Sep 09 06:06:52.422842 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:52.422817    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fccee0a-43ae-4f08-9c7a-013128180c11-tls-certs\") pod \"noten-model-f8bdbb-kserve-69b9b4987f-6fpwp\" (UID: \"2fccee0a-43ae-4f08-9c7a-013128180c11\") " pod="ai-tenant-e2e-noten-f8bdbb/noten-model-f8bdbb-kserve-69b9b4987f-6fpwp"
Sep 09 06:06:52.422938 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:52.422847    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5bfg\" (UniqueName: \"kubernetes.io/projected/2fccee0a-43ae-4f08-9c7a-013128180c11-kube-api-access-z5bfg\") pod \"noten-model-f8bdbb-kserve-69b9b4987f-6fpwp\" (UID: \"2fccee0a-43ae-4f08-9c7a-013128180c11\") " pod="ai-tenant-e2e-noten-f8bdbb/noten-model-f8bdbb-kserve-69b9b4987f-6fpwp"
Sep 09 06:06:52.422938 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:52.422872    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fccee0a-43ae-4f08-9c7a-013128180c11-dshm\") pod \"noten-model-f8bdbb-kserve-69b9b4987f-6fpwp\" (UID: \"2fccee0a-43ae-4f08-9c7a-013128180c11\") " pod="ai-tenant-e2e-noten-f8bdbb/noten-model-f8bdbb-kserve-69b9b4987f-6fpwp"
Sep 09 06:06:52.422938 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:52.422887    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fccee0a-43ae-4f08-9c7a-013128180c11-model-cache\") pod \"noten-model-f8bdbb-kserve-69b9b4987f-6fpwp\" (UID: \"2fccee0a-43ae-4f08-9c7a-013128180c11\") " pod="ai-tenant-e2e-noten-f8bdbb/noten-model-f8bdbb-kserve-69b9b4987f-6fpwp"
Sep 09 06:06:52.422938 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:52.422904    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fccee0a-43ae-4f08-9c7a-013128180c11-tmp-dir\") pod \"noten-model-f8bdbb-kserve-69b9b4987f-6fpwp\" (UID: \"2fccee0a-43ae-4f08-9c7a-013128180c11\") " pod="ai-tenant-e2e-noten-f8bdbb/noten-model-f8bdbb-kserve-69b9b4987f-6fpwp"
Sep 09 06:06:52.423151 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:52.422956    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fccee0a-43ae-4f08-9c7a-013128180c11-home\") pod \"noten-model-f8bdbb-kserve-69b9b4987f-6fpwp\" (UID: \"2fccee0a-43ae-4f08-9c7a-013128180c11\") " pod="ai-tenant-e2e-noten-f8bdbb/noten-model-f8bdbb-kserve-69b9b4987f-6fpwp"
Sep 09 06:06:52.423327 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:52.423298    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fccee0a-43ae-4f08-9c7a-013128180c11-model-cache\") pod \"noten-model-f8bdbb-kserve-69b9b4987f-6fpwp\" (UID: \"2fccee0a-43ae-4f08-9c7a-013128180c11\") " pod="ai-tenant-e2e-noten-f8bdbb/noten-model-f8bdbb-kserve-69b9b4987f-6fpwp"
Sep 09 06:06:52.423442 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:52.423354    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fccee0a-43ae-4f08-9c7a-013128180c11-tmp-dir\") pod \"noten-model-f8bdbb-kserve-69b9b4987f-6fpwp\" (UID: \"2fccee0a-43ae-4f08-9c7a-013128180c11\") " pod="ai-tenant-e2e-noten-f8bdbb/noten-model-f8bdbb-kserve-69b9b4987f-6fpwp"
Sep 09 06:06:52.423442 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:52.423357    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fccee0a-43ae-4f08-9c7a-013128180c11-home\") pod \"noten-model-f8bdbb-kserve-69b9b4987f-6fpwp\" (UID: \"2fccee0a-43ae-4f08-9c7a-013128180c11\") " pod="ai-tenant-e2e-noten-f8bdbb/noten-model-f8bdbb-kserve-69b9b4987f-6fpwp"
Sep 09 06:06:52.425109 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:52.425087    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fccee0a-43ae-4f08-9c7a-013128180c11-dshm\") pod \"noten-model-f8bdbb-kserve-69b9b4987f-6fpwp\" (UID: \"2fccee0a-43ae-4f08-9c7a-013128180c11\") " pod="ai-tenant-e2e-noten-f8bdbb/noten-model-f8bdbb-kserve-69b9b4987f-6fpwp"
Sep 09 06:06:52.425348 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:52.425331    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fccee0a-43ae-4f08-9c7a-013128180c11-tls-certs\") pod \"noten-model-f8bdbb-kserve-69b9b4987f-6fpwp\" (UID: \"2fccee0a-43ae-4f08-9c7a-013128180c11\") " pod="ai-tenant-e2e-noten-f8bdbb/noten-model-f8bdbb-kserve-69b9b4987f-6fpwp"
Sep 09 06:06:52.430855 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:52.430838    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5bfg\" (UniqueName: \"kubernetes.io/projected/2fccee0a-43ae-4f08-9c7a-013128180c11-kube-api-access-z5bfg\") pod \"noten-model-f8bdbb-kserve-69b9b4987f-6fpwp\" (UID: \"2fccee0a-43ae-4f08-9c7a-013128180c11\") " pod="ai-tenant-e2e-noten-f8bdbb/noten-model-f8bdbb-kserve-69b9b4987f-6fpwp"
Sep 09 06:06:52.583660 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:52.583621    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-f8bdbb/noten-model-f8bdbb-kserve-69b9b4987f-6fpwp"
Sep 09 06:06:52.855472 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:52.855448    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-f8bdbb/noten-model-f8bdbb-kserve-69b9b4987f-6fpwp"]
Sep 09 06:06:52.857985 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:06:52.857958    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2fccee0a_43ae_4f08_9c7a_013128180c11.slice/crio-40200ee9c545c8f4f627b3a07f97a60283d4cb8f7a5f57af40df8e7014fb8ded WatchSource:0}: Error finding container 40200ee9c545c8f4f627b3a07f97a60283d4cb8f7a5f57af40df8e7014fb8ded: Status 404 returned error can't find the container with id 40200ee9c545c8f4f627b3a07f97a60283d4cb8f7a5f57af40df8e7014fb8ded
Sep 09 06:06:53.463485 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:53.462659    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-f8bdbb/noten-model-f8bdbb-kserve-69b9b4987f-6fpwp" event={"ID":"2fccee0a-43ae-4f08-9c7a-013128180c11","Type":"ContainerStarted","Data":"c087a6a689cde7d06ee4ae4495f77c2ae5217d2e33f1e187b262328f9d944623"}
Sep 09 06:06:53.463485 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:53.462722    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-f8bdbb/noten-model-f8bdbb-kserve-69b9b4987f-6fpwp" event={"ID":"2fccee0a-43ae-4f08-9c7a-013128180c11","Type":"ContainerStarted","Data":"40200ee9c545c8f4f627b3a07f97a60283d4cb8f7a5f57af40df8e7014fb8ded"}
Sep 09 06:06:53.481116 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:53.481070    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-f8bdbb/noten-model-f8bdbb-kserve-69b9b4987f-6fpwp" podStartSLOduration=1.481055786 podStartE2EDuration="1.481055786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:06:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:06:53.478668786 +0000 UTC m=+1038.819156700" watchObservedRunningTime="2026-09-09 06:06:53.481055786 +0000 UTC m=+1038.821543695"
Sep 09 06:06:57.296169 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:06:57.296139    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:07:02.584601 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:02.584566    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-f8bdbb/noten-model-f8bdbb-kserve-69b9b4987f-6fpwp"
Sep 09 06:07:02.585025 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:02.584855    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-f8bdbb/noten-model-f8bdbb-kserve-69b9b4987f-6fpwp"
Sep 09 06:07:02.597373 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:02.597353    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-f8bdbb/noten-model-f8bdbb-kserve-69b9b4987f-6fpwp"
Sep 09 06:07:02.864393 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:02.864324    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-f8bdbb/noten-model-f8bdbb-kserve-69b9b4987f-6fpwp"]
Sep 09 06:07:03.520500 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:03.520474    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-f8bdbb/noten-model-f8bdbb-kserve-69b9b4987f-6fpwp"
Sep 09 06:07:03.611762 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:07:03.611732    2579 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-f8bdbb/noten-model-f8bdbb-kserve-self-signed-certs: secret "noten-model-f8bdbb-kserve-self-signed-certs" not found
Sep 09 06:07:03.612112 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:07:03.611819    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fccee0a-43ae-4f08-9c7a-013128180c11-tls-certs podName:2fccee0a-43ae-4f08-9c7a-013128180c11 nodeName:}" failed. No retries permitted until 2026-09-09 06:07:04.111795403 +0000 UTC m=+1049.452283312 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2fccee0a-43ae-4f08-9c7a-013128180c11-tls-certs") pod "noten-model-f8bdbb-kserve-69b9b4987f-6fpwp" (UID: "2fccee0a-43ae-4f08-9c7a-013128180c11") : secret "noten-model-f8bdbb-kserve-self-signed-certs" not found
Sep 09 06:07:04.114914 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:07:04.114876    2579 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-f8bdbb/noten-model-f8bdbb-kserve-self-signed-certs: secret "noten-model-f8bdbb-kserve-self-signed-certs" not found
Sep 09 06:07:04.115076 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:07:04.114958    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fccee0a-43ae-4f08-9c7a-013128180c11-tls-certs podName:2fccee0a-43ae-4f08-9c7a-013128180c11 nodeName:}" failed. No retries permitted until 2026-09-09 06:07:05.114940963 +0000 UTC m=+1050.455428851 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2fccee0a-43ae-4f08-9c7a-013128180c11-tls-certs") pod "noten-model-f8bdbb-kserve-69b9b4987f-6fpwp" (UID: "2fccee0a-43ae-4f08-9c7a-013128180c11") : secret "noten-model-f8bdbb-kserve-self-signed-certs" not found
Sep 09 06:07:05.122216 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:07:05.122186    2579 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-f8bdbb/noten-model-f8bdbb-kserve-self-signed-certs: secret "noten-model-f8bdbb-kserve-self-signed-certs" not found
Sep 09 06:07:05.122576 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:07:05.122272    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fccee0a-43ae-4f08-9c7a-013128180c11-tls-certs podName:2fccee0a-43ae-4f08-9c7a-013128180c11 nodeName:}" failed. No retries permitted until 2026-09-09 06:07:07.12225727 +0000 UTC m=+1052.462745157 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2fccee0a-43ae-4f08-9c7a-013128180c11-tls-certs") pod "noten-model-f8bdbb-kserve-69b9b4987f-6fpwp" (UID: "2fccee0a-43ae-4f08-9c7a-013128180c11") : secret "noten-model-f8bdbb-kserve-self-signed-certs" not found
Sep 09 06:07:07.139423 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:07:07.139391    2579 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-f8bdbb/noten-model-f8bdbb-kserve-self-signed-certs: secret "noten-model-f8bdbb-kserve-self-signed-certs" not found
Sep 09 06:07:07.139900 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:07:07.139459    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fccee0a-43ae-4f08-9c7a-013128180c11-tls-certs podName:2fccee0a-43ae-4f08-9c7a-013128180c11 nodeName:}" failed. No retries permitted until 2026-09-09 06:07:11.139444867 +0000 UTC m=+1056.479932759 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2fccee0a-43ae-4f08-9c7a-013128180c11-tls-certs") pod "noten-model-f8bdbb-kserve-69b9b4987f-6fpwp" (UID: "2fccee0a-43ae-4f08-9c7a-013128180c11") : secret "noten-model-f8bdbb-kserve-self-signed-certs" not found
Sep 09 06:07:11.176205 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:07:11.176169    2579 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-f8bdbb/noten-model-f8bdbb-kserve-self-signed-certs: secret "noten-model-f8bdbb-kserve-self-signed-certs" not found
Sep 09 06:07:11.176685 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:07:11.176276    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fccee0a-43ae-4f08-9c7a-013128180c11-tls-certs podName:2fccee0a-43ae-4f08-9c7a-013128180c11 nodeName:}" failed. No retries permitted until 2026-09-09 06:07:19.17625716 +0000 UTC m=+1064.516745063 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2fccee0a-43ae-4f08-9c7a-013128180c11-tls-certs") pod "noten-model-f8bdbb-kserve-69b9b4987f-6fpwp" (UID: "2fccee0a-43ae-4f08-9c7a-013128180c11") : secret "noten-model-f8bdbb-kserve-self-signed-certs" not found
Sep 09 06:07:19.240763 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:07:19.240732    2579 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-f8bdbb/noten-model-f8bdbb-kserve-self-signed-certs: secret "noten-model-f8bdbb-kserve-self-signed-certs" not found
Sep 09 06:07:19.241153 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:07:19.240803    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fccee0a-43ae-4f08-9c7a-013128180c11-tls-certs podName:2fccee0a-43ae-4f08-9c7a-013128180c11 nodeName:}" failed. No retries permitted until 2026-09-09 06:07:35.240789619 +0000 UTC m=+1080.581277507 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2fccee0a-43ae-4f08-9c7a-013128180c11-tls-certs") pod "noten-model-f8bdbb-kserve-69b9b4987f-6fpwp" (UID: "2fccee0a-43ae-4f08-9c7a-013128180c11") : secret "noten-model-f8bdbb-kserve-self-signed-certs" not found
Sep 09 06:07:19.532846 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:19.532767    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-f8bdbb/noten-model-f8bdbb-kserve-69b9b4987f-6fpwp" podUID="2fccee0a-43ae-4f08-9c7a-013128180c11" containerName="main" containerID="cri-o://c087a6a689cde7d06ee4ae4495f77c2ae5217d2e33f1e187b262328f9d944623" gracePeriod=45
Sep 09 06:07:19.774571 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:19.774550    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-f8bdbb/noten-model-f8bdbb-kserve-69b9b4987f-6fpwp"
Sep 09 06:07:19.946167 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:19.946136    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fccee0a-43ae-4f08-9c7a-013128180c11-dshm\") pod \"2fccee0a-43ae-4f08-9c7a-013128180c11\" (UID: \"2fccee0a-43ae-4f08-9c7a-013128180c11\") "
Sep 09 06:07:19.946316 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:19.946185    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fccee0a-43ae-4f08-9c7a-013128180c11-tls-certs\") pod \"2fccee0a-43ae-4f08-9c7a-013128180c11\" (UID: \"2fccee0a-43ae-4f08-9c7a-013128180c11\") "
Sep 09 06:07:19.946316 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:19.946238    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fccee0a-43ae-4f08-9c7a-013128180c11-model-cache\") pod \"2fccee0a-43ae-4f08-9c7a-013128180c11\" (UID: \"2fccee0a-43ae-4f08-9c7a-013128180c11\") "
Sep 09 06:07:19.946316 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:19.946285    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5bfg\" (UniqueName: \"kubernetes.io/projected/2fccee0a-43ae-4f08-9c7a-013128180c11-kube-api-access-z5bfg\") pod \"2fccee0a-43ae-4f08-9c7a-013128180c11\" (UID: \"2fccee0a-43ae-4f08-9c7a-013128180c11\") "
Sep 09 06:07:19.946491 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:19.946331    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fccee0a-43ae-4f08-9c7a-013128180c11-tmp-dir\") pod \"2fccee0a-43ae-4f08-9c7a-013128180c11\" (UID: \"2fccee0a-43ae-4f08-9c7a-013128180c11\") "
Sep 09 06:07:19.946491 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:19.946351    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fccee0a-43ae-4f08-9c7a-013128180c11-home\") pod \"2fccee0a-43ae-4f08-9c7a-013128180c11\" (UID: \"2fccee0a-43ae-4f08-9c7a-013128180c11\") "
Sep 09 06:07:19.946591 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:19.946514    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fccee0a-43ae-4f08-9c7a-013128180c11-model-cache" (OuterVolumeSpecName: "model-cache") pod "2fccee0a-43ae-4f08-9c7a-013128180c11" (UID: "2fccee0a-43ae-4f08-9c7a-013128180c11"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:07:19.946739 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:19.946708    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fccee0a-43ae-4f08-9c7a-013128180c11-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2fccee0a-43ae-4f08-9c7a-013128180c11" (UID: "2fccee0a-43ae-4f08-9c7a-013128180c11"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:07:19.946739 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:19.946724    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fccee0a-43ae-4f08-9c7a-013128180c11-home" (OuterVolumeSpecName: "home") pod "2fccee0a-43ae-4f08-9c7a-013128180c11" (UID: "2fccee0a-43ae-4f08-9c7a-013128180c11"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:07:19.946875 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:19.946725    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fccee0a-43ae-4f08-9c7a-013128180c11-model-cache\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:07:19.948519 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:19.948496    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fccee0a-43ae-4f08-9c7a-013128180c11-dshm" (OuterVolumeSpecName: "dshm") pod "2fccee0a-43ae-4f08-9c7a-013128180c11" (UID: "2fccee0a-43ae-4f08-9c7a-013128180c11"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:07:19.948602 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:19.948561    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fccee0a-43ae-4f08-9c7a-013128180c11-kube-api-access-z5bfg" (OuterVolumeSpecName: "kube-api-access-z5bfg") pod "2fccee0a-43ae-4f08-9c7a-013128180c11" (UID: "2fccee0a-43ae-4f08-9c7a-013128180c11"). InnerVolumeSpecName "kube-api-access-z5bfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:07:19.948602 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:19.948575    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fccee0a-43ae-4f08-9c7a-013128180c11-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2fccee0a-43ae-4f08-9c7a-013128180c11" (UID: "2fccee0a-43ae-4f08-9c7a-013128180c11"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:07:20.047471 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:20.047447    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5bfg\" (UniqueName: \"kubernetes.io/projected/2fccee0a-43ae-4f08-9c7a-013128180c11-kube-api-access-z5bfg\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:07:20.047471 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:20.047470    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fccee0a-43ae-4f08-9c7a-013128180c11-tmp-dir\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:07:20.047699 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:20.047480    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fccee0a-43ae-4f08-9c7a-013128180c11-home\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:07:20.047699 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:20.047487    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fccee0a-43ae-4f08-9c7a-013128180c11-dshm\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:07:20.047699 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:20.047494    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fccee0a-43ae-4f08-9c7a-013128180c11-tls-certs\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:07:20.587565 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:20.587538    2579 generic.go:356] "Generic (PLEG): container finished" podID="2fccee0a-43ae-4f08-9c7a-013128180c11" containerID="c087a6a689cde7d06ee4ae4495f77c2ae5217d2e33f1e187b262328f9d944623" exitCode=0
Sep 09 06:07:20.587974 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:20.587581    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-f8bdbb/noten-model-f8bdbb-kserve-69b9b4987f-6fpwp"
Sep 09 06:07:20.587974 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:20.587645    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-f8bdbb/noten-model-f8bdbb-kserve-69b9b4987f-6fpwp" event={"ID":"2fccee0a-43ae-4f08-9c7a-013128180c11","Type":"ContainerDied","Data":"c087a6a689cde7d06ee4ae4495f77c2ae5217d2e33f1e187b262328f9d944623"}
Sep 09 06:07:20.587974 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:20.587686    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-f8bdbb/noten-model-f8bdbb-kserve-69b9b4987f-6fpwp" event={"ID":"2fccee0a-43ae-4f08-9c7a-013128180c11","Type":"ContainerDied","Data":"40200ee9c545c8f4f627b3a07f97a60283d4cb8f7a5f57af40df8e7014fb8ded"}
Sep 09 06:07:20.587974 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:20.587703    2579 scope.go:117] "RemoveContainer" containerID="c087a6a689cde7d06ee4ae4495f77c2ae5217d2e33f1e187b262328f9d944623"
Sep 09 06:07:20.596875 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:20.596861    2579 scope.go:117] "RemoveContainer" containerID="c087a6a689cde7d06ee4ae4495f77c2ae5217d2e33f1e187b262328f9d944623"
Sep 09 06:07:20.597113 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:07:20.597094    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c087a6a689cde7d06ee4ae4495f77c2ae5217d2e33f1e187b262328f9d944623\": container with ID starting with c087a6a689cde7d06ee4ae4495f77c2ae5217d2e33f1e187b262328f9d944623 not found: ID does not exist" containerID="c087a6a689cde7d06ee4ae4495f77c2ae5217d2e33f1e187b262328f9d944623"
Sep 09 06:07:20.597160 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:20.597122    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c087a6a689cde7d06ee4ae4495f77c2ae5217d2e33f1e187b262328f9d944623"} err="failed to get container status \"c087a6a689cde7d06ee4ae4495f77c2ae5217d2e33f1e187b262328f9d944623\": rpc error: code = NotFound desc = could not find container \"c087a6a689cde7d06ee4ae4495f77c2ae5217d2e33f1e187b262328f9d944623\": container with ID starting with c087a6a689cde7d06ee4ae4495f77c2ae5217d2e33f1e187b262328f9d944623 not found: ID does not exist"
Sep 09 06:07:20.608006 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:20.607984    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-f8bdbb/noten-model-f8bdbb-kserve-69b9b4987f-6fpwp"]
Sep 09 06:07:20.611348 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:20.611332    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-f8bdbb/noten-model-f8bdbb-kserve-69b9b4987f-6fpwp"]
Sep 09 06:07:21.148511 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:21.148479    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fccee0a-43ae-4f08-9c7a-013128180c11" path="/var/lib/kubelet/pods/2fccee0a-43ae-4f08-9c7a-013128180c11/volumes"
Sep 09 06:07:23.401336 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:23.401301    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:07:43.285300 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:43.285260    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-750c68-6b9b48496f-wkb4k"]
Sep 09 06:07:43.285778 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:43.285752    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fccee0a-43ae-4f08-9c7a-013128180c11" containerName="main"
Sep 09 06:07:43.285778 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:43.285769    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fccee0a-43ae-4f08-9c7a-013128180c11" containerName="main"
Sep 09 06:07:43.285893 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:43.285849    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fccee0a-43ae-4f08-9c7a-013128180c11" containerName="main"
Sep 09 06:07:43.290051 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:43.290028    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-750c68-6b9b48496f-wkb4k"
Sep 09 06:07:43.292212 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:43.292192    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-750c68-dockercfg-klknn\""
Sep 09 06:07:43.292333 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:43.292248    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-750c68\""
Sep 09 06:07:43.301079 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:43.301024    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-750c68-6b9b48496f-wkb4k"]
Sep 09 06:07:43.321427 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:43.321396    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05730de2-6196-49fc-af43-cb81e8ec0160-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-750c68-6b9b48496f-wkb4k\" (UID: \"05730de2-6196-49fc-af43-cb81e8ec0160\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-750c68-6b9b48496f-wkb4k"
Sep 09 06:07:43.321524 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:43.321475    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn8tr\" (UniqueName: \"kubernetes.io/projected/05730de2-6196-49fc-af43-cb81e8ec0160-kube-api-access-qn8tr\") pod \"payload-processing-e2e-shared-b-w2-750c68-6b9b48496f-wkb4k\" (UID: \"05730de2-6196-49fc-af43-cb81e8ec0160\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-750c68-6b9b48496f-wkb4k"
Sep 09 06:07:43.422704 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:43.422676    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn8tr\" (UniqueName: \"kubernetes.io/projected/05730de2-6196-49fc-af43-cb81e8ec0160-kube-api-access-qn8tr\") pod \"payload-processing-e2e-shared-b-w2-750c68-6b9b48496f-wkb4k\" (UID: \"05730de2-6196-49fc-af43-cb81e8ec0160\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-750c68-6b9b48496f-wkb4k"
Sep 09 06:07:43.422806 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:43.422726    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05730de2-6196-49fc-af43-cb81e8ec0160-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-750c68-6b9b48496f-wkb4k\" (UID: \"05730de2-6196-49fc-af43-cb81e8ec0160\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-750c68-6b9b48496f-wkb4k"
Sep 09 06:07:43.423345 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:43.423329    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05730de2-6196-49fc-af43-cb81e8ec0160-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-750c68-6b9b48496f-wkb4k\" (UID: \"05730de2-6196-49fc-af43-cb81e8ec0160\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-750c68-6b9b48496f-wkb4k"
Sep 09 06:07:43.430285 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:43.430265    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn8tr\" (UniqueName: \"kubernetes.io/projected/05730de2-6196-49fc-af43-cb81e8ec0160-kube-api-access-qn8tr\") pod \"payload-processing-e2e-shared-b-w2-750c68-6b9b48496f-wkb4k\" (UID: \"05730de2-6196-49fc-af43-cb81e8ec0160\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-750c68-6b9b48496f-wkb4k"
Sep 09 06:07:43.603437 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:43.603414    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-750c68-6b9b48496f-wkb4k"
Sep 09 06:07:43.871292 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:43.871243    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-750c68-6b9b48496f-wkb4k"]
Sep 09 06:07:43.873894 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:07:43.873868    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05730de2_6196_49fc_af43_cb81e8ec0160.slice/crio-b9de9cb2941d054adca7ff21efb9d78a97ae7f6b2494e90a951b3880157ce539 WatchSource:0}: Error finding container b9de9cb2941d054adca7ff21efb9d78a97ae7f6b2494e90a951b3880157ce539: Status 404 returned error can't find the container with id b9de9cb2941d054adca7ff21efb9d78a97ae7f6b2494e90a951b3880157ce539
Sep 09 06:07:44.407825 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:44.407798    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:07:44.708216 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:44.708126    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-750c68-6b9b48496f-wkb4k" event={"ID":"05730de2-6196-49fc-af43-cb81e8ec0160","Type":"ContainerStarted","Data":"8b28864fd361887619d1a4d6256369620d2583917afed555c0e3bb1af62b192a"}
Sep 09 06:07:44.708216 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:44.708180    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-750c68-6b9b48496f-wkb4k" event={"ID":"05730de2-6196-49fc-af43-cb81e8ec0160","Type":"ContainerStarted","Data":"b9de9cb2941d054adca7ff21efb9d78a97ae7f6b2494e90a951b3880157ce539"}
Sep 09 06:07:44.708406 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:44.708230    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-750c68-6b9b48496f-wkb4k"
Sep 09 06:07:44.725159 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:44.725109    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-750c68-6b9b48496f-wkb4k" podStartSLOduration=1.725090886 podStartE2EDuration="1.725090886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:07:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:07:44.724900776 +0000 UTC m=+1090.065388686" watchObservedRunningTime="2026-09-09 06:07:44.725090886 +0000 UTC m=+1090.065578796"
Sep 09 06:07:46.917765 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:46.917734    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-43eb2d/distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7"]
Sep 09 06:07:46.921091 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:46.921076    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-43eb2d/distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7"
Sep 09 06:07:46.923155 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:46.923137    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-43eb2d\"/\"distinct-2-w4-43eb2d-kserve-self-signed-certs\""
Sep 09 06:07:46.928740 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:46.928720    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-43eb2d/distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7"]
Sep 09 06:07:46.945267 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:46.945246    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa204694-c894-4f13-b422-f6726b3db3c8-home\") pod \"distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7\" (UID: \"fa204694-c894-4f13-b422-f6726b3db3c8\") " pod="e2e-models-e2e-worker-w4-43eb2d/distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7"
Sep 09 06:07:46.945367 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:46.945276    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzjpc\" (UniqueName: \"kubernetes.io/projected/fa204694-c894-4f13-b422-f6726b3db3c8-kube-api-access-mzjpc\") pod \"distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7\" (UID: \"fa204694-c894-4f13-b422-f6726b3db3c8\") " pod="e2e-models-e2e-worker-w4-43eb2d/distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7"
Sep 09 06:07:46.945367 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:46.945313    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa204694-c894-4f13-b422-f6726b3db3c8-dshm\") pod \"distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7\" (UID: \"fa204694-c894-4f13-b422-f6726b3db3c8\") " pod="e2e-models-e2e-worker-w4-43eb2d/distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7"
Sep 09 06:07:46.945450 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:46.945365    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa204694-c894-4f13-b422-f6726b3db3c8-tmp-dir\") pod \"distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7\" (UID: \"fa204694-c894-4f13-b422-f6726b3db3c8\") " pod="e2e-models-e2e-worker-w4-43eb2d/distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7"
Sep 09 06:07:46.945450 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:46.945392    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa204694-c894-4f13-b422-f6726b3db3c8-model-cache\") pod \"distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7\" (UID: \"fa204694-c894-4f13-b422-f6726b3db3c8\") " pod="e2e-models-e2e-worker-w4-43eb2d/distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7"
Sep 09 06:07:46.945450 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:46.945409    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa204694-c894-4f13-b422-f6726b3db3c8-tls-certs\") pod \"distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7\" (UID: \"fa204694-c894-4f13-b422-f6726b3db3c8\") " pod="e2e-models-e2e-worker-w4-43eb2d/distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7"
Sep 09 06:07:47.045908 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:47.045876    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa204694-c894-4f13-b422-f6726b3db3c8-home\") pod \"distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7\" (UID: \"fa204694-c894-4f13-b422-f6726b3db3c8\") " pod="e2e-models-e2e-worker-w4-43eb2d/distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7"
Sep 09 06:07:47.045908 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:47.045909    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzjpc\" (UniqueName: \"kubernetes.io/projected/fa204694-c894-4f13-b422-f6726b3db3c8-kube-api-access-mzjpc\") pod \"distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7\" (UID: \"fa204694-c894-4f13-b422-f6726b3db3c8\") " pod="e2e-models-e2e-worker-w4-43eb2d/distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7"
Sep 09 06:07:47.046101 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:47.046029    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa204694-c894-4f13-b422-f6726b3db3c8-dshm\") pod \"distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7\" (UID: \"fa204694-c894-4f13-b422-f6726b3db3c8\") " pod="e2e-models-e2e-worker-w4-43eb2d/distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7"
Sep 09 06:07:47.046143 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:47.046102    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa204694-c894-4f13-b422-f6726b3db3c8-tmp-dir\") pod \"distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7\" (UID: \"fa204694-c894-4f13-b422-f6726b3db3c8\") " pod="e2e-models-e2e-worker-w4-43eb2d/distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7"
Sep 09 06:07:47.046143 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:47.046133    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa204694-c894-4f13-b422-f6726b3db3c8-model-cache\") pod \"distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7\" (UID: \"fa204694-c894-4f13-b422-f6726b3db3c8\") " pod="e2e-models-e2e-worker-w4-43eb2d/distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7"
Sep 09 06:07:47.046309 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:47.046149    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa204694-c894-4f13-b422-f6726b3db3c8-tls-certs\") pod \"distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7\" (UID: \"fa204694-c894-4f13-b422-f6726b3db3c8\") " pod="e2e-models-e2e-worker-w4-43eb2d/distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7"
Sep 09 06:07:47.046309 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:47.046232    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa204694-c894-4f13-b422-f6726b3db3c8-home\") pod \"distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7\" (UID: \"fa204694-c894-4f13-b422-f6726b3db3c8\") " pod="e2e-models-e2e-worker-w4-43eb2d/distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7"
Sep 09 06:07:47.046478 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:47.046459    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa204694-c894-4f13-b422-f6726b3db3c8-tmp-dir\") pod \"distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7\" (UID: \"fa204694-c894-4f13-b422-f6726b3db3c8\") " pod="e2e-models-e2e-worker-w4-43eb2d/distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7"
Sep 09 06:07:47.046534 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:47.046493    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa204694-c894-4f13-b422-f6726b3db3c8-model-cache\") pod \"distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7\" (UID: \"fa204694-c894-4f13-b422-f6726b3db3c8\") " pod="e2e-models-e2e-worker-w4-43eb2d/distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7"
Sep 09 06:07:47.048316 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:47.048298    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa204694-c894-4f13-b422-f6726b3db3c8-dshm\") pod \"distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7\" (UID: \"fa204694-c894-4f13-b422-f6726b3db3c8\") " pod="e2e-models-e2e-worker-w4-43eb2d/distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7"
Sep 09 06:07:47.048639 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:47.048609    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa204694-c894-4f13-b422-f6726b3db3c8-tls-certs\") pod \"distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7\" (UID: \"fa204694-c894-4f13-b422-f6726b3db3c8\") " pod="e2e-models-e2e-worker-w4-43eb2d/distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7"
Sep 09 06:07:47.057057 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:47.057036    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzjpc\" (UniqueName: \"kubernetes.io/projected/fa204694-c894-4f13-b422-f6726b3db3c8-kube-api-access-mzjpc\") pod \"distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7\" (UID: \"fa204694-c894-4f13-b422-f6726b3db3c8\") " pod="e2e-models-e2e-worker-w4-43eb2d/distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7"
Sep 09 06:07:47.230420 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:47.230364    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-43eb2d/distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7"
Sep 09 06:07:47.499970 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:47.499906    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-43eb2d/distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7"]
Sep 09 06:07:47.502071 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:07:47.502041    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa204694_c894_4f13_b422_f6726b3db3c8.slice/crio-c9d56e132c06ce2e928e0664bfe186193bb4d0064f361cb30eca9032763c489d WatchSource:0}: Error finding container c9d56e132c06ce2e928e0664bfe186193bb4d0064f361cb30eca9032763c489d: Status 404 returned error can't find the container with id c9d56e132c06ce2e928e0664bfe186193bb4d0064f361cb30eca9032763c489d
Sep 09 06:07:47.724134 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:47.724104    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-43eb2d/distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7" event={"ID":"fa204694-c894-4f13-b422-f6726b3db3c8","Type":"ContainerStarted","Data":"e36a83e9a47ea4d6982f95c45cf90ee124e71ffbcab12ef6fdabbfb54d07e79f"}
Sep 09 06:07:47.724286 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:47.724142    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-43eb2d/distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7" event={"ID":"fa204694-c894-4f13-b422-f6726b3db3c8","Type":"ContainerStarted","Data":"c9d56e132c06ce2e928e0664bfe186193bb4d0064f361cb30eca9032763c489d"}
Sep 09 06:07:47.739791 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:47.739744    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-43eb2d/distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7" podStartSLOduration=1.739731302 podStartE2EDuration="1.739731302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:07:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:07:47.738960536 +0000 UTC m=+1093.079448447" watchObservedRunningTime="2026-09-09 06:07:47.739731302 +0000 UTC m=+1093.080219212"
Sep 09 06:07:55.714698 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:55.714670    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-750c68-6b9b48496f-wkb4k"
Sep 09 06:07:57.230467 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:57.230435    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-43eb2d/distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7"
Sep 09 06:07:57.230901 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:57.230485    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-43eb2d/distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7"
Sep 09 06:07:57.247285 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:57.247260    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-43eb2d/distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7"
Sep 09 06:07:57.784635 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:07:57.784608    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-43eb2d/distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7"
Sep 09 06:08:01.181849 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:01.181819    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ecd13e-hpwqz"]
Sep 09 06:08:01.185243 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:01.185225    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ecd13e-hpwqz"
Sep 09 06:08:01.187425 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:01.187406    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ksjk6\""
Sep 09 06:08:01.191810 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:01.191787    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ecd13e-hpwqz"]
Sep 09 06:08:01.266283 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:01.266255    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1cf1896-ee8f-41a8-a7bb-628e35b1c4fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ecd13e-hpwqz\" (UID: \"e1cf1896-ee8f-41a8-a7bb-628e35b1c4fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ecd13e-hpwqz"
Sep 09 06:08:01.367191 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:01.367160    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1cf1896-ee8f-41a8-a7bb-628e35b1c4fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ecd13e-hpwqz\" (UID: \"e1cf1896-ee8f-41a8-a7bb-628e35b1c4fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ecd13e-hpwqz"
Sep 09 06:08:01.367754 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:01.367736    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1cf1896-ee8f-41a8-a7bb-628e35b1c4fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ecd13e-hpwqz\" (UID: \"e1cf1896-ee8f-41a8-a7bb-628e35b1c4fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ecd13e-hpwqz"
Sep 09 06:08:01.497121 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:01.497049    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ecd13e-hpwqz"
Sep 09 06:08:01.764685 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:01.764517    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ecd13e-hpwqz"]
Sep 09 06:08:01.767195 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:08:01.767166    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1cf1896_ee8f_41a8_a7bb_628e35b1c4fd.slice/crio-7250afb8364fdbfcb58f853d84ccbed4dad95dae7a144107977c400cd6aa81a7 WatchSource:0}: Error finding container 7250afb8364fdbfcb58f853d84ccbed4dad95dae7a144107977c400cd6aa81a7: Status 404 returned error can't find the container with id 7250afb8364fdbfcb58f853d84ccbed4dad95dae7a144107977c400cd6aa81a7
Sep 09 06:08:01.794834 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:01.794810    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ecd13e-hpwqz" event={"ID":"e1cf1896-ee8f-41a8-a7bb-628e35b1c4fd","Type":"ContainerStarted","Data":"7250afb8364fdbfcb58f853d84ccbed4dad95dae7a144107977c400cd6aa81a7"}
Sep 09 06:08:02.801770 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:02.801675    2579 generic.go:356] "Generic (PLEG): container finished" podID="e1cf1896-ee8f-41a8-a7bb-628e35b1c4fd" containerID="bbe6d90e91eac90908b18d88125501397a57f27b2061a05aa9f4ba5f4e0e35e3" exitCode=0
Sep 09 06:08:02.802137 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:02.801768    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ecd13e-hpwqz" event={"ID":"e1cf1896-ee8f-41a8-a7bb-628e35b1c4fd","Type":"ContainerDied","Data":"bbe6d90e91eac90908b18d88125501397a57f27b2061a05aa9f4ba5f4e0e35e3"}
Sep 09 06:08:03.938173 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:03.938154    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ecd13e-hpwqz"
Sep 09 06:08:03.985802 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:03.985770    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1cf1896-ee8f-41a8-a7bb-628e35b1c4fd-maas-api-service-ca\") pod \"e1cf1896-ee8f-41a8-a7bb-628e35b1c4fd\" (UID: \"e1cf1896-ee8f-41a8-a7bb-628e35b1c4fd\") "
Sep 09 06:08:03.986111 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:03.986088    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1cf1896-ee8f-41a8-a7bb-628e35b1c4fd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e1cf1896-ee8f-41a8-a7bb-628e35b1c4fd" (UID: "e1cf1896-ee8f-41a8-a7bb-628e35b1c4fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:08:04.086362 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:04.086332    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1cf1896-ee8f-41a8-a7bb-628e35b1c4fd-maas-api-service-ca\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:08:04.812599 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:04.812569    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ecd13e-hpwqz"
Sep 09 06:08:04.812599 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:04.812584    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ecd13e-hpwqz" event={"ID":"e1cf1896-ee8f-41a8-a7bb-628e35b1c4fd","Type":"ContainerDied","Data":"7250afb8364fdbfcb58f853d84ccbed4dad95dae7a144107977c400cd6aa81a7"}
Sep 09 06:08:04.812883 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:04.812651    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7250afb8364fdbfcb58f853d84ccbed4dad95dae7a144107977c400cd6aa81a7"
Sep 09 06:08:07.139603 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:07.139558    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7979-139872053911600"]
Sep 09 06:08:07.140017 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:07.139909    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1cf1896-ee8f-41a8-a7bb-628e35b1c4fd" containerName="revoke-keys"
Sep 09 06:08:07.140017 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:07.139921    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1cf1896-ee8f-41a8-a7bb-628e35b1c4fd" containerName="revoke-keys"
Sep 09 06:08:07.140017 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:07.139976    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1cf1896-ee8f-41a8-a7bb-628e35b1c4fd" containerName="revoke-keys"
Sep 09 06:08:07.142957 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:07.142940    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7979-139872053911600"
Sep 09 06:08:07.145300 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:07.145279    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vljc5\""
Sep 09 06:08:07.149253 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:07.149235    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7979-139872053911600"]
Sep 09 06:08:07.211588 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:07.211559    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjh9k\" (UniqueName: \"kubernetes.io/projected/1fe9eb33-1062-47b9-9e5f-24d214b35768-kube-api-access-qjh9k\") pod \"test-curl-7979-139872053911600\" (UID: \"1fe9eb33-1062-47b9-9e5f-24d214b35768\") " pod="opendatahub/test-curl-7979-139872053911600"
Sep 09 06:08:07.297950 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:07.297915    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:08:07.312706 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:07.312675    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjh9k\" (UniqueName: \"kubernetes.io/projected/1fe9eb33-1062-47b9-9e5f-24d214b35768-kube-api-access-qjh9k\") pod \"test-curl-7979-139872053911600\" (UID: \"1fe9eb33-1062-47b9-9e5f-24d214b35768\") " pod="opendatahub/test-curl-7979-139872053911600"
Sep 09 06:08:07.323328 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:07.323297    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjh9k\" (UniqueName: \"kubernetes.io/projected/1fe9eb33-1062-47b9-9e5f-24d214b35768-kube-api-access-qjh9k\") pod \"test-curl-7979-139872053911600\" (UID: \"1fe9eb33-1062-47b9-9e5f-24d214b35768\") " pod="opendatahub/test-curl-7979-139872053911600"
Sep 09 06:08:07.453664 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:07.453577    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7979-139872053911600"
Sep 09 06:08:07.723895 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:07.723824    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7979-139872053911600"]
Sep 09 06:08:07.727193 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:08:07.727171    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe9eb33_1062_47b9_9e5f_24d214b35768.slice/crio-934e2675e270774062f32084497fd7de76b0f34218b453ed3541513abad42c78 WatchSource:0}: Error finding container 934e2675e270774062f32084497fd7de76b0f34218b453ed3541513abad42c78: Status 404 returned error can't find the container with id 934e2675e270774062f32084497fd7de76b0f34218b453ed3541513abad42c78
Sep 09 06:08:07.728820 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:07.728803    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 06:08:07.828439 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:07.828406    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7979-139872053911600" event={"ID":"1fe9eb33-1062-47b9-9e5f-24d214b35768","Type":"ContainerStarted","Data":"934e2675e270774062f32084497fd7de76b0f34218b453ed3541513abad42c78"}
Sep 09 06:08:08.835050 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:08.835021    2579 generic.go:356] "Generic (PLEG): container finished" podID="1fe9eb33-1062-47b9-9e5f-24d214b35768" containerID="c984e7bb3cdcf054b01903cede1cc6cc06c1d462089b53413c7b9e94200fb2be" exitCode=0
Sep 09 06:08:08.835423 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:08.835064    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7979-139872053911600" event={"ID":"1fe9eb33-1062-47b9-9e5f-24d214b35768","Type":"ContainerDied","Data":"c984e7bb3cdcf054b01903cede1cc6cc06c1d462089b53413c7b9e94200fb2be"}
Sep 09 06:08:09.969104 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:09.969081    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7979-139872053911600"
Sep 09 06:08:10.002170 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:10.002145    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7979-139872053911600_1fe9eb33-1062-47b9-9e5f-24d214b35768/test-curl-7979-139872053911600/0.log"
Sep 09 06:08:10.023639 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:10.023605    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7979-139872053911600"]
Sep 09 06:08:10.027497 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:10.027468    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7979-139872053911600"]
Sep 09 06:08:10.039088 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:10.039066    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjh9k\" (UniqueName: \"kubernetes.io/projected/1fe9eb33-1062-47b9-9e5f-24d214b35768-kube-api-access-qjh9k\") pod \"1fe9eb33-1062-47b9-9e5f-24d214b35768\" (UID: \"1fe9eb33-1062-47b9-9e5f-24d214b35768\") "
Sep 09 06:08:10.041265 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:10.041236    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe9eb33-1062-47b9-9e5f-24d214b35768-kube-api-access-qjh9k" (OuterVolumeSpecName: "kube-api-access-qjh9k") pod "1fe9eb33-1062-47b9-9e5f-24d214b35768" (UID: "1fe9eb33-1062-47b9-9e5f-24d214b35768"). InnerVolumeSpecName "kube-api-access-qjh9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:08:10.140671 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:10.140570    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjh9k\" (UniqueName: \"kubernetes.io/projected/1fe9eb33-1062-47b9-9e5f-24d214b35768-kube-api-access-qjh9k\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:08:10.149433 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:10.149405    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7979-139872053911600"]
Sep 09 06:08:10.149783 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:10.149769    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fe9eb33-1062-47b9-9e5f-24d214b35768" containerName="test-curl-7979-139872053911600"
Sep 09 06:08:10.149783 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:10.149788    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fe9eb33-1062-47b9-9e5f-24d214b35768" containerName="test-curl-7979-139872053911600"
Sep 09 06:08:10.149881 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:10.149866    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fe9eb33-1062-47b9-9e5f-24d214b35768" containerName="test-curl-7979-139872053911600"
Sep 09 06:08:10.152769 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:10.152755    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7979-139872053911600"
Sep 09 06:08:10.158827 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:10.158802    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7979-139872053911600"]
Sep 09 06:08:10.241162 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:10.241132    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfw8g\" (UniqueName: \"kubernetes.io/projected/2e0e73e7-70e9-4e81-b4e1-8724dba21e39-kube-api-access-tfw8g\") pod \"test-curl-7979-139872053911600\" (UID: \"2e0e73e7-70e9-4e81-b4e1-8724dba21e39\") " pod="opendatahub/test-curl-7979-139872053911600"
Sep 09 06:08:10.341955 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:10.341925    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfw8g\" (UniqueName: \"kubernetes.io/projected/2e0e73e7-70e9-4e81-b4e1-8724dba21e39-kube-api-access-tfw8g\") pod \"test-curl-7979-139872053911600\" (UID: \"2e0e73e7-70e9-4e81-b4e1-8724dba21e39\") " pod="opendatahub/test-curl-7979-139872053911600"
Sep 09 06:08:10.350392 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:10.350359    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfw8g\" (UniqueName: \"kubernetes.io/projected/2e0e73e7-70e9-4e81-b4e1-8724dba21e39-kube-api-access-tfw8g\") pod \"test-curl-7979-139872053911600\" (UID: \"2e0e73e7-70e9-4e81-b4e1-8724dba21e39\") " pod="opendatahub/test-curl-7979-139872053911600"
Sep 09 06:08:10.462586 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:10.462519    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7979-139872053911600"
Sep 09 06:08:10.729146 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:10.729123    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7979-139872053911600"]
Sep 09 06:08:10.731146 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:08:10.731118    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e0e73e7_70e9_4e81_b4e1_8724dba21e39.slice/crio-0e6fbd9b56b13a64a4acd39c609c10480b1127af1ff496da07e88e56d6b97213 WatchSource:0}: Error finding container 0e6fbd9b56b13a64a4acd39c609c10480b1127af1ff496da07e88e56d6b97213: Status 404 returned error can't find the container with id 0e6fbd9b56b13a64a4acd39c609c10480b1127af1ff496da07e88e56d6b97213
Sep 09 06:08:10.846473 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:10.846450    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="934e2675e270774062f32084497fd7de76b0f34218b453ed3541513abad42c78"
Sep 09 06:08:10.846473 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:10.846452    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7979-139872053911600"
Sep 09 06:08:10.847772 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:10.847743    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7979-139872053911600" event={"ID":"2e0e73e7-70e9-4e81-b4e1-8724dba21e39","Type":"ContainerStarted","Data":"0e6fbd9b56b13a64a4acd39c609c10480b1127af1ff496da07e88e56d6b97213"}
Sep 09 06:08:10.857322 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:10.857298    2579 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7979-139872053911600" oldPodUID="1fe9eb33-1062-47b9-9e5f-24d214b35768" podUID="2e0e73e7-70e9-4e81-b4e1-8724dba21e39"
Sep 09 06:08:11.148053 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:11.148017    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fe9eb33-1062-47b9-9e5f-24d214b35768" path="/var/lib/kubelet/pods/1fe9eb33-1062-47b9-9e5f-24d214b35768/volumes"
Sep 09 06:08:11.857228 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:11.857195    2579 generic.go:356] "Generic (PLEG): container finished" podID="2e0e73e7-70e9-4e81-b4e1-8724dba21e39" containerID="ca823a459efea11787af4653f1a7c99ac6098d93ffe5038b9fd3c27925d92f45" exitCode=0
Sep 09 06:08:11.857433 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:11.857264    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7979-139872053911600" event={"ID":"2e0e73e7-70e9-4e81-b4e1-8724dba21e39","Type":"ContainerDied","Data":"ca823a459efea11787af4653f1a7c99ac6098d93ffe5038b9fd3c27925d92f45"}
Sep 09 06:08:11.912334 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:11.912306    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ecd13e-6bc8846b59-vq8gl"]
Sep 09 06:08:11.912571 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:11.912502    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-ecd13e-6bc8846b59-vq8gl" podUID="c727c74f-dc8b-4efd-9bf2-10bdc66c2c08" containerName="payload-processing" containerID="cri-o://8ed563b0d004084bbc95f0854c0f84ee0cc554b70af56b33a5e4d2888ae0d518" gracePeriod=30
Sep 09 06:08:12.081496 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:12.081464    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ecd13e-hpwqz"]
Sep 09 06:08:12.085378 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:12.085356    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ecd13e-hpwqz"]
Sep 09 06:08:12.167400 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:12.167371    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ecd13e-6bc8846b59-vq8gl"
Sep 09 06:08:12.258061 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:12.258028    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rv56\" (UniqueName: \"kubernetes.io/projected/c727c74f-dc8b-4efd-9bf2-10bdc66c2c08-kube-api-access-8rv56\") pod \"c727c74f-dc8b-4efd-9bf2-10bdc66c2c08\" (UID: \"c727c74f-dc8b-4efd-9bf2-10bdc66c2c08\") "
Sep 09 06:08:12.258219 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:12.258151    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c727c74f-dc8b-4efd-9bf2-10bdc66c2c08-plugins-config-volume\") pod \"c727c74f-dc8b-4efd-9bf2-10bdc66c2c08\" (UID: \"c727c74f-dc8b-4efd-9bf2-10bdc66c2c08\") "
Sep 09 06:08:12.258526 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:12.258503    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c727c74f-dc8b-4efd-9bf2-10bdc66c2c08-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c727c74f-dc8b-4efd-9bf2-10bdc66c2c08" (UID: "c727c74f-dc8b-4efd-9bf2-10bdc66c2c08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:08:12.260303 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:12.260285    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c727c74f-dc8b-4efd-9bf2-10bdc66c2c08-kube-api-access-8rv56" (OuterVolumeSpecName: "kube-api-access-8rv56") pod "c727c74f-dc8b-4efd-9bf2-10bdc66c2c08" (UID: "c727c74f-dc8b-4efd-9bf2-10bdc66c2c08"). InnerVolumeSpecName "kube-api-access-8rv56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:08:12.359215 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:12.359186    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c727c74f-dc8b-4efd-9bf2-10bdc66c2c08-plugins-config-volume\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:08:12.359215 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:12.359213    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rv56\" (UniqueName: \"kubernetes.io/projected/c727c74f-dc8b-4efd-9bf2-10bdc66c2c08-kube-api-access-8rv56\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:08:12.863449 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:12.863414    2579 generic.go:356] "Generic (PLEG): container finished" podID="c727c74f-dc8b-4efd-9bf2-10bdc66c2c08" containerID="8ed563b0d004084bbc95f0854c0f84ee0cc554b70af56b33a5e4d2888ae0d518" exitCode=0
Sep 09 06:08:12.863667 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:12.863456    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ecd13e-6bc8846b59-vq8gl"
Sep 09 06:08:12.863667 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:12.863499    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ecd13e-6bc8846b59-vq8gl" event={"ID":"c727c74f-dc8b-4efd-9bf2-10bdc66c2c08","Type":"ContainerDied","Data":"8ed563b0d004084bbc95f0854c0f84ee0cc554b70af56b33a5e4d2888ae0d518"}
Sep 09 06:08:12.863667 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:12.863552    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ecd13e-6bc8846b59-vq8gl" event={"ID":"c727c74f-dc8b-4efd-9bf2-10bdc66c2c08","Type":"ContainerDied","Data":"d02d577c3f7798042fce4ef92ac16b14f0c74a85bdb31669703f0554a5889ab1"}
Sep 09 06:08:12.863667 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:12.863580    2579 scope.go:117] "RemoveContainer" containerID="8ed563b0d004084bbc95f0854c0f84ee0cc554b70af56b33a5e4d2888ae0d518"
Sep 09 06:08:12.878837 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:12.878810    2579 scope.go:117] "RemoveContainer" containerID="8ed563b0d004084bbc95f0854c0f84ee0cc554b70af56b33a5e4d2888ae0d518"
Sep 09 06:08:12.879168 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:08:12.879147    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ed563b0d004084bbc95f0854c0f84ee0cc554b70af56b33a5e4d2888ae0d518\": container with ID starting with 8ed563b0d004084bbc95f0854c0f84ee0cc554b70af56b33a5e4d2888ae0d518 not found: ID does not exist" containerID="8ed563b0d004084bbc95f0854c0f84ee0cc554b70af56b33a5e4d2888ae0d518"
Sep 09 06:08:12.879249 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:12.879181    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ed563b0d004084bbc95f0854c0f84ee0cc554b70af56b33a5e4d2888ae0d518"} err="failed to get container status \"8ed563b0d004084bbc95f0854c0f84ee0cc554b70af56b33a5e4d2888ae0d518\": rpc error: code = NotFound desc = could not find container \"8ed563b0d004084bbc95f0854c0f84ee0cc554b70af56b33a5e4d2888ae0d518\": container with ID starting with 8ed563b0d004084bbc95f0854c0f84ee0cc554b70af56b33a5e4d2888ae0d518 not found: ID does not exist"
Sep 09 06:08:12.893175 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:12.893155    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ecd13e-6bc8846b59-vq8gl"]
Sep 09 06:08:12.902594 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:12.902568    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ecd13e-6bc8846b59-vq8gl"]
Sep 09 06:08:12.999261 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:12.999237    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7979-139872053911600"
Sep 09 06:08:13.017781 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:13.017756    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7979-139872053911600_2e0e73e7-70e9-4e81-b4e1-8724dba21e39/test-curl-7979-139872053911600/0.log"
Sep 09 06:08:13.037576 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:13.037547    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7979-139872053911600"]
Sep 09 06:08:13.043417 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:13.043394    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7979-139872053911600"]
Sep 09 06:08:13.064884 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:13.064855    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfw8g\" (UniqueName: \"kubernetes.io/projected/2e0e73e7-70e9-4e81-b4e1-8724dba21e39-kube-api-access-tfw8g\") pod \"2e0e73e7-70e9-4e81-b4e1-8724dba21e39\" (UID: \"2e0e73e7-70e9-4e81-b4e1-8724dba21e39\") "
Sep 09 06:08:13.066989 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:13.066969    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e0e73e7-70e9-4e81-b4e1-8724dba21e39-kube-api-access-tfw8g" (OuterVolumeSpecName: "kube-api-access-tfw8g") pod "2e0e73e7-70e9-4e81-b4e1-8724dba21e39" (UID: "2e0e73e7-70e9-4e81-b4e1-8724dba21e39"). InnerVolumeSpecName "kube-api-access-tfw8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:08:13.148702 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:13.148608    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e0e73e7-70e9-4e81-b4e1-8724dba21e39" path="/var/lib/kubelet/pods/2e0e73e7-70e9-4e81-b4e1-8724dba21e39/volumes"
Sep 09 06:08:13.149009 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:13.148996    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c727c74f-dc8b-4efd-9bf2-10bdc66c2c08" path="/var/lib/kubelet/pods/c727c74f-dc8b-4efd-9bf2-10bdc66c2c08/volumes"
Sep 09 06:08:13.149274 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:13.149264    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1cf1896-ee8f-41a8-a7bb-628e35b1c4fd" path="/var/lib/kubelet/pods/e1cf1896-ee8f-41a8-a7bb-628e35b1c4fd/volumes"
Sep 09 06:08:13.166297 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:13.166274    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfw8g\" (UniqueName: \"kubernetes.io/projected/2e0e73e7-70e9-4e81-b4e1-8724dba21e39-kube-api-access-tfw8g\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:08:13.181168 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:13.181145    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7979-139872053912320"]
Sep 09 06:08:13.181446 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:13.181434    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e0e73e7-70e9-4e81-b4e1-8724dba21e39" containerName="test-curl-7979-139872053911600"
Sep 09 06:08:13.181484 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:13.181447    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e0e73e7-70e9-4e81-b4e1-8724dba21e39" containerName="test-curl-7979-139872053911600"
Sep 09 06:08:13.181484 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:13.181461    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c727c74f-dc8b-4efd-9bf2-10bdc66c2c08" containerName="payload-processing"
Sep 09 06:08:13.181484 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:13.181466    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c727c74f-dc8b-4efd-9bf2-10bdc66c2c08" containerName="payload-processing"
Sep 09 06:08:13.181579 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:13.181522    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e0e73e7-70e9-4e81-b4e1-8724dba21e39" containerName="test-curl-7979-139872053911600"
Sep 09 06:08:13.181579 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:13.181530    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c727c74f-dc8b-4efd-9bf2-10bdc66c2c08" containerName="payload-processing"
Sep 09 06:08:13.184594 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:13.184580    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7979-139872053912320"
Sep 09 06:08:13.195606 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:13.195588    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7979-139872053912320"]
Sep 09 06:08:13.267063 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:13.267035    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2sd2\" (UniqueName: \"kubernetes.io/projected/8e4e1543-5a52-43a6-8702-4d5b535095a7-kube-api-access-s2sd2\") pod \"test-curl-7979-139872053912320\" (UID: \"8e4e1543-5a52-43a6-8702-4d5b535095a7\") " pod="opendatahub/test-curl-7979-139872053912320"
Sep 09 06:08:13.368455 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:13.368425    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2sd2\" (UniqueName: \"kubernetes.io/projected/8e4e1543-5a52-43a6-8702-4d5b535095a7-kube-api-access-s2sd2\") pod \"test-curl-7979-139872053912320\" (UID: \"8e4e1543-5a52-43a6-8702-4d5b535095a7\") " pod="opendatahub/test-curl-7979-139872053912320"
Sep 09 06:08:13.376164 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:13.376136    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2sd2\" (UniqueName: \"kubernetes.io/projected/8e4e1543-5a52-43a6-8702-4d5b535095a7-kube-api-access-s2sd2\") pod \"test-curl-7979-139872053912320\" (UID: \"8e4e1543-5a52-43a6-8702-4d5b535095a7\") " pod="opendatahub/test-curl-7979-139872053912320"
Sep 09 06:08:13.492333 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:13.492249    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7979-139872053912320"
Sep 09 06:08:13.759611 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:13.759534    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7979-139872053912320"]
Sep 09 06:08:13.761812 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:08:13.761780    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e4e1543_5a52_43a6_8702_4d5b535095a7.slice/crio-d5e85dcd73d44c30afe849de88a1706bac5894a8c5a2ee83b6d65a0231c31ac8 WatchSource:0}: Error finding container d5e85dcd73d44c30afe849de88a1706bac5894a8c5a2ee83b6d65a0231c31ac8: Status 404 returned error can't find the container with id d5e85dcd73d44c30afe849de88a1706bac5894a8c5a2ee83b6d65a0231c31ac8
Sep 09 06:08:13.870356 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:13.870321    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7979-139872053911600"
Sep 09 06:08:13.870356 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:13.870342    2579 scope.go:117] "RemoveContainer" containerID="ca823a459efea11787af4653f1a7c99ac6098d93ffe5038b9fd3c27925d92f45"
Sep 09 06:08:13.872002 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:13.871793    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7979-139872053912320" event={"ID":"8e4e1543-5a52-43a6-8702-4d5b535095a7","Type":"ContainerStarted","Data":"d5e85dcd73d44c30afe849de88a1706bac5894a8c5a2ee83b6d65a0231c31ac8"}
Sep 09 06:08:14.880829 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:14.880799    2579 generic.go:356] "Generic (PLEG): container finished" podID="8e4e1543-5a52-43a6-8702-4d5b535095a7" containerID="214ecd98386aac7c21a85ac2cc97fc6849de794afaa5490321548dff45c93045" exitCode=0
Sep 09 06:08:14.881355 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:14.880878    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7979-139872053912320" event={"ID":"8e4e1543-5a52-43a6-8702-4d5b535095a7","Type":"ContainerDied","Data":"214ecd98386aac7c21a85ac2cc97fc6849de794afaa5490321548dff45c93045"}
Sep 09 06:08:16.020079 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:16.020056    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7979-139872053912320"
Sep 09 06:08:16.037714 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:16.037689    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7979-139872053912320_8e4e1543-5a52-43a6-8702-4d5b535095a7/test-curl-7979-139872053912320/0.log"
Sep 09 06:08:16.059351 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:16.059322    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7979-139872053912320"]
Sep 09 06:08:16.064518 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:16.064497    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7979-139872053912320"]
Sep 09 06:08:16.093359 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:16.093342    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2sd2\" (UniqueName: \"kubernetes.io/projected/8e4e1543-5a52-43a6-8702-4d5b535095a7-kube-api-access-s2sd2\") pod \"8e4e1543-5a52-43a6-8702-4d5b535095a7\" (UID: \"8e4e1543-5a52-43a6-8702-4d5b535095a7\") "
Sep 09 06:08:16.095445 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:16.095425    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e4e1543-5a52-43a6-8702-4d5b535095a7-kube-api-access-s2sd2" (OuterVolumeSpecName: "kube-api-access-s2sd2") pod "8e4e1543-5a52-43a6-8702-4d5b535095a7" (UID: "8e4e1543-5a52-43a6-8702-4d5b535095a7"). InnerVolumeSpecName "kube-api-access-s2sd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:08:16.184293 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:16.184219    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7979-139872053912320"]
Sep 09 06:08:16.184570 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:16.184549    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e4e1543-5a52-43a6-8702-4d5b535095a7" containerName="test-curl-7979-139872053912320"
Sep 09 06:08:16.184616 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:16.184571    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e4e1543-5a52-43a6-8702-4d5b535095a7" containerName="test-curl-7979-139872053912320"
Sep 09 06:08:16.184690 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:16.184649    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e4e1543-5a52-43a6-8702-4d5b535095a7" containerName="test-curl-7979-139872053912320"
Sep 09 06:08:16.187493 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:16.187479    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7979-139872053912320"
Sep 09 06:08:16.193216 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:16.193195    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7979-139872053912320"]
Sep 09 06:08:16.194549 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:16.194448    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2sd2\" (UniqueName: \"kubernetes.io/projected/8e4e1543-5a52-43a6-8702-4d5b535095a7-kube-api-access-s2sd2\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:08:16.295734 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:16.295701    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9x67\" (UniqueName: \"kubernetes.io/projected/2ca15fc7-db4e-46b4-9bbc-0778f75b4e43-kube-api-access-d9x67\") pod \"test-curl-7979-139872053912320\" (UID: \"2ca15fc7-db4e-46b4-9bbc-0778f75b4e43\") " pod="opendatahub/test-curl-7979-139872053912320"
Sep 09 06:08:16.396360 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:16.396323    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9x67\" (UniqueName: \"kubernetes.io/projected/2ca15fc7-db4e-46b4-9bbc-0778f75b4e43-kube-api-access-d9x67\") pod \"test-curl-7979-139872053912320\" (UID: \"2ca15fc7-db4e-46b4-9bbc-0778f75b4e43\") " pod="opendatahub/test-curl-7979-139872053912320"
Sep 09 06:08:16.404867 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:16.404834    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9x67\" (UniqueName: \"kubernetes.io/projected/2ca15fc7-db4e-46b4-9bbc-0778f75b4e43-kube-api-access-d9x67\") pod \"test-curl-7979-139872053912320\" (UID: \"2ca15fc7-db4e-46b4-9bbc-0778f75b4e43\") " pod="opendatahub/test-curl-7979-139872053912320"
Sep 09 06:08:16.497036 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:16.496951    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7979-139872053912320"
Sep 09 06:08:16.762989 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:16.762912    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7979-139872053912320"]
Sep 09 06:08:16.765855 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:08:16.765823    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ca15fc7_db4e_46b4_9bbc_0778f75b4e43.slice/crio-7560bcfd9bd114ad7a5d3351f6aca28dde840295debf4def261ab824cc276c99 WatchSource:0}: Error finding container 7560bcfd9bd114ad7a5d3351f6aca28dde840295debf4def261ab824cc276c99: Status 404 returned error can't find the container with id 7560bcfd9bd114ad7a5d3351f6aca28dde840295debf4def261ab824cc276c99
Sep 09 06:08:16.898965 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:16.898932    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7979-139872053912320" event={"ID":"2ca15fc7-db4e-46b4-9bbc-0778f75b4e43","Type":"ContainerStarted","Data":"7560bcfd9bd114ad7a5d3351f6aca28dde840295debf4def261ab824cc276c99"}
Sep 09 06:08:16.900228 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:16.900209    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7979-139872053912320"
Sep 09 06:08:16.900332 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:16.900213    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5e85dcd73d44c30afe849de88a1706bac5894a8c5a2ee83b6d65a0231c31ac8"
Sep 09 06:08:16.911260 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:16.911235    2579 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7979-139872053912320" oldPodUID="8e4e1543-5a52-43a6-8702-4d5b535095a7" podUID="2ca15fc7-db4e-46b4-9bbc-0778f75b4e43"
Sep 09 06:08:17.148218 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:17.148185    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e4e1543-5a52-43a6-8702-4d5b535095a7" path="/var/lib/kubelet/pods/8e4e1543-5a52-43a6-8702-4d5b535095a7/volumes"
Sep 09 06:08:17.906475 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:17.906447    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7979-139872053912320" event={"ID":"2ca15fc7-db4e-46b4-9bbc-0778f75b4e43","Type":"ContainerStarted","Data":"f794c748e93da30e6c83fe8f2acb2a334db3608a3f83fc6321bb64e14f330332"}
Sep 09 06:08:17.919374 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:17.919331    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7979-139872053912320" podStartSLOduration=0.844742892 podStartE2EDuration="1.919315794s" totalImagesPullingTime="1.074572902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:08:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:08:17.917832481 +0000 UTC m=+1123.258320389" watchObservedRunningTime="2026-09-09 06:08:17.919315794 +0000 UTC m=+1123.259803703"
Sep 09 06:08:18.012198 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:18.012175    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7979-139872053912320_2ca15fc7-db4e-46b4-9bbc-0778f75b4e43/test-curl-7979-139872053912320/0.log"
Sep 09 06:08:18.786195 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:18.786165    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe8eaa-9bw4t"]
Sep 09 06:08:18.789497 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:18.789482    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe8eaa-9bw4t"
Sep 09 06:08:18.791655 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:18.791619    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ksjk6\""
Sep 09 06:08:18.797018 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:18.796859    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe8eaa-9bw4t"]
Sep 09 06:08:18.912968 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:18.912935    2579 generic.go:356] "Generic (PLEG): container finished" podID="2ca15fc7-db4e-46b4-9bbc-0778f75b4e43" containerID="f794c748e93da30e6c83fe8f2acb2a334db3608a3f83fc6321bb64e14f330332" exitCode=0
Sep 09 06:08:18.913132 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:18.912999    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7979-139872053912320" event={"ID":"2ca15fc7-db4e-46b4-9bbc-0778f75b4e43","Type":"ContainerDied","Data":"f794c748e93da30e6c83fe8f2acb2a334db3608a3f83fc6321bb64e14f330332"}
Sep 09 06:08:18.918492 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:18.918470    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ba28f2b-d2f4-4fd5-a953-cdd07268ae47-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fe8eaa-9bw4t\" (UID: \"3ba28f2b-d2f4-4fd5-a953-cdd07268ae47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe8eaa-9bw4t"
Sep 09 06:08:19.019179 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:19.019146    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ba28f2b-d2f4-4fd5-a953-cdd07268ae47-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fe8eaa-9bw4t\" (UID: \"3ba28f2b-d2f4-4fd5-a953-cdd07268ae47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe8eaa-9bw4t"
Sep 09 06:08:19.019815 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:19.019799    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ba28f2b-d2f4-4fd5-a953-cdd07268ae47-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fe8eaa-9bw4t\" (UID: \"3ba28f2b-d2f4-4fd5-a953-cdd07268ae47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe8eaa-9bw4t"
Sep 09 06:08:19.101470 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:19.101440    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe8eaa-9bw4t"
Sep 09 06:08:19.369601 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:19.369576    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe8eaa-9bw4t"]
Sep 09 06:08:19.372163 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:08:19.372138    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ba28f2b_d2f4_4fd5_a953_cdd07268ae47.slice/crio-709c9f6850ae392a9ee6df9c5d6113b8729475c20af87338f2e3aa76b5b7b005 WatchSource:0}: Error finding container 709c9f6850ae392a9ee6df9c5d6113b8729475c20af87338f2e3aa76b5b7b005: Status 404 returned error can't find the container with id 709c9f6850ae392a9ee6df9c5d6113b8729475c20af87338f2e3aa76b5b7b005
Sep 09 06:08:19.922814 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:19.922695    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe8eaa-9bw4t" event={"ID":"3ba28f2b-d2f4-4fd5-a953-cdd07268ae47","Type":"ContainerStarted","Data":"152d24b981de79f230157e40eaee9ea630f710fb983409ed2ef7cf937e2ccc81"}
Sep 09 06:08:19.923200 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:19.922841    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe8eaa-9bw4t" event={"ID":"3ba28f2b-d2f4-4fd5-a953-cdd07268ae47","Type":"ContainerStarted","Data":"709c9f6850ae392a9ee6df9c5d6113b8729475c20af87338f2e3aa76b5b7b005"}
Sep 09 06:08:19.937938 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:19.937887    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe8eaa-9bw4t" podStartSLOduration=1.9378716329999999 podStartE2EDuration="1.937871633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:08:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:08:19.936537477 +0000 UTC m=+1125.277025400" watchObservedRunningTime="2026-09-09 06:08:19.937871633 +0000 UTC m=+1125.278359541"
Sep 09 06:08:20.080226 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:20.080205    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7979-139872053912320"
Sep 09 06:08:20.114539 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:20.114507    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7979-139872053912320"]
Sep 09 06:08:20.119698 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:20.119674    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7979-139872053912320"]
Sep 09 06:08:20.229269 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:20.229181    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9x67\" (UniqueName: \"kubernetes.io/projected/2ca15fc7-db4e-46b4-9bbc-0778f75b4e43-kube-api-access-d9x67\") pod \"2ca15fc7-db4e-46b4-9bbc-0778f75b4e43\" (UID: \"2ca15fc7-db4e-46b4-9bbc-0778f75b4e43\") "
Sep 09 06:08:20.231323 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:20.231300    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ca15fc7-db4e-46b4-9bbc-0778f75b4e43-kube-api-access-d9x67" (OuterVolumeSpecName: "kube-api-access-d9x67") pod "2ca15fc7-db4e-46b4-9bbc-0778f75b4e43" (UID: "2ca15fc7-db4e-46b4-9bbc-0778f75b4e43"). InnerVolumeSpecName "kube-api-access-d9x67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:08:20.329902 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:20.329863    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9x67\" (UniqueName: \"kubernetes.io/projected/2ca15fc7-db4e-46b4-9bbc-0778f75b4e43-kube-api-access-d9x67\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:08:20.928707 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:20.928678    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7979-139872053912320"
Sep 09 06:08:20.929188 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:20.928680    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7560bcfd9bd114ad7a5d3351f6aca28dde840295debf4def261ab824cc276c99"
Sep 09 06:08:20.930253 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:20.930225    2579 generic.go:356] "Generic (PLEG): container finished" podID="3ba28f2b-d2f4-4fd5-a953-cdd07268ae47" containerID="152d24b981de79f230157e40eaee9ea630f710fb983409ed2ef7cf937e2ccc81" exitCode=0
Sep 09 06:08:20.930354 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:20.930314    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe8eaa-9bw4t" event={"ID":"3ba28f2b-d2f4-4fd5-a953-cdd07268ae47","Type":"ContainerDied","Data":"152d24b981de79f230157e40eaee9ea630f710fb983409ed2ef7cf937e2ccc81"}
Sep 09 06:08:20.945422 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:08:20.945397    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7979-139872053912320\" is forbidden: User \"system:node:ip-10-0-131-142.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-142.ec2.internal' and this object" podUID="2ca15fc7-db4e-46b4-9bbc-0778f75b4e43" pod="opendatahub/test-curl-7979-139872053912320"
Sep 09 06:08:21.148223 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:21.148187    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ca15fc7-db4e-46b4-9bbc-0778f75b4e43" path="/var/lib/kubelet/pods/2ca15fc7-db4e-46b4-9bbc-0778f75b4e43/volumes"
Sep 09 06:08:22.058666 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:22.058618    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe8eaa-9bw4t"
Sep 09 06:08:22.146781 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:22.146755    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ba28f2b-d2f4-4fd5-a953-cdd07268ae47-maas-api-service-ca\") pod \"3ba28f2b-d2f4-4fd5-a953-cdd07268ae47\" (UID: \"3ba28f2b-d2f4-4fd5-a953-cdd07268ae47\") "
Sep 09 06:08:22.147055 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:22.147034    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ba28f2b-d2f4-4fd5-a953-cdd07268ae47-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3ba28f2b-d2f4-4fd5-a953-cdd07268ae47" (UID: "3ba28f2b-d2f4-4fd5-a953-cdd07268ae47"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:08:22.248129 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:22.248070    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ba28f2b-d2f4-4fd5-a953-cdd07268ae47-maas-api-service-ca\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:08:22.940959 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:22.940926    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe8eaa-9bw4t" event={"ID":"3ba28f2b-d2f4-4fd5-a953-cdd07268ae47","Type":"ContainerDied","Data":"709c9f6850ae392a9ee6df9c5d6113b8729475c20af87338f2e3aa76b5b7b005"}
Sep 09 06:08:22.940959 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:22.940958    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe8eaa-9bw4t"
Sep 09 06:08:22.941159 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:22.940972    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="709c9f6850ae392a9ee6df9c5d6113b8729475c20af87338f2e3aa76b5b7b005"
Sep 09 06:08:25.711869 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:25.711835    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-389cd1da-7cddb7cbfc-x2nn9"]
Sep 09 06:08:25.712218 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:25.712159    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ba28f2b-d2f4-4fd5-a953-cdd07268ae47" containerName="revoke-keys"
Sep 09 06:08:25.712218 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:25.712169    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ba28f2b-d2f4-4fd5-a953-cdd07268ae47" containerName="revoke-keys"
Sep 09 06:08:25.712218 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:25.712180    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ca15fc7-db4e-46b4-9bbc-0778f75b4e43" containerName="test-curl-7979-139872053912320"
Sep 09 06:08:25.712218 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:25.712186    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ca15fc7-db4e-46b4-9bbc-0778f75b4e43" containerName="test-curl-7979-139872053912320"
Sep 09 06:08:25.712356 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:25.712257    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ba28f2b-d2f4-4fd5-a953-cdd07268ae47" containerName="revoke-keys"
Sep 09 06:08:25.712356 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:25.712266    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ca15fc7-db4e-46b4-9bbc-0778f75b4e43" containerName="test-curl-7979-139872053912320"
Sep 09 06:08:25.715300 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:25.715281    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-389cd1da-7cddb7cbfc-x2nn9"
Sep 09 06:08:25.717706 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:25.717689    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-389cd1da-dockercfg-ntgbw\""
Sep 09 06:08:25.717783 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:25.717689    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-389cd1da\""
Sep 09 06:08:25.725784 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:25.725763    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-389cd1da-7cddb7cbfc-x2nn9"]
Sep 09 06:08:25.886493 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:25.886454    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52l7z\" (UniqueName: \"kubernetes.io/projected/9f66df18-7b8a-4a9c-bdc3-8bc17061c133-kube-api-access-52l7z\") pod \"payload-processing-e2e-mt-389cd1da-7cddb7cbfc-x2nn9\" (UID: \"9f66df18-7b8a-4a9c-bdc3-8bc17061c133\") " pod="openshift-ingress/payload-processing-e2e-mt-389cd1da-7cddb7cbfc-x2nn9"
Sep 09 06:08:25.886690 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:25.886544    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f66df18-7b8a-4a9c-bdc3-8bc17061c133-plugins-config-volume\") pod \"payload-processing-e2e-mt-389cd1da-7cddb7cbfc-x2nn9\" (UID: \"9f66df18-7b8a-4a9c-bdc3-8bc17061c133\") " pod="openshift-ingress/payload-processing-e2e-mt-389cd1da-7cddb7cbfc-x2nn9"
Sep 09 06:08:25.987192 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:25.987100    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f66df18-7b8a-4a9c-bdc3-8bc17061c133-plugins-config-volume\") pod \"payload-processing-e2e-mt-389cd1da-7cddb7cbfc-x2nn9\" (UID: \"9f66df18-7b8a-4a9c-bdc3-8bc17061c133\") " pod="openshift-ingress/payload-processing-e2e-mt-389cd1da-7cddb7cbfc-x2nn9"
Sep 09 06:08:25.987192 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:25.987164    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52l7z\" (UniqueName: \"kubernetes.io/projected/9f66df18-7b8a-4a9c-bdc3-8bc17061c133-kube-api-access-52l7z\") pod \"payload-processing-e2e-mt-389cd1da-7cddb7cbfc-x2nn9\" (UID: \"9f66df18-7b8a-4a9c-bdc3-8bc17061c133\") " pod="openshift-ingress/payload-processing-e2e-mt-389cd1da-7cddb7cbfc-x2nn9"
Sep 09 06:08:25.987850 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:25.987832    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f66df18-7b8a-4a9c-bdc3-8bc17061c133-plugins-config-volume\") pod \"payload-processing-e2e-mt-389cd1da-7cddb7cbfc-x2nn9\" (UID: \"9f66df18-7b8a-4a9c-bdc3-8bc17061c133\") " pod="openshift-ingress/payload-processing-e2e-mt-389cd1da-7cddb7cbfc-x2nn9"
Sep 09 06:08:25.995389 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:25.995373    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52l7z\" (UniqueName: \"kubernetes.io/projected/9f66df18-7b8a-4a9c-bdc3-8bc17061c133-kube-api-access-52l7z\") pod \"payload-processing-e2e-mt-389cd1da-7cddb7cbfc-x2nn9\" (UID: \"9f66df18-7b8a-4a9c-bdc3-8bc17061c133\") " pod="openshift-ingress/payload-processing-e2e-mt-389cd1da-7cddb7cbfc-x2nn9"
Sep 09 06:08:26.026354 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:26.026325    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-389cd1da-7cddb7cbfc-x2nn9"
Sep 09 06:08:26.295482 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:26.295459    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-389cd1da-7cddb7cbfc-x2nn9"]
Sep 09 06:08:26.297743 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:08:26.297709    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f66df18_7b8a_4a9c_bdc3_8bc17061c133.slice/crio-4662092661be20ffda129f0d22f31b1dd40f0a8e710e1c2a87e8da5ceec5f2da WatchSource:0}: Error finding container 4662092661be20ffda129f0d22f31b1dd40f0a8e710e1c2a87e8da5ceec5f2da: Status 404 returned error can't find the container with id 4662092661be20ffda129f0d22f31b1dd40f0a8e710e1c2a87e8da5ceec5f2da
Sep 09 06:08:26.961647 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:26.961597    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-389cd1da-7cddb7cbfc-x2nn9" event={"ID":"9f66df18-7b8a-4a9c-bdc3-8bc17061c133","Type":"ContainerStarted","Data":"9c77dca60f99617a4362c15b4241c44e8e69c1d7038238830b61314193a69f01"}
Sep 09 06:08:26.962110 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:26.961675    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-389cd1da-7cddb7cbfc-x2nn9" event={"ID":"9f66df18-7b8a-4a9c-bdc3-8bc17061c133","Type":"ContainerStarted","Data":"4662092661be20ffda129f0d22f31b1dd40f0a8e710e1c2a87e8da5ceec5f2da"}
Sep 09 06:08:26.962110 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:26.961745    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-389cd1da-7cddb7cbfc-x2nn9"
Sep 09 06:08:26.977146 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:26.977099    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-389cd1da-7cddb7cbfc-x2nn9" podStartSLOduration=1.9770857830000002 podStartE2EDuration="1.977085783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:08:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:08:26.975588395 +0000 UTC m=+1132.316076316" watchObservedRunningTime="2026-09-09 06:08:26.977085783 +0000 UTC m=+1132.317573693"
Sep 09 06:08:29.568677 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:29.568644    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh"]
Sep 09 06:08:29.569130 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:29.568860    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh" podUID="0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4" containerName="maas-api" containerID="cri-o://e64578a3fda3f0d41bc8faaf3998a95b2e3f8a0aba9a504079acf7c1fa495fb7" gracePeriod=30
Sep 09 06:08:29.624581 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:29.624541    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fe8eaa-568f87ccc5-5z4qp"]
Sep 09 06:08:29.624781 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:29.624758    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-fe8eaa-568f87ccc5-5z4qp" podUID="a4d64e57-8756-43c5-a798-1690572ade99" containerName="payload-processing" containerID="cri-o://a2004bc32f90cebc920db828454f54223da1030f061716844fed97b24c3934dd" gracePeriod=30
Sep 09 06:08:29.630707 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:29.630674    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fe8eaa-76889d76c4-mphkg"]
Sep 09 06:08:29.630950 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:29.630885    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fe8eaa-76889d76c4-mphkg" podUID="6ada3927-7884-47a4-8827-3cdb556fb999" containerName="payload-pre-processing" containerID="cri-o://dfcdb5ea07c5f66040abd77c9eac21f70c7ac10f508b4f7af7f2cabc5afe04db" gracePeriod=30
Sep 09 06:08:29.784182 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:29.781878    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe8eaa-9bw4t"]
Sep 09 06:08:29.785835 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:29.785802    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe8eaa-9bw4t"]
Sep 09 06:08:29.906950 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:29.906924    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fe8eaa-568f87ccc5-5z4qp"
Sep 09 06:08:29.910806 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:29.910784    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fe8eaa-76889d76c4-mphkg"
Sep 09 06:08:29.977909 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:29.977882    2579 generic.go:356] "Generic (PLEG): container finished" podID="a4d64e57-8756-43c5-a798-1690572ade99" containerID="a2004bc32f90cebc920db828454f54223da1030f061716844fed97b24c3934dd" exitCode=0
Sep 09 06:08:29.978062 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:29.977937    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fe8eaa-568f87ccc5-5z4qp"
Sep 09 06:08:29.978062 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:29.977956    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fe8eaa-568f87ccc5-5z4qp" event={"ID":"a4d64e57-8756-43c5-a798-1690572ade99","Type":"ContainerDied","Data":"a2004bc32f90cebc920db828454f54223da1030f061716844fed97b24c3934dd"}
Sep 09 06:08:29.978190 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:29.978059    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fe8eaa-568f87ccc5-5z4qp" event={"ID":"a4d64e57-8756-43c5-a798-1690572ade99","Type":"ContainerDied","Data":"bc8549e2b1005155e71b767defe22a0e4d2ae3d9ecc75a356b57674ca0558e25"}
Sep 09 06:08:29.978190 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:29.978099    2579 scope.go:117] "RemoveContainer" containerID="a2004bc32f90cebc920db828454f54223da1030f061716844fed97b24c3934dd"
Sep 09 06:08:29.979894 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:29.979869    2579 generic.go:356] "Generic (PLEG): container finished" podID="0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4" containerID="e64578a3fda3f0d41bc8faaf3998a95b2e3f8a0aba9a504079acf7c1fa495fb7" exitCode=0
Sep 09 06:08:29.979998 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:29.979942    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh" event={"ID":"0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4","Type":"ContainerDied","Data":"e64578a3fda3f0d41bc8faaf3998a95b2e3f8a0aba9a504079acf7c1fa495fb7"}
Sep 09 06:08:29.981695 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:29.981661    2579 generic.go:356] "Generic (PLEG): container finished" podID="6ada3927-7884-47a4-8827-3cdb556fb999" containerID="dfcdb5ea07c5f66040abd77c9eac21f70c7ac10f508b4f7af7f2cabc5afe04db" exitCode=0
Sep 09 06:08:29.981799 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:29.981703    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fe8eaa-76889d76c4-mphkg"
Sep 09 06:08:29.981799 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:29.981772    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fe8eaa-76889d76c4-mphkg" event={"ID":"6ada3927-7884-47a4-8827-3cdb556fb999","Type":"ContainerDied","Data":"dfcdb5ea07c5f66040abd77c9eac21f70c7ac10f508b4f7af7f2cabc5afe04db"}
Sep 09 06:08:29.981920 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:29.981801    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fe8eaa-76889d76c4-mphkg" event={"ID":"6ada3927-7884-47a4-8827-3cdb556fb999","Type":"ContainerDied","Data":"4cf7e0f3cde7b4da68b1c63fb26a7aceea85bce6c85803ae0932c48eec064caf"}
Sep 09 06:08:29.990820 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:29.990795    2579 scope.go:117] "RemoveContainer" containerID="a2004bc32f90cebc920db828454f54223da1030f061716844fed97b24c3934dd"
Sep 09 06:08:29.991376 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:08:29.991334    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2004bc32f90cebc920db828454f54223da1030f061716844fed97b24c3934dd\": container with ID starting with a2004bc32f90cebc920db828454f54223da1030f061716844fed97b24c3934dd not found: ID does not exist" containerID="a2004bc32f90cebc920db828454f54223da1030f061716844fed97b24c3934dd"
Sep 09 06:08:29.991461 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:29.991386    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2004bc32f90cebc920db828454f54223da1030f061716844fed97b24c3934dd"} err="failed to get container status \"a2004bc32f90cebc920db828454f54223da1030f061716844fed97b24c3934dd\": rpc error: code = NotFound desc = could not find container \"a2004bc32f90cebc920db828454f54223da1030f061716844fed97b24c3934dd\": container with ID starting with a2004bc32f90cebc920db828454f54223da1030f061716844fed97b24c3934dd not found: ID does not exist"
Sep 09 06:08:29.991461 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:29.991408    2579 scope.go:117] "RemoveContainer" containerID="dfcdb5ea07c5f66040abd77c9eac21f70c7ac10f508b4f7af7f2cabc5afe04db"
Sep 09 06:08:30.000518 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:30.000502    2579 scope.go:117] "RemoveContainer" containerID="dfcdb5ea07c5f66040abd77c9eac21f70c7ac10f508b4f7af7f2cabc5afe04db"
Sep 09 06:08:30.000814 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:08:30.000798    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfcdb5ea07c5f66040abd77c9eac21f70c7ac10f508b4f7af7f2cabc5afe04db\": container with ID starting with dfcdb5ea07c5f66040abd77c9eac21f70c7ac10f508b4f7af7f2cabc5afe04db not found: ID does not exist" containerID="dfcdb5ea07c5f66040abd77c9eac21f70c7ac10f508b4f7af7f2cabc5afe04db"
Sep 09 06:08:30.000875 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:30.000823    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfcdb5ea07c5f66040abd77c9eac21f70c7ac10f508b4f7af7f2cabc5afe04db"} err="failed to get container status \"dfcdb5ea07c5f66040abd77c9eac21f70c7ac10f508b4f7af7f2cabc5afe04db\": rpc error: code = NotFound desc = could not find container \"dfcdb5ea07c5f66040abd77c9eac21f70c7ac10f508b4f7af7f2cabc5afe04db\": container with ID starting with dfcdb5ea07c5f66040abd77c9eac21f70c7ac10f508b4f7af7f2cabc5afe04db not found: ID does not exist"
Sep 09 06:08:30.024677 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:30.024660    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ada3927-7884-47a4-8827-3cdb556fb999-plugins-config-volume\") pod \"6ada3927-7884-47a4-8827-3cdb556fb999\" (UID: \"6ada3927-7884-47a4-8827-3cdb556fb999\") "
Sep 09 06:08:30.024787 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:30.024726    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4d64e57-8756-43c5-a798-1690572ade99-plugins-config-volume\") pod \"a4d64e57-8756-43c5-a798-1690572ade99\" (UID: \"a4d64e57-8756-43c5-a798-1690572ade99\") "
Sep 09 06:08:30.024787 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:30.024770    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jclp2\" (UniqueName: \"kubernetes.io/projected/6ada3927-7884-47a4-8827-3cdb556fb999-kube-api-access-jclp2\") pod \"6ada3927-7884-47a4-8827-3cdb556fb999\" (UID: \"6ada3927-7884-47a4-8827-3cdb556fb999\") "
Sep 09 06:08:30.024787 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:30.024784    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7txw\" (UniqueName: \"kubernetes.io/projected/a4d64e57-8756-43c5-a798-1690572ade99-kube-api-access-f7txw\") pod \"a4d64e57-8756-43c5-a798-1690572ade99\" (UID: \"a4d64e57-8756-43c5-a798-1690572ade99\") "
Sep 09 06:08:30.025094 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:30.025068    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ada3927-7884-47a4-8827-3cdb556fb999-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6ada3927-7884-47a4-8827-3cdb556fb999" (UID: "6ada3927-7884-47a4-8827-3cdb556fb999"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:08:30.025180 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:30.025128    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4d64e57-8756-43c5-a798-1690572ade99-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a4d64e57-8756-43c5-a798-1690572ade99" (UID: "a4d64e57-8756-43c5-a798-1690572ade99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:08:30.027055 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:30.027032    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ada3927-7884-47a4-8827-3cdb556fb999-kube-api-access-jclp2" (OuterVolumeSpecName: "kube-api-access-jclp2") pod "6ada3927-7884-47a4-8827-3cdb556fb999" (UID: "6ada3927-7884-47a4-8827-3cdb556fb999"). InnerVolumeSpecName "kube-api-access-jclp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:08:30.027199 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:30.027179    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4d64e57-8756-43c5-a798-1690572ade99-kube-api-access-f7txw" (OuterVolumeSpecName: "kube-api-access-f7txw") pod "a4d64e57-8756-43c5-a798-1690572ade99" (UID: "a4d64e57-8756-43c5-a798-1690572ade99"). InnerVolumeSpecName "kube-api-access-f7txw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:08:30.030922 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:30.030906    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh"
Sep 09 06:08:30.126187 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:30.126119    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4-maas-api-tls\") pod \"0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4\" (UID: \"0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4\") "
Sep 09 06:08:30.126187 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:30.126156    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f47qp\" (UniqueName: \"kubernetes.io/projected/0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4-kube-api-access-f47qp\") pod \"0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4\" (UID: \"0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4\") "
Sep 09 06:08:30.126363 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:30.126193    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4-metrics-certs\") pod \"0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4\" (UID: \"0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4\") "
Sep 09 06:08:30.126363 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:30.126237    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4-openshift-service-ca\") pod \"0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4\" (UID: \"0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4\") "
Sep 09 06:08:30.126494 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:30.126478    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4d64e57-8756-43c5-a798-1690572ade99-plugins-config-volume\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:08:30.126560 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:30.126499    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jclp2\" (UniqueName: \"kubernetes.io/projected/6ada3927-7884-47a4-8827-3cdb556fb999-kube-api-access-jclp2\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:08:30.126560 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:30.126514    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7txw\" (UniqueName: \"kubernetes.io/projected/a4d64e57-8756-43c5-a798-1690572ade99-kube-api-access-f7txw\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:08:30.126560 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:30.126529    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ada3927-7884-47a4-8827-3cdb556fb999-plugins-config-volume\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:08:30.126695 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:30.126664    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4" (UID: "0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:08:30.128331 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:30.128305    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4" (UID: "0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:08:30.128439 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:30.128355    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4" (UID: "0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:08:30.128439 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:30.128374    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4-kube-api-access-f47qp" (OuterVolumeSpecName: "kube-api-access-f47qp") pod "0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4" (UID: "0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4"). InnerVolumeSpecName "kube-api-access-f47qp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:08:30.227539 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:30.227505    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4-maas-api-tls\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:08:30.227539 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:30.227532    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f47qp\" (UniqueName: \"kubernetes.io/projected/0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4-kube-api-access-f47qp\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:08:30.227539 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:30.227542    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4-metrics-certs\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:08:30.227731 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:30.227551    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4-openshift-service-ca\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:08:30.313891 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:30.313869    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fe8eaa-568f87ccc5-5z4qp"]
Sep 09 06:08:30.327473 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:30.327453    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fe8eaa-568f87ccc5-5z4qp"]
Sep 09 06:08:30.347886 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:30.347866    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fe8eaa-76889d76c4-mphkg"]
Sep 09 06:08:30.361427 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:30.361407    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fe8eaa-76889d76c4-mphkg"]
Sep 09 06:08:30.988992 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:30.988908    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh"
Sep 09 06:08:30.989418 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:30.988908    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh" event={"ID":"0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4","Type":"ContainerDied","Data":"67e6734faae53296bbc942b1b938811641d4b33f1874516cfb2a9d12a65be6b3"}
Sep 09 06:08:30.989418 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:30.989072    2579 scope.go:117] "RemoveContainer" containerID="e64578a3fda3f0d41bc8faaf3998a95b2e3f8a0aba9a504079acf7c1fa495fb7"
Sep 09 06:08:31.009475 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:31.009450    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh"]
Sep 09 06:08:31.013029 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:31.013009    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fe8eaa-75d9cbf94f-q76xh"]
Sep 09 06:08:31.148117 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:31.148091    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4" path="/var/lib/kubelet/pods/0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4/volumes"
Sep 09 06:08:31.148391 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:31.148379    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ba28f2b-d2f4-4fd5-a953-cdd07268ae47" path="/var/lib/kubelet/pods/3ba28f2b-d2f4-4fd5-a953-cdd07268ae47/volumes"
Sep 09 06:08:31.148729 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:31.148714    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ada3927-7884-47a4-8827-3cdb556fb999" path="/var/lib/kubelet/pods/6ada3927-7884-47a4-8827-3cdb556fb999/volumes"
Sep 09 06:08:31.149028 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:31.149017    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4d64e57-8756-43c5-a798-1690572ade99" path="/var/lib/kubelet/pods/a4d64e57-8756-43c5-a798-1690572ade99/volumes"
Sep 09 06:08:35.242214 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:35.242187    2579 scope.go:117] "RemoveContainer" containerID="9ea92371a34b93a05e4c07c7678b4e7d020d64c85504bf65469a595ba1817fff"
Sep 09 06:08:35.250286 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:35.250265    2579 scope.go:117] "RemoveContainer" containerID="032f390afb3d0de7b71fc02d950367b44e8b48dcdd6241b3e0b3b46e2df28a90"
Sep 09 06:08:36.798425 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:36.798388    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:08:36.974300 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:36.974266    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"]
Sep 09 06:08:36.974765 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:36.974745    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4d64e57-8756-43c5-a798-1690572ade99" containerName="payload-processing"
Sep 09 06:08:36.974765 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:36.974766    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4d64e57-8756-43c5-a798-1690572ade99" containerName="payload-processing"
Sep 09 06:08:36.974946 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:36.974783    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ada3927-7884-47a4-8827-3cdb556fb999" containerName="payload-pre-processing"
Sep 09 06:08:36.974946 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:36.974791    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ada3927-7884-47a4-8827-3cdb556fb999" containerName="payload-pre-processing"
Sep 09 06:08:36.974946 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:36.974801    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4" containerName="maas-api"
Sep 09 06:08:36.974946 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:36.974809    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4" containerName="maas-api"
Sep 09 06:08:36.974946 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:36.974881    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0762cdd5-6db6-4e44-b2be-ddbd5ff6a0c4" containerName="maas-api"
Sep 09 06:08:36.974946 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:36.974892    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ada3927-7884-47a4-8827-3cdb556fb999" containerName="payload-pre-processing"
Sep 09 06:08:36.974946 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:36.974903    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4d64e57-8756-43c5-a798-1690572ade99" containerName="payload-processing"
Sep 09 06:08:36.979759 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:36.979737    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"
Sep 09 06:08:36.984500 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:36.984474    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 09 06:08:36.984782 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:36.984764    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-6b5d89-openshift-default-dockercfg-xgcxl\""
Sep 09 06:08:36.995923 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:36.995904    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"]
Sep 09 06:08:37.083104 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:37.083065    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a027fe8-30f7-424a-9634-5341f728d4b0-istiod-ca-cert\") pod \"e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4\" (UID: \"0a027fe8-30f7-424a-9634-5341f728d4b0\") " pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"
Sep 09 06:08:37.083273 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:37.083112    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a027fe8-30f7-424a-9634-5341f728d4b0-istio-data\") pod \"e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4\" (UID: \"0a027fe8-30f7-424a-9634-5341f728d4b0\") " pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"
Sep 09 06:08:37.083273 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:37.083182    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a027fe8-30f7-424a-9634-5341f728d4b0-workload-certs\") pod \"e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4\" (UID: \"0a027fe8-30f7-424a-9634-5341f728d4b0\") " pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"
Sep 09 06:08:37.083273 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:37.083240    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a027fe8-30f7-424a-9634-5341f728d4b0-credential-socket\") pod \"e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4\" (UID: \"0a027fe8-30f7-424a-9634-5341f728d4b0\") " pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"
Sep 09 06:08:37.083273 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:37.083262    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcx49\" (UniqueName: \"kubernetes.io/projected/0a027fe8-30f7-424a-9634-5341f728d4b0-kube-api-access-jcx49\") pod \"e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4\" (UID: \"0a027fe8-30f7-424a-9634-5341f728d4b0\") " pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"
Sep 09 06:08:37.083429 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:37.083281    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a027fe8-30f7-424a-9634-5341f728d4b0-workload-socket\") pod \"e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4\" (UID: \"0a027fe8-30f7-424a-9634-5341f728d4b0\") " pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"
Sep 09 06:08:37.083429 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:37.083334    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a027fe8-30f7-424a-9634-5341f728d4b0-istio-envoy\") pod \"e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4\" (UID: \"0a027fe8-30f7-424a-9634-5341f728d4b0\") " pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"
Sep 09 06:08:37.083429 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:37.083352    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a027fe8-30f7-424a-9634-5341f728d4b0-istio-token\") pod \"e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4\" (UID: \"0a027fe8-30f7-424a-9634-5341f728d4b0\") " pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"
Sep 09 06:08:37.083429 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:37.083373    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a027fe8-30f7-424a-9634-5341f728d4b0-istio-podinfo\") pod \"e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4\" (UID: \"0a027fe8-30f7-424a-9634-5341f728d4b0\") " pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"
Sep 09 06:08:37.184877 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:37.184839    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a027fe8-30f7-424a-9634-5341f728d4b0-credential-socket\") pod \"e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4\" (UID: \"0a027fe8-30f7-424a-9634-5341f728d4b0\") " pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"
Sep 09 06:08:37.184877 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:37.184881    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcx49\" (UniqueName: \"kubernetes.io/projected/0a027fe8-30f7-424a-9634-5341f728d4b0-kube-api-access-jcx49\") pod \"e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4\" (UID: \"0a027fe8-30f7-424a-9634-5341f728d4b0\") " pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"
Sep 09 06:08:37.185120 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:37.184909    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a027fe8-30f7-424a-9634-5341f728d4b0-workload-socket\") pod \"e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4\" (UID: \"0a027fe8-30f7-424a-9634-5341f728d4b0\") " pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"
Sep 09 06:08:37.185120 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:37.184948    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a027fe8-30f7-424a-9634-5341f728d4b0-istio-envoy\") pod \"e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4\" (UID: \"0a027fe8-30f7-424a-9634-5341f728d4b0\") " pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"
Sep 09 06:08:37.185120 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:37.184970    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a027fe8-30f7-424a-9634-5341f728d4b0-istio-token\") pod \"e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4\" (UID: \"0a027fe8-30f7-424a-9634-5341f728d4b0\") " pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"
Sep 09 06:08:37.185120 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:37.184997    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a027fe8-30f7-424a-9634-5341f728d4b0-istio-podinfo\") pod \"e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4\" (UID: \"0a027fe8-30f7-424a-9634-5341f728d4b0\") " pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"
Sep 09 06:08:37.185120 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:37.185050    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a027fe8-30f7-424a-9634-5341f728d4b0-istiod-ca-cert\") pod \"e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4\" (UID: \"0a027fe8-30f7-424a-9634-5341f728d4b0\") " pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"
Sep 09 06:08:37.185120 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:37.185084    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a027fe8-30f7-424a-9634-5341f728d4b0-istio-data\") pod \"e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4\" (UID: \"0a027fe8-30f7-424a-9634-5341f728d4b0\") " pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"
Sep 09 06:08:37.185455 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:37.185129    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a027fe8-30f7-424a-9634-5341f728d4b0-workload-certs\") pod \"e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4\" (UID: \"0a027fe8-30f7-424a-9634-5341f728d4b0\") " pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"
Sep 09 06:08:37.185455 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:37.185260    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a027fe8-30f7-424a-9634-5341f728d4b0-credential-socket\") pod \"e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4\" (UID: \"0a027fe8-30f7-424a-9634-5341f728d4b0\") " pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"
Sep 09 06:08:37.185455 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:37.185320    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a027fe8-30f7-424a-9634-5341f728d4b0-workload-socket\") pod \"e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4\" (UID: \"0a027fe8-30f7-424a-9634-5341f728d4b0\") " pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"
Sep 09 06:08:37.185580 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:37.185478    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a027fe8-30f7-424a-9634-5341f728d4b0-workload-certs\") pod \"e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4\" (UID: \"0a027fe8-30f7-424a-9634-5341f728d4b0\") " pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"
Sep 09 06:08:37.185652 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:37.185606    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a027fe8-30f7-424a-9634-5341f728d4b0-istio-data\") pod \"e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4\" (UID: \"0a027fe8-30f7-424a-9634-5341f728d4b0\") " pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"
Sep 09 06:08:37.186043 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:37.186018    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a027fe8-30f7-424a-9634-5341f728d4b0-istiod-ca-cert\") pod \"e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4\" (UID: \"0a027fe8-30f7-424a-9634-5341f728d4b0\") " pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"
Sep 09 06:08:37.187484 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:37.187459    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a027fe8-30f7-424a-9634-5341f728d4b0-istio-envoy\") pod \"e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4\" (UID: \"0a027fe8-30f7-424a-9634-5341f728d4b0\") " pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"
Sep 09 06:08:37.187719 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:37.187701    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a027fe8-30f7-424a-9634-5341f728d4b0-istio-podinfo\") pod \"e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4\" (UID: \"0a027fe8-30f7-424a-9634-5341f728d4b0\") " pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"
Sep 09 06:08:37.193097 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:37.193078    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a027fe8-30f7-424a-9634-5341f728d4b0-istio-token\") pod \"e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4\" (UID: \"0a027fe8-30f7-424a-9634-5341f728d4b0\") " pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"
Sep 09 06:08:37.193251 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:37.193233    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcx49\" (UniqueName: \"kubernetes.io/projected/0a027fe8-30f7-424a-9634-5341f728d4b0-kube-api-access-jcx49\") pod \"e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4\" (UID: \"0a027fe8-30f7-424a-9634-5341f728d4b0\") " pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"
Sep 09 06:08:37.291956 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:37.291931    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"
Sep 09 06:08:37.566401 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:37.566264    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"]
Sep 09 06:08:37.568979 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:08:37.568951    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a027fe8_30f7_424a_9634_5341f728d4b0.slice/crio-9fe6774d8c37ebc97716ae79a87347ffb0d45c986a4ad630b6af79d6d8ee6bde WatchSource:0}: Error finding container 9fe6774d8c37ebc97716ae79a87347ffb0d45c986a4ad630b6af79d6d8ee6bde: Status 404 returned error can't find the container with id 9fe6774d8c37ebc97716ae79a87347ffb0d45c986a4ad630b6af79d6d8ee6bde
Sep 09 06:08:37.968472 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:37.968441    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-389cd1da-7cddb7cbfc-x2nn9"
Sep 09 06:08:38.024393 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:38.024366    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4" event={"ID":"0a027fe8-30f7-424a-9634-5341f728d4b0","Type":"ContainerStarted","Data":"9fe6774d8c37ebc97716ae79a87347ffb0d45c986a4ad630b6af79d6d8ee6bde"}
Sep 09 06:08:40.325558 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:40.325519    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Sep 09 06:08:40.325827 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:40.325601    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Sep 09 06:08:40.325827 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:40.325652    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Sep 09 06:08:41.043058 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:41.043025    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4" event={"ID":"0a027fe8-30f7-424a-9634-5341f728d4b0","Type":"ContainerStarted","Data":"18aac7bddf92ff25354888ff84c7661a007064c924a765d44c8dc802cf6e56d8"}
Sep 09 06:08:41.061606 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:41.061565    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4" podStartSLOduration=2.307504926 podStartE2EDuration="5.061552205s" totalImagesPullingTime="2.754047279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:08:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:08:41.060395244 +0000 UTC m=+1146.400883153" watchObservedRunningTime="2026-09-09 06:08:41.061552205 +0000 UTC m=+1146.402040114"
Sep 09 06:08:41.292365 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:41.292338    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"
Sep 09 06:08:41.297823 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:41.297753    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"
Sep 09 06:08:42.047993 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:42.047962    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"
Sep 09 06:08:42.048951 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:08:42.048931    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"
Sep 09 06:09:00.594211 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:00.594129    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:09:06.310750 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:06.310714    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-389cd1da/e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk"]
Sep 09 06:09:06.313886 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:06.313867    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-389cd1da/e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk"
Sep 09 06:09:06.315869 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:06.315845    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-389cd1da\"/\"kube-root-ca.crt\""
Sep 09 06:09:06.315975 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:06.315957    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-389cd1da\"/\"openshift-service-ca.crt\""
Sep 09 06:09:06.316881 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:06.316857    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-389cd1da\"/\"default-dockercfg-rjhkm\""
Sep 09 06:09:06.316964 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:06.316880    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-389cd1da\"/\"e2e-lifecycle-model-389cd1da-kserve-self-signed-certs\""
Sep 09 06:09:06.325599 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:06.325572    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-389cd1da/e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk"]
Sep 09 06:09:06.410449 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:06.410414    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b3e7231-ce7d-43d0-b13a-ffda818a3ebf-tmp-dir\") pod \"e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk\" (UID: \"5b3e7231-ce7d-43d0-b13a-ffda818a3ebf\") " pod="ai-tenant-e2e-mt-389cd1da/e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk"
Sep 09 06:09:06.410449 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:06.410453    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b3e7231-ce7d-43d0-b13a-ffda818a3ebf-dshm\") pod \"e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk\" (UID: \"5b3e7231-ce7d-43d0-b13a-ffda818a3ebf\") " pod="ai-tenant-e2e-mt-389cd1da/e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk"
Sep 09 06:09:06.410663 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:06.410473    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b3e7231-ce7d-43d0-b13a-ffda818a3ebf-home\") pod \"e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk\" (UID: \"5b3e7231-ce7d-43d0-b13a-ffda818a3ebf\") " pod="ai-tenant-e2e-mt-389cd1da/e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk"
Sep 09 06:09:06.410663 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:06.410552    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7ncv\" (UniqueName: \"kubernetes.io/projected/5b3e7231-ce7d-43d0-b13a-ffda818a3ebf-kube-api-access-l7ncv\") pod \"e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk\" (UID: \"5b3e7231-ce7d-43d0-b13a-ffda818a3ebf\") " pod="ai-tenant-e2e-mt-389cd1da/e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk"
Sep 09 06:09:06.410749 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:06.410667    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b3e7231-ce7d-43d0-b13a-ffda818a3ebf-tls-certs\") pod \"e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk\" (UID: \"5b3e7231-ce7d-43d0-b13a-ffda818a3ebf\") " pod="ai-tenant-e2e-mt-389cd1da/e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk"
Sep 09 06:09:06.410749 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:06.410711    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b3e7231-ce7d-43d0-b13a-ffda818a3ebf-model-cache\") pod \"e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk\" (UID: \"5b3e7231-ce7d-43d0-b13a-ffda818a3ebf\") " pod="ai-tenant-e2e-mt-389cd1da/e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk"
Sep 09 06:09:06.511411 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:06.511380    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b3e7231-ce7d-43d0-b13a-ffda818a3ebf-tmp-dir\") pod \"e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk\" (UID: \"5b3e7231-ce7d-43d0-b13a-ffda818a3ebf\") " pod="ai-tenant-e2e-mt-389cd1da/e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk"
Sep 09 06:09:06.511411 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:06.511413    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b3e7231-ce7d-43d0-b13a-ffda818a3ebf-dshm\") pod \"e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk\" (UID: \"5b3e7231-ce7d-43d0-b13a-ffda818a3ebf\") " pod="ai-tenant-e2e-mt-389cd1da/e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk"
Sep 09 06:09:06.511595 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:06.511430    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b3e7231-ce7d-43d0-b13a-ffda818a3ebf-home\") pod \"e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk\" (UID: \"5b3e7231-ce7d-43d0-b13a-ffda818a3ebf\") " pod="ai-tenant-e2e-mt-389cd1da/e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk"
Sep 09 06:09:06.511595 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:06.511548    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7ncv\" (UniqueName: \"kubernetes.io/projected/5b3e7231-ce7d-43d0-b13a-ffda818a3ebf-kube-api-access-l7ncv\") pod \"e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk\" (UID: \"5b3e7231-ce7d-43d0-b13a-ffda818a3ebf\") " pod="ai-tenant-e2e-mt-389cd1da/e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk"
Sep 09 06:09:06.511729 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:06.511672    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b3e7231-ce7d-43d0-b13a-ffda818a3ebf-tls-certs\") pod \"e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk\" (UID: \"5b3e7231-ce7d-43d0-b13a-ffda818a3ebf\") " pod="ai-tenant-e2e-mt-389cd1da/e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk"
Sep 09 06:09:06.511790 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:06.511738    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b3e7231-ce7d-43d0-b13a-ffda818a3ebf-model-cache\") pod \"e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk\" (UID: \"5b3e7231-ce7d-43d0-b13a-ffda818a3ebf\") " pod="ai-tenant-e2e-mt-389cd1da/e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk"
Sep 09 06:09:06.511848 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:06.511832    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b3e7231-ce7d-43d0-b13a-ffda818a3ebf-tmp-dir\") pod \"e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk\" (UID: \"5b3e7231-ce7d-43d0-b13a-ffda818a3ebf\") " pod="ai-tenant-e2e-mt-389cd1da/e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk"
Sep 09 06:09:06.511910 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:06.511841    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b3e7231-ce7d-43d0-b13a-ffda818a3ebf-home\") pod \"e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk\" (UID: \"5b3e7231-ce7d-43d0-b13a-ffda818a3ebf\") " pod="ai-tenant-e2e-mt-389cd1da/e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk"
Sep 09 06:09:06.512068 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:06.512050    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b3e7231-ce7d-43d0-b13a-ffda818a3ebf-model-cache\") pod \"e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk\" (UID: \"5b3e7231-ce7d-43d0-b13a-ffda818a3ebf\") " pod="ai-tenant-e2e-mt-389cd1da/e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk"
Sep 09 06:09:06.513990 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:06.513970    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b3e7231-ce7d-43d0-b13a-ffda818a3ebf-dshm\") pod \"e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk\" (UID: \"5b3e7231-ce7d-43d0-b13a-ffda818a3ebf\") " pod="ai-tenant-e2e-mt-389cd1da/e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk"
Sep 09 06:09:06.514225 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:06.514207    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b3e7231-ce7d-43d0-b13a-ffda818a3ebf-tls-certs\") pod \"e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk\" (UID: \"5b3e7231-ce7d-43d0-b13a-ffda818a3ebf\") " pod="ai-tenant-e2e-mt-389cd1da/e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk"
Sep 09 06:09:06.522302 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:06.522258    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7ncv\" (UniqueName: \"kubernetes.io/projected/5b3e7231-ce7d-43d0-b13a-ffda818a3ebf-kube-api-access-l7ncv\") pod \"e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk\" (UID: \"5b3e7231-ce7d-43d0-b13a-ffda818a3ebf\") " pod="ai-tenant-e2e-mt-389cd1da/e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk"
Sep 09 06:09:06.628177 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:06.628091    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-389cd1da/e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk"
Sep 09 06:09:06.902951 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:06.902924    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-389cd1da/e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk"]
Sep 09 06:09:06.905434 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:09:06.905404    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b3e7231_ce7d_43d0_b13a_ffda818a3ebf.slice/crio-0123b02e25bba9a1b95490b4205a339807904a810d2cfc20bc6abb3a68e3b524 WatchSource:0}: Error finding container 0123b02e25bba9a1b95490b4205a339807904a810d2cfc20bc6abb3a68e3b524: Status 404 returned error can't find the container with id 0123b02e25bba9a1b95490b4205a339807904a810d2cfc20bc6abb3a68e3b524
Sep 09 06:09:07.169754 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:07.169677    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-389cd1da/e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk" event={"ID":"5b3e7231-ce7d-43d0-b13a-ffda818a3ebf","Type":"ContainerStarted","Data":"356ec11b9ba5b4e44bd7868753c65c3ecf75c95104ec4d18304825830c6ef180"}
Sep 09 06:09:07.169754 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:07.169715    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-389cd1da/e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk" event={"ID":"5b3e7231-ce7d-43d0-b13a-ffda818a3ebf","Type":"ContainerStarted","Data":"0123b02e25bba9a1b95490b4205a339807904a810d2cfc20bc6abb3a68e3b524"}
Sep 09 06:09:07.186050 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:07.185985    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-389cd1da/e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk" podStartSLOduration=1.1859717 podStartE2EDuration="1.1859717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:09:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:09:07.184547567 +0000 UTC m=+1172.525035475" watchObservedRunningTime="2026-09-09 06:09:07.1859717 +0000 UTC m=+1172.526459608"
Sep 09 06:09:11.191121 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:11.191089    2579 generic.go:356] "Generic (PLEG): container finished" podID="0a027fe8-30f7-424a-9634-5341f728d4b0" containerID="18aac7bddf92ff25354888ff84c7661a007064c924a765d44c8dc802cf6e56d8" exitCode=0
Sep 09 06:09:11.191589 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:11.191173    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4" event={"ID":"0a027fe8-30f7-424a-9634-5341f728d4b0","Type":"ContainerDied","Data":"18aac7bddf92ff25354888ff84c7661a007064c924a765d44c8dc802cf6e56d8"}
Sep 09 06:09:11.191589 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:11.191544    2579 scope.go:117] "RemoveContainer" containerID="18aac7bddf92ff25354888ff84c7661a007064c924a765d44c8dc802cf6e56d8"
Sep 09 06:09:11.192686 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:11.192657    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Sep 09 06:09:11.192747 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:11.192730    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Sep 09 06:09:11.192784 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:11.192761    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Sep 09 06:09:12.048843 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:12.048808    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"
Sep 09 06:09:12.197141 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:12.197106    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4" event={"ID":"0a027fe8-30f7-424a-9634-5341f728d4b0","Type":"ContainerStarted","Data":"a648646b8865fafc9345de206318e04436a65c8874e756a5b26b794101950bb9"}
Sep 09 06:09:12.292461 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:12.292428    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"
Sep 09 06:09:12.293757 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:12.293735    2579 patch_prober.go:28] interesting pod/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Sep 09 06:09:12.293877 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:12.293790    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4" podUID="0a027fe8-30f7-424a-9634-5341f728d4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Sep 09 06:09:13.278221 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:13.278194    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cf6be-7f5595974-mqgd8"]
Sep 09 06:09:13.281501 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:13.281481    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cf6be-7f5595974-mqgd8"
Sep 09 06:09:13.283674 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:13.283648    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-7cf6be\""
Sep 09 06:09:13.284371 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:13.284252    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-7cf6be-dockercfg-4md7g\""
Sep 09 06:09:13.292898 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:13.292422    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cf6be-7f5595974-mqgd8"]
Sep 09 06:09:13.293062 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:13.292988    2579 patch_prober.go:28] interesting pod/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Sep 09 06:09:13.294351 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:13.294318    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4" podUID="0a027fe8-30f7-424a-9634-5341f728d4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Sep 09 06:09:13.325649 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:13.324969    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7cf6be-59fdf7cbf5-fv8wn"]
Sep 09 06:09:13.352525 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:13.352478    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7cf6be-59fdf7cbf5-fv8wn"]
Sep 09 06:09:13.352722 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:13.352653    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7cf6be-59fdf7cbf5-fv8wn"
Sep 09 06:09:13.373852 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:13.373824    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7pmr\" (UniqueName: \"kubernetes.io/projected/2666d09c-dba9-4a67-8b69-df034c55ce9a-kube-api-access-s7pmr\") pod \"payload-pre-processing-e2e-shared-b-w3-7cf6be-7f5595974-mqgd8\" (UID: \"2666d09c-dba9-4a67-8b69-df034c55ce9a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cf6be-7f5595974-mqgd8"
Sep 09 06:09:13.373962 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:13.373860    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2666d09c-dba9-4a67-8b69-df034c55ce9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7cf6be-7f5595974-mqgd8\" (UID: \"2666d09c-dba9-4a67-8b69-df034c55ce9a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cf6be-7f5595974-mqgd8"
Sep 09 06:09:13.474264 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:13.474231    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6qr4\" (UniqueName: \"kubernetes.io/projected/3f1bc250-9dcd-437f-ae35-af778ebdad1e-kube-api-access-c6qr4\") pod \"payload-processing-e2e-shared-b-w3-7cf6be-59fdf7cbf5-fv8wn\" (UID: \"3f1bc250-9dcd-437f-ae35-af778ebdad1e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7cf6be-59fdf7cbf5-fv8wn"
Sep 09 06:09:13.474264 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:13.474268    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f1bc250-9dcd-437f-ae35-af778ebdad1e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7cf6be-59fdf7cbf5-fv8wn\" (UID: \"3f1bc250-9dcd-437f-ae35-af778ebdad1e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7cf6be-59fdf7cbf5-fv8wn"
Sep 09 06:09:13.474494 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:13.474354    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7pmr\" (UniqueName: \"kubernetes.io/projected/2666d09c-dba9-4a67-8b69-df034c55ce9a-kube-api-access-s7pmr\") pod \"payload-pre-processing-e2e-shared-b-w3-7cf6be-7f5595974-mqgd8\" (UID: \"2666d09c-dba9-4a67-8b69-df034c55ce9a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cf6be-7f5595974-mqgd8"
Sep 09 06:09:13.474494 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:13.474392    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2666d09c-dba9-4a67-8b69-df034c55ce9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7cf6be-7f5595974-mqgd8\" (UID: \"2666d09c-dba9-4a67-8b69-df034c55ce9a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cf6be-7f5595974-mqgd8"
Sep 09 06:09:13.475048 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:13.475031    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2666d09c-dba9-4a67-8b69-df034c55ce9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7cf6be-7f5595974-mqgd8\" (UID: \"2666d09c-dba9-4a67-8b69-df034c55ce9a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cf6be-7f5595974-mqgd8"
Sep 09 06:09:13.484468 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:13.484447    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7pmr\" (UniqueName: \"kubernetes.io/projected/2666d09c-dba9-4a67-8b69-df034c55ce9a-kube-api-access-s7pmr\") pod \"payload-pre-processing-e2e-shared-b-w3-7cf6be-7f5595974-mqgd8\" (UID: \"2666d09c-dba9-4a67-8b69-df034c55ce9a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cf6be-7f5595974-mqgd8"
Sep 09 06:09:13.575583 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:13.575556    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6qr4\" (UniqueName: \"kubernetes.io/projected/3f1bc250-9dcd-437f-ae35-af778ebdad1e-kube-api-access-c6qr4\") pod \"payload-processing-e2e-shared-b-w3-7cf6be-59fdf7cbf5-fv8wn\" (UID: \"3f1bc250-9dcd-437f-ae35-af778ebdad1e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7cf6be-59fdf7cbf5-fv8wn"
Sep 09 06:09:13.575729 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:13.575592    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f1bc250-9dcd-437f-ae35-af778ebdad1e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7cf6be-59fdf7cbf5-fv8wn\" (UID: \"3f1bc250-9dcd-437f-ae35-af778ebdad1e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7cf6be-59fdf7cbf5-fv8wn"
Sep 09 06:09:13.576252 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:13.576235    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f1bc250-9dcd-437f-ae35-af778ebdad1e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7cf6be-59fdf7cbf5-fv8wn\" (UID: \"3f1bc250-9dcd-437f-ae35-af778ebdad1e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7cf6be-59fdf7cbf5-fv8wn"
Sep 09 06:09:13.583511 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:13.583482    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6qr4\" (UniqueName: \"kubernetes.io/projected/3f1bc250-9dcd-437f-ae35-af778ebdad1e-kube-api-access-c6qr4\") pod \"payload-processing-e2e-shared-b-w3-7cf6be-59fdf7cbf5-fv8wn\" (UID: \"3f1bc250-9dcd-437f-ae35-af778ebdad1e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7cf6be-59fdf7cbf5-fv8wn"
Sep 09 06:09:13.600268 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:13.600230    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cf6be-7f5595974-mqgd8"
Sep 09 06:09:13.662901 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:13.662855    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7cf6be-59fdf7cbf5-fv8wn"
Sep 09 06:09:13.892316 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:13.892284    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cf6be-7f5595974-mqgd8"]
Sep 09 06:09:13.894804 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:09:13.894777    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2666d09c_dba9_4a67_8b69_df034c55ce9a.slice/crio-c74404be11a2ef7562348ada500e8b1cec6a49ee51c7837118be82556cfe2241 WatchSource:0}: Error finding container c74404be11a2ef7562348ada500e8b1cec6a49ee51c7837118be82556cfe2241: Status 404 returned error can't find the container with id c74404be11a2ef7562348ada500e8b1cec6a49ee51c7837118be82556cfe2241
Sep 09 06:09:13.971404 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:13.971382    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7cf6be-59fdf7cbf5-fv8wn"]
Sep 09 06:09:13.975692 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:09:13.975667    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f1bc250_9dcd_437f_ae35_af778ebdad1e.slice/crio-80dac74ac731dbb0296a89067e517ddda269aafe18f33cfcf41e3c2ca64580d1 WatchSource:0}: Error finding container 80dac74ac731dbb0296a89067e517ddda269aafe18f33cfcf41e3c2ca64580d1: Status 404 returned error can't find the container with id 80dac74ac731dbb0296a89067e517ddda269aafe18f33cfcf41e3c2ca64580d1
Sep 09 06:09:14.209239 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:14.209147    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7cf6be-59fdf7cbf5-fv8wn" event={"ID":"3f1bc250-9dcd-437f-ae35-af778ebdad1e","Type":"ContainerStarted","Data":"0e35f676b177b456c654ba7b00855caebd6952bf69a83b3608bba6d229828958"}
Sep 09 06:09:14.209239 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:14.209205    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7cf6be-59fdf7cbf5-fv8wn" event={"ID":"3f1bc250-9dcd-437f-ae35-af778ebdad1e","Type":"ContainerStarted","Data":"80dac74ac731dbb0296a89067e517ddda269aafe18f33cfcf41e3c2ca64580d1"}
Sep 09 06:09:14.209239 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:14.209236    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7cf6be-59fdf7cbf5-fv8wn"
Sep 09 06:09:14.210810 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:14.210784    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cf6be-7f5595974-mqgd8" event={"ID":"2666d09c-dba9-4a67-8b69-df034c55ce9a","Type":"ContainerStarted","Data":"a4cc44ad704a3ecc9b85bd3b591e87edd2335505a12c04e6a1c318b9c7a29e00"}
Sep 09 06:09:14.210923 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:14.210832    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cf6be-7f5595974-mqgd8" event={"ID":"2666d09c-dba9-4a67-8b69-df034c55ce9a","Type":"ContainerStarted","Data":"c74404be11a2ef7562348ada500e8b1cec6a49ee51c7837118be82556cfe2241"}
Sep 09 06:09:14.211039 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:14.211023    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cf6be-7f5595974-mqgd8"
Sep 09 06:09:14.245141 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:14.245071    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cf6be-7f5595974-mqgd8" podStartSLOduration=1.245050623 podStartE2EDuration="1.245050623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:09:14.243829182 +0000 UTC m=+1179.584317094" watchObservedRunningTime="2026-09-09 06:09:14.245050623 +0000 UTC m=+1179.585538536"
Sep 09 06:09:14.245827 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:14.245781    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7cf6be-59fdf7cbf5-fv8wn" podStartSLOduration=1.2457666729999999 podStartE2EDuration="1.245766673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:09:14.22785829 +0000 UTC m=+1179.568346201" watchObservedRunningTime="2026-09-09 06:09:14.245766673 +0000 UTC m=+1179.586254588"
Sep 09 06:09:14.292888 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:14.292857    2579 patch_prober.go:28] interesting pod/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Sep 09 06:09:14.293285 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:14.292920    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4" podUID="0a027fe8-30f7-424a-9634-5341f728d4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Sep 09 06:09:15.292589 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:15.292560    2579 patch_prober.go:28] interesting pod/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Sep 09 06:09:15.292782 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:15.292640    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4" podUID="0a027fe8-30f7-424a-9634-5341f728d4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Sep 09 06:09:16.293300 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:16.293262    2579 patch_prober.go:28] interesting pod/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Sep 09 06:09:16.293702 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:16.293330    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4" podUID="0a027fe8-30f7-424a-9634-5341f728d4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Sep 09 06:09:16.629208 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:16.629166    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-389cd1da/e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk"
Sep 09 06:09:16.629930 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:16.629902    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-389cd1da/e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk"
Sep 09 06:09:16.655284 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:16.655259    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-389cd1da/e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk"
Sep 09 06:09:17.257813 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:17.257783    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-389cd1da/e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk"
Sep 09 06:09:17.292594 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:17.292560    2579 patch_prober.go:28] interesting pod/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Sep 09 06:09:17.292785 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:17.292649    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4" podUID="0a027fe8-30f7-424a-9634-5341f728d4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Sep 09 06:09:18.296822 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:18.296797    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"
Sep 09 06:09:18.297214 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:18.297160    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"
Sep 09 06:09:18.297772 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:18.297755    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"
Sep 09 06:09:19.614047 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:19.614014    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:09:23.019450 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:23.019414    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg"]
Sep 09 06:09:23.023369 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:23.023354    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg"
Sep 09 06:09:23.025511 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:23.025490    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-43eb2d\"/\"e2e-conflict-a-431541ed-kserve-self-signed-certs\""
Sep 09 06:09:23.032719 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:23.032697    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg"]
Sep 09 06:09:23.159938 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:23.159910    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a5192af-cf47-4d86-a163-8e45882e0fb3-tls-certs\") pod \"e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg\" (UID: \"9a5192af-cf47-4d86-a163-8e45882e0fb3\") " pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg"
Sep 09 06:09:23.160074 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:23.159941    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a5192af-cf47-4d86-a163-8e45882e0fb3-model-cache\") pod \"e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg\" (UID: \"9a5192af-cf47-4d86-a163-8e45882e0fb3\") " pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg"
Sep 09 06:09:23.160074 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:23.159965    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a5192af-cf47-4d86-a163-8e45882e0fb3-dshm\") pod \"e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg\" (UID: \"9a5192af-cf47-4d86-a163-8e45882e0fb3\") " pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg"
Sep 09 06:09:23.160074 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:23.160039    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a5192af-cf47-4d86-a163-8e45882e0fb3-tmp-dir\") pod \"e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg\" (UID: \"9a5192af-cf47-4d86-a163-8e45882e0fb3\") " pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg"
Sep 09 06:09:23.160233 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:23.160079    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a5192af-cf47-4d86-a163-8e45882e0fb3-home\") pod \"e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg\" (UID: \"9a5192af-cf47-4d86-a163-8e45882e0fb3\") " pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg"
Sep 09 06:09:23.160233 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:23.160126    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcvgg\" (UniqueName: \"kubernetes.io/projected/9a5192af-cf47-4d86-a163-8e45882e0fb3-kube-api-access-jcvgg\") pod \"e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg\" (UID: \"9a5192af-cf47-4d86-a163-8e45882e0fb3\") " pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg"
Sep 09 06:09:23.260766 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:23.260738    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a5192af-cf47-4d86-a163-8e45882e0fb3-tls-certs\") pod \"e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg\" (UID: \"9a5192af-cf47-4d86-a163-8e45882e0fb3\") " pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg"
Sep 09 06:09:23.260924 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:23.260787    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a5192af-cf47-4d86-a163-8e45882e0fb3-model-cache\") pod \"e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg\" (UID: \"9a5192af-cf47-4d86-a163-8e45882e0fb3\") " pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg"
Sep 09 06:09:23.260924 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:23.260831    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a5192af-cf47-4d86-a163-8e45882e0fb3-dshm\") pod \"e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg\" (UID: \"9a5192af-cf47-4d86-a163-8e45882e0fb3\") " pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg"
Sep 09 06:09:23.260924 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:23.260898    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a5192af-cf47-4d86-a163-8e45882e0fb3-tmp-dir\") pod \"e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg\" (UID: \"9a5192af-cf47-4d86-a163-8e45882e0fb3\") " pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg"
Sep 09 06:09:23.261102 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:23.260957    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a5192af-cf47-4d86-a163-8e45882e0fb3-home\") pod \"e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg\" (UID: \"9a5192af-cf47-4d86-a163-8e45882e0fb3\") " pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg"
Sep 09 06:09:23.261102 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:23.261091    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcvgg\" (UniqueName: \"kubernetes.io/projected/9a5192af-cf47-4d86-a163-8e45882e0fb3-kube-api-access-jcvgg\") pod \"e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg\" (UID: \"9a5192af-cf47-4d86-a163-8e45882e0fb3\") " pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg"
Sep 09 06:09:23.261225 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:23.261204    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a5192af-cf47-4d86-a163-8e45882e0fb3-model-cache\") pod \"e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg\" (UID: \"9a5192af-cf47-4d86-a163-8e45882e0fb3\") " pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg"
Sep 09 06:09:23.261287 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:23.261266    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a5192af-cf47-4d86-a163-8e45882e0fb3-tmp-dir\") pod \"e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg\" (UID: \"9a5192af-cf47-4d86-a163-8e45882e0fb3\") " pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg"
Sep 09 06:09:23.261455 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:23.261433    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a5192af-cf47-4d86-a163-8e45882e0fb3-home\") pod \"e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg\" (UID: \"9a5192af-cf47-4d86-a163-8e45882e0fb3\") " pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg"
Sep 09 06:09:23.265655 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:23.263956    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a5192af-cf47-4d86-a163-8e45882e0fb3-tls-certs\") pod \"e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg\" (UID: \"9a5192af-cf47-4d86-a163-8e45882e0fb3\") " pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg"
Sep 09 06:09:23.265655 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:23.264303    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a5192af-cf47-4d86-a163-8e45882e0fb3-dshm\") pod \"e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg\" (UID: \"9a5192af-cf47-4d86-a163-8e45882e0fb3\") " pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg"
Sep 09 06:09:23.270096 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:23.270042    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcvgg\" (UniqueName: \"kubernetes.io/projected/9a5192af-cf47-4d86-a163-8e45882e0fb3-kube-api-access-jcvgg\") pod \"e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg\" (UID: \"9a5192af-cf47-4d86-a163-8e45882e0fb3\") " pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg"
Sep 09 06:09:23.333119 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:23.333094    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg"
Sep 09 06:09:23.603675 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:23.603646    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg"]
Sep 09 06:09:23.606081 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:09:23.606048    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a5192af_cf47_4d86_a163_8e45882e0fb3.slice/crio-26c56f4947285e1a1c00b09171c98eb6426ec7ad0bdc25470f0492971f360d27 WatchSource:0}: Error finding container 26c56f4947285e1a1c00b09171c98eb6426ec7ad0bdc25470f0492971f360d27: Status 404 returned error can't find the container with id 26c56f4947285e1a1c00b09171c98eb6426ec7ad0bdc25470f0492971f360d27
Sep 09 06:09:24.262957 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:24.262923    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg" event={"ID":"9a5192af-cf47-4d86-a163-8e45882e0fb3","Type":"ContainerStarted","Data":"ca76262fad03d82cf6451a41df4bc7135ee79178dffa1a7243e398d74bc9e457"}
Sep 09 06:09:24.263325 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:24.262970    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg" event={"ID":"9a5192af-cf47-4d86-a163-8e45882e0fb3","Type":"ContainerStarted","Data":"26c56f4947285e1a1c00b09171c98eb6426ec7ad0bdc25470f0492971f360d27"}
Sep 09 06:09:24.278738 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:24.278689    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg" podStartSLOduration=1.278674751 podStartE2EDuration="1.278674751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:09:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:09:24.278177569 +0000 UTC m=+1189.618665478" watchObservedRunningTime="2026-09-09 06:09:24.278674751 +0000 UTC m=+1189.619162659"
Sep 09 06:09:25.217215 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:25.217186    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cf6be-7f5595974-mqgd8"
Sep 09 06:09:25.217403 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:25.217260    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7cf6be-59fdf7cbf5-fv8wn"
Sep 09 06:09:33.334101 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:33.334066    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg"
Sep 09 06:09:33.334611 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:33.334111    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg"
Sep 09 06:09:33.346391 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:33.346367    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg"
Sep 09 06:09:34.324421 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:34.324395    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg"
Sep 09 06:09:45.194801 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:45.194768    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:09:59.406053 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:59.406020    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-389cd1da-v8xqq"]
Sep 09 06:09:59.410712 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:59.410692    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-389cd1da-v8xqq"
Sep 09 06:09:59.413018 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:59.412997    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 06:09:59.413086 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:59.413013    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ksjk6\""
Sep 09 06:09:59.416547 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:59.416522    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-389cd1da-v8xqq"]
Sep 09 06:09:59.468866 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:59.468840    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34d43acd-ea31-468e-ad87-c30e9006f3ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-389cd1da-v8xqq\" (UID: \"34d43acd-ea31-468e-ad87-c30e9006f3ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-389cd1da-v8xqq"
Sep 09 06:09:59.569443 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:59.569412    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34d43acd-ea31-468e-ad87-c30e9006f3ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-389cd1da-v8xqq\" (UID: \"34d43acd-ea31-468e-ad87-c30e9006f3ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-389cd1da-v8xqq"
Sep 09 06:09:59.570062 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:59.570038    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34d43acd-ea31-468e-ad87-c30e9006f3ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-389cd1da-v8xqq\" (UID: \"34d43acd-ea31-468e-ad87-c30e9006f3ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-389cd1da-v8xqq"
Sep 09 06:09:59.722257 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:59.722176    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-389cd1da-v8xqq"
Sep 09 06:09:59.991112 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:09:59.991013    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-389cd1da-v8xqq"]
Sep 09 06:09:59.993802 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:09:59.993771    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34d43acd_ea31_468e_ad87_c30e9006f3ad.slice/crio-33c9157358940d8b42e6e4820961facde993c374d08a4d728f7c2b5c3f4f04a6 WatchSource:0}: Error finding container 33c9157358940d8b42e6e4820961facde993c374d08a4d728f7c2b5c3f4f04a6: Status 404 returned error can't find the container with id 33c9157358940d8b42e6e4820961facde993c374d08a4d728f7c2b5c3f4f04a6
Sep 09 06:10:00.438089 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:00.438057    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-389cd1da-v8xqq" event={"ID":"34d43acd-ea31-468e-ad87-c30e9006f3ad","Type":"ContainerStarted","Data":"cf6f1ae0dc04e945c85c1d00407c0a375e2d190994a722c3cf00da002bf4b132"}
Sep 09 06:10:00.438525 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:00.438110    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-389cd1da-v8xqq" event={"ID":"34d43acd-ea31-468e-ad87-c30e9006f3ad","Type":"ContainerStarted","Data":"33c9157358940d8b42e6e4820961facde993c374d08a4d728f7c2b5c3f4f04a6"}
Sep 09 06:10:00.453085 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:00.453043    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-389cd1da-v8xqq" podStartSLOduration=1.453027546 podStartE2EDuration="1.453027546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:09:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:10:00.451732201 +0000 UTC m=+1225.792220110" watchObservedRunningTime="2026-09-09 06:10:00.453027546 +0000 UTC m=+1225.793515506"
Sep 09 06:10:01.445109 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:01.445073    2579 generic.go:356] "Generic (PLEG): container finished" podID="34d43acd-ea31-468e-ad87-c30e9006f3ad" containerID="cf6f1ae0dc04e945c85c1d00407c0a375e2d190994a722c3cf00da002bf4b132" exitCode=0
Sep 09 06:10:01.445481 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:01.445164    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-389cd1da-v8xqq" event={"ID":"34d43acd-ea31-468e-ad87-c30e9006f3ad","Type":"ContainerDied","Data":"cf6f1ae0dc04e945c85c1d00407c0a375e2d190994a722c3cf00da002bf4b132"}
Sep 09 06:10:02.579900 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:02.579877    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-389cd1da-v8xqq"
Sep 09 06:10:02.592153 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:02.592133    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34d43acd-ea31-468e-ad87-c30e9006f3ad-maas-api-service-ca\") pod \"34d43acd-ea31-468e-ad87-c30e9006f3ad\" (UID: \"34d43acd-ea31-468e-ad87-c30e9006f3ad\") "
Sep 09 06:10:02.592449 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:02.592426    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34d43acd-ea31-468e-ad87-c30e9006f3ad-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "34d43acd-ea31-468e-ad87-c30e9006f3ad" (UID: "34d43acd-ea31-468e-ad87-c30e9006f3ad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:10:02.693484 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:02.693454    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34d43acd-ea31-468e-ad87-c30e9006f3ad-maas-api-service-ca\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:10:03.457564 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:03.457474    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-389cd1da-v8xqq"
Sep 09 06:10:03.457564 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:03.457513    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-389cd1da-v8xqq" event={"ID":"34d43acd-ea31-468e-ad87-c30e9006f3ad","Type":"ContainerDied","Data":"33c9157358940d8b42e6e4820961facde993c374d08a4d728f7c2b5c3f4f04a6"}
Sep 09 06:10:03.457809 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:03.457569    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33c9157358940d8b42e6e4820961facde993c374d08a4d728f7c2b5c3f4f04a6"
Sep 09 06:10:04.404174 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:04.404141    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:10:15.007124 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:15.007086    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-389cd1da-7cddb7cbfc-x2nn9"]
Sep 09 06:10:15.007685 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:15.007315    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-389cd1da-7cddb7cbfc-x2nn9" podUID="9f66df18-7b8a-4a9c-bdc3-8bc17061c133" containerName="payload-processing" containerID="cri-o://9c77dca60f99617a4362c15b4241c44e8e69c1d7038238830b61314193a69f01" gracePeriod=30
Sep 09 06:10:15.161951 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:15.161913    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-389cd1da-v8xqq"]
Sep 09 06:10:15.163665 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:15.163620    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-389cd1da-v8xqq"]
Sep 09 06:10:15.258090 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:15.258024    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-389cd1da-7cddb7cbfc-x2nn9"
Sep 09 06:10:15.301308 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:15.301272    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52l7z\" (UniqueName: \"kubernetes.io/projected/9f66df18-7b8a-4a9c-bdc3-8bc17061c133-kube-api-access-52l7z\") pod \"9f66df18-7b8a-4a9c-bdc3-8bc17061c133\" (UID: \"9f66df18-7b8a-4a9c-bdc3-8bc17061c133\") "
Sep 09 06:10:15.301500 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:15.301330    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f66df18-7b8a-4a9c-bdc3-8bc17061c133-plugins-config-volume\") pod \"9f66df18-7b8a-4a9c-bdc3-8bc17061c133\" (UID: \"9f66df18-7b8a-4a9c-bdc3-8bc17061c133\") "
Sep 09 06:10:15.301697 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:15.301669    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f66df18-7b8a-4a9c-bdc3-8bc17061c133-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9f66df18-7b8a-4a9c-bdc3-8bc17061c133" (UID: "9f66df18-7b8a-4a9c-bdc3-8bc17061c133"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:10:15.303705 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:15.303679    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f66df18-7b8a-4a9c-bdc3-8bc17061c133-kube-api-access-52l7z" (OuterVolumeSpecName: "kube-api-access-52l7z") pod "9f66df18-7b8a-4a9c-bdc3-8bc17061c133" (UID: "9f66df18-7b8a-4a9c-bdc3-8bc17061c133"). InnerVolumeSpecName "kube-api-access-52l7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:10:15.402792 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:15.402762    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52l7z\" (UniqueName: \"kubernetes.io/projected/9f66df18-7b8a-4a9c-bdc3-8bc17061c133-kube-api-access-52l7z\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:10:15.402792 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:15.402785    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f66df18-7b8a-4a9c-bdc3-8bc17061c133-plugins-config-volume\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:10:15.522873 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:15.522779    2579 generic.go:356] "Generic (PLEG): container finished" podID="9f66df18-7b8a-4a9c-bdc3-8bc17061c133" containerID="9c77dca60f99617a4362c15b4241c44e8e69c1d7038238830b61314193a69f01" exitCode=0
Sep 09 06:10:15.522873 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:15.522828    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-389cd1da-7cddb7cbfc-x2nn9"
Sep 09 06:10:15.523060 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:15.522867    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-389cd1da-7cddb7cbfc-x2nn9" event={"ID":"9f66df18-7b8a-4a9c-bdc3-8bc17061c133","Type":"ContainerDied","Data":"9c77dca60f99617a4362c15b4241c44e8e69c1d7038238830b61314193a69f01"}
Sep 09 06:10:15.523060 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:15.522922    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-389cd1da-7cddb7cbfc-x2nn9" event={"ID":"9f66df18-7b8a-4a9c-bdc3-8bc17061c133","Type":"ContainerDied","Data":"4662092661be20ffda129f0d22f31b1dd40f0a8e710e1c2a87e8da5ceec5f2da"}
Sep 09 06:10:15.523060 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:15.522945    2579 scope.go:117] "RemoveContainer" containerID="9c77dca60f99617a4362c15b4241c44e8e69c1d7038238830b61314193a69f01"
Sep 09 06:10:15.532123 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:15.532104    2579 scope.go:117] "RemoveContainer" containerID="9c77dca60f99617a4362c15b4241c44e8e69c1d7038238830b61314193a69f01"
Sep 09 06:10:15.532400 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:10:15.532377    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c77dca60f99617a4362c15b4241c44e8e69c1d7038238830b61314193a69f01\": container with ID starting with 9c77dca60f99617a4362c15b4241c44e8e69c1d7038238830b61314193a69f01 not found: ID does not exist" containerID="9c77dca60f99617a4362c15b4241c44e8e69c1d7038238830b61314193a69f01"
Sep 09 06:10:15.532453 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:15.532415    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c77dca60f99617a4362c15b4241c44e8e69c1d7038238830b61314193a69f01"} err="failed to get container status \"9c77dca60f99617a4362c15b4241c44e8e69c1d7038238830b61314193a69f01\": rpc error: code = NotFound desc = could not find container \"9c77dca60f99617a4362c15b4241c44e8e69c1d7038238830b61314193a69f01\": container with ID starting with 9c77dca60f99617a4362c15b4241c44e8e69c1d7038238830b61314193a69f01 not found: ID does not exist"
Sep 09 06:10:15.543436 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:15.543412    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-389cd1da-7cddb7cbfc-x2nn9"]
Sep 09 06:10:15.547273 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:15.547251    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-389cd1da-7cddb7cbfc-x2nn9"]
Sep 09 06:10:17.148998 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:17.148966    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34d43acd-ea31-468e-ad87-c30e9006f3ad" path="/var/lib/kubelet/pods/34d43acd-ea31-468e-ad87-c30e9006f3ad/volumes"
Sep 09 06:10:17.149361 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:17.149294    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f66df18-7b8a-4a9c-bdc3-8bc17061c133" path="/var/lib/kubelet/pods/9f66df18-7b8a-4a9c-bdc3-8bc17061c133/volumes"
Sep 09 06:10:21.431001 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:21.430972    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-b-431541ed-kserve-7447484876-8dh8b"]
Sep 09 06:10:21.431464 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:21.431445    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f66df18-7b8a-4a9c-bdc3-8bc17061c133" containerName="payload-processing"
Sep 09 06:10:21.431551 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:21.431469    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f66df18-7b8a-4a9c-bdc3-8bc17061c133" containerName="payload-processing"
Sep 09 06:10:21.431551 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:21.431495    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34d43acd-ea31-468e-ad87-c30e9006f3ad" containerName="revoke-keys"
Sep 09 06:10:21.431551 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:21.431504    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="34d43acd-ea31-468e-ad87-c30e9006f3ad" containerName="revoke-keys"
Sep 09 06:10:21.431786 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:21.431592    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="34d43acd-ea31-468e-ad87-c30e9006f3ad" containerName="revoke-keys"
Sep 09 06:10:21.431786 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:21.431607    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f66df18-7b8a-4a9c-bdc3-8bc17061c133" containerName="payload-processing"
Sep 09 06:10:21.436053 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:21.436032    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-b-431541ed-kserve-7447484876-8dh8b"
Sep 09 06:10:21.438454 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:21.438430    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-43eb2d\"/\"e2e-conflict-b-431541ed-kserve-self-signed-certs\""
Sep 09 06:10:21.445036 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:21.445015    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-b-431541ed-kserve-7447484876-8dh8b"]
Sep 09 06:10:21.557545 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:21.557514    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f7231bd-6db5-4c56-abc4-c45d5a23edd2-tls-certs\") pod \"e2e-conflict-b-431541ed-kserve-7447484876-8dh8b\" (UID: \"2f7231bd-6db5-4c56-abc4-c45d5a23edd2\") " pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-b-431541ed-kserve-7447484876-8dh8b"
Sep 09 06:10:21.557713 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:21.557586    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f7231bd-6db5-4c56-abc4-c45d5a23edd2-home\") pod \"e2e-conflict-b-431541ed-kserve-7447484876-8dh8b\" (UID: \"2f7231bd-6db5-4c56-abc4-c45d5a23edd2\") " pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-b-431541ed-kserve-7447484876-8dh8b"
Sep 09 06:10:21.557713 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:21.557606    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f7231bd-6db5-4c56-abc4-c45d5a23edd2-dshm\") pod \"e2e-conflict-b-431541ed-kserve-7447484876-8dh8b\" (UID: \"2f7231bd-6db5-4c56-abc4-c45d5a23edd2\") " pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-b-431541ed-kserve-7447484876-8dh8b"
Sep 09 06:10:21.557713 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:21.557648    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjtzt\" (UniqueName: \"kubernetes.io/projected/2f7231bd-6db5-4c56-abc4-c45d5a23edd2-kube-api-access-cjtzt\") pod \"e2e-conflict-b-431541ed-kserve-7447484876-8dh8b\" (UID: \"2f7231bd-6db5-4c56-abc4-c45d5a23edd2\") " pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-b-431541ed-kserve-7447484876-8dh8b"
Sep 09 06:10:21.557831 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:21.557728    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f7231bd-6db5-4c56-abc4-c45d5a23edd2-model-cache\") pod \"e2e-conflict-b-431541ed-kserve-7447484876-8dh8b\" (UID: \"2f7231bd-6db5-4c56-abc4-c45d5a23edd2\") " pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-b-431541ed-kserve-7447484876-8dh8b"
Sep 09 06:10:21.557831 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:21.557759    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f7231bd-6db5-4c56-abc4-c45d5a23edd2-tmp-dir\") pod \"e2e-conflict-b-431541ed-kserve-7447484876-8dh8b\" (UID: \"2f7231bd-6db5-4c56-abc4-c45d5a23edd2\") " pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-b-431541ed-kserve-7447484876-8dh8b"
Sep 09 06:10:21.658793 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:21.658765    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f7231bd-6db5-4c56-abc4-c45d5a23edd2-home\") pod \"e2e-conflict-b-431541ed-kserve-7447484876-8dh8b\" (UID: \"2f7231bd-6db5-4c56-abc4-c45d5a23edd2\") " pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-b-431541ed-kserve-7447484876-8dh8b"
Sep 09 06:10:21.658793 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:21.658795    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f7231bd-6db5-4c56-abc4-c45d5a23edd2-dshm\") pod \"e2e-conflict-b-431541ed-kserve-7447484876-8dh8b\" (UID: \"2f7231bd-6db5-4c56-abc4-c45d5a23edd2\") " pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-b-431541ed-kserve-7447484876-8dh8b"
Sep 09 06:10:21.659048 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:21.658816    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjtzt\" (UniqueName: \"kubernetes.io/projected/2f7231bd-6db5-4c56-abc4-c45d5a23edd2-kube-api-access-cjtzt\") pod \"e2e-conflict-b-431541ed-kserve-7447484876-8dh8b\" (UID: \"2f7231bd-6db5-4c56-abc4-c45d5a23edd2\") " pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-b-431541ed-kserve-7447484876-8dh8b"
Sep 09 06:10:21.659048 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:21.658957    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f7231bd-6db5-4c56-abc4-c45d5a23edd2-model-cache\") pod \"e2e-conflict-b-431541ed-kserve-7447484876-8dh8b\" (UID: \"2f7231bd-6db5-4c56-abc4-c45d5a23edd2\") " pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-b-431541ed-kserve-7447484876-8dh8b"
Sep 09 06:10:21.659048 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:21.659002    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f7231bd-6db5-4c56-abc4-c45d5a23edd2-tmp-dir\") pod \"e2e-conflict-b-431541ed-kserve-7447484876-8dh8b\" (UID: \"2f7231bd-6db5-4c56-abc4-c45d5a23edd2\") " pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-b-431541ed-kserve-7447484876-8dh8b"
Sep 09 06:10:21.659207 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:21.659088    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f7231bd-6db5-4c56-abc4-c45d5a23edd2-tls-certs\") pod \"e2e-conflict-b-431541ed-kserve-7447484876-8dh8b\" (UID: \"2f7231bd-6db5-4c56-abc4-c45d5a23edd2\") " pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-b-431541ed-kserve-7447484876-8dh8b"
Sep 09 06:10:21.659207 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:21.659148    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f7231bd-6db5-4c56-abc4-c45d5a23edd2-home\") pod \"e2e-conflict-b-431541ed-kserve-7447484876-8dh8b\" (UID: \"2f7231bd-6db5-4c56-abc4-c45d5a23edd2\") " pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-b-431541ed-kserve-7447484876-8dh8b"
Sep 09 06:10:21.659319 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:21.659265    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f7231bd-6db5-4c56-abc4-c45d5a23edd2-model-cache\") pod \"e2e-conflict-b-431541ed-kserve-7447484876-8dh8b\" (UID: \"2f7231bd-6db5-4c56-abc4-c45d5a23edd2\") " pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-b-431541ed-kserve-7447484876-8dh8b"
Sep 09 06:10:21.659377 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:21.659326    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f7231bd-6db5-4c56-abc4-c45d5a23edd2-tmp-dir\") pod \"e2e-conflict-b-431541ed-kserve-7447484876-8dh8b\" (UID: \"2f7231bd-6db5-4c56-abc4-c45d5a23edd2\") " pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-b-431541ed-kserve-7447484876-8dh8b"
Sep 09 06:10:21.661202 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:21.661179    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f7231bd-6db5-4c56-abc4-c45d5a23edd2-dshm\") pod \"e2e-conflict-b-431541ed-kserve-7447484876-8dh8b\" (UID: \"2f7231bd-6db5-4c56-abc4-c45d5a23edd2\") " pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-b-431541ed-kserve-7447484876-8dh8b"
Sep 09 06:10:21.661654 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:21.661617    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f7231bd-6db5-4c56-abc4-c45d5a23edd2-tls-certs\") pod \"e2e-conflict-b-431541ed-kserve-7447484876-8dh8b\" (UID: \"2f7231bd-6db5-4c56-abc4-c45d5a23edd2\") " pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-b-431541ed-kserve-7447484876-8dh8b"
Sep 09 06:10:21.666309 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:21.666288    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjtzt\" (UniqueName: \"kubernetes.io/projected/2f7231bd-6db5-4c56-abc4-c45d5a23edd2-kube-api-access-cjtzt\") pod \"e2e-conflict-b-431541ed-kserve-7447484876-8dh8b\" (UID: \"2f7231bd-6db5-4c56-abc4-c45d5a23edd2\") " pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-b-431541ed-kserve-7447484876-8dh8b"
Sep 09 06:10:21.747047 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:21.746962    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-b-431541ed-kserve-7447484876-8dh8b"
Sep 09 06:10:22.001745 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:22.001663    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:10:22.021330 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:22.021304    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-b-431541ed-kserve-7447484876-8dh8b"]
Sep 09 06:10:22.023332 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:10:22.023295    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f7231bd_6db5_4c56_abc4_c45d5a23edd2.slice/crio-3bf735712aed5534379c405a67ec6483d29349fe8584d3436b337652bf479c74 WatchSource:0}: Error finding container 3bf735712aed5534379c405a67ec6483d29349fe8584d3436b337652bf479c74: Status 404 returned error can't find the container with id 3bf735712aed5534379c405a67ec6483d29349fe8584d3436b337652bf479c74
Sep 09 06:10:22.567059 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:22.567020    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-b-431541ed-kserve-7447484876-8dh8b" event={"ID":"2f7231bd-6db5-4c56-abc4-c45d5a23edd2","Type":"ContainerStarted","Data":"5f14b0762d797c0b30d00022609fd484ab900c18ef836b1f18b9cab6050e8023"}
Sep 09 06:10:22.567559 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:22.567068    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-b-431541ed-kserve-7447484876-8dh8b" event={"ID":"2f7231bd-6db5-4c56-abc4-c45d5a23edd2","Type":"ContainerStarted","Data":"3bf735712aed5534379c405a67ec6483d29349fe8584d3436b337652bf479c74"}
Sep 09 06:10:22.587336 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:22.587281    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-b-431541ed-kserve-7447484876-8dh8b" podStartSLOduration=1.5872605530000001 podStartE2EDuration="1.587260553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:10:22.585081572 +0000 UTC m=+1247.925569484" watchObservedRunningTime="2026-09-09 06:10:22.587260553 +0000 UTC m=+1247.927748463"
Sep 09 06:10:23.463086 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:23.463049    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-389cd1da/e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk"]
Sep 09 06:10:31.747213 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:31.747178    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-b-431541ed-kserve-7447484876-8dh8b"
Sep 09 06:10:31.747213 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:31.747222    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-b-431541ed-kserve-7447484876-8dh8b"
Sep 09 06:10:31.760031 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:31.760002    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-b-431541ed-kserve-7447484876-8dh8b"
Sep 09 06:10:32.629678 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:32.629650    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-b-431541ed-kserve-7447484876-8dh8b"
Sep 09 06:10:35.297509 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:35.297481    2579 scope.go:117] "RemoveContainer" containerID="fc0c80c60bbd1ca9e7c18d27e8e43acc3a21a0cd4346d320ad5cd75d85ad7459"
Sep 09 06:10:35.305602 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:35.305584    2579 scope.go:117] "RemoveContainer" containerID="26cc04b3f97062ebfd51689fbb1f776b16153a07dbd326f0e84ef390cb24576b"
Sep 09 06:10:38.483368 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:38.483327    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-389cd1da/e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk" podUID="5b3e7231-ce7d-43d0-b13a-ffda818a3ebf" containerName="main" containerID="cri-o://356ec11b9ba5b4e44bd7868753c65c3ecf75c95104ec4d18304825830c6ef180" gracePeriod=45
Sep 09 06:10:38.649441 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:38.649412    2579 generic.go:356] "Generic (PLEG): container finished" podID="5b3e7231-ce7d-43d0-b13a-ffda818a3ebf" containerID="356ec11b9ba5b4e44bd7868753c65c3ecf75c95104ec4d18304825830c6ef180" exitCode=0
Sep 09 06:10:38.649592 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:38.649486    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-389cd1da/e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk" event={"ID":"5b3e7231-ce7d-43d0-b13a-ffda818a3ebf","Type":"ContainerDied","Data":"356ec11b9ba5b4e44bd7868753c65c3ecf75c95104ec4d18304825830c6ef180"}
Sep 09 06:10:38.732056 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:38.732036    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-389cd1da/e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk"
Sep 09 06:10:38.908733 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:38.908697    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b3e7231-ce7d-43d0-b13a-ffda818a3ebf-dshm\") pod \"5b3e7231-ce7d-43d0-b13a-ffda818a3ebf\" (UID: \"5b3e7231-ce7d-43d0-b13a-ffda818a3ebf\") "
Sep 09 06:10:38.908941 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:38.908739    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b3e7231-ce7d-43d0-b13a-ffda818a3ebf-tls-certs\") pod \"5b3e7231-ce7d-43d0-b13a-ffda818a3ebf\" (UID: \"5b3e7231-ce7d-43d0-b13a-ffda818a3ebf\") "
Sep 09 06:10:38.908941 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:38.908778    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b3e7231-ce7d-43d0-b13a-ffda818a3ebf-model-cache\") pod \"5b3e7231-ce7d-43d0-b13a-ffda818a3ebf\" (UID: \"5b3e7231-ce7d-43d0-b13a-ffda818a3ebf\") "
Sep 09 06:10:38.908941 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:38.908808    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b3e7231-ce7d-43d0-b13a-ffda818a3ebf-tmp-dir\") pod \"5b3e7231-ce7d-43d0-b13a-ffda818a3ebf\" (UID: \"5b3e7231-ce7d-43d0-b13a-ffda818a3ebf\") "
Sep 09 06:10:38.909120 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:38.908977    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b3e7231-ce7d-43d0-b13a-ffda818a3ebf-home\") pod \"5b3e7231-ce7d-43d0-b13a-ffda818a3ebf\" (UID: \"5b3e7231-ce7d-43d0-b13a-ffda818a3ebf\") "
Sep 09 06:10:38.909120 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:38.909045    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7ncv\" (UniqueName: \"kubernetes.io/projected/5b3e7231-ce7d-43d0-b13a-ffda818a3ebf-kube-api-access-l7ncv\") pod \"5b3e7231-ce7d-43d0-b13a-ffda818a3ebf\" (UID: \"5b3e7231-ce7d-43d0-b13a-ffda818a3ebf\") "
Sep 09 06:10:38.909120 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:38.909079    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b3e7231-ce7d-43d0-b13a-ffda818a3ebf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5b3e7231-ce7d-43d0-b13a-ffda818a3ebf" (UID: "5b3e7231-ce7d-43d0-b13a-ffda818a3ebf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:10:38.909120 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:38.909092    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b3e7231-ce7d-43d0-b13a-ffda818a3ebf-model-cache" (OuterVolumeSpecName: "model-cache") pod "5b3e7231-ce7d-43d0-b13a-ffda818a3ebf" (UID: "5b3e7231-ce7d-43d0-b13a-ffda818a3ebf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:10:38.909316 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:38.909210    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b3e7231-ce7d-43d0-b13a-ffda818a3ebf-home" (OuterVolumeSpecName: "home") pod "5b3e7231-ce7d-43d0-b13a-ffda818a3ebf" (UID: "5b3e7231-ce7d-43d0-b13a-ffda818a3ebf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:10:38.909460 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:38.909422    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b3e7231-ce7d-43d0-b13a-ffda818a3ebf-model-cache\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:10:38.909460 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:38.909443    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b3e7231-ce7d-43d0-b13a-ffda818a3ebf-tmp-dir\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:10:38.909460 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:38.909450    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b3e7231-ce7d-43d0-b13a-ffda818a3ebf-home\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:10:38.911215 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:38.911181    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b3e7231-ce7d-43d0-b13a-ffda818a3ebf-dshm" (OuterVolumeSpecName: "dshm") pod "5b3e7231-ce7d-43d0-b13a-ffda818a3ebf" (UID: "5b3e7231-ce7d-43d0-b13a-ffda818a3ebf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:10:38.911310 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:38.911258    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b3e7231-ce7d-43d0-b13a-ffda818a3ebf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5b3e7231-ce7d-43d0-b13a-ffda818a3ebf" (UID: "5b3e7231-ce7d-43d0-b13a-ffda818a3ebf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:10:38.911310 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:38.911261    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b3e7231-ce7d-43d0-b13a-ffda818a3ebf-kube-api-access-l7ncv" (OuterVolumeSpecName: "kube-api-access-l7ncv") pod "5b3e7231-ce7d-43d0-b13a-ffda818a3ebf" (UID: "5b3e7231-ce7d-43d0-b13a-ffda818a3ebf"). InnerVolumeSpecName "kube-api-access-l7ncv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:10:39.010771 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:39.010733    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b3e7231-ce7d-43d0-b13a-ffda818a3ebf-dshm\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:10:39.010771 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:39.010768    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b3e7231-ce7d-43d0-b13a-ffda818a3ebf-tls-certs\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:10:39.010973 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:39.010784    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7ncv\" (UniqueName: \"kubernetes.io/projected/5b3e7231-ce7d-43d0-b13a-ffda818a3ebf-kube-api-access-l7ncv\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:10:39.656719 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:39.656681    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-389cd1da/e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk" event={"ID":"5b3e7231-ce7d-43d0-b13a-ffda818a3ebf","Type":"ContainerDied","Data":"0123b02e25bba9a1b95490b4205a339807904a810d2cfc20bc6abb3a68e3b524"}
Sep 09 06:10:39.657141 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:39.656734    2579 scope.go:117] "RemoveContainer" containerID="356ec11b9ba5b4e44bd7868753c65c3ecf75c95104ec4d18304825830c6ef180"
Sep 09 06:10:39.657141 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:39.656732    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-389cd1da/e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk"
Sep 09 06:10:39.672509 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:39.672486    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-389cd1da/e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk"]
Sep 09 06:10:39.674510 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:39.674488    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-389cd1da/e2e-lifecycle-model-389cd1da-kserve-54f97c6659-lrtqk"]
Sep 09 06:10:41.149172 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:41.149128    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b3e7231-ce7d-43d0-b13a-ffda818a3ebf" path="/var/lib/kubelet/pods/5b3e7231-ce7d-43d0-b13a-ffda818a3ebf/volumes"
Sep 09 06:10:50.194916 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:50.194884    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:10:52.636122 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:52.636091    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-7cf6be/auth-test-model-7cf6be-kserve-579d696f59-hhqwt"]
Sep 09 06:10:52.636543 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:52.636526    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b3e7231-ce7d-43d0-b13a-ffda818a3ebf" containerName="main"
Sep 09 06:10:52.636600 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:52.636546    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b3e7231-ce7d-43d0-b13a-ffda818a3ebf" containerName="main"
Sep 09 06:10:52.636672 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:52.636639    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b3e7231-ce7d-43d0-b13a-ffda818a3ebf" containerName="main"
Sep 09 06:10:52.640956 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:52.640936    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7cf6be/auth-test-model-7cf6be-kserve-579d696f59-hhqwt"
Sep 09 06:10:52.644083 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:52.644056    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7cf6be\"/\"kube-root-ca.crt\""
Sep 09 06:10:52.644207 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:52.644104    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7cf6be\"/\"openshift-service-ca.crt\""
Sep 09 06:10:52.644207 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:52.644113    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7cf6be\"/\"default-dockercfg-7wv4f\""
Sep 09 06:10:52.644207 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:52.644104    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7cf6be\"/\"auth-test-model-7cf6be-kserve-self-signed-certs\""
Sep 09 06:10:52.651505 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:52.651464    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7cf6be/auth-test-model-7cf6be-kserve-579d696f59-hhqwt"]
Sep 09 06:10:52.720423 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:52.720393    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccdb1451-54db-4ac4-ad11-0deb92293513-tmp-dir\") pod \"auth-test-model-7cf6be-kserve-579d696f59-hhqwt\" (UID: \"ccdb1451-54db-4ac4-ad11-0deb92293513\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/auth-test-model-7cf6be-kserve-579d696f59-hhqwt"
Sep 09 06:10:52.720553 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:52.720430    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccdb1451-54db-4ac4-ad11-0deb92293513-tls-certs\") pod \"auth-test-model-7cf6be-kserve-579d696f59-hhqwt\" (UID: \"ccdb1451-54db-4ac4-ad11-0deb92293513\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/auth-test-model-7cf6be-kserve-579d696f59-hhqwt"
Sep 09 06:10:52.720553 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:52.720448    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccdb1451-54db-4ac4-ad11-0deb92293513-model-cache\") pod \"auth-test-model-7cf6be-kserve-579d696f59-hhqwt\" (UID: \"ccdb1451-54db-4ac4-ad11-0deb92293513\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/auth-test-model-7cf6be-kserve-579d696f59-hhqwt"
Sep 09 06:10:52.720553 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:52.720487    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccdb1451-54db-4ac4-ad11-0deb92293513-home\") pod \"auth-test-model-7cf6be-kserve-579d696f59-hhqwt\" (UID: \"ccdb1451-54db-4ac4-ad11-0deb92293513\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/auth-test-model-7cf6be-kserve-579d696f59-hhqwt"
Sep 09 06:10:52.720553 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:52.720509    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4kpd\" (UniqueName: \"kubernetes.io/projected/ccdb1451-54db-4ac4-ad11-0deb92293513-kube-api-access-w4kpd\") pod \"auth-test-model-7cf6be-kserve-579d696f59-hhqwt\" (UID: \"ccdb1451-54db-4ac4-ad11-0deb92293513\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/auth-test-model-7cf6be-kserve-579d696f59-hhqwt"
Sep 09 06:10:52.720553 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:52.720532    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccdb1451-54db-4ac4-ad11-0deb92293513-dshm\") pod \"auth-test-model-7cf6be-kserve-579d696f59-hhqwt\" (UID: \"ccdb1451-54db-4ac4-ad11-0deb92293513\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/auth-test-model-7cf6be-kserve-579d696f59-hhqwt"
Sep 09 06:10:52.821153 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:52.821124    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccdb1451-54db-4ac4-ad11-0deb92293513-tmp-dir\") pod \"auth-test-model-7cf6be-kserve-579d696f59-hhqwt\" (UID: \"ccdb1451-54db-4ac4-ad11-0deb92293513\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/auth-test-model-7cf6be-kserve-579d696f59-hhqwt"
Sep 09 06:10:52.821287 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:52.821161    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccdb1451-54db-4ac4-ad11-0deb92293513-tls-certs\") pod \"auth-test-model-7cf6be-kserve-579d696f59-hhqwt\" (UID: \"ccdb1451-54db-4ac4-ad11-0deb92293513\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/auth-test-model-7cf6be-kserve-579d696f59-hhqwt"
Sep 09 06:10:52.821287 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:52.821181    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccdb1451-54db-4ac4-ad11-0deb92293513-model-cache\") pod \"auth-test-model-7cf6be-kserve-579d696f59-hhqwt\" (UID: \"ccdb1451-54db-4ac4-ad11-0deb92293513\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/auth-test-model-7cf6be-kserve-579d696f59-hhqwt"
Sep 09 06:10:52.821287 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:52.821206    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccdb1451-54db-4ac4-ad11-0deb92293513-home\") pod \"auth-test-model-7cf6be-kserve-579d696f59-hhqwt\" (UID: \"ccdb1451-54db-4ac4-ad11-0deb92293513\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/auth-test-model-7cf6be-kserve-579d696f59-hhqwt"
Sep 09 06:10:52.821287 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:52.821241    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4kpd\" (UniqueName: \"kubernetes.io/projected/ccdb1451-54db-4ac4-ad11-0deb92293513-kube-api-access-w4kpd\") pod \"auth-test-model-7cf6be-kserve-579d696f59-hhqwt\" (UID: \"ccdb1451-54db-4ac4-ad11-0deb92293513\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/auth-test-model-7cf6be-kserve-579d696f59-hhqwt"
Sep 09 06:10:52.821287 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:52.821282    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccdb1451-54db-4ac4-ad11-0deb92293513-dshm\") pod \"auth-test-model-7cf6be-kserve-579d696f59-hhqwt\" (UID: \"ccdb1451-54db-4ac4-ad11-0deb92293513\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/auth-test-model-7cf6be-kserve-579d696f59-hhqwt"
Sep 09 06:10:52.821570 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:52.821547    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccdb1451-54db-4ac4-ad11-0deb92293513-tmp-dir\") pod \"auth-test-model-7cf6be-kserve-579d696f59-hhqwt\" (UID: \"ccdb1451-54db-4ac4-ad11-0deb92293513\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/auth-test-model-7cf6be-kserve-579d696f59-hhqwt"
Sep 09 06:10:52.821663 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:52.821568    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccdb1451-54db-4ac4-ad11-0deb92293513-model-cache\") pod \"auth-test-model-7cf6be-kserve-579d696f59-hhqwt\" (UID: \"ccdb1451-54db-4ac4-ad11-0deb92293513\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/auth-test-model-7cf6be-kserve-579d696f59-hhqwt"
Sep 09 06:10:52.821663 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:52.821645    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccdb1451-54db-4ac4-ad11-0deb92293513-home\") pod \"auth-test-model-7cf6be-kserve-579d696f59-hhqwt\" (UID: \"ccdb1451-54db-4ac4-ad11-0deb92293513\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/auth-test-model-7cf6be-kserve-579d696f59-hhqwt"
Sep 09 06:10:52.823599 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:52.823578    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccdb1451-54db-4ac4-ad11-0deb92293513-dshm\") pod \"auth-test-model-7cf6be-kserve-579d696f59-hhqwt\" (UID: \"ccdb1451-54db-4ac4-ad11-0deb92293513\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/auth-test-model-7cf6be-kserve-579d696f59-hhqwt"
Sep 09 06:10:52.823898 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:52.823882    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccdb1451-54db-4ac4-ad11-0deb92293513-tls-certs\") pod \"auth-test-model-7cf6be-kserve-579d696f59-hhqwt\" (UID: \"ccdb1451-54db-4ac4-ad11-0deb92293513\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/auth-test-model-7cf6be-kserve-579d696f59-hhqwt"
Sep 09 06:10:52.829425 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:52.829408    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4kpd\" (UniqueName: \"kubernetes.io/projected/ccdb1451-54db-4ac4-ad11-0deb92293513-kube-api-access-w4kpd\") pod \"auth-test-model-7cf6be-kserve-579d696f59-hhqwt\" (UID: \"ccdb1451-54db-4ac4-ad11-0deb92293513\") " pod="ai-tenant-e2e-shared-b-w3-7cf6be/auth-test-model-7cf6be-kserve-579d696f59-hhqwt"
Sep 09 06:10:52.950728 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:52.950670    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7cf6be/auth-test-model-7cf6be-kserve-579d696f59-hhqwt"
Sep 09 06:10:53.220506 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:53.220482    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7cf6be/auth-test-model-7cf6be-kserve-579d696f59-hhqwt"]
Sep 09 06:10:53.222534 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:10:53.222508    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccdb1451_54db_4ac4_ad11_0deb92293513.slice/crio-14742f206b55ae0579de0a5b0f9bf212a8bfb57e7560f71cf225d6d2628a6ce2 WatchSource:0}: Error finding container 14742f206b55ae0579de0a5b0f9bf212a8bfb57e7560f71cf225d6d2628a6ce2: Status 404 returned error can't find the container with id 14742f206b55ae0579de0a5b0f9bf212a8bfb57e7560f71cf225d6d2628a6ce2
Sep 09 06:10:53.726164 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:53.726124    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7cf6be/auth-test-model-7cf6be-kserve-579d696f59-hhqwt" event={"ID":"ccdb1451-54db-4ac4-ad11-0deb92293513","Type":"ContainerStarted","Data":"7bb91744a021ecc5e31b4b8c4f762f9d518a922597b9cac62b0f885631004258"}
Sep 09 06:10:53.726164 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:53.726163    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7cf6be/auth-test-model-7cf6be-kserve-579d696f59-hhqwt" event={"ID":"ccdb1451-54db-4ac4-ad11-0deb92293513","Type":"ContainerStarted","Data":"14742f206b55ae0579de0a5b0f9bf212a8bfb57e7560f71cf225d6d2628a6ce2"}
Sep 09 06:10:53.741773 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:53.741734    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-7cf6be/auth-test-model-7cf6be-kserve-579d696f59-hhqwt" podStartSLOduration=1.741719085 podStartE2EDuration="1.741719085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:10:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:10:53.740873986 +0000 UTC m=+1279.081361906" watchObservedRunningTime="2026-09-09 06:10:53.741719085 +0000 UTC m=+1279.082206993"
Sep 09 06:10:54.826040 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:10:54.826002    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-b-431541ed-kserve-7447484876-8dh8b"]
Sep 09 06:11:02.950946 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:02.950911    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-7cf6be/auth-test-model-7cf6be-kserve-579d696f59-hhqwt"
Sep 09 06:11:02.951316 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:02.950964    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-7cf6be/auth-test-model-7cf6be-kserve-579d696f59-hhqwt"
Sep 09 06:11:02.963526 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:02.963503    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-7cf6be/auth-test-model-7cf6be-kserve-579d696f59-hhqwt"
Sep 09 06:11:03.786767 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:03.786741    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-7cf6be/auth-test-model-7cf6be-kserve-579d696f59-hhqwt"
Sep 09 06:11:09.846340 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:09.846292    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-b-431541ed-kserve-7447484876-8dh8b" podUID="2f7231bd-6db5-4c56-abc4-c45d5a23edd2" containerName="main" containerID="cri-o://5f14b0762d797c0b30d00022609fd484ab900c18ef836b1f18b9cab6050e8023" gracePeriod=45
Sep 09 06:11:10.089539 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:10.089514    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-b-431541ed-kserve-7447484876-8dh8b"
Sep 09 06:11:10.160079 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:10.160023    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f7231bd-6db5-4c56-abc4-c45d5a23edd2-tls-certs\") pod \"2f7231bd-6db5-4c56-abc4-c45d5a23edd2\" (UID: \"2f7231bd-6db5-4c56-abc4-c45d5a23edd2\") "
Sep 09 06:11:10.160079 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:10.160075    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f7231bd-6db5-4c56-abc4-c45d5a23edd2-home\") pod \"2f7231bd-6db5-4c56-abc4-c45d5a23edd2\" (UID: \"2f7231bd-6db5-4c56-abc4-c45d5a23edd2\") "
Sep 09 06:11:10.160267 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:10.160125    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f7231bd-6db5-4c56-abc4-c45d5a23edd2-model-cache\") pod \"2f7231bd-6db5-4c56-abc4-c45d5a23edd2\" (UID: \"2f7231bd-6db5-4c56-abc4-c45d5a23edd2\") "
Sep 09 06:11:10.160267 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:10.160146    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f7231bd-6db5-4c56-abc4-c45d5a23edd2-tmp-dir\") pod \"2f7231bd-6db5-4c56-abc4-c45d5a23edd2\" (UID: \"2f7231bd-6db5-4c56-abc4-c45d5a23edd2\") "
Sep 09 06:11:10.160267 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:10.160198    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjtzt\" (UniqueName: \"kubernetes.io/projected/2f7231bd-6db5-4c56-abc4-c45d5a23edd2-kube-api-access-cjtzt\") pod \"2f7231bd-6db5-4c56-abc4-c45d5a23edd2\" (UID: \"2f7231bd-6db5-4c56-abc4-c45d5a23edd2\") "
Sep 09 06:11:10.160267 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:10.160244    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f7231bd-6db5-4c56-abc4-c45d5a23edd2-dshm\") pod \"2f7231bd-6db5-4c56-abc4-c45d5a23edd2\" (UID: \"2f7231bd-6db5-4c56-abc4-c45d5a23edd2\") "
Sep 09 06:11:10.160454 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:10.160378    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f7231bd-6db5-4c56-abc4-c45d5a23edd2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2f7231bd-6db5-4c56-abc4-c45d5a23edd2" (UID: "2f7231bd-6db5-4c56-abc4-c45d5a23edd2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:11:10.160454 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:10.160391    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f7231bd-6db5-4c56-abc4-c45d5a23edd2-home" (OuterVolumeSpecName: "home") pod "2f7231bd-6db5-4c56-abc4-c45d5a23edd2" (UID: "2f7231bd-6db5-4c56-abc4-c45d5a23edd2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:11:10.160454 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:10.160403    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f7231bd-6db5-4c56-abc4-c45d5a23edd2-model-cache" (OuterVolumeSpecName: "model-cache") pod "2f7231bd-6db5-4c56-abc4-c45d5a23edd2" (UID: "2f7231bd-6db5-4c56-abc4-c45d5a23edd2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:11:10.160952 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:10.160933    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f7231bd-6db5-4c56-abc4-c45d5a23edd2-home\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:11:10.161031 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:10.160957    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f7231bd-6db5-4c56-abc4-c45d5a23edd2-model-cache\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:11:10.161031 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:10.160972    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f7231bd-6db5-4c56-abc4-c45d5a23edd2-tmp-dir\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:11:10.162406 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:10.162379    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f7231bd-6db5-4c56-abc4-c45d5a23edd2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2f7231bd-6db5-4c56-abc4-c45d5a23edd2" (UID: "2f7231bd-6db5-4c56-abc4-c45d5a23edd2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:11:10.162498 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:10.162474    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f7231bd-6db5-4c56-abc4-c45d5a23edd2-kube-api-access-cjtzt" (OuterVolumeSpecName: "kube-api-access-cjtzt") pod "2f7231bd-6db5-4c56-abc4-c45d5a23edd2" (UID: "2f7231bd-6db5-4c56-abc4-c45d5a23edd2"). InnerVolumeSpecName "kube-api-access-cjtzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:11:10.162672 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:10.162657    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f7231bd-6db5-4c56-abc4-c45d5a23edd2-dshm" (OuterVolumeSpecName: "dshm") pod "2f7231bd-6db5-4c56-abc4-c45d5a23edd2" (UID: "2f7231bd-6db5-4c56-abc4-c45d5a23edd2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:11:10.261690 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:10.261665    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjtzt\" (UniqueName: \"kubernetes.io/projected/2f7231bd-6db5-4c56-abc4-c45d5a23edd2-kube-api-access-cjtzt\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:11:10.261690 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:10.261686    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f7231bd-6db5-4c56-abc4-c45d5a23edd2-dshm\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:11:10.261825 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:10.261695    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f7231bd-6db5-4c56-abc4-c45d5a23edd2-tls-certs\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:11:10.813634 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:10.813588    2579 generic.go:356] "Generic (PLEG): container finished" podID="2f7231bd-6db5-4c56-abc4-c45d5a23edd2" containerID="5f14b0762d797c0b30d00022609fd484ab900c18ef836b1f18b9cab6050e8023" exitCode=0
Sep 09 06:11:10.813826 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:10.813647    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-b-431541ed-kserve-7447484876-8dh8b"
Sep 09 06:11:10.813826 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:10.813677    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-b-431541ed-kserve-7447484876-8dh8b" event={"ID":"2f7231bd-6db5-4c56-abc4-c45d5a23edd2","Type":"ContainerDied","Data":"5f14b0762d797c0b30d00022609fd484ab900c18ef836b1f18b9cab6050e8023"}
Sep 09 06:11:10.813826 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:10.813731    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-b-431541ed-kserve-7447484876-8dh8b" event={"ID":"2f7231bd-6db5-4c56-abc4-c45d5a23edd2","Type":"ContainerDied","Data":"3bf735712aed5534379c405a67ec6483d29349fe8584d3436b337652bf479c74"}
Sep 09 06:11:10.813826 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:10.813754    2579 scope.go:117] "RemoveContainer" containerID="5f14b0762d797c0b30d00022609fd484ab900c18ef836b1f18b9cab6050e8023"
Sep 09 06:11:10.822819 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:10.822800    2579 scope.go:117] "RemoveContainer" containerID="5f14b0762d797c0b30d00022609fd484ab900c18ef836b1f18b9cab6050e8023"
Sep 09 06:11:10.823072 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:11:10.823054    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f14b0762d797c0b30d00022609fd484ab900c18ef836b1f18b9cab6050e8023\": container with ID starting with 5f14b0762d797c0b30d00022609fd484ab900c18ef836b1f18b9cab6050e8023 not found: ID does not exist" containerID="5f14b0762d797c0b30d00022609fd484ab900c18ef836b1f18b9cab6050e8023"
Sep 09 06:11:10.823118 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:10.823084    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f14b0762d797c0b30d00022609fd484ab900c18ef836b1f18b9cab6050e8023"} err="failed to get container status \"5f14b0762d797c0b30d00022609fd484ab900c18ef836b1f18b9cab6050e8023\": rpc error: code = NotFound desc = could not find container \"5f14b0762d797c0b30d00022609fd484ab900c18ef836b1f18b9cab6050e8023\": container with ID starting with 5f14b0762d797c0b30d00022609fd484ab900c18ef836b1f18b9cab6050e8023 not found: ID does not exist"
Sep 09 06:11:10.833146 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:10.833124    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-b-431541ed-kserve-7447484876-8dh8b"]
Sep 09 06:11:10.834736 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:10.834718    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-b-431541ed-kserve-7447484876-8dh8b"]
Sep 09 06:11:11.148780 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:11.148716    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f7231bd-6db5-4c56-abc4-c45d5a23edd2" path="/var/lib/kubelet/pods/2f7231bd-6db5-4c56-abc4-c45d5a23edd2/volumes"
Sep 09 06:11:11.241943 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:11.241913    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg"]
Sep 09 06:11:18.999786 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:18.999752    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:11:26.262328 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:26.262268    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg" podUID="9a5192af-cf47-4d86-a163-8e45882e0fb3" containerName="main" containerID="cri-o://ca76262fad03d82cf6451a41df4bc7135ee79178dffa1a7243e398d74bc9e457" gracePeriod=45
Sep 09 06:11:26.508130 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:26.508105    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg"
Sep 09 06:11:26.597390 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:26.597364    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a5192af-cf47-4d86-a163-8e45882e0fb3-tmp-dir\") pod \"9a5192af-cf47-4d86-a163-8e45882e0fb3\" (UID: \"9a5192af-cf47-4d86-a163-8e45882e0fb3\") "
Sep 09 06:11:26.597536 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:26.597404    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a5192af-cf47-4d86-a163-8e45882e0fb3-tls-certs\") pod \"9a5192af-cf47-4d86-a163-8e45882e0fb3\" (UID: \"9a5192af-cf47-4d86-a163-8e45882e0fb3\") "
Sep 09 06:11:26.597536 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:26.597433    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcvgg\" (UniqueName: \"kubernetes.io/projected/9a5192af-cf47-4d86-a163-8e45882e0fb3-kube-api-access-jcvgg\") pod \"9a5192af-cf47-4d86-a163-8e45882e0fb3\" (UID: \"9a5192af-cf47-4d86-a163-8e45882e0fb3\") "
Sep 09 06:11:26.597536 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:26.597455    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a5192af-cf47-4d86-a163-8e45882e0fb3-home\") pod \"9a5192af-cf47-4d86-a163-8e45882e0fb3\" (UID: \"9a5192af-cf47-4d86-a163-8e45882e0fb3\") "
Sep 09 06:11:26.597536 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:26.597495    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a5192af-cf47-4d86-a163-8e45882e0fb3-dshm\") pod \"9a5192af-cf47-4d86-a163-8e45882e0fb3\" (UID: \"9a5192af-cf47-4d86-a163-8e45882e0fb3\") "
Sep 09 06:11:26.597791 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:26.597566    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a5192af-cf47-4d86-a163-8e45882e0fb3-model-cache\") pod \"9a5192af-cf47-4d86-a163-8e45882e0fb3\" (UID: \"9a5192af-cf47-4d86-a163-8e45882e0fb3\") "
Sep 09 06:11:26.597791 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:26.597675    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5192af-cf47-4d86-a163-8e45882e0fb3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9a5192af-cf47-4d86-a163-8e45882e0fb3" (UID: "9a5192af-cf47-4d86-a163-8e45882e0fb3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:11:26.597791 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:26.597761    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5192af-cf47-4d86-a163-8e45882e0fb3-home" (OuterVolumeSpecName: "home") pod "9a5192af-cf47-4d86-a163-8e45882e0fb3" (UID: "9a5192af-cf47-4d86-a163-8e45882e0fb3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:11:26.597956 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:26.597931    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5192af-cf47-4d86-a163-8e45882e0fb3-model-cache" (OuterVolumeSpecName: "model-cache") pod "9a5192af-cf47-4d86-a163-8e45882e0fb3" (UID: "9a5192af-cf47-4d86-a163-8e45882e0fb3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:11:26.598014 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:26.597943    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a5192af-cf47-4d86-a163-8e45882e0fb3-tmp-dir\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:11:26.598014 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:26.597991    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a5192af-cf47-4d86-a163-8e45882e0fb3-home\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:11:26.599597 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:26.599574    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a5192af-cf47-4d86-a163-8e45882e0fb3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9a5192af-cf47-4d86-a163-8e45882e0fb3" (UID: "9a5192af-cf47-4d86-a163-8e45882e0fb3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:11:26.599704 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:26.599682    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a5192af-cf47-4d86-a163-8e45882e0fb3-kube-api-access-jcvgg" (OuterVolumeSpecName: "kube-api-access-jcvgg") pod "9a5192af-cf47-4d86-a163-8e45882e0fb3" (UID: "9a5192af-cf47-4d86-a163-8e45882e0fb3"). InnerVolumeSpecName "kube-api-access-jcvgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:11:26.599760 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:26.599742    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5192af-cf47-4d86-a163-8e45882e0fb3-dshm" (OuterVolumeSpecName: "dshm") pod "9a5192af-cf47-4d86-a163-8e45882e0fb3" (UID: "9a5192af-cf47-4d86-a163-8e45882e0fb3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:11:26.699289 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:26.699266    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcvgg\" (UniqueName: \"kubernetes.io/projected/9a5192af-cf47-4d86-a163-8e45882e0fb3-kube-api-access-jcvgg\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:11:26.699289 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:26.699289    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a5192af-cf47-4d86-a163-8e45882e0fb3-dshm\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:11:26.699413 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:26.699299    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a5192af-cf47-4d86-a163-8e45882e0fb3-model-cache\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:11:26.699413 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:26.699307    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a5192af-cf47-4d86-a163-8e45882e0fb3-tls-certs\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:11:26.900086 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:26.900006    2579 generic.go:356] "Generic (PLEG): container finished" podID="9a5192af-cf47-4d86-a163-8e45882e0fb3" containerID="ca76262fad03d82cf6451a41df4bc7135ee79178dffa1a7243e398d74bc9e457" exitCode=0
Sep 09 06:11:26.900086 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:26.900067    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg"
Sep 09 06:11:26.900245 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:26.900093    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg" event={"ID":"9a5192af-cf47-4d86-a163-8e45882e0fb3","Type":"ContainerDied","Data":"ca76262fad03d82cf6451a41df4bc7135ee79178dffa1a7243e398d74bc9e457"}
Sep 09 06:11:26.900245 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:26.900133    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg" event={"ID":"9a5192af-cf47-4d86-a163-8e45882e0fb3","Type":"ContainerDied","Data":"26c56f4947285e1a1c00b09171c98eb6426ec7ad0bdc25470f0492971f360d27"}
Sep 09 06:11:26.900245 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:26.900151    2579 scope.go:117] "RemoveContainer" containerID="ca76262fad03d82cf6451a41df4bc7135ee79178dffa1a7243e398d74bc9e457"
Sep 09 06:11:26.909158 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:26.909144    2579 scope.go:117] "RemoveContainer" containerID="ca76262fad03d82cf6451a41df4bc7135ee79178dffa1a7243e398d74bc9e457"
Sep 09 06:11:26.909416 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:11:26.909398    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca76262fad03d82cf6451a41df4bc7135ee79178dffa1a7243e398d74bc9e457\": container with ID starting with ca76262fad03d82cf6451a41df4bc7135ee79178dffa1a7243e398d74bc9e457 not found: ID does not exist" containerID="ca76262fad03d82cf6451a41df4bc7135ee79178dffa1a7243e398d74bc9e457"
Sep 09 06:11:26.909459 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:26.909423    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca76262fad03d82cf6451a41df4bc7135ee79178dffa1a7243e398d74bc9e457"} err="failed to get container status \"ca76262fad03d82cf6451a41df4bc7135ee79178dffa1a7243e398d74bc9e457\": rpc error: code = NotFound desc = could not find container \"ca76262fad03d82cf6451a41df4bc7135ee79178dffa1a7243e398d74bc9e457\": container with ID starting with ca76262fad03d82cf6451a41df4bc7135ee79178dffa1a7243e398d74bc9e457 not found: ID does not exist"
Sep 09 06:11:26.919115 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:26.919096    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg"]
Sep 09 06:11:26.922418 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:26.922401    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-43eb2d/e2e-conflict-a-431541ed-kserve-58cd5d99db-968vg"]
Sep 09 06:11:27.148408 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:27.148381    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a5192af-cf47-4d86-a163-8e45882e0fb3" path="/var/lib/kubelet/pods/9a5192af-cf47-4d86-a163-8e45882e0fb3/volumes"
Sep 09 06:11:45.202228 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:45.202191    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:11:59.149157 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:59.149076    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"]
Sep 09 06:11:59.149594 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:59.149448    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a5192af-cf47-4d86-a163-8e45882e0fb3" containerName="main"
Sep 09 06:11:59.149594 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:59.149462    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a5192af-cf47-4d86-a163-8e45882e0fb3" containerName="main"
Sep 09 06:11:59.149594 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:59.149469    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f7231bd-6db5-4c56-abc4-c45d5a23edd2" containerName="main"
Sep 09 06:11:59.149594 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:59.149475    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f7231bd-6db5-4c56-abc4-c45d5a23edd2" containerName="main"
Sep 09 06:11:59.149594 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:59.149525    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a5192af-cf47-4d86-a163-8e45882e0fb3" containerName="main"
Sep 09 06:11:59.149594 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:59.149535    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f7231bd-6db5-4c56-abc4-c45d5a23edd2" containerName="main"
Sep 09 06:11:59.153992 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:59.153970    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"
Sep 09 06:11:59.156494 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:59.156475    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-e1ed02-openshift-default-dockercfg-qndd6\""
Sep 09 06:11:59.163394 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:59.163369    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"]
Sep 09 06:11:59.270596 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:59.270563    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-workload-certs\") pod \"e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc\" (UID: \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\") " pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"
Sep 09 06:11:59.270596 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:59.270594    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-istio-envoy\") pod \"e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc\" (UID: \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\") " pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"
Sep 09 06:11:59.270850 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:59.270637    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-credential-socket\") pod \"e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc\" (UID: \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\") " pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"
Sep 09 06:11:59.270850 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:59.270698    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-istio-token\") pod \"e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc\" (UID: \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\") " pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"
Sep 09 06:11:59.270850 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:59.270762    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-istiod-ca-cert\") pod \"e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc\" (UID: \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\") " pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"
Sep 09 06:11:59.270850 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:59.270812    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-workload-socket\") pod \"e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc\" (UID: \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\") " pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"
Sep 09 06:11:59.271057 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:59.270906    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-istio-data\") pod \"e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc\" (UID: \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\") " pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"
Sep 09 06:11:59.271057 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:59.270929    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-istio-podinfo\") pod \"e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc\" (UID: \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\") " pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"
Sep 09 06:11:59.271057 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:59.270957    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgm7b\" (UniqueName: \"kubernetes.io/projected/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-kube-api-access-fgm7b\") pod \"e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc\" (UID: \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\") " pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"
Sep 09 06:11:59.371373 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:59.371342    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-workload-certs\") pod \"e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc\" (UID: \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\") " pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"
Sep 09 06:11:59.371520 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:59.371379    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-istio-envoy\") pod \"e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc\" (UID: \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\") " pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"
Sep 09 06:11:59.371520 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:59.371496    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-credential-socket\") pod \"e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc\" (UID: \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\") " pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"
Sep 09 06:11:59.371659 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:59.371534    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-istio-token\") pod \"e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc\" (UID: \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\") " pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"
Sep 09 06:11:59.371659 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:59.371560    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-istiod-ca-cert\") pod \"e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc\" (UID: \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\") " pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"
Sep 09 06:11:59.371784 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:59.371673    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-workload-socket\") pod \"e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc\" (UID: \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\") " pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"
Sep 09 06:11:59.371784 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:59.371736    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-workload-certs\") pod \"e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc\" (UID: \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\") " pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"
Sep 09 06:11:59.371784 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:59.371771    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-istio-data\") pod \"e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc\" (UID: \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\") " pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"
Sep 09 06:11:59.371948 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:59.371800    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-istio-podinfo\") pod \"e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc\" (UID: \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\") " pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"
Sep 09 06:11:59.371948 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:59.371828    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgm7b\" (UniqueName: \"kubernetes.io/projected/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-kube-api-access-fgm7b\") pod \"e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc\" (UID: \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\") " pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"
Sep 09 06:11:59.371948 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:59.371893    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-credential-socket\") pod \"e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc\" (UID: \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\") " pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"
Sep 09 06:11:59.372071 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:59.371976    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-workload-socket\") pod \"e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc\" (UID: \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\") " pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"
Sep 09 06:11:59.372071 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:59.372061    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-istio-data\") pod \"e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc\" (UID: \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\") " pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"
Sep 09 06:11:59.372271 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:59.372247    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-istiod-ca-cert\") pod \"e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc\" (UID: \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\") " pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"
Sep 09 06:11:59.374080 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:59.374052    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-istio-envoy\") pod \"e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc\" (UID: \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\") " pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"
Sep 09 06:11:59.374404 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:59.374381    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-istio-podinfo\") pod \"e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc\" (UID: \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\") " pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"
Sep 09 06:11:59.386925 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:59.386905    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-istio-token\") pod \"e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc\" (UID: \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\") " pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"
Sep 09 06:11:59.394276 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:59.394236    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgm7b\" (UniqueName: \"kubernetes.io/projected/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-kube-api-access-fgm7b\") pod \"e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc\" (UID: \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\") " pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"
Sep 09 06:11:59.465824 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:59.465750    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"
Sep 09 06:11:59.743790 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:59.743766    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"]
Sep 09 06:11:59.745884 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:11:59.745861    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec1cd9d9_051b_4641_8b07_7d0b8984dcfb.slice/crio-be29670e2803bf66391ebf755d7e12ac73ed3bf912c429da85f5551a1e9e55a9 WatchSource:0}: Error finding container be29670e2803bf66391ebf755d7e12ac73ed3bf912c429da85f5551a1e9e55a9: Status 404 returned error can't find the container with id be29670e2803bf66391ebf755d7e12ac73ed3bf912c429da85f5551a1e9e55a9
Sep 09 06:11:59.748166 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:59.748133    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Sep 09 06:11:59.748232 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:59.748204    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Sep 09 06:11:59.748319 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:11:59.748234    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Sep 09 06:12:00.071978 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:00.071948    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc" event={"ID":"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb","Type":"ContainerStarted","Data":"a76c777a5f2e576566de3011e920b59907146f9d2c821a9abef20cf29f8c9da4"}
Sep 09 06:12:00.072154 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:00.072004    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc" event={"ID":"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb","Type":"ContainerStarted","Data":"be29670e2803bf66391ebf755d7e12ac73ed3bf912c429da85f5551a1e9e55a9"}
Sep 09 06:12:00.094071 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:00.094028    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc" podStartSLOduration=1.094012579 podStartE2EDuration="1.094012579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:11:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:12:00.092504959 +0000 UTC m=+1345.432992910" watchObservedRunningTime="2026-09-09 06:12:00.094012579 +0000 UTC m=+1345.434500489"
Sep 09 06:12:00.467002 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:00.466926    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"
Sep 09 06:12:00.471541 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:00.471516    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"
Sep 09 06:12:01.076645 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:01.076586    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"
Sep 09 06:12:01.077666 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:01.077644    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"
Sep 09 06:12:17.005538 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:17.005502    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:12:27.633272 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:27.633238    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"]
Sep 09 06:12:27.636935 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:27.636910    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"
Sep 09 06:12:27.639301 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:27.639133    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-d907ff-openshift-default-dockercfg-68fnb\""
Sep 09 06:12:27.649142 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:27.649106    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"]
Sep 09 06:12:27.722463 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:27.722432    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-istio-data\") pod \"e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794\" (UID: \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\") " pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"
Sep 09 06:12:27.722675 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:27.722485    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-workload-socket\") pod \"e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794\" (UID: \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\") " pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"
Sep 09 06:12:27.722675 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:27.722514    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-istio-envoy\") pod \"e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794\" (UID: \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\") " pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"
Sep 09 06:12:27.722675 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:27.722538    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xnpl\" (UniqueName: \"kubernetes.io/projected/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-kube-api-access-8xnpl\") pod \"e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794\" (UID: \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\") " pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"
Sep 09 06:12:27.722675 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:27.722618    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-istio-podinfo\") pod \"e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794\" (UID: \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\") " pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"
Sep 09 06:12:27.722886 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:27.722715    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-istiod-ca-cert\") pod \"e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794\" (UID: \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\") " pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"
Sep 09 06:12:27.722886 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:27.722779    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-istio-token\") pod \"e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794\" (UID: \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\") " pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"
Sep 09 06:12:27.722886 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:27.722835    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-credential-socket\") pod \"e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794\" (UID: \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\") " pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"
Sep 09 06:12:27.722886 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:27.722864    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-workload-certs\") pod \"e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794\" (UID: \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\") " pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"
Sep 09 06:12:27.823792 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:27.823758    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-istiod-ca-cert\") pod \"e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794\" (UID: \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\") " pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"
Sep 09 06:12:27.826191 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:27.823811    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-istio-token\") pod \"e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794\" (UID: \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\") " pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"
Sep 09 06:12:27.826191 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:27.823843    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-credential-socket\") pod \"e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794\" (UID: \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\") " pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"
Sep 09 06:12:27.826191 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:27.823880    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-workload-certs\") pod \"e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794\" (UID: \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\") " pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"
Sep 09 06:12:27.826191 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:27.823922    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-istio-data\") pod \"e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794\" (UID: \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\") " pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"
Sep 09 06:12:27.826191 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:27.823968    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-workload-socket\") pod \"e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794\" (UID: \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\") " pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"
Sep 09 06:12:27.826191 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:27.824191    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-istio-envoy\") pod \"e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794\" (UID: \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\") " pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"
Sep 09 06:12:27.826191 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:27.824264    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xnpl\" (UniqueName: \"kubernetes.io/projected/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-kube-api-access-8xnpl\") pod \"e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794\" (UID: \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\") " pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"
Sep 09 06:12:27.826191 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:27.824325    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-istio-podinfo\") pod \"e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794\" (UID: \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\") " pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"
Sep 09 06:12:27.826191 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:27.824260    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-workload-certs\") pod \"e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794\" (UID: \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\") " pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"
Sep 09 06:12:27.826191 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:27.824377    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-credential-socket\") pod \"e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794\" (UID: \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\") " pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"
Sep 09 06:12:27.826191 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:27.824399    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-istio-data\") pod \"e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794\" (UID: \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\") " pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"
Sep 09 06:12:27.826191 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:27.824530    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-workload-socket\") pod \"e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794\" (UID: \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\") " pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"
Sep 09 06:12:27.826191 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:27.824539    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-istiod-ca-cert\") pod \"e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794\" (UID: \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\") " pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"
Sep 09 06:12:27.827214 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:27.827183    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-istio-envoy\") pod \"e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794\" (UID: \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\") " pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"
Sep 09 06:12:27.827467 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:27.827447    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-istio-podinfo\") pod \"e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794\" (UID: \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\") " pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"
Sep 09 06:12:27.831736 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:27.831717    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xnpl\" (UniqueName: \"kubernetes.io/projected/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-kube-api-access-8xnpl\") pod \"e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794\" (UID: \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\") " pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"
Sep 09 06:12:27.832232 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:27.832212    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-istio-token\") pod \"e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794\" (UID: \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\") " pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"
Sep 09 06:12:27.949486 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:27.949447    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"
Sep 09 06:12:28.214613 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:28.214534    2579 generic.go:356] "Generic (PLEG): container finished" podID="ec1cd9d9-051b-4641-8b07-7d0b8984dcfb" containerID="a76c777a5f2e576566de3011e920b59907146f9d2c821a9abef20cf29f8c9da4" exitCode=0
Sep 09 06:12:28.214803 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:28.214610    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc" event={"ID":"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb","Type":"ContainerDied","Data":"a76c777a5f2e576566de3011e920b59907146f9d2c821a9abef20cf29f8c9da4"}
Sep 09 06:12:28.215111 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:28.215093    2579 scope.go:117] "RemoveContainer" containerID="a76c777a5f2e576566de3011e920b59907146f9d2c821a9abef20cf29f8c9da4"
Sep 09 06:12:28.216267 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:28.216241    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Sep 09 06:12:28.216336 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:28.216306    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Sep 09 06:12:28.216374 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:28.216336    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Sep 09 06:12:28.229206 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:28.229190    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"]
Sep 09 06:12:28.231917 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:12:28.231878    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod619ba57a_cb9b_46e8_ae40_6c2b8ebef515.slice/crio-533f641b6bb48f906df0267658e2226f3f3ba5dc55376a7dab3276893c35d546 WatchSource:0}: Error finding container 533f641b6bb48f906df0267658e2226f3f3ba5dc55376a7dab3276893c35d546: Status 404 returned error can't find the container with id 533f641b6bb48f906df0267658e2226f3f3ba5dc55376a7dab3276893c35d546
Sep 09 06:12:28.234412 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:28.234378    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Sep 09 06:12:28.234488 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:28.234453    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Sep 09 06:12:28.234527 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:28.234497    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Sep 09 06:12:29.222664 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:29.222602    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc" event={"ID":"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb","Type":"ContainerStarted","Data":"0f6bbf3fc53b06ea88191bc33b351282926f24c969bbdf8d7ddc93dccdbdabee"}
Sep 09 06:12:29.225781 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:29.225744    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794" event={"ID":"619ba57a-cb9b-46e8-ae40-6c2b8ebef515","Type":"ContainerStarted","Data":"b5261f8fe692cae5c0f3d41024cca76331ad802ba9ec175b39be270b92e8fa13"}
Sep 09 06:12:29.225945 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:29.225902    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794" event={"ID":"619ba57a-cb9b-46e8-ae40-6c2b8ebef515","Type":"ContainerStarted","Data":"533f641b6bb48f906df0267658e2226f3f3ba5dc55376a7dab3276893c35d546"}
Sep 09 06:12:29.264418 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:29.264368    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794" podStartSLOduration=2.26435395 podStartE2EDuration="2.26435395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:12:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:12:29.263244844 +0000 UTC m=+1374.603732755" watchObservedRunningTime="2026-09-09 06:12:29.26435395 +0000 UTC m=+1374.604841858"
Sep 09 06:12:29.466005 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:29.465968    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"
Sep 09 06:12:29.467295 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:29.467267    2579 patch_prober.go:28] interesting pod/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 09 06:12:29.467426 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:29.467315    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc" podUID="ec1cd9d9-051b-4641-8b07-7d0b8984dcfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 09 06:12:29.949872 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:29.949833    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"
Sep 09 06:12:29.954968 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:29.954942    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"
Sep 09 06:12:30.234840 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:30.234754    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"
Sep 09 06:12:30.240134 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:30.240110    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"
Sep 09 06:12:30.467139 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:30.467108    2579 patch_prober.go:28] interesting pod/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 09 06:12:30.467392 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:30.467161    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc" podUID="ec1cd9d9-051b-4641-8b07-7d0b8984dcfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 09 06:12:31.077496 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:31.077463    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"
Sep 09 06:12:31.466254 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:31.466173    2579 patch_prober.go:28] interesting pod/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 09 06:12:31.466254 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:31.466229    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc" podUID="ec1cd9d9-051b-4641-8b07-7d0b8984dcfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 09 06:12:32.466099 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:32.466067    2579 patch_prober.go:28] interesting pod/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 09 06:12:32.466282 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:32.466168    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc" podUID="ec1cd9d9-051b-4641-8b07-7d0b8984dcfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 09 06:12:33.039345 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:33.039304    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d907ff-75b7557474-tql5j"]
Sep 09 06:12:33.043687 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:33.043657    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d907ff-75b7557474-tql5j"
Sep 09 06:12:33.046160 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:33.046140    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-d907ff-dockercfg-2hngv\""
Sep 09 06:12:33.046473 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:33.046183    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-d907ff\""
Sep 09 06:12:33.059940 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:33.059918    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d907ff-75b7557474-tql5j"]
Sep 09 06:12:33.177765 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:33.177731    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmps8\" (UniqueName: \"kubernetes.io/projected/25168fb6-d620-4016-a464-42e83eb3bb53-kube-api-access-rmps8\") pod \"payload-pre-processing-e2e-api-b-d907ff-75b7557474-tql5j\" (UID: \"25168fb6-d620-4016-a464-42e83eb3bb53\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d907ff-75b7557474-tql5j"
Sep 09 06:12:33.177929 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:33.177828    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25168fb6-d620-4016-a464-42e83eb3bb53-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d907ff-75b7557474-tql5j\" (UID: \"25168fb6-d620-4016-a464-42e83eb3bb53\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d907ff-75b7557474-tql5j"
Sep 09 06:12:33.279148 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:33.279105    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25168fb6-d620-4016-a464-42e83eb3bb53-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d907ff-75b7557474-tql5j\" (UID: \"25168fb6-d620-4016-a464-42e83eb3bb53\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d907ff-75b7557474-tql5j"
Sep 09 06:12:33.279316 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:33.279191    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmps8\" (UniqueName: \"kubernetes.io/projected/25168fb6-d620-4016-a464-42e83eb3bb53-kube-api-access-rmps8\") pod \"payload-pre-processing-e2e-api-b-d907ff-75b7557474-tql5j\" (UID: \"25168fb6-d620-4016-a464-42e83eb3bb53\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d907ff-75b7557474-tql5j"
Sep 09 06:12:33.279888 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:33.279867    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25168fb6-d620-4016-a464-42e83eb3bb53-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d907ff-75b7557474-tql5j\" (UID: \"25168fb6-d620-4016-a464-42e83eb3bb53\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d907ff-75b7557474-tql5j"
Sep 09 06:12:33.287198 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:33.287176    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmps8\" (UniqueName: \"kubernetes.io/projected/25168fb6-d620-4016-a464-42e83eb3bb53-kube-api-access-rmps8\") pod \"payload-pre-processing-e2e-api-b-d907ff-75b7557474-tql5j\" (UID: \"25168fb6-d620-4016-a464-42e83eb3bb53\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d907ff-75b7557474-tql5j"
Sep 09 06:12:33.355963 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:33.355930    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d907ff-75b7557474-tql5j"
Sep 09 06:12:33.466397 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:33.466362    2579 patch_prober.go:28] interesting pod/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 09 06:12:33.466748 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:33.466433    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc" podUID="ec1cd9d9-051b-4641-8b07-7d0b8984dcfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 09 06:12:33.630789 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:33.628264    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d907ff-75b7557474-tql5j"]
Sep 09 06:12:34.264036 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:34.263997    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d907ff-75b7557474-tql5j" event={"ID":"25168fb6-d620-4016-a464-42e83eb3bb53","Type":"ContainerStarted","Data":"178233d3ca2dc089cdc11b52a8636984b811986a8223fc81f014253711597b21"}
Sep 09 06:12:34.264254 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:34.264056    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d907ff-75b7557474-tql5j" event={"ID":"25168fb6-d620-4016-a464-42e83eb3bb53","Type":"ContainerStarted","Data":"d961b9b0615f200c6152571e26f4bdfb4085d277b96be8e11a4d4a5476f5379a"}
Sep 09 06:12:34.264254 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:34.264138    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d907ff-75b7557474-tql5j"
Sep 09 06:12:34.281262 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:34.281208    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d907ff-75b7557474-tql5j" podStartSLOduration=1.281187438 podStartE2EDuration="1.281187438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:12:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:12:34.279243868 +0000 UTC m=+1379.619731769" watchObservedRunningTime="2026-09-09 06:12:34.281187438 +0000 UTC m=+1379.621675348"
Sep 09 06:12:34.470528 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:34.470496    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"
Sep 09 06:12:34.471426 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:34.471408    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"
Sep 09 06:12:35.348786 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:35.348758    2579 scope.go:117] "RemoveContainer" containerID="eaee0ca4be1a6861898f6fe2e38c37ce959549043e76d54a46a1a94e4229cb20"
Sep 09 06:12:43.805572 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:43.805529    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:12:45.270387 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:45.270358    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d907ff-75b7557474-tql5j"
Sep 09 06:12:55.383130 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:55.383096    2579 generic.go:356] "Generic (PLEG): container finished" podID="619ba57a-cb9b-46e8-ae40-6c2b8ebef515" containerID="b5261f8fe692cae5c0f3d41024cca76331ad802ba9ec175b39be270b92e8fa13" exitCode=0
Sep 09 06:12:55.383677 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:55.383179    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794" event={"ID":"619ba57a-cb9b-46e8-ae40-6c2b8ebef515","Type":"ContainerDied","Data":"b5261f8fe692cae5c0f3d41024cca76331ad802ba9ec175b39be270b92e8fa13"}
Sep 09 06:12:55.383677 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:55.383660    2579 scope.go:117] "RemoveContainer" containerID="b5261f8fe692cae5c0f3d41024cca76331ad802ba9ec175b39be270b92e8fa13"
Sep 09 06:12:55.390943 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:55.385718    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Sep 09 06:12:55.390943 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:55.385932    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Sep 09 06:12:55.390943 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:55.386032    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Sep 09 06:12:56.390497 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:56.390461    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794" event={"ID":"619ba57a-cb9b-46e8-ae40-6c2b8ebef515","Type":"ContainerStarted","Data":"1c616c15e9b57191139056cfb5e04a2c1760ffc8b4a30a9fed0cbd4f77717a25"}
Sep 09 06:12:56.949837 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:56.949797    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"
Sep 09 06:12:56.951008 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:56.950980    2579 patch_prober.go:28] interesting pod/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 09 06:12:56.951144 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:56.951046    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794" podUID="619ba57a-cb9b-46e8-ae40-6c2b8ebef515" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 09 06:12:57.950214 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:57.950179    2579 patch_prober.go:28] interesting pod/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 09 06:12:57.950596 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:57.950244    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794" podUID="619ba57a-cb9b-46e8-ae40-6c2b8ebef515" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 09 06:12:58.950207 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:58.950159    2579 patch_prober.go:28] interesting pod/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 09 06:12:58.950416 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:58.950231    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794" podUID="619ba57a-cb9b-46e8-ae40-6c2b8ebef515" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 09 06:12:59.050668 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:59.050636    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e1ed02-hpxqv"]
Sep 09 06:12:59.053969 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:59.053953    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e1ed02-hpxqv"
Sep 09 06:12:59.056036 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:59.056017    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 06:12:59.056123 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:59.056017    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ksjk6\""
Sep 09 06:12:59.059909 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:59.059888    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e1ed02-hpxqv"]
Sep 09 06:12:59.220220 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:59.220127    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a709128-ac21-4986-89af-bf32ec8cf0df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e1ed02-hpxqv\" (UID: \"0a709128-ac21-4986-89af-bf32ec8cf0df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e1ed02-hpxqv"
Sep 09 06:12:59.321298 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:59.321264    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a709128-ac21-4986-89af-bf32ec8cf0df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e1ed02-hpxqv\" (UID: \"0a709128-ac21-4986-89af-bf32ec8cf0df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e1ed02-hpxqv"
Sep 09 06:12:59.322206 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:59.322172    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a709128-ac21-4986-89af-bf32ec8cf0df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e1ed02-hpxqv\" (UID: \"0a709128-ac21-4986-89af-bf32ec8cf0df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e1ed02-hpxqv"
Sep 09 06:12:59.366854 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:59.366829    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e1ed02-hpxqv"
Sep 09 06:12:59.659226 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:59.659205    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e1ed02-hpxqv"]
Sep 09 06:12:59.661397 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:12:59.661361    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a709128_ac21_4986_89af_bf32ec8cf0df.slice/crio-2f65b0c6a8eea48931d313b3fe71b7cd2fd7cccf97e91eb1394df364b79c54de WatchSource:0}: Error finding container 2f65b0c6a8eea48931d313b3fe71b7cd2fd7cccf97e91eb1394df364b79c54de: Status 404 returned error can't find the container with id 2f65b0c6a8eea48931d313b3fe71b7cd2fd7cccf97e91eb1394df364b79c54de
Sep 09 06:12:59.950105 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:59.950082    2579 patch_prober.go:28] interesting pod/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 09 06:12:59.950256 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:12:59.950235    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794" podUID="619ba57a-cb9b-46e8-ae40-6c2b8ebef515" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 09 06:13:00.235312 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:00.235221    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"
Sep 09 06:13:00.412581 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:00.412534    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e1ed02-hpxqv" event={"ID":"0a709128-ac21-4986-89af-bf32ec8cf0df","Type":"ContainerStarted","Data":"8e156decab8b63888831ac002675e1dbf908aaa693c6a64d325578f3524cad70"}
Sep 09 06:13:00.412766 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:00.412593    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e1ed02-hpxqv" event={"ID":"0a709128-ac21-4986-89af-bf32ec8cf0df","Type":"ContainerStarted","Data":"2f65b0c6a8eea48931d313b3fe71b7cd2fd7cccf97e91eb1394df364b79c54de"}
Sep 09 06:13:00.950875 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:00.950794    2579 patch_prober.go:28] interesting pod/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 09 06:13:00.950875 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:00.950850    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794" podUID="619ba57a-cb9b-46e8-ae40-6c2b8ebef515" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 09 06:13:01.420248 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:01.420208    2579 generic.go:356] "Generic (PLEG): container finished" podID="0a709128-ac21-4986-89af-bf32ec8cf0df" containerID="8e156decab8b63888831ac002675e1dbf908aaa693c6a64d325578f3524cad70" exitCode=0
Sep 09 06:13:01.420898 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:01.420313    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e1ed02-hpxqv" event={"ID":"0a709128-ac21-4986-89af-bf32ec8cf0df","Type":"ContainerDied","Data":"8e156decab8b63888831ac002675e1dbf908aaa693c6a64d325578f3524cad70"}
Sep 09 06:13:01.950778 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:01.950743    2579 patch_prober.go:28] interesting pod/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 09 06:13:01.950971 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:01.950799    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794" podUID="619ba57a-cb9b-46e8-ae40-6c2b8ebef515" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 09 06:13:02.559794 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:02.559767    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e1ed02-hpxqv"
Sep 09 06:13:02.570719 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:02.570697    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a709128-ac21-4986-89af-bf32ec8cf0df-maas-api-service-ca\") pod \"0a709128-ac21-4986-89af-bf32ec8cf0df\" (UID: \"0a709128-ac21-4986-89af-bf32ec8cf0df\") "
Sep 09 06:13:02.570994 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:02.570974    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a709128-ac21-4986-89af-bf32ec8cf0df-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0a709128-ac21-4986-89af-bf32ec8cf0df" (UID: "0a709128-ac21-4986-89af-bf32ec8cf0df"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:13:02.671524 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:02.671495    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a709128-ac21-4986-89af-bf32ec8cf0df-maas-api-service-ca\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:02.954145 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:02.954120    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"
Sep 09 06:13:02.955271 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:02.955253    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"
Sep 09 06:13:03.431871 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:03.431847    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e1ed02-hpxqv"
Sep 09 06:13:03.432044 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:03.431865    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e1ed02-hpxqv" event={"ID":"0a709128-ac21-4986-89af-bf32ec8cf0df","Type":"ContainerDied","Data":"2f65b0c6a8eea48931d313b3fe71b7cd2fd7cccf97e91eb1394df364b79c54de"}
Sep 09 06:13:03.432044 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:03.431915    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f65b0c6a8eea48931d313b3fe71b7cd2fd7cccf97e91eb1394df364b79c54de"
Sep 09 06:13:13.193601 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:13.193567    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:13:15.064918 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:15.064885    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e1ed02-hpxqv"]
Sep 09 06:13:15.067666 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:15.067641    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e1ed02-hpxqv"]
Sep 09 06:13:15.149786 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:15.149752    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a709128-ac21-4986-89af-bf32ec8cf0df" path="/var/lib/kubelet/pods/0a709128-ac21-4986-89af-bf32ec8cf0df/volumes"
Sep 09 06:13:19.792762 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:19.792731    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-43eb2d-xtlh5"]
Sep 09 06:13:19.793134 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:19.793119    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a709128-ac21-4986-89af-bf32ec8cf0df" containerName="revoke-keys"
Sep 09 06:13:19.793134 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:19.793132    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a709128-ac21-4986-89af-bf32ec8cf0df" containerName="revoke-keys"
Sep 09 06:13:19.793208 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:19.793198    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a709128-ac21-4986-89af-bf32ec8cf0df" containerName="revoke-keys"
Sep 09 06:13:19.796385 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:19.796364    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-43eb2d-xtlh5"
Sep 09 06:13:19.798976 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:19.798960    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ksjk6\""
Sep 09 06:13:19.799071 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:19.798996    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 06:13:19.804460 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:19.804435    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-43eb2d-xtlh5"]
Sep 09 06:13:19.920830 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:19.920800    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e73dee7d-2739-445f-bc52-afca338084fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-43eb2d-xtlh5\" (UID: \"e73dee7d-2739-445f-bc52-afca338084fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-43eb2d-xtlh5"
Sep 09 06:13:20.022272 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:20.022241    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e73dee7d-2739-445f-bc52-afca338084fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-43eb2d-xtlh5\" (UID: \"e73dee7d-2739-445f-bc52-afca338084fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-43eb2d-xtlh5"
Sep 09 06:13:20.022887 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:20.022866    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e73dee7d-2739-445f-bc52-afca338084fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-43eb2d-xtlh5\" (UID: \"e73dee7d-2739-445f-bc52-afca338084fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-43eb2d-xtlh5"
Sep 09 06:13:20.108057 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:20.108032    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-43eb2d-xtlh5"
Sep 09 06:13:20.379098 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:20.379074    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-43eb2d-xtlh5"]
Sep 09 06:13:20.381181 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:13:20.381152    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode73dee7d_2739_445f_bc52_afca338084fd.slice/crio-a68bf09b0eecec6984fc4f0b79df3e854c916471d03e311b5af32c20ab6821e1 WatchSource:0}: Error finding container a68bf09b0eecec6984fc4f0b79df3e854c916471d03e311b5af32c20ab6821e1: Status 404 returned error can't find the container with id a68bf09b0eecec6984fc4f0b79df3e854c916471d03e311b5af32c20ab6821e1
Sep 09 06:13:20.518097 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:20.518071    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-43eb2d-xtlh5" event={"ID":"e73dee7d-2739-445f-bc52-afca338084fd","Type":"ContainerStarted","Data":"a68bf09b0eecec6984fc4f0b79df3e854c916471d03e311b5af32c20ab6821e1"}
Sep 09 06:13:21.413252 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:21.413225    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"]
Sep 09 06:13:21.413717 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:21.413420    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc" podUID="ec1cd9d9-051b-4641-8b07-7d0b8984dcfb" containerName="istio-proxy" containerID="cri-o://0f6bbf3fc53b06ea88191bc33b351282926f24c969bbdf8d7ddc93dccdbdabee" gracePeriod=30
Sep 09 06:13:21.525582 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:21.525545    2579 generic.go:356] "Generic (PLEG): container finished" podID="e73dee7d-2739-445f-bc52-afca338084fd" containerID="d63c588d1180e2c634a32465aaedb1213606d6275bed98ab02c0bb38d73d3901" exitCode=0
Sep 09 06:13:21.525794 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:21.525692    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-43eb2d-xtlh5" event={"ID":"e73dee7d-2739-445f-bc52-afca338084fd","Type":"ContainerDied","Data":"d63c588d1180e2c634a32465aaedb1213606d6275bed98ab02c0bb38d73d3901"}
Sep 09 06:13:21.733359 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:21.733285    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d907ff-jnf42"]
Sep 09 06:13:21.736478 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:21.736461    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d907ff-jnf42"
Sep 09 06:13:21.743972 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:21.743820    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d907ff-jnf42"]
Sep 09 06:13:21.838416 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:21.838386    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf59ce57-8063-4a0b-97bf-3170a2d5483e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d907ff-jnf42\" (UID: \"bf59ce57-8063-4a0b-97bf-3170a2d5483e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d907ff-jnf42"
Sep 09 06:13:21.939690 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:21.939660    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf59ce57-8063-4a0b-97bf-3170a2d5483e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d907ff-jnf42\" (UID: \"bf59ce57-8063-4a0b-97bf-3170a2d5483e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d907ff-jnf42"
Sep 09 06:13:21.940291 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:21.940264    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf59ce57-8063-4a0b-97bf-3170a2d5483e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d907ff-jnf42\" (UID: \"bf59ce57-8063-4a0b-97bf-3170a2d5483e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d907ff-jnf42"
Sep 09 06:13:22.048649 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:22.048545    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d907ff-jnf42"
Sep 09 06:13:22.319379 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:22.319346    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d907ff-jnf42"]
Sep 09 06:13:22.321611 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:13:22.321581    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf59ce57_8063_4a0b_97bf_3170a2d5483e.slice/crio-b44db83d25d459b58212d70eaba30c6f9b3b8bac513f1f473e40361348a3a947 WatchSource:0}: Error finding container b44db83d25d459b58212d70eaba30c6f9b3b8bac513f1f473e40361348a3a947: Status 404 returned error can't find the container with id b44db83d25d459b58212d70eaba30c6f9b3b8bac513f1f473e40361348a3a947
Sep 09 06:13:22.531378 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:22.531353    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d907ff-jnf42" event={"ID":"bf59ce57-8063-4a0b-97bf-3170a2d5483e","Type":"ContainerStarted","Data":"b44db83d25d459b58212d70eaba30c6f9b3b8bac513f1f473e40361348a3a947"}
Sep 09 06:13:22.722994 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:22.722970    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-43eb2d-xtlh5"
Sep 09 06:13:22.847735 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:22.847706    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e73dee7d-2739-445f-bc52-afca338084fd-maas-api-service-ca\") pod \"e73dee7d-2739-445f-bc52-afca338084fd\" (UID: \"e73dee7d-2739-445f-bc52-afca338084fd\") "
Sep 09 06:13:22.848049 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:22.848026    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e73dee7d-2739-445f-bc52-afca338084fd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e73dee7d-2739-445f-bc52-afca338084fd" (UID: "e73dee7d-2739-445f-bc52-afca338084fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:13:22.948842 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:22.948773    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e73dee7d-2739-445f-bc52-afca338084fd-maas-api-service-ca\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:23.537791 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:23.537752    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-43eb2d-xtlh5"
Sep 09 06:13:23.538199 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:23.537752    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-43eb2d-xtlh5" event={"ID":"e73dee7d-2739-445f-bc52-afca338084fd","Type":"ContainerDied","Data":"a68bf09b0eecec6984fc4f0b79df3e854c916471d03e311b5af32c20ab6821e1"}
Sep 09 06:13:23.538199 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:23.537882    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a68bf09b0eecec6984fc4f0b79df3e854c916471d03e311b5af32c20ab6821e1"
Sep 09 06:13:23.539385 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:23.539362    2579 generic.go:356] "Generic (PLEG): container finished" podID="bf59ce57-8063-4a0b-97bf-3170a2d5483e" containerID="494aa76251c124b1cf8ab6d9c3770c5170b8606430c1820ab3099337a5084b59" exitCode=0
Sep 09 06:13:23.539503 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:23.539413    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d907ff-jnf42" event={"ID":"bf59ce57-8063-4a0b-97bf-3170a2d5483e","Type":"ContainerDied","Data":"494aa76251c124b1cf8ab6d9c3770c5170b8606430c1820ab3099337a5084b59"}
Sep 09 06:13:24.663267 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:24.663234    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-43eb2d/distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7"]
Sep 09 06:13:24.668525 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:24.668500    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-43eb2d/distinct-w4-43eb2d-kserve-596559c7f-8x4kz"]
Sep 09 06:13:24.676721 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:24.676695    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d907ff-jnf42"
Sep 09 06:13:24.678987 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:24.678964    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-43eb2d/premium-w4-43eb2d-kserve-84b57f44-tlrbl"]
Sep 09 06:13:24.687306 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:24.687282    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-43eb2d/simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk"]
Sep 09 06:13:24.763719 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:24.763694    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf59ce57-8063-4a0b-97bf-3170a2d5483e-maas-api-service-ca\") pod \"bf59ce57-8063-4a0b-97bf-3170a2d5483e\" (UID: \"bf59ce57-8063-4a0b-97bf-3170a2d5483e\") "
Sep 09 06:13:24.764047 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:24.764024    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf59ce57-8063-4a0b-97bf-3170a2d5483e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bf59ce57-8063-4a0b-97bf-3170a2d5483e" (UID: "bf59ce57-8063-4a0b-97bf-3170a2d5483e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:13:24.864571 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:24.864530    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf59ce57-8063-4a0b-97bf-3170a2d5483e-maas-api-service-ca\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:25.550983 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:25.550891    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d907ff-jnf42" event={"ID":"bf59ce57-8063-4a0b-97bf-3170a2d5483e","Type":"ContainerDied","Data":"b44db83d25d459b58212d70eaba30c6f9b3b8bac513f1f473e40361348a3a947"}
Sep 09 06:13:25.550983 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:25.550921    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d907ff-jnf42"
Sep 09 06:13:25.550983 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:25.550949    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b44db83d25d459b58212d70eaba30c6f9b3b8bac513f1f473e40361348a3a947"
Sep 09 06:13:26.561612 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:26.561577    2579 generic.go:356] "Generic (PLEG): container finished" podID="ec1cd9d9-051b-4641-8b07-7d0b8984dcfb" containerID="0f6bbf3fc53b06ea88191bc33b351282926f24c969bbdf8d7ddc93dccdbdabee" exitCode=0
Sep 09 06:13:26.561968 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:26.561664    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc" event={"ID":"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb","Type":"ContainerDied","Data":"0f6bbf3fc53b06ea88191bc33b351282926f24c969bbdf8d7ddc93dccdbdabee"}
Sep 09 06:13:26.561968 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:26.561742    2579 scope.go:117] "RemoveContainer" containerID="a76c777a5f2e576566de3011e920b59907146f9d2c821a9abef20cf29f8c9da4"
Sep 09 06:13:26.662851 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:26.662831    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"
Sep 09 06:13:26.780366 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:26.780277    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgm7b\" (UniqueName: \"kubernetes.io/projected/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-kube-api-access-fgm7b\") pod \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\" (UID: \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\") "
Sep 09 06:13:26.780366 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:26.780318    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-istio-podinfo\") pod \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\" (UID: \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\") "
Sep 09 06:13:26.780366 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:26.780355    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-workload-socket\") pod \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\" (UID: \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\") "
Sep 09 06:13:26.780685 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:26.780409    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-istio-token\") pod \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\" (UID: \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\") "
Sep 09 06:13:26.780685 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:26.780457    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-credential-socket\") pod \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\" (UID: \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\") "
Sep 09 06:13:26.780685 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:26.780495    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-istiod-ca-cert\") pod \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\" (UID: \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\") "
Sep 09 06:13:26.780685 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:26.780552    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-istio-envoy\") pod \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\" (UID: \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\") "
Sep 09 06:13:26.780685 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:26.780619    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-istio-data\") pod \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\" (UID: \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\") "
Sep 09 06:13:26.780685 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:26.780671    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-workload-certs\") pod \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\" (UID: \"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb\") "
Sep 09 06:13:26.780993 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:26.780733    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ec1cd9d9-051b-4641-8b07-7d0b8984dcfb" (UID: "ec1cd9d9-051b-4641-8b07-7d0b8984dcfb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:13:26.780993 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:26.780753    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ec1cd9d9-051b-4641-8b07-7d0b8984dcfb" (UID: "ec1cd9d9-051b-4641-8b07-7d0b8984dcfb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:13:26.780993 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:26.780953    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-istio-data" (OuterVolumeSpecName: "istio-data") pod "ec1cd9d9-051b-4641-8b07-7d0b8984dcfb" (UID: "ec1cd9d9-051b-4641-8b07-7d0b8984dcfb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:13:26.780993 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:26.780969    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ec1cd9d9-051b-4641-8b07-7d0b8984dcfb" (UID: "ec1cd9d9-051b-4641-8b07-7d0b8984dcfb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:13:26.781279 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:26.781196    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-credential-socket\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:26.781279 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:26.781280    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-istio-data\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:26.781472 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:26.781295    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-workload-certs\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:26.781472 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:26.781307    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-workload-socket\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:26.781472 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:26.781242    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ec1cd9d9-051b-4641-8b07-7d0b8984dcfb" (UID: "ec1cd9d9-051b-4641-8b07-7d0b8984dcfb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:13:26.782787 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:26.782764    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ec1cd9d9-051b-4641-8b07-7d0b8984dcfb" (UID: "ec1cd9d9-051b-4641-8b07-7d0b8984dcfb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 06:13:26.782879 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:26.782855    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ec1cd9d9-051b-4641-8b07-7d0b8984dcfb" (UID: "ec1cd9d9-051b-4641-8b07-7d0b8984dcfb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:13:26.782879 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:26.782862    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-kube-api-access-fgm7b" (OuterVolumeSpecName: "kube-api-access-fgm7b") pod "ec1cd9d9-051b-4641-8b07-7d0b8984dcfb" (UID: "ec1cd9d9-051b-4641-8b07-7d0b8984dcfb"). InnerVolumeSpecName "kube-api-access-fgm7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:13:26.783158 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:26.783139    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-istio-token" (OuterVolumeSpecName: "istio-token") pod "ec1cd9d9-051b-4641-8b07-7d0b8984dcfb" (UID: "ec1cd9d9-051b-4641-8b07-7d0b8984dcfb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:13:26.881890 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:26.881863    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-istio-envoy\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:26.881890 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:26.881885    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgm7b\" (UniqueName: \"kubernetes.io/projected/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-kube-api-access-fgm7b\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:26.881890 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:26.881896    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-istio-podinfo\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:26.882092 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:26.881906    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-istio-token\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:26.882092 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:26.881914    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb-istiod-ca-cert\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:27.569045 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:27.569008    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc" event={"ID":"ec1cd9d9-051b-4641-8b07-7d0b8984dcfb","Type":"ContainerDied","Data":"be29670e2803bf66391ebf755d7e12ac73ed3bf912c429da85f5551a1e9e55a9"}
Sep 09 06:13:27.569434 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:27.569019    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"
Sep 09 06:13:27.569434 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:27.569098    2579 scope.go:117] "RemoveContainer" containerID="0f6bbf3fc53b06ea88191bc33b351282926f24c969bbdf8d7ddc93dccdbdabee"
Sep 09 06:13:27.586464 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:27.586440    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"]
Sep 09 06:13:27.590295 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:27.590274    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e1ed02-openshift-default-6974d5668b-2hdxc"]
Sep 09 06:13:29.147751 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:29.147725    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec1cd9d9-051b-4641-8b07-7d0b8984dcfb" path="/var/lib/kubelet/pods/ec1cd9d9-051b-4641-8b07-7d0b8984dcfb/volumes"
Sep 09 06:13:35.368740 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:35.368709    2579 scope.go:117] "RemoveContainer" containerID="4fdbf581da9e6efc61a921b0bd07f66edaeacd4dc21e29fdb251d927913f4b06"
Sep 09 06:13:35.804863 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:35.804787    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-43eb2d-xtlh5"]
Sep 09 06:13:35.807304 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:35.807275    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-43eb2d-xtlh5"]
Sep 09 06:13:37.148638 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:37.148585    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e73dee7d-2739-445f-bc52-afca338084fd" path="/var/lib/kubelet/pods/e73dee7d-2739-445f-bc52-afca338084fd/volumes"
Sep 09 06:13:37.455387 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:37.455310    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d907ff-75b7557474-tql5j"]
Sep 09 06:13:37.455695 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:37.455546    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d907ff-75b7557474-tql5j" podUID="25168fb6-d620-4016-a464-42e83eb3bb53" containerName="payload-pre-processing" containerID="cri-o://178233d3ca2dc089cdc11b52a8636984b811986a8223fc81f014253711597b21" gracePeriod=30
Sep 09 06:13:37.600809 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:37.600782    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d907ff-jnf42"]
Sep 09 06:13:37.604833 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:37.604809    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d907ff-jnf42"]
Sep 09 06:13:37.623665 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:37.623610    2579 generic.go:356] "Generic (PLEG): container finished" podID="25168fb6-d620-4016-a464-42e83eb3bb53" containerID="178233d3ca2dc089cdc11b52a8636984b811986a8223fc81f014253711597b21" exitCode=0
Sep 09 06:13:37.623775 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:37.623668    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d907ff-75b7557474-tql5j" event={"ID":"25168fb6-d620-4016-a464-42e83eb3bb53","Type":"ContainerDied","Data":"178233d3ca2dc089cdc11b52a8636984b811986a8223fc81f014253711597b21"}
Sep 09 06:13:37.702942 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:37.702914    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d907ff-75b7557474-tql5j"
Sep 09 06:13:37.874966 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:37.874936    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25168fb6-d620-4016-a464-42e83eb3bb53-plugins-config-volume\") pod \"25168fb6-d620-4016-a464-42e83eb3bb53\" (UID: \"25168fb6-d620-4016-a464-42e83eb3bb53\") "
Sep 09 06:13:37.875144 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:37.875059    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmps8\" (UniqueName: \"kubernetes.io/projected/25168fb6-d620-4016-a464-42e83eb3bb53-kube-api-access-rmps8\") pod \"25168fb6-d620-4016-a464-42e83eb3bb53\" (UID: \"25168fb6-d620-4016-a464-42e83eb3bb53\") "
Sep 09 06:13:37.875296 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:37.875273    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25168fb6-d620-4016-a464-42e83eb3bb53-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "25168fb6-d620-4016-a464-42e83eb3bb53" (UID: "25168fb6-d620-4016-a464-42e83eb3bb53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:13:37.877272 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:37.877245    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25168fb6-d620-4016-a464-42e83eb3bb53-kube-api-access-rmps8" (OuterVolumeSpecName: "kube-api-access-rmps8") pod "25168fb6-d620-4016-a464-42e83eb3bb53" (UID: "25168fb6-d620-4016-a464-42e83eb3bb53"). InnerVolumeSpecName "kube-api-access-rmps8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:13:37.976158 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:37.976058    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmps8\" (UniqueName: \"kubernetes.io/projected/25168fb6-d620-4016-a464-42e83eb3bb53-kube-api-access-rmps8\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:37.976158 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:37.976082    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25168fb6-d620-4016-a464-42e83eb3bb53-plugins-config-volume\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:38.630283 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:38.630245    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d907ff-75b7557474-tql5j" event={"ID":"25168fb6-d620-4016-a464-42e83eb3bb53","Type":"ContainerDied","Data":"d961b9b0615f200c6152571e26f4bdfb4085d277b96be8e11a4d4a5476f5379a"}
Sep 09 06:13:38.630283 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:38.630271    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d907ff-75b7557474-tql5j"
Sep 09 06:13:38.630828 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:38.630340    2579 scope.go:117] "RemoveContainer" containerID="178233d3ca2dc089cdc11b52a8636984b811986a8223fc81f014253711597b21"
Sep 09 06:13:38.654136 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:38.654113    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d907ff-75b7557474-tql5j"]
Sep 09 06:13:38.661100 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:38.661075    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d907ff-75b7557474-tql5j"]
Sep 09 06:13:39.148469 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:39.148440    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25168fb6-d620-4016-a464-42e83eb3bb53" path="/var/lib/kubelet/pods/25168fb6-d620-4016-a464-42e83eb3bb53/volumes"
Sep 09 06:13:39.148782 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:39.148768    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf59ce57-8063-4a0b-97bf-3170a2d5483e" path="/var/lib/kubelet/pods/bf59ce57-8063-4a0b-97bf-3170a2d5483e/volumes"
Sep 09 06:13:39.684992 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:39.684950    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-43eb2d/distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7" podUID="fa204694-c894-4f13-b422-f6726b3db3c8" containerName="main" containerID="cri-o://e36a83e9a47ea4d6982f95c45cf90ee124e71ffbcab12ef6fdabbfb54d07e79f" gracePeriod=45
Sep 09 06:13:39.690405 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:39.690373    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-43eb2d/distinct-w4-43eb2d-kserve-596559c7f-8x4kz" podUID="6c4be116-1d2a-4d8a-8bf2-9aa980a954c4" containerName="main" containerID="cri-o://3598743b6532f9d688daf0c608edb923edf4769496d12d7dfab4f0739ffab48a" gracePeriod=45
Sep 09 06:13:39.691986 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:39.691962    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:13:39.699208 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:39.699084    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-43eb2d/premium-w4-43eb2d-kserve-84b57f44-tlrbl" podUID="de67233d-58eb-43dc-8473-f5dd1ef69b05" containerName="main" containerID="cri-o://0f399fc050cc3a452af9697155c6d859046d4f4e8b3b805d6632bd6df1237614" gracePeriod=45
Sep 09 06:13:39.706115 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:39.706081    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-43eb2d/simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk" podUID="82a478a9-69bb-4e81-b5bd-e30a56f2733d" containerName="main" containerID="cri-o://d87861771eb71c73f10abadc4d2ea767eec7506378f0d4fad15c14c911edb235" gracePeriod=45
Sep 09 06:13:39.848116 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:39.847769    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43eb2d/premium-w4-43eb2d-kserve-84b57f44-tlrbl" podUID="de67233d-58eb-43dc-8473-f5dd1ef69b05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 09 06:13:39.947542 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:39.947507    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-43eb2d/premium-w4-43eb2d-kserve-84b57f44-tlrbl"
Sep 09 06:13:40.023951 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.023928    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-43eb2d/distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7"
Sep 09 06:13:40.027999 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.027978    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-43eb2d/simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk"
Sep 09 06:13:40.031351 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.031332    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-43eb2d/distinct-w4-43eb2d-kserve-596559c7f-8x4kz"
Sep 09 06:13:40.099433 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.099397    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de67233d-58eb-43dc-8473-f5dd1ef69b05-tmp-dir\") pod \"de67233d-58eb-43dc-8473-f5dd1ef69b05\" (UID: \"de67233d-58eb-43dc-8473-f5dd1ef69b05\") "
Sep 09 06:13:40.099433 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.099433    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de67233d-58eb-43dc-8473-f5dd1ef69b05-dshm\") pod \"de67233d-58eb-43dc-8473-f5dd1ef69b05\" (UID: \"de67233d-58eb-43dc-8473-f5dd1ef69b05\") "
Sep 09 06:13:40.099652 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.099506    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de67233d-58eb-43dc-8473-f5dd1ef69b05-home\") pod \"de67233d-58eb-43dc-8473-f5dd1ef69b05\" (UID: \"de67233d-58eb-43dc-8473-f5dd1ef69b05\") "
Sep 09 06:13:40.099652 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.099532    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de67233d-58eb-43dc-8473-f5dd1ef69b05-tls-certs\") pod \"de67233d-58eb-43dc-8473-f5dd1ef69b05\" (UID: \"de67233d-58eb-43dc-8473-f5dd1ef69b05\") "
Sep 09 06:13:40.099652 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.099609    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5kkk\" (UniqueName: \"kubernetes.io/projected/de67233d-58eb-43dc-8473-f5dd1ef69b05-kube-api-access-v5kkk\") pod \"de67233d-58eb-43dc-8473-f5dd1ef69b05\" (UID: \"de67233d-58eb-43dc-8473-f5dd1ef69b05\") "
Sep 09 06:13:40.099813 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.099654    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de67233d-58eb-43dc-8473-f5dd1ef69b05-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "de67233d-58eb-43dc-8473-f5dd1ef69b05" (UID: "de67233d-58eb-43dc-8473-f5dd1ef69b05"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:13:40.099813 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.099663    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de67233d-58eb-43dc-8473-f5dd1ef69b05-model-cache\") pod \"de67233d-58eb-43dc-8473-f5dd1ef69b05\" (UID: \"de67233d-58eb-43dc-8473-f5dd1ef69b05\") "
Sep 09 06:13:40.099813 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.099801    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de67233d-58eb-43dc-8473-f5dd1ef69b05-home" (OuterVolumeSpecName: "home") pod "de67233d-58eb-43dc-8473-f5dd1ef69b05" (UID: "de67233d-58eb-43dc-8473-f5dd1ef69b05"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:13:40.099974 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.099862    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de67233d-58eb-43dc-8473-f5dd1ef69b05-model-cache" (OuterVolumeSpecName: "model-cache") pod "de67233d-58eb-43dc-8473-f5dd1ef69b05" (UID: "de67233d-58eb-43dc-8473-f5dd1ef69b05"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:13:40.100147 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.100127    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de67233d-58eb-43dc-8473-f5dd1ef69b05-tmp-dir\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:40.100231 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.100152    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de67233d-58eb-43dc-8473-f5dd1ef69b05-home\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:40.100231 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.100164    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de67233d-58eb-43dc-8473-f5dd1ef69b05-model-cache\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:40.101783 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.101760    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de67233d-58eb-43dc-8473-f5dd1ef69b05-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "de67233d-58eb-43dc-8473-f5dd1ef69b05" (UID: "de67233d-58eb-43dc-8473-f5dd1ef69b05"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:13:40.101879 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.101810    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de67233d-58eb-43dc-8473-f5dd1ef69b05-kube-api-access-v5kkk" (OuterVolumeSpecName: "kube-api-access-v5kkk") pod "de67233d-58eb-43dc-8473-f5dd1ef69b05" (UID: "de67233d-58eb-43dc-8473-f5dd1ef69b05"). InnerVolumeSpecName "kube-api-access-v5kkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:13:40.102021 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.102008    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de67233d-58eb-43dc-8473-f5dd1ef69b05-dshm" (OuterVolumeSpecName: "dshm") pod "de67233d-58eb-43dc-8473-f5dd1ef69b05" (UID: "de67233d-58eb-43dc-8473-f5dd1ef69b05"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:13:40.200903 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.200828    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c4be116-1d2a-4d8a-8bf2-9aa980a954c4-model-cache\") pod \"6c4be116-1d2a-4d8a-8bf2-9aa980a954c4\" (UID: \"6c4be116-1d2a-4d8a-8bf2-9aa980a954c4\") "
Sep 09 06:13:40.200903 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.200871    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6bzf\" (UniqueName: \"kubernetes.io/projected/6c4be116-1d2a-4d8a-8bf2-9aa980a954c4-kube-api-access-w6bzf\") pod \"6c4be116-1d2a-4d8a-8bf2-9aa980a954c4\" (UID: \"6c4be116-1d2a-4d8a-8bf2-9aa980a954c4\") "
Sep 09 06:13:40.200903 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.200901    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82a478a9-69bb-4e81-b5bd-e30a56f2733d-tmp-dir\") pod \"82a478a9-69bb-4e81-b5bd-e30a56f2733d\" (UID: \"82a478a9-69bb-4e81-b5bd-e30a56f2733d\") "
Sep 09 06:13:40.201152 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.200924    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c4be116-1d2a-4d8a-8bf2-9aa980a954c4-tls-certs\") pod \"6c4be116-1d2a-4d8a-8bf2-9aa980a954c4\" (UID: \"6c4be116-1d2a-4d8a-8bf2-9aa980a954c4\") "
Sep 09 06:13:40.201152 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.200949    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c4be116-1d2a-4d8a-8bf2-9aa980a954c4-dshm\") pod \"6c4be116-1d2a-4d8a-8bf2-9aa980a954c4\" (UID: \"6c4be116-1d2a-4d8a-8bf2-9aa980a954c4\") "
Sep 09 06:13:40.201152 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.200973    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82a478a9-69bb-4e81-b5bd-e30a56f2733d-home\") pod \"82a478a9-69bb-4e81-b5bd-e30a56f2733d\" (UID: \"82a478a9-69bb-4e81-b5bd-e30a56f2733d\") "
Sep 09 06:13:40.201152 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.200996    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa204694-c894-4f13-b422-f6726b3db3c8-model-cache\") pod \"fa204694-c894-4f13-b422-f6726b3db3c8\" (UID: \"fa204694-c894-4f13-b422-f6726b3db3c8\") "
Sep 09 06:13:40.201152 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.201025    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82a478a9-69bb-4e81-b5bd-e30a56f2733d-model-cache\") pod \"82a478a9-69bb-4e81-b5bd-e30a56f2733d\" (UID: \"82a478a9-69bb-4e81-b5bd-e30a56f2733d\") "
Sep 09 06:13:40.201152 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.201080    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa204694-c894-4f13-b422-f6726b3db3c8-tmp-dir\") pod \"fa204694-c894-4f13-b422-f6726b3db3c8\" (UID: \"fa204694-c894-4f13-b422-f6726b3db3c8\") "
Sep 09 06:13:40.201152 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.201110    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82a478a9-69bb-4e81-b5bd-e30a56f2733d-tls-certs\") pod \"82a478a9-69bb-4e81-b5bd-e30a56f2733d\" (UID: \"82a478a9-69bb-4e81-b5bd-e30a56f2733d\") "
Sep 09 06:13:40.201152 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.201137    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa204694-c894-4f13-b422-f6726b3db3c8-home\") pod \"fa204694-c894-4f13-b422-f6726b3db3c8\" (UID: \"fa204694-c894-4f13-b422-f6726b3db3c8\") "
Sep 09 06:13:40.201547 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.201169    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzjpc\" (UniqueName: \"kubernetes.io/projected/fa204694-c894-4f13-b422-f6726b3db3c8-kube-api-access-mzjpc\") pod \"fa204694-c894-4f13-b422-f6726b3db3c8\" (UID: \"fa204694-c894-4f13-b422-f6726b3db3c8\") "
Sep 09 06:13:40.201547 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.201216    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa204694-c894-4f13-b422-f6726b3db3c8-tls-certs\") pod \"fa204694-c894-4f13-b422-f6726b3db3c8\" (UID: \"fa204694-c894-4f13-b422-f6726b3db3c8\") "
Sep 09 06:13:40.201547 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.201215    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a478a9-69bb-4e81-b5bd-e30a56f2733d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "82a478a9-69bb-4e81-b5bd-e30a56f2733d" (UID: "82a478a9-69bb-4e81-b5bd-e30a56f2733d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:13:40.201547 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.201249    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82a478a9-69bb-4e81-b5bd-e30a56f2733d-dshm\") pod \"82a478a9-69bb-4e81-b5bd-e30a56f2733d\" (UID: \"82a478a9-69bb-4e81-b5bd-e30a56f2733d\") "
Sep 09 06:13:40.201547 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.201294    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9nkj\" (UniqueName: \"kubernetes.io/projected/82a478a9-69bb-4e81-b5bd-e30a56f2733d-kube-api-access-v9nkj\") pod \"82a478a9-69bb-4e81-b5bd-e30a56f2733d\" (UID: \"82a478a9-69bb-4e81-b5bd-e30a56f2733d\") "
Sep 09 06:13:40.201547 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.201335    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c4be116-1d2a-4d8a-8bf2-9aa980a954c4-home\") pod \"6c4be116-1d2a-4d8a-8bf2-9aa980a954c4\" (UID: \"6c4be116-1d2a-4d8a-8bf2-9aa980a954c4\") "
Sep 09 06:13:40.201547 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.201384    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa204694-c894-4f13-b422-f6726b3db3c8-dshm\") pod \"fa204694-c894-4f13-b422-f6726b3db3c8\" (UID: \"fa204694-c894-4f13-b422-f6726b3db3c8\") "
Sep 09 06:13:40.201547 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.201412    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c4be116-1d2a-4d8a-8bf2-9aa980a954c4-tmp-dir\") pod \"6c4be116-1d2a-4d8a-8bf2-9aa980a954c4\" (UID: \"6c4be116-1d2a-4d8a-8bf2-9aa980a954c4\") "
Sep 09 06:13:40.201963 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.201226    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a478a9-69bb-4e81-b5bd-e30a56f2733d-home" (OuterVolumeSpecName: "home") pod "82a478a9-69bb-4e81-b5bd-e30a56f2733d" (UID: "82a478a9-69bb-4e81-b5bd-e30a56f2733d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:13:40.201963 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.201273    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa204694-c894-4f13-b422-f6726b3db3c8-model-cache" (OuterVolumeSpecName: "model-cache") pod "fa204694-c894-4f13-b422-f6726b3db3c8" (UID: "fa204694-c894-4f13-b422-f6726b3db3c8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:13:40.201963 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.201336    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa204694-c894-4f13-b422-f6726b3db3c8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fa204694-c894-4f13-b422-f6726b3db3c8" (UID: "fa204694-c894-4f13-b422-f6726b3db3c8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:13:40.201963 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.201385    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa204694-c894-4f13-b422-f6726b3db3c8-home" (OuterVolumeSpecName: "home") pod "fa204694-c894-4f13-b422-f6726b3db3c8" (UID: "fa204694-c894-4f13-b422-f6726b3db3c8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:13:40.201963 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.201501    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a478a9-69bb-4e81-b5bd-e30a56f2733d-model-cache" (OuterVolumeSpecName: "model-cache") pod "82a478a9-69bb-4e81-b5bd-e30a56f2733d" (UID: "82a478a9-69bb-4e81-b5bd-e30a56f2733d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:13:40.201963 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.201605    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c4be116-1d2a-4d8a-8bf2-9aa980a954c4-model-cache" (OuterVolumeSpecName: "model-cache") pod "6c4be116-1d2a-4d8a-8bf2-9aa980a954c4" (UID: "6c4be116-1d2a-4d8a-8bf2-9aa980a954c4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:13:40.201963 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.201891    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c4be116-1d2a-4d8a-8bf2-9aa980a954c4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6c4be116-1d2a-4d8a-8bf2-9aa980a954c4" (UID: "6c4be116-1d2a-4d8a-8bf2-9aa980a954c4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:13:40.201963 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.201908    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c4be116-1d2a-4d8a-8bf2-9aa980a954c4-home" (OuterVolumeSpecName: "home") pod "6c4be116-1d2a-4d8a-8bf2-9aa980a954c4" (UID: "6c4be116-1d2a-4d8a-8bf2-9aa980a954c4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:13:40.202433 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.201967    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de67233d-58eb-43dc-8473-f5dd1ef69b05-tls-certs\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:40.202433 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.201988    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c4be116-1d2a-4d8a-8bf2-9aa980a954c4-model-cache\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:40.202433 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.202001    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82a478a9-69bb-4e81-b5bd-e30a56f2733d-tmp-dir\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:40.202433 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.202013    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82a478a9-69bb-4e81-b5bd-e30a56f2733d-home\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:40.202433 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.202034    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa204694-c894-4f13-b422-f6726b3db3c8-model-cache\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:40.202433 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.202046    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82a478a9-69bb-4e81-b5bd-e30a56f2733d-model-cache\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:40.202433 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.202059    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5kkk\" (UniqueName: \"kubernetes.io/projected/de67233d-58eb-43dc-8473-f5dd1ef69b05-kube-api-access-v5kkk\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:40.202433 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.202073    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa204694-c894-4f13-b422-f6726b3db3c8-tmp-dir\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:40.202433 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.202111    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa204694-c894-4f13-b422-f6726b3db3c8-home\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:40.202433 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.202123    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de67233d-58eb-43dc-8473-f5dd1ef69b05-dshm\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:40.203773 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.203731    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c4be116-1d2a-4d8a-8bf2-9aa980a954c4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6c4be116-1d2a-4d8a-8bf2-9aa980a954c4" (UID: "6c4be116-1d2a-4d8a-8bf2-9aa980a954c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:13:40.204256 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.204219    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c4be116-1d2a-4d8a-8bf2-9aa980a954c4-kube-api-access-w6bzf" (OuterVolumeSpecName: "kube-api-access-w6bzf") pod "6c4be116-1d2a-4d8a-8bf2-9aa980a954c4" (UID: "6c4be116-1d2a-4d8a-8bf2-9aa980a954c4"). InnerVolumeSpecName "kube-api-access-w6bzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:13:40.204566 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.204537    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82a478a9-69bb-4e81-b5bd-e30a56f2733d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "82a478a9-69bb-4e81-b5bd-e30a56f2733d" (UID: "82a478a9-69bb-4e81-b5bd-e30a56f2733d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:13:40.204693 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.204658    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa204694-c894-4f13-b422-f6726b3db3c8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fa204694-c894-4f13-b422-f6726b3db3c8" (UID: "fa204694-c894-4f13-b422-f6726b3db3c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:13:40.204796 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.204770    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa204694-c894-4f13-b422-f6726b3db3c8-kube-api-access-mzjpc" (OuterVolumeSpecName: "kube-api-access-mzjpc") pod "fa204694-c894-4f13-b422-f6726b3db3c8" (UID: "fa204694-c894-4f13-b422-f6726b3db3c8"). InnerVolumeSpecName "kube-api-access-mzjpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:13:40.204915 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.204822    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa204694-c894-4f13-b422-f6726b3db3c8-dshm" (OuterVolumeSpecName: "dshm") pod "fa204694-c894-4f13-b422-f6726b3db3c8" (UID: "fa204694-c894-4f13-b422-f6726b3db3c8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:13:40.205063 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.205042    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a478a9-69bb-4e81-b5bd-e30a56f2733d-dshm" (OuterVolumeSpecName: "dshm") pod "82a478a9-69bb-4e81-b5bd-e30a56f2733d" (UID: "82a478a9-69bb-4e81-b5bd-e30a56f2733d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:13:40.205147 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.205087    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c4be116-1d2a-4d8a-8bf2-9aa980a954c4-dshm" (OuterVolumeSpecName: "dshm") pod "6c4be116-1d2a-4d8a-8bf2-9aa980a954c4" (UID: "6c4be116-1d2a-4d8a-8bf2-9aa980a954c4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:13:40.205302 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.205285    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82a478a9-69bb-4e81-b5bd-e30a56f2733d-kube-api-access-v9nkj" (OuterVolumeSpecName: "kube-api-access-v9nkj") pod "82a478a9-69bb-4e81-b5bd-e30a56f2733d" (UID: "82a478a9-69bb-4e81-b5bd-e30a56f2733d"). InnerVolumeSpecName "kube-api-access-v9nkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:13:40.302445 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.302428    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82a478a9-69bb-4e81-b5bd-e30a56f2733d-tls-certs\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:40.302543 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.302448    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzjpc\" (UniqueName: \"kubernetes.io/projected/fa204694-c894-4f13-b422-f6726b3db3c8-kube-api-access-mzjpc\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:40.302543 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.302457    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa204694-c894-4f13-b422-f6726b3db3c8-tls-certs\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:40.302543 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.302466    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82a478a9-69bb-4e81-b5bd-e30a56f2733d-dshm\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:40.302543 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.302474    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9nkj\" (UniqueName: \"kubernetes.io/projected/82a478a9-69bb-4e81-b5bd-e30a56f2733d-kube-api-access-v9nkj\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:40.302543 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.302482    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c4be116-1d2a-4d8a-8bf2-9aa980a954c4-home\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:40.302543 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.302490    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa204694-c894-4f13-b422-f6726b3db3c8-dshm\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:40.302543 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.302497    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c4be116-1d2a-4d8a-8bf2-9aa980a954c4-tmp-dir\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:40.302543 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.302505    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6bzf\" (UniqueName: \"kubernetes.io/projected/6c4be116-1d2a-4d8a-8bf2-9aa980a954c4-kube-api-access-w6bzf\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:40.302543 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.302512    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c4be116-1d2a-4d8a-8bf2-9aa980a954c4-tls-certs\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:40.302543 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.302519    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c4be116-1d2a-4d8a-8bf2-9aa980a954c4-dshm\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:40.645803 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.645772    2579 generic.go:356] "Generic (PLEG): container finished" podID="de67233d-58eb-43dc-8473-f5dd1ef69b05" containerID="0f399fc050cc3a452af9697155c6d859046d4f4e8b3b805d6632bd6df1237614" exitCode=0
Sep 09 06:13:40.645937 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.645821    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-43eb2d/premium-w4-43eb2d-kserve-84b57f44-tlrbl"
Sep 09 06:13:40.645937 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.645864    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-43eb2d/premium-w4-43eb2d-kserve-84b57f44-tlrbl" event={"ID":"de67233d-58eb-43dc-8473-f5dd1ef69b05","Type":"ContainerDied","Data":"0f399fc050cc3a452af9697155c6d859046d4f4e8b3b805d6632bd6df1237614"}
Sep 09 06:13:40.645937 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.645908    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-43eb2d/premium-w4-43eb2d-kserve-84b57f44-tlrbl" event={"ID":"de67233d-58eb-43dc-8473-f5dd1ef69b05","Type":"ContainerDied","Data":"a214f41992a48c06cee61b21851fb31448c0e515d080710773a6b12a3ff946bd"}
Sep 09 06:13:40.645937 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.645927    2579 scope.go:117] "RemoveContainer" containerID="0f399fc050cc3a452af9697155c6d859046d4f4e8b3b805d6632bd6df1237614"
Sep 09 06:13:40.647361 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.647338    2579 generic.go:356] "Generic (PLEG): container finished" podID="6c4be116-1d2a-4d8a-8bf2-9aa980a954c4" containerID="3598743b6532f9d688daf0c608edb923edf4769496d12d7dfab4f0739ffab48a" exitCode=0
Sep 09 06:13:40.647458 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.647364    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-43eb2d/distinct-w4-43eb2d-kserve-596559c7f-8x4kz" event={"ID":"6c4be116-1d2a-4d8a-8bf2-9aa980a954c4","Type":"ContainerDied","Data":"3598743b6532f9d688daf0c608edb923edf4769496d12d7dfab4f0739ffab48a"}
Sep 09 06:13:40.647458 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.647397    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-43eb2d/distinct-w4-43eb2d-kserve-596559c7f-8x4kz"
Sep 09 06:13:40.647458 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.647417    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-43eb2d/distinct-w4-43eb2d-kserve-596559c7f-8x4kz" event={"ID":"6c4be116-1d2a-4d8a-8bf2-9aa980a954c4","Type":"ContainerDied","Data":"041bc02e4e0f69ac2e98a3560b13ae4fbf57ce525d55d12b5ff4ecdde1421c50"}
Sep 09 06:13:40.648925 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.648901    2579 generic.go:356] "Generic (PLEG): container finished" podID="fa204694-c894-4f13-b422-f6726b3db3c8" containerID="e36a83e9a47ea4d6982f95c45cf90ee124e71ffbcab12ef6fdabbfb54d07e79f" exitCode=0
Sep 09 06:13:40.649043 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.648942    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-43eb2d/distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7"
Sep 09 06:13:40.649043 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.648988    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-43eb2d/distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7" event={"ID":"fa204694-c894-4f13-b422-f6726b3db3c8","Type":"ContainerDied","Data":"e36a83e9a47ea4d6982f95c45cf90ee124e71ffbcab12ef6fdabbfb54d07e79f"}
Sep 09 06:13:40.649043 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.649022    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-43eb2d/distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7" event={"ID":"fa204694-c894-4f13-b422-f6726b3db3c8","Type":"ContainerDied","Data":"c9d56e132c06ce2e928e0664bfe186193bb4d0064f361cb30eca9032763c489d"}
Sep 09 06:13:40.650431 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.650410    2579 generic.go:356] "Generic (PLEG): container finished" podID="82a478a9-69bb-4e81-b5bd-e30a56f2733d" containerID="d87861771eb71c73f10abadc4d2ea767eec7506378f0d4fad15c14c911edb235" exitCode=0
Sep 09 06:13:40.650533 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.650443    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-43eb2d/simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk" event={"ID":"82a478a9-69bb-4e81-b5bd-e30a56f2733d","Type":"ContainerDied","Data":"d87861771eb71c73f10abadc4d2ea767eec7506378f0d4fad15c14c911edb235"}
Sep 09 06:13:40.650533 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.650464    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-43eb2d/simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk" event={"ID":"82a478a9-69bb-4e81-b5bd-e30a56f2733d","Type":"ContainerDied","Data":"51bbc891f96d89ebb40ae07d5d61f0e4cbfdb97c97d7e8b04ea6f73dc101cf74"}
Sep 09 06:13:40.650533 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.650466    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-43eb2d/simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk"
Sep 09 06:13:40.655405 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.655387    2579 scope.go:117] "RemoveContainer" containerID="0f399fc050cc3a452af9697155c6d859046d4f4e8b3b805d6632bd6df1237614"
Sep 09 06:13:40.655664 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:13:40.655641    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f399fc050cc3a452af9697155c6d859046d4f4e8b3b805d6632bd6df1237614\": container with ID starting with 0f399fc050cc3a452af9697155c6d859046d4f4e8b3b805d6632bd6df1237614 not found: ID does not exist" containerID="0f399fc050cc3a452af9697155c6d859046d4f4e8b3b805d6632bd6df1237614"
Sep 09 06:13:40.655751 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.655676    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f399fc050cc3a452af9697155c6d859046d4f4e8b3b805d6632bd6df1237614"} err="failed to get container status \"0f399fc050cc3a452af9697155c6d859046d4f4e8b3b805d6632bd6df1237614\": rpc error: code = NotFound desc = could not find container \"0f399fc050cc3a452af9697155c6d859046d4f4e8b3b805d6632bd6df1237614\": container with ID starting with 0f399fc050cc3a452af9697155c6d859046d4f4e8b3b805d6632bd6df1237614 not found: ID does not exist"
Sep 09 06:13:40.655751 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.655699    2579 scope.go:117] "RemoveContainer" containerID="3598743b6532f9d688daf0c608edb923edf4769496d12d7dfab4f0739ffab48a"
Sep 09 06:13:40.670106 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.669994    2579 scope.go:117] "RemoveContainer" containerID="3598743b6532f9d688daf0c608edb923edf4769496d12d7dfab4f0739ffab48a"
Sep 09 06:13:40.670332 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:13:40.670304    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3598743b6532f9d688daf0c608edb923edf4769496d12d7dfab4f0739ffab48a\": container with ID starting with 3598743b6532f9d688daf0c608edb923edf4769496d12d7dfab4f0739ffab48a not found: ID does not exist" containerID="3598743b6532f9d688daf0c608edb923edf4769496d12d7dfab4f0739ffab48a"
Sep 09 06:13:40.670405 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.670341    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3598743b6532f9d688daf0c608edb923edf4769496d12d7dfab4f0739ffab48a"} err="failed to get container status \"3598743b6532f9d688daf0c608edb923edf4769496d12d7dfab4f0739ffab48a\": rpc error: code = NotFound desc = could not find container \"3598743b6532f9d688daf0c608edb923edf4769496d12d7dfab4f0739ffab48a\": container with ID starting with 3598743b6532f9d688daf0c608edb923edf4769496d12d7dfab4f0739ffab48a not found: ID does not exist"
Sep 09 06:13:40.670405 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.670363    2579 scope.go:117] "RemoveContainer" containerID="e36a83e9a47ea4d6982f95c45cf90ee124e71ffbcab12ef6fdabbfb54d07e79f"
Sep 09 06:13:40.677898 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.677879    2579 scope.go:117] "RemoveContainer" containerID="e36a83e9a47ea4d6982f95c45cf90ee124e71ffbcab12ef6fdabbfb54d07e79f"
Sep 09 06:13:40.678130 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:13:40.678114    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e36a83e9a47ea4d6982f95c45cf90ee124e71ffbcab12ef6fdabbfb54d07e79f\": container with ID starting with e36a83e9a47ea4d6982f95c45cf90ee124e71ffbcab12ef6fdabbfb54d07e79f not found: ID does not exist" containerID="e36a83e9a47ea4d6982f95c45cf90ee124e71ffbcab12ef6fdabbfb54d07e79f"
Sep 09 06:13:40.678168 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.678137    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e36a83e9a47ea4d6982f95c45cf90ee124e71ffbcab12ef6fdabbfb54d07e79f"} err="failed to get container status \"e36a83e9a47ea4d6982f95c45cf90ee124e71ffbcab12ef6fdabbfb54d07e79f\": rpc error: code = NotFound desc = could not find container \"e36a83e9a47ea4d6982f95c45cf90ee124e71ffbcab12ef6fdabbfb54d07e79f\": container with ID starting with e36a83e9a47ea4d6982f95c45cf90ee124e71ffbcab12ef6fdabbfb54d07e79f not found: ID does not exist"
Sep 09 06:13:40.678168 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.678151    2579 scope.go:117] "RemoveContainer" containerID="d87861771eb71c73f10abadc4d2ea767eec7506378f0d4fad15c14c911edb235"
Sep 09 06:13:40.683910 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.683892    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-43eb2d/premium-w4-43eb2d-kserve-84b57f44-tlrbl"]
Sep 09 06:13:40.686877 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.686571    2579 scope.go:117] "RemoveContainer" containerID="d87861771eb71c73f10abadc4d2ea767eec7506378f0d4fad15c14c911edb235"
Sep 09 06:13:40.686877 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:13:40.686851    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d87861771eb71c73f10abadc4d2ea767eec7506378f0d4fad15c14c911edb235\": container with ID starting with d87861771eb71c73f10abadc4d2ea767eec7506378f0d4fad15c14c911edb235 not found: ID does not exist" containerID="d87861771eb71c73f10abadc4d2ea767eec7506378f0d4fad15c14c911edb235"
Sep 09 06:13:40.687139 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.686873    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d87861771eb71c73f10abadc4d2ea767eec7506378f0d4fad15c14c911edb235"} err="failed to get container status \"d87861771eb71c73f10abadc4d2ea767eec7506378f0d4fad15c14c911edb235\": rpc error: code = NotFound desc = could not find container \"d87861771eb71c73f10abadc4d2ea767eec7506378f0d4fad15c14c911edb235\": container with ID starting with d87861771eb71c73f10abadc4d2ea767eec7506378f0d4fad15c14c911edb235 not found: ID does not exist"
Sep 09 06:13:40.687342 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.687325    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-43eb2d/premium-w4-43eb2d-kserve-84b57f44-tlrbl"]
Sep 09 06:13:40.696788 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.696765    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-43eb2d/distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7"]
Sep 09 06:13:40.698438 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.698415    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-43eb2d/distinct-2-w4-43eb2d-kserve-74c5c778cb-76kr7"]
Sep 09 06:13:40.708387 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.708369    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-43eb2d/distinct-w4-43eb2d-kserve-596559c7f-8x4kz"]
Sep 09 06:13:40.713121 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.713102    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-43eb2d/distinct-w4-43eb2d-kserve-596559c7f-8x4kz"]
Sep 09 06:13:40.724102 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.724081    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-43eb2d/simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk"]
Sep 09 06:13:40.728013 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:40.727995    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-43eb2d/simulator-w4-43eb2d-kserve-6b6c6868d5-q42wk"]
Sep 09 06:13:41.148398 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:41.148366    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c4be116-1d2a-4d8a-8bf2-9aa980a954c4" path="/var/lib/kubelet/pods/6c4be116-1d2a-4d8a-8bf2-9aa980a954c4/volumes"
Sep 09 06:13:41.148812 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:41.148798    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82a478a9-69bb-4e81-b5bd-e30a56f2733d" path="/var/lib/kubelet/pods/82a478a9-69bb-4e81-b5bd-e30a56f2733d/volumes"
Sep 09 06:13:41.149140 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:41.149128    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de67233d-58eb-43dc-8473-f5dd1ef69b05" path="/var/lib/kubelet/pods/de67233d-58eb-43dc-8473-f5dd1ef69b05/volumes"
Sep 09 06:13:41.149771 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:41.149757    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa204694-c894-4f13-b422-f6726b3db3c8" path="/var/lib/kubelet/pods/fa204694-c894-4f13-b422-f6726b3db3c8/volumes"
Sep 09 06:13:44.261346 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:44.261317    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"]
Sep 09 06:13:44.261740 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:44.261520    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794" podUID="619ba57a-cb9b-46e8-ae40-6c2b8ebef515" containerName="istio-proxy" containerID="cri-o://1c616c15e9b57191139056cfb5e04a2c1760ffc8b4a30a9fed0cbd4f77717a25" gracePeriod=30
Sep 09 06:13:44.572332 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:44.572300    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f3d736-dr6w7"]
Sep 09 06:13:44.572632 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:44.572608    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec1cd9d9-051b-4641-8b07-7d0b8984dcfb" containerName="istio-proxy"
Sep 09 06:13:44.572682 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:44.572634    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec1cd9d9-051b-4641-8b07-7d0b8984dcfb" containerName="istio-proxy"
Sep 09 06:13:44.572682 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:44.572644    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf59ce57-8063-4a0b-97bf-3170a2d5483e" containerName="revoke-keys"
Sep 09 06:13:44.572682 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:44.572650    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf59ce57-8063-4a0b-97bf-3170a2d5483e" containerName="revoke-keys"
Sep 09 06:13:44.572682 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:44.572661    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec1cd9d9-051b-4641-8b07-7d0b8984dcfb" containerName="istio-proxy"
Sep 09 06:13:44.572682 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:44.572667    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec1cd9d9-051b-4641-8b07-7d0b8984dcfb" containerName="istio-proxy"
Sep 09 06:13:44.572682 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:44.572674    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e73dee7d-2739-445f-bc52-afca338084fd" containerName="revoke-keys"
Sep 09 06:13:44.572682 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:44.572678    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e73dee7d-2739-445f-bc52-afca338084fd" containerName="revoke-keys"
Sep 09 06:13:44.572886 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:44.572691    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82a478a9-69bb-4e81-b5bd-e30a56f2733d" containerName="main"
Sep 09 06:13:44.572886 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:44.572698    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="82a478a9-69bb-4e81-b5bd-e30a56f2733d" containerName="main"
Sep 09 06:13:44.572886 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:44.572706    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de67233d-58eb-43dc-8473-f5dd1ef69b05" containerName="main"
Sep 09 06:13:44.572886 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:44.572710    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="de67233d-58eb-43dc-8473-f5dd1ef69b05" containerName="main"
Sep 09 06:13:44.572886 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:44.572720    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c4be116-1d2a-4d8a-8bf2-9aa980a954c4" containerName="main"
Sep 09 06:13:44.572886 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:44.572725    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c4be116-1d2a-4d8a-8bf2-9aa980a954c4" containerName="main"
Sep 09 06:13:44.572886 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:44.572731    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25168fb6-d620-4016-a464-42e83eb3bb53" containerName="payload-pre-processing"
Sep 09 06:13:44.572886 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:44.572736    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="25168fb6-d620-4016-a464-42e83eb3bb53" containerName="payload-pre-processing"
Sep 09 06:13:44.572886 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:44.572741    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa204694-c894-4f13-b422-f6726b3db3c8" containerName="main"
Sep 09 06:13:44.572886 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:44.572746    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa204694-c894-4f13-b422-f6726b3db3c8" containerName="main"
Sep 09 06:13:44.572886 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:44.572800    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e73dee7d-2739-445f-bc52-afca338084fd" containerName="revoke-keys"
Sep 09 06:13:44.572886 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:44.572808    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec1cd9d9-051b-4641-8b07-7d0b8984dcfb" containerName="istio-proxy"
Sep 09 06:13:44.572886 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:44.572813    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="25168fb6-d620-4016-a464-42e83eb3bb53" containerName="payload-pre-processing"
Sep 09 06:13:44.572886 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:44.572819    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="82a478a9-69bb-4e81-b5bd-e30a56f2733d" containerName="main"
Sep 09 06:13:44.572886 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:44.572824    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf59ce57-8063-4a0b-97bf-3170a2d5483e" containerName="revoke-keys"
Sep 09 06:13:44.572886 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:44.572828    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa204694-c894-4f13-b422-f6726b3db3c8" containerName="main"
Sep 09 06:13:44.572886 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:44.572833    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec1cd9d9-051b-4641-8b07-7d0b8984dcfb" containerName="istio-proxy"
Sep 09 06:13:44.572886 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:44.572838    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="de67233d-58eb-43dc-8473-f5dd1ef69b05" containerName="main"
Sep 09 06:13:44.572886 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:44.572844    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c4be116-1d2a-4d8a-8bf2-9aa980a954c4" containerName="main"
Sep 09 06:13:44.577583 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:44.577566    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f3d736-dr6w7"
Sep 09 06:13:44.579903 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:44.579886    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 06:13:44.579980 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:44.579927    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ksjk6\""
Sep 09 06:13:44.583429 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:44.583406    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f3d736-dr6w7"]
Sep 09 06:13:44.632621 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:44.632592    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c306f740-df93-4c82-a4f2-7ec083bb8cad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f3d736-dr6w7\" (UID: \"c306f740-df93-4c82-a4f2-7ec083bb8cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f3d736-dr6w7"
Sep 09 06:13:44.733092 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:44.733067    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c306f740-df93-4c82-a4f2-7ec083bb8cad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f3d736-dr6w7\" (UID: \"c306f740-df93-4c82-a4f2-7ec083bb8cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f3d736-dr6w7"
Sep 09 06:13:44.733592 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:44.733575    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c306f740-df93-4c82-a4f2-7ec083bb8cad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f3d736-dr6w7\" (UID: \"c306f740-df93-4c82-a4f2-7ec083bb8cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f3d736-dr6w7"
Sep 09 06:13:44.889129 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:44.889059    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f3d736-dr6w7"
Sep 09 06:13:45.158611 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:45.158583    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f3d736-dr6w7"]
Sep 09 06:13:45.160788 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:13:45.160760    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc306f740_df93_4c82_a4f2_7ec083bb8cad.slice/crio-873c4e27dcc2b285df1135dcda5f330b9698889a485f1a3bf8cf03981ba9e9ec WatchSource:0}: Error finding container 873c4e27dcc2b285df1135dcda5f330b9698889a485f1a3bf8cf03981ba9e9ec: Status 404 returned error can't find the container with id 873c4e27dcc2b285df1135dcda5f330b9698889a485f1a3bf8cf03981ba9e9ec
Sep 09 06:13:45.681116 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:45.681086    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f3d736-dr6w7" event={"ID":"c306f740-df93-4c82-a4f2-7ec083bb8cad","Type":"ContainerStarted","Data":"042e6f6724b97df117dabde2bff052fa082ed08c25d09a630ec46446a0631a20"}
Sep 09 06:13:45.681439 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:45.681135    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f3d736-dr6w7" event={"ID":"c306f740-df93-4c82-a4f2-7ec083bb8cad","Type":"ContainerStarted","Data":"873c4e27dcc2b285df1135dcda5f330b9698889a485f1a3bf8cf03981ba9e9ec"}
Sep 09 06:13:45.726552 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:45.726508    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f3d736-dr6w7" podStartSLOduration=1.726493852 podStartE2EDuration="1.726493852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:13:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:13:45.7254059 +0000 UTC m=+1451.065893809" watchObservedRunningTime="2026-09-09 06:13:45.726493852 +0000 UTC m=+1451.066981761"
Sep 09 06:13:46.687757 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:46.687727    2579 generic.go:356] "Generic (PLEG): container finished" podID="c306f740-df93-4c82-a4f2-7ec083bb8cad" containerID="042e6f6724b97df117dabde2bff052fa082ed08c25d09a630ec46446a0631a20" exitCode=0
Sep 09 06:13:46.688107 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:46.687822    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f3d736-dr6w7" event={"ID":"c306f740-df93-4c82-a4f2-7ec083bb8cad","Type":"ContainerDied","Data":"042e6f6724b97df117dabde2bff052fa082ed08c25d09a630ec46446a0631a20"}
Sep 09 06:13:47.824083 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:47.824060    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f3d736-dr6w7"
Sep 09 06:13:47.853449 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:47.853425    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c306f740-df93-4c82-a4f2-7ec083bb8cad-maas-api-service-ca\") pod \"c306f740-df93-4c82-a4f2-7ec083bb8cad\" (UID: \"c306f740-df93-4c82-a4f2-7ec083bb8cad\") "
Sep 09 06:13:47.853798 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:47.853779    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c306f740-df93-4c82-a4f2-7ec083bb8cad-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c306f740-df93-4c82-a4f2-7ec083bb8cad" (UID: "c306f740-df93-4c82-a4f2-7ec083bb8cad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:13:47.955048 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:47.954992    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c306f740-df93-4c82-a4f2-7ec083bb8cad-maas-api-service-ca\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:47.956618 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:47.956588    2579 patch_prober.go:28] interesting pod/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 06:13:47.956717 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:47.956681    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794" podUID="619ba57a-cb9b-46e8-ae40-6c2b8ebef515" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 06:13:48.701990 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:48.701956    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f3d736-dr6w7"
Sep 09 06:13:48.701990 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:48.701975    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f3d736-dr6w7" event={"ID":"c306f740-df93-4c82-a4f2-7ec083bb8cad","Type":"ContainerDied","Data":"873c4e27dcc2b285df1135dcda5f330b9698889a485f1a3bf8cf03981ba9e9ec"}
Sep 09 06:13:48.702181 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:48.702017    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="873c4e27dcc2b285df1135dcda5f330b9698889a485f1a3bf8cf03981ba9e9ec"
Sep 09 06:13:49.518872 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:49.518845    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"
Sep 09 06:13:49.566993 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:49.566965    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xnpl\" (UniqueName: \"kubernetes.io/projected/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-kube-api-access-8xnpl\") pod \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\" (UID: \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\") "
Sep 09 06:13:49.567131 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:49.567049    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-istio-token\") pod \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\" (UID: \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\") "
Sep 09 06:13:49.567131 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:49.567099    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-istio-data\") pod \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\" (UID: \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\") "
Sep 09 06:13:49.567234 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:49.567144    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-workload-socket\") pod \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\" (UID: \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\") "
Sep 09 06:13:49.567234 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:49.567219    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-istio-envoy\") pod \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\" (UID: \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\") "
Sep 09 06:13:49.567333 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:49.567277    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-istiod-ca-cert\") pod \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\" (UID: \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\") "
Sep 09 06:13:49.567389 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:49.567332    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-workload-certs\") pod \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\" (UID: \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\") "
Sep 09 06:13:49.567389 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:49.567362    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-credential-socket\") pod \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\" (UID: \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\") "
Sep 09 06:13:49.567502 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:49.567393    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-istio-podinfo\") pod \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\" (UID: \"619ba57a-cb9b-46e8-ae40-6c2b8ebef515\") "
Sep 09 06:13:49.567502 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:49.567413    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "619ba57a-cb9b-46e8-ae40-6c2b8ebef515" (UID: "619ba57a-cb9b-46e8-ae40-6c2b8ebef515"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:13:49.567607 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:49.567593    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "619ba57a-cb9b-46e8-ae40-6c2b8ebef515" (UID: "619ba57a-cb9b-46e8-ae40-6c2b8ebef515"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:13:49.567721 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:49.567691    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-istio-data" (OuterVolumeSpecName: "istio-data") pod "619ba57a-cb9b-46e8-ae40-6c2b8ebef515" (UID: "619ba57a-cb9b-46e8-ae40-6c2b8ebef515"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:13:49.567883 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:49.567791    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "619ba57a-cb9b-46e8-ae40-6c2b8ebef515" (UID: "619ba57a-cb9b-46e8-ae40-6c2b8ebef515"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:13:49.568156 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:49.567682    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "619ba57a-cb9b-46e8-ae40-6c2b8ebef515" (UID: "619ba57a-cb9b-46e8-ae40-6c2b8ebef515"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:13:49.568257 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:49.567715    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-workload-socket\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:49.568368 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:49.568344    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-workload-certs\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:49.569456 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:49.569424    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-kube-api-access-8xnpl" (OuterVolumeSpecName: "kube-api-access-8xnpl") pod "619ba57a-cb9b-46e8-ae40-6c2b8ebef515" (UID: "619ba57a-cb9b-46e8-ae40-6c2b8ebef515"). InnerVolumeSpecName "kube-api-access-8xnpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:13:49.569600 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:49.569578    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "619ba57a-cb9b-46e8-ae40-6c2b8ebef515" (UID: "619ba57a-cb9b-46e8-ae40-6c2b8ebef515"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:13:49.569698 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:49.569582    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-istio-token" (OuterVolumeSpecName: "istio-token") pod "619ba57a-cb9b-46e8-ae40-6c2b8ebef515" (UID: "619ba57a-cb9b-46e8-ae40-6c2b8ebef515"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:13:49.570144 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:49.570125    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "619ba57a-cb9b-46e8-ae40-6c2b8ebef515" (UID: "619ba57a-cb9b-46e8-ae40-6c2b8ebef515"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 06:13:49.668810 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:49.668783    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-istio-envoy\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:49.668810 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:49.668805    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-istiod-ca-cert\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:49.668955 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:49.668817    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-credential-socket\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:49.668955 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:49.668829    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-istio-podinfo\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:49.668955 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:49.668842    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xnpl\" (UniqueName: \"kubernetes.io/projected/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-kube-api-access-8xnpl\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:49.668955 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:49.668856    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-istio-token\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:49.668955 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:49.668869    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/619ba57a-cb9b-46e8-ae40-6c2b8ebef515-istio-data\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:13:49.708689 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:49.708662    2579 generic.go:356] "Generic (PLEG): container finished" podID="619ba57a-cb9b-46e8-ae40-6c2b8ebef515" containerID="1c616c15e9b57191139056cfb5e04a2c1760ffc8b4a30a9fed0cbd4f77717a25" exitCode=0
Sep 09 06:13:49.708803 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:49.708722    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"
Sep 09 06:13:49.708803 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:49.708756    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794" event={"ID":"619ba57a-cb9b-46e8-ae40-6c2b8ebef515","Type":"ContainerDied","Data":"1c616c15e9b57191139056cfb5e04a2c1760ffc8b4a30a9fed0cbd4f77717a25"}
Sep 09 06:13:49.708918 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:49.708806    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794" event={"ID":"619ba57a-cb9b-46e8-ae40-6c2b8ebef515","Type":"ContainerDied","Data":"533f641b6bb48f906df0267658e2226f3f3ba5dc55376a7dab3276893c35d546"}
Sep 09 06:13:49.708918 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:49.708828    2579 scope.go:117] "RemoveContainer" containerID="1c616c15e9b57191139056cfb5e04a2c1760ffc8b4a30a9fed0cbd4f77717a25"
Sep 09 06:13:49.717769 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:49.717755    2579 scope.go:117] "RemoveContainer" containerID="b5261f8fe692cae5c0f3d41024cca76331ad802ba9ec175b39be270b92e8fa13"
Sep 09 06:13:49.725559 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:49.725541    2579 scope.go:117] "RemoveContainer" containerID="1c616c15e9b57191139056cfb5e04a2c1760ffc8b4a30a9fed0cbd4f77717a25"
Sep 09 06:13:49.725823 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:13:49.725805    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c616c15e9b57191139056cfb5e04a2c1760ffc8b4a30a9fed0cbd4f77717a25\": container with ID starting with 1c616c15e9b57191139056cfb5e04a2c1760ffc8b4a30a9fed0cbd4f77717a25 not found: ID does not exist" containerID="1c616c15e9b57191139056cfb5e04a2c1760ffc8b4a30a9fed0cbd4f77717a25"
Sep 09 06:13:49.725871 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:49.725831    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c616c15e9b57191139056cfb5e04a2c1760ffc8b4a30a9fed0cbd4f77717a25"} err="failed to get container status \"1c616c15e9b57191139056cfb5e04a2c1760ffc8b4a30a9fed0cbd4f77717a25\": rpc error: code = NotFound desc = could not find container \"1c616c15e9b57191139056cfb5e04a2c1760ffc8b4a30a9fed0cbd4f77717a25\": container with ID starting with 1c616c15e9b57191139056cfb5e04a2c1760ffc8b4a30a9fed0cbd4f77717a25 not found: ID does not exist"
Sep 09 06:13:49.725871 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:49.725847    2579 scope.go:117] "RemoveContainer" containerID="b5261f8fe692cae5c0f3d41024cca76331ad802ba9ec175b39be270b92e8fa13"
Sep 09 06:13:49.726110 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:13:49.726091    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5261f8fe692cae5c0f3d41024cca76331ad802ba9ec175b39be270b92e8fa13\": container with ID starting with b5261f8fe692cae5c0f3d41024cca76331ad802ba9ec175b39be270b92e8fa13 not found: ID does not exist" containerID="b5261f8fe692cae5c0f3d41024cca76331ad802ba9ec175b39be270b92e8fa13"
Sep 09 06:13:49.726173 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:49.726118    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5261f8fe692cae5c0f3d41024cca76331ad802ba9ec175b39be270b92e8fa13"} err="failed to get container status \"b5261f8fe692cae5c0f3d41024cca76331ad802ba9ec175b39be270b92e8fa13\": rpc error: code = NotFound desc = could not find container \"b5261f8fe692cae5c0f3d41024cca76331ad802ba9ec175b39be270b92e8fa13\": container with ID starting with b5261f8fe692cae5c0f3d41024cca76331ad802ba9ec175b39be270b92e8fa13 not found: ID does not exist"
Sep 09 06:13:49.730048 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:49.730025    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"]
Sep 09 06:13:49.733565 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:49.733547    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d907ff-openshift-default-5b66f7b58-nm794"]
Sep 09 06:13:51.147815 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:51.147784    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="619ba57a-cb9b-46e8-ae40-6c2b8ebef515" path="/var/lib/kubelet/pods/619ba57a-cb9b-46e8-ae40-6c2b8ebef515/volumes"
Sep 09 06:13:55.299282 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:55.299250    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f3d736-dr6w7"]
Sep 09 06:13:55.302517 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:55.302495    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f3d736-dr6w7"]
Sep 09 06:13:57.147509 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:13:57.147474    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c306f740-df93-4c82-a4f2-7ec083bb8cad" path="/var/lib/kubelet/pods/c306f740-df93-4c82-a4f2-7ec083bb8cad/volumes"
Sep 09 06:14:02.212490 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:02.212453    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:14:03.587272 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:03.587236    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-750c68-cpv9m"]
Sep 09 06:14:03.588490 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:03.588464    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="619ba57a-cb9b-46e8-ae40-6c2b8ebef515" containerName="istio-proxy"
Sep 09 06:14:03.588662 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:03.588649    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="619ba57a-cb9b-46e8-ae40-6c2b8ebef515" containerName="istio-proxy"
Sep 09 06:14:03.588774 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:03.588763    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="619ba57a-cb9b-46e8-ae40-6c2b8ebef515" containerName="istio-proxy"
Sep 09 06:14:03.588856 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:03.588848    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="619ba57a-cb9b-46e8-ae40-6c2b8ebef515" containerName="istio-proxy"
Sep 09 06:14:03.588969 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:03.588959    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c306f740-df93-4c82-a4f2-7ec083bb8cad" containerName="revoke-keys"
Sep 09 06:14:03.589038 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:03.589030    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c306f740-df93-4c82-a4f2-7ec083bb8cad" containerName="revoke-keys"
Sep 09 06:14:03.589262 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:03.589250    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="619ba57a-cb9b-46e8-ae40-6c2b8ebef515" containerName="istio-proxy"
Sep 09 06:14:03.589348 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:03.589338    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c306f740-df93-4c82-a4f2-7ec083bb8cad" containerName="revoke-keys"
Sep 09 06:14:03.594578 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:03.594546    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-750c68-cpv9m"
Sep 09 06:14:03.597036 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:03.596997    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 06:14:03.597142 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:03.597054    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-750c68-cpv9m"]
Sep 09 06:14:03.597142 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:03.597001    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ksjk6\""
Sep 09 06:14:03.679916 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:03.679889    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc456f65-a237-4934-9ba9-088bdb38f0b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-750c68-cpv9m\" (UID: \"dc456f65-a237-4934-9ba9-088bdb38f0b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-750c68-cpv9m"
Sep 09 06:14:03.780705 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:03.780666    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc456f65-a237-4934-9ba9-088bdb38f0b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-750c68-cpv9m\" (UID: \"dc456f65-a237-4934-9ba9-088bdb38f0b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-750c68-cpv9m"
Sep 09 06:14:03.781246 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:03.781228    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc456f65-a237-4934-9ba9-088bdb38f0b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-750c68-cpv9m\" (UID: \"dc456f65-a237-4934-9ba9-088bdb38f0b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-750c68-cpv9m"
Sep 09 06:14:03.906656 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:03.906573    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-750c68-cpv9m"
Sep 09 06:14:04.177815 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:04.177732    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-750c68-cpv9m"]
Sep 09 06:14:04.181285 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:14:04.181257    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc456f65_a237_4934_9ba9_088bdb38f0b4.slice/crio-ad000e300333ef8efbbbe2119d801d7d2a64c9ed32b48e06753ccaff34e479e5 WatchSource:0}: Error finding container ad000e300333ef8efbbbe2119d801d7d2a64c9ed32b48e06753ccaff34e479e5: Status 404 returned error can't find the container with id ad000e300333ef8efbbbe2119d801d7d2a64c9ed32b48e06753ccaff34e479e5
Sep 09 06:14:04.782548 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:04.782470    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-750c68-cpv9m" event={"ID":"dc456f65-a237-4934-9ba9-088bdb38f0b4","Type":"ContainerStarted","Data":"64edcf84c041b79d00b7ebfdb44f1a37d8dcecdd7b7d7ea93da343137512516b"}
Sep 09 06:14:04.782548 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:04.782518    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-750c68-cpv9m" event={"ID":"dc456f65-a237-4934-9ba9-088bdb38f0b4","Type":"ContainerStarted","Data":"ad000e300333ef8efbbbe2119d801d7d2a64c9ed32b48e06753ccaff34e479e5"}
Sep 09 06:14:04.798589 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:04.798542    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-750c68-cpv9m" podStartSLOduration=1.798523895 podStartE2EDuration="1.798523895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:14:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:14:04.796237403 +0000 UTC m=+1470.136725311" watchObservedRunningTime="2026-09-09 06:14:04.798523895 +0000 UTC m=+1470.139011854"
Sep 09 06:14:05.788983 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:05.788946    2579 generic.go:356] "Generic (PLEG): container finished" podID="dc456f65-a237-4934-9ba9-088bdb38f0b4" containerID="64edcf84c041b79d00b7ebfdb44f1a37d8dcecdd7b7d7ea93da343137512516b" exitCode=0
Sep 09 06:14:05.789348 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:05.789036    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-750c68-cpv9m" event={"ID":"dc456f65-a237-4934-9ba9-088bdb38f0b4","Type":"ContainerDied","Data":"64edcf84c041b79d00b7ebfdb44f1a37d8dcecdd7b7d7ea93da343137512516b"}
Sep 09 06:14:06.917387 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:06.917365    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-750c68-cpv9m"
Sep 09 06:14:07.005471 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:07.005443    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc456f65-a237-4934-9ba9-088bdb38f0b4-maas-api-service-ca\") pod \"dc456f65-a237-4934-9ba9-088bdb38f0b4\" (UID: \"dc456f65-a237-4934-9ba9-088bdb38f0b4\") "
Sep 09 06:14:07.005844 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:07.005826    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc456f65-a237-4934-9ba9-088bdb38f0b4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dc456f65-a237-4934-9ba9-088bdb38f0b4" (UID: "dc456f65-a237-4934-9ba9-088bdb38f0b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:14:07.106058 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:07.106033    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc456f65-a237-4934-9ba9-088bdb38f0b4-maas-api-service-ca\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:14:07.800004 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:07.799972    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-750c68-cpv9m"
Sep 09 06:14:07.800004 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:07.799991    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-750c68-cpv9m" event={"ID":"dc456f65-a237-4934-9ba9-088bdb38f0b4","Type":"ContainerDied","Data":"ad000e300333ef8efbbbe2119d801d7d2a64c9ed32b48e06753ccaff34e479e5"}
Sep 09 06:14:07.800219 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:07.800044    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad000e300333ef8efbbbe2119d801d7d2a64c9ed32b48e06753ccaff34e479e5"
Sep 09 06:14:09.367183 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:09.367151    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7982-55d0b4"]
Sep 09 06:14:09.367555 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:09.367450    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc456f65-a237-4934-9ba9-088bdb38f0b4" containerName="revoke-keys"
Sep 09 06:14:09.367555 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:09.367460    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc456f65-a237-4934-9ba9-088bdb38f0b4" containerName="revoke-keys"
Sep 09 06:14:09.367555 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:09.367539    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc456f65-a237-4934-9ba9-088bdb38f0b4" containerName="revoke-keys"
Sep 09 06:14:09.367555 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:09.367549    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="619ba57a-cb9b-46e8-ae40-6c2b8ebef515" containerName="istio-proxy"
Sep 09 06:14:09.370603 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:09.370588    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7982-55d0b4"
Sep 09 06:14:09.373031 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:09.373012    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vljc5\""
Sep 09 06:14:09.378571 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:09.378552    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7982-55d0b4"]
Sep 09 06:14:09.424216 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:09.424183    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnvkf\" (UniqueName: \"kubernetes.io/projected/35fab56a-902b-4ce1-b48e-ec76b99735f6-kube-api-access-rnvkf\") pod \"mt-curl-7982-55d0b4\" (UID: \"35fab56a-902b-4ce1-b48e-ec76b99735f6\") " pod="opendatahub/mt-curl-7982-55d0b4"
Sep 09 06:14:09.525421 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:09.525394    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnvkf\" (UniqueName: \"kubernetes.io/projected/35fab56a-902b-4ce1-b48e-ec76b99735f6-kube-api-access-rnvkf\") pod \"mt-curl-7982-55d0b4\" (UID: \"35fab56a-902b-4ce1-b48e-ec76b99735f6\") " pod="opendatahub/mt-curl-7982-55d0b4"
Sep 09 06:14:09.534111 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:09.534089    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnvkf\" (UniqueName: \"kubernetes.io/projected/35fab56a-902b-4ce1-b48e-ec76b99735f6-kube-api-access-rnvkf\") pod \"mt-curl-7982-55d0b4\" (UID: \"35fab56a-902b-4ce1-b48e-ec76b99735f6\") " pod="opendatahub/mt-curl-7982-55d0b4"
Sep 09 06:14:09.680303 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:09.680227    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7982-55d0b4"
Sep 09 06:14:09.949539 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:09.949463    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7982-55d0b4"]
Sep 09 06:14:09.952209 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:14:09.952175    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35fab56a_902b_4ce1_b48e_ec76b99735f6.slice/crio-324f19f15cdc2588e030d468ca078a775a31d1c3e966e365e334e62d2bb7e79d WatchSource:0}: Error finding container 324f19f15cdc2588e030d468ca078a775a31d1c3e966e365e334e62d2bb7e79d: Status 404 returned error can't find the container with id 324f19f15cdc2588e030d468ca078a775a31d1c3e966e365e334e62d2bb7e79d
Sep 09 06:14:09.953497 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:09.953478    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 06:14:10.816329 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:10.816296    2579 generic.go:356] "Generic (PLEG): container finished" podID="35fab56a-902b-4ce1-b48e-ec76b99735f6" containerID="99a7388760d6060d3cfe05af701bd687828c0059e77fc38d6f78e1b12557e1ba" exitCode=0
Sep 09 06:14:10.816727 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:10.816384    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7982-55d0b4" event={"ID":"35fab56a-902b-4ce1-b48e-ec76b99735f6","Type":"ContainerDied","Data":"99a7388760d6060d3cfe05af701bd687828c0059e77fc38d6f78e1b12557e1ba"}
Sep 09 06:14:10.816727 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:10.816425    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7982-55d0b4" event={"ID":"35fab56a-902b-4ce1-b48e-ec76b99735f6","Type":"ContainerStarted","Data":"324f19f15cdc2588e030d468ca078a775a31d1c3e966e365e334e62d2bb7e79d"}
Sep 09 06:14:11.953592 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:11.953569    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7982-55d0b4"
Sep 09 06:14:11.994002 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:11.993963    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7982-55d0b4_35fab56a-902b-4ce1-b48e-ec76b99735f6/mt-curl-7982-55d0b4/0.log"
Sep 09 06:14:12.018686 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:12.018659    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7982-55d0b4"]
Sep 09 06:14:12.022417 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:12.022391    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7982-55d0b4"]
Sep 09 06:14:12.046776 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:12.046752    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnvkf\" (UniqueName: \"kubernetes.io/projected/35fab56a-902b-4ce1-b48e-ec76b99735f6-kube-api-access-rnvkf\") pod \"35fab56a-902b-4ce1-b48e-ec76b99735f6\" (UID: \"35fab56a-902b-4ce1-b48e-ec76b99735f6\") "
Sep 09 06:14:12.048967 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:12.048947    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35fab56a-902b-4ce1-b48e-ec76b99735f6-kube-api-access-rnvkf" (OuterVolumeSpecName: "kube-api-access-rnvkf") pod "35fab56a-902b-4ce1-b48e-ec76b99735f6" (UID: "35fab56a-902b-4ce1-b48e-ec76b99735f6"). InnerVolumeSpecName "kube-api-access-rnvkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:14:12.147678 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:12.147590    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnvkf\" (UniqueName: \"kubernetes.io/projected/35fab56a-902b-4ce1-b48e-ec76b99735f6-kube-api-access-rnvkf\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:14:12.828056 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:12.828025    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7982-55d0b4"
Sep 09 06:14:12.828229 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:12.828066    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="324f19f15cdc2588e030d468ca078a775a31d1c3e966e365e334e62d2bb7e79d"
Sep 09 06:14:12.837471 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:14:12.837445    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7982-55d0b4\" is forbidden: User \"system:node:ip-10-0-131-142.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-142.ec2.internal' and this object" podUID="35fab56a-902b-4ce1-b48e-ec76b99735f6" pod="opendatahub/mt-curl-7982-55d0b4"
Sep 09 06:14:13.148074 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:13.147992    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35fab56a-902b-4ce1-b48e-ec76b99735f6" path="/var/lib/kubelet/pods/35fab56a-902b-4ce1-b48e-ec76b99735f6/volumes"
Sep 09 06:14:14.246994 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:14.246961    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7cf6be/auth-test-model-7cf6be-kserve-579d696f59-hhqwt"]
Sep 09 06:14:14.431832 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:14.431801    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-750c68-6b9b48496f-wkb4k"]
Sep 09 06:14:14.432137 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:14.432089    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-750c68-6b9b48496f-wkb4k" podUID="05730de2-6196-49fc-af43-cb81e8ec0160" containerName="payload-processing" containerID="cri-o://8b28864fd361887619d1a4d6256369620d2583917afed555c0e3bb1af62b192a" gracePeriod=30
Sep 09 06:14:14.682137 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:14.682116    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-750c68-6b9b48496f-wkb4k"
Sep 09 06:14:14.770425 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:14.770397    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qn8tr\" (UniqueName: \"kubernetes.io/projected/05730de2-6196-49fc-af43-cb81e8ec0160-kube-api-access-qn8tr\") pod \"05730de2-6196-49fc-af43-cb81e8ec0160\" (UID: \"05730de2-6196-49fc-af43-cb81e8ec0160\") "
Sep 09 06:14:14.770612 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:14.770459    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05730de2-6196-49fc-af43-cb81e8ec0160-plugins-config-volume\") pod \"05730de2-6196-49fc-af43-cb81e8ec0160\" (UID: \"05730de2-6196-49fc-af43-cb81e8ec0160\") "
Sep 09 06:14:14.770842 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:14.770821    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05730de2-6196-49fc-af43-cb81e8ec0160-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "05730de2-6196-49fc-af43-cb81e8ec0160" (UID: "05730de2-6196-49fc-af43-cb81e8ec0160"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:14:14.772689 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:14.772665    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05730de2-6196-49fc-af43-cb81e8ec0160-kube-api-access-qn8tr" (OuterVolumeSpecName: "kube-api-access-qn8tr") pod "05730de2-6196-49fc-af43-cb81e8ec0160" (UID: "05730de2-6196-49fc-af43-cb81e8ec0160"). InnerVolumeSpecName "kube-api-access-qn8tr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:14:14.839944 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:14.839916    2579 generic.go:356] "Generic (PLEG): container finished" podID="05730de2-6196-49fc-af43-cb81e8ec0160" containerID="8b28864fd361887619d1a4d6256369620d2583917afed555c0e3bb1af62b192a" exitCode=0
Sep 09 06:14:14.840046 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:14.839965    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-750c68-6b9b48496f-wkb4k" event={"ID":"05730de2-6196-49fc-af43-cb81e8ec0160","Type":"ContainerDied","Data":"8b28864fd361887619d1a4d6256369620d2583917afed555c0e3bb1af62b192a"}
Sep 09 06:14:14.840046 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:14.839975    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-750c68-6b9b48496f-wkb4k"
Sep 09 06:14:14.840046 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:14.840018    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-750c68-6b9b48496f-wkb4k" event={"ID":"05730de2-6196-49fc-af43-cb81e8ec0160","Type":"ContainerDied","Data":"b9de9cb2941d054adca7ff21efb9d78a97ae7f6b2494e90a951b3880157ce539"}
Sep 09 06:14:14.840046 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:14.840042    2579 scope.go:117] "RemoveContainer" containerID="8b28864fd361887619d1a4d6256369620d2583917afed555c0e3bb1af62b192a"
Sep 09 06:14:14.849124 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:14.849108    2579 scope.go:117] "RemoveContainer" containerID="8b28864fd361887619d1a4d6256369620d2583917afed555c0e3bb1af62b192a"
Sep 09 06:14:14.849379 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:14:14.849360    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b28864fd361887619d1a4d6256369620d2583917afed555c0e3bb1af62b192a\": container with ID starting with 8b28864fd361887619d1a4d6256369620d2583917afed555c0e3bb1af62b192a not found: ID does not exist" containerID="8b28864fd361887619d1a4d6256369620d2583917afed555c0e3bb1af62b192a"
Sep 09 06:14:14.849426 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:14.849387    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b28864fd361887619d1a4d6256369620d2583917afed555c0e3bb1af62b192a"} err="failed to get container status \"8b28864fd361887619d1a4d6256369620d2583917afed555c0e3bb1af62b192a\": rpc error: code = NotFound desc = could not find container \"8b28864fd361887619d1a4d6256369620d2583917afed555c0e3bb1af62b192a\": container with ID starting with 8b28864fd361887619d1a4d6256369620d2583917afed555c0e3bb1af62b192a not found: ID does not exist"
Sep 09 06:14:14.859916 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:14.859896    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-750c68-cpv9m"]
Sep 09 06:14:14.863526 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:14.863509    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-750c68-cpv9m"]
Sep 09 06:14:14.871394 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:14.871378    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn8tr\" (UniqueName: \"kubernetes.io/projected/05730de2-6196-49fc-af43-cb81e8ec0160-kube-api-access-qn8tr\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:14:14.871455 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:14.871396    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05730de2-6196-49fc-af43-cb81e8ec0160-plugins-config-volume\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:14:14.875213 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:14.875196    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-750c68-6b9b48496f-wkb4k"]
Sep 09 06:14:14.882071 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:14.882054    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-750c68-6b9b48496f-wkb4k"]
Sep 09 06:14:15.147520 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:15.147442    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05730de2-6196-49fc-af43-cb81e8ec0160" path="/var/lib/kubelet/pods/05730de2-6196-49fc-af43-cb81e8ec0160/volumes"
Sep 09 06:14:15.147790 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:15.147777    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc456f65-a237-4934-9ba9-088bdb38f0b4" path="/var/lib/kubelet/pods/dc456f65-a237-4934-9ba9-088bdb38f0b4/volumes"
Sep 09 06:14:16.425079 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:16.425047    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:14:19.888804 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:19.888774    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-6b5d89/sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7"]
Sep 09 06:14:19.889201 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:19.889074    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05730de2-6196-49fc-af43-cb81e8ec0160" containerName="payload-processing"
Sep 09 06:14:19.889201 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:19.889084    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="05730de2-6196-49fc-af43-cb81e8ec0160" containerName="payload-processing"
Sep 09 06:14:19.889201 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:19.889094    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35fab56a-902b-4ce1-b48e-ec76b99735f6" containerName="mt-curl-7982-55d0b4"
Sep 09 06:14:19.889201 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:19.889099    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="35fab56a-902b-4ce1-b48e-ec76b99735f6" containerName="mt-curl-7982-55d0b4"
Sep 09 06:14:19.889201 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:19.889153    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="35fab56a-902b-4ce1-b48e-ec76b99735f6" containerName="mt-curl-7982-55d0b4"
Sep 09 06:14:19.889201 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:19.889164    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="05730de2-6196-49fc-af43-cb81e8ec0160" containerName="payload-processing"
Sep 09 06:14:19.893544 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:19.893521    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6b5d89/sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7"
Sep 09 06:14:19.897670 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:19.897648    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-6b5d89\"/\"default-dockercfg-cjhg2\""
Sep 09 06:14:19.898441 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:19.898420    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-6b5d89\"/\"sub-test-model-6b5d89-kserve-self-signed-certs\""
Sep 09 06:14:19.898538 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:19.898454    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-6b5d89\"/\"kube-root-ca.crt\""
Sep 09 06:14:19.898538 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:19.898422    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-6b5d89\"/\"openshift-service-ca.crt\""
Sep 09 06:14:19.917119 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:19.917091    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6b5d89/sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7"]
Sep 09 06:14:20.015981 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:20.015950    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17b3a247-4c4b-4378-b5f2-f04e9740aa77-tmp-dir\") pod \"sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7\" (UID: \"17b3a247-4c4b-4378-b5f2-f04e9740aa77\") " pod="ai-tenant-e2e-shared-a-w3-6b5d89/sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7"
Sep 09 06:14:20.015981 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:20.015985    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17b3a247-4c4b-4378-b5f2-f04e9740aa77-model-cache\") pod \"sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7\" (UID: \"17b3a247-4c4b-4378-b5f2-f04e9740aa77\") " pod="ai-tenant-e2e-shared-a-w3-6b5d89/sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7"
Sep 09 06:14:20.016190 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:20.016011    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvz27\" (UniqueName: \"kubernetes.io/projected/17b3a247-4c4b-4378-b5f2-f04e9740aa77-kube-api-access-fvz27\") pod \"sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7\" (UID: \"17b3a247-4c4b-4378-b5f2-f04e9740aa77\") " pod="ai-tenant-e2e-shared-a-w3-6b5d89/sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7"
Sep 09 06:14:20.016190 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:20.016068    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17b3a247-4c4b-4378-b5f2-f04e9740aa77-tls-certs\") pod \"sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7\" (UID: \"17b3a247-4c4b-4378-b5f2-f04e9740aa77\") " pod="ai-tenant-e2e-shared-a-w3-6b5d89/sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7"
Sep 09 06:14:20.016190 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:20.016096    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17b3a247-4c4b-4378-b5f2-f04e9740aa77-dshm\") pod \"sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7\" (UID: \"17b3a247-4c4b-4378-b5f2-f04e9740aa77\") " pod="ai-tenant-e2e-shared-a-w3-6b5d89/sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7"
Sep 09 06:14:20.016190 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:20.016152    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17b3a247-4c4b-4378-b5f2-f04e9740aa77-home\") pod \"sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7\" (UID: \"17b3a247-4c4b-4378-b5f2-f04e9740aa77\") " pod="ai-tenant-e2e-shared-a-w3-6b5d89/sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7"
Sep 09 06:14:20.117112 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:20.117080    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17b3a247-4c4b-4378-b5f2-f04e9740aa77-tls-certs\") pod \"sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7\" (UID: \"17b3a247-4c4b-4378-b5f2-f04e9740aa77\") " pod="ai-tenant-e2e-shared-a-w3-6b5d89/sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7"
Sep 09 06:14:20.117305 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:20.117120    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17b3a247-4c4b-4378-b5f2-f04e9740aa77-dshm\") pod \"sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7\" (UID: \"17b3a247-4c4b-4378-b5f2-f04e9740aa77\") " pod="ai-tenant-e2e-shared-a-w3-6b5d89/sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7"
Sep 09 06:14:20.117305 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:20.117136    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17b3a247-4c4b-4378-b5f2-f04e9740aa77-home\") pod \"sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7\" (UID: \"17b3a247-4c4b-4378-b5f2-f04e9740aa77\") " pod="ai-tenant-e2e-shared-a-w3-6b5d89/sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7"
Sep 09 06:14:20.117305 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:20.117216    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17b3a247-4c4b-4378-b5f2-f04e9740aa77-tmp-dir\") pod \"sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7\" (UID: \"17b3a247-4c4b-4378-b5f2-f04e9740aa77\") " pod="ai-tenant-e2e-shared-a-w3-6b5d89/sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7"
Sep 09 06:14:20.117305 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:20.117244    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17b3a247-4c4b-4378-b5f2-f04e9740aa77-model-cache\") pod \"sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7\" (UID: \"17b3a247-4c4b-4378-b5f2-f04e9740aa77\") " pod="ai-tenant-e2e-shared-a-w3-6b5d89/sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7"
Sep 09 06:14:20.117305 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:20.117269    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvz27\" (UniqueName: \"kubernetes.io/projected/17b3a247-4c4b-4378-b5f2-f04e9740aa77-kube-api-access-fvz27\") pod \"sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7\" (UID: \"17b3a247-4c4b-4378-b5f2-f04e9740aa77\") " pod="ai-tenant-e2e-shared-a-w3-6b5d89/sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7"
Sep 09 06:14:20.117617 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:20.117584    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17b3a247-4c4b-4378-b5f2-f04e9740aa77-model-cache\") pod \"sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7\" (UID: \"17b3a247-4c4b-4378-b5f2-f04e9740aa77\") " pod="ai-tenant-e2e-shared-a-w3-6b5d89/sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7"
Sep 09 06:14:20.117729 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:20.117647    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17b3a247-4c4b-4378-b5f2-f04e9740aa77-home\") pod \"sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7\" (UID: \"17b3a247-4c4b-4378-b5f2-f04e9740aa77\") " pod="ai-tenant-e2e-shared-a-w3-6b5d89/sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7"
Sep 09 06:14:20.117729 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:20.117657    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17b3a247-4c4b-4378-b5f2-f04e9740aa77-tmp-dir\") pod \"sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7\" (UID: \"17b3a247-4c4b-4378-b5f2-f04e9740aa77\") " pod="ai-tenant-e2e-shared-a-w3-6b5d89/sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7"
Sep 09 06:14:20.119700 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:20.119681    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17b3a247-4c4b-4378-b5f2-f04e9740aa77-dshm\") pod \"sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7\" (UID: \"17b3a247-4c4b-4378-b5f2-f04e9740aa77\") " pod="ai-tenant-e2e-shared-a-w3-6b5d89/sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7"
Sep 09 06:14:20.119857 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:20.119842    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17b3a247-4c4b-4378-b5f2-f04e9740aa77-tls-certs\") pod \"sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7\" (UID: \"17b3a247-4c4b-4378-b5f2-f04e9740aa77\") " pod="ai-tenant-e2e-shared-a-w3-6b5d89/sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7"
Sep 09 06:14:20.125925 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:20.125906    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvz27\" (UniqueName: \"kubernetes.io/projected/17b3a247-4c4b-4378-b5f2-f04e9740aa77-kube-api-access-fvz27\") pod \"sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7\" (UID: \"17b3a247-4c4b-4378-b5f2-f04e9740aa77\") " pod="ai-tenant-e2e-shared-a-w3-6b5d89/sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7"
Sep 09 06:14:20.204152 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:20.204084    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6b5d89/sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7"
Sep 09 06:14:20.484418 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:20.484305    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6b5d89/sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7"]
Sep 09 06:14:20.488312 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:14:20.488286    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17b3a247_4c4b_4378_b5f2_f04e9740aa77.slice/crio-59d1a7173d61e619fc638a5a5bec28c7de7cd2c980c2f28c838717199e3d1306 WatchSource:0}: Error finding container 59d1a7173d61e619fc638a5a5bec28c7de7cd2c980c2f28c838717199e3d1306: Status 404 returned error can't find the container with id 59d1a7173d61e619fc638a5a5bec28c7de7cd2c980c2f28c838717199e3d1306
Sep 09 06:14:20.872522 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:20.872487    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6b5d89/sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7" event={"ID":"17b3a247-4c4b-4378-b5f2-f04e9740aa77","Type":"ContainerStarted","Data":"7dffa7a746a1a15a6a9c62a252a2f288b338327a045a680c98ce66f5a0626793"}
Sep 09 06:14:20.872749 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:20.872531    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6b5d89/sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7" event={"ID":"17b3a247-4c4b-4378-b5f2-f04e9740aa77","Type":"ContainerStarted","Data":"59d1a7173d61e619fc638a5a5bec28c7de7cd2c980c2f28c838717199e3d1306"}
Sep 09 06:14:20.889903 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:20.889846    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-6b5d89/sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7" podStartSLOduration=1.88983144 podStartE2EDuration="1.88983144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:14:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:14:20.889573232 +0000 UTC m=+1486.230061148" watchObservedRunningTime="2026-09-09 06:14:20.88983144 +0000 UTC m=+1486.230319349"
Sep 09 06:14:26.719665 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:26.719615    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:14:29.266911 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:29.266871    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-7cf6be/auth-test-model-7cf6be-kserve-579d696f59-hhqwt" podUID="ccdb1451-54db-4ac4-ad11-0deb92293513" containerName="main" containerID="cri-o://7bb91744a021ecc5e31b4b8c4f762f9d518a922597b9cac62b0f885631004258" gracePeriod=45
Sep 09 06:14:29.506595 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:29.506570    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7cf6be/auth-test-model-7cf6be-kserve-579d696f59-hhqwt"
Sep 09 06:14:29.607187 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:29.607158    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccdb1451-54db-4ac4-ad11-0deb92293513-tls-certs\") pod \"ccdb1451-54db-4ac4-ad11-0deb92293513\" (UID: \"ccdb1451-54db-4ac4-ad11-0deb92293513\") "
Sep 09 06:14:29.607187 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:29.607188    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccdb1451-54db-4ac4-ad11-0deb92293513-dshm\") pod \"ccdb1451-54db-4ac4-ad11-0deb92293513\" (UID: \"ccdb1451-54db-4ac4-ad11-0deb92293513\") "
Sep 09 06:14:29.607393 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:29.607245    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4kpd\" (UniqueName: \"kubernetes.io/projected/ccdb1451-54db-4ac4-ad11-0deb92293513-kube-api-access-w4kpd\") pod \"ccdb1451-54db-4ac4-ad11-0deb92293513\" (UID: \"ccdb1451-54db-4ac4-ad11-0deb92293513\") "
Sep 09 06:14:29.607393 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:29.607271    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccdb1451-54db-4ac4-ad11-0deb92293513-tmp-dir\") pod \"ccdb1451-54db-4ac4-ad11-0deb92293513\" (UID: \"ccdb1451-54db-4ac4-ad11-0deb92293513\") "
Sep 09 06:14:29.607393 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:29.607308    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccdb1451-54db-4ac4-ad11-0deb92293513-home\") pod \"ccdb1451-54db-4ac4-ad11-0deb92293513\" (UID: \"ccdb1451-54db-4ac4-ad11-0deb92293513\") "
Sep 09 06:14:29.607393 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:29.607350    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccdb1451-54db-4ac4-ad11-0deb92293513-model-cache\") pod \"ccdb1451-54db-4ac4-ad11-0deb92293513\" (UID: \"ccdb1451-54db-4ac4-ad11-0deb92293513\") "
Sep 09 06:14:29.607640 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:29.607591    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccdb1451-54db-4ac4-ad11-0deb92293513-home" (OuterVolumeSpecName: "home") pod "ccdb1451-54db-4ac4-ad11-0deb92293513" (UID: "ccdb1451-54db-4ac4-ad11-0deb92293513"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:14:29.607712 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:29.607575    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccdb1451-54db-4ac4-ad11-0deb92293513-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ccdb1451-54db-4ac4-ad11-0deb92293513" (UID: "ccdb1451-54db-4ac4-ad11-0deb92293513"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:14:29.607712 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:29.607660    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccdb1451-54db-4ac4-ad11-0deb92293513-model-cache" (OuterVolumeSpecName: "model-cache") pod "ccdb1451-54db-4ac4-ad11-0deb92293513" (UID: "ccdb1451-54db-4ac4-ad11-0deb92293513"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:14:29.609615 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:29.609590    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccdb1451-54db-4ac4-ad11-0deb92293513-dshm" (OuterVolumeSpecName: "dshm") pod "ccdb1451-54db-4ac4-ad11-0deb92293513" (UID: "ccdb1451-54db-4ac4-ad11-0deb92293513"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:14:29.609724 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:29.609683    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccdb1451-54db-4ac4-ad11-0deb92293513-kube-api-access-w4kpd" (OuterVolumeSpecName: "kube-api-access-w4kpd") pod "ccdb1451-54db-4ac4-ad11-0deb92293513" (UID: "ccdb1451-54db-4ac4-ad11-0deb92293513"). InnerVolumeSpecName "kube-api-access-w4kpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:14:29.609724 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:29.609705    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccdb1451-54db-4ac4-ad11-0deb92293513-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ccdb1451-54db-4ac4-ad11-0deb92293513" (UID: "ccdb1451-54db-4ac4-ad11-0deb92293513"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:14:29.708012 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:29.707983    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccdb1451-54db-4ac4-ad11-0deb92293513-tmp-dir\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:14:29.708012 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:29.708012    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccdb1451-54db-4ac4-ad11-0deb92293513-home\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:14:29.708178 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:29.708025    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccdb1451-54db-4ac4-ad11-0deb92293513-model-cache\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:14:29.708178 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:29.708036    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccdb1451-54db-4ac4-ad11-0deb92293513-tls-certs\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:14:29.708178 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:29.708047    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccdb1451-54db-4ac4-ad11-0deb92293513-dshm\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:14:29.708178 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:29.708058    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4kpd\" (UniqueName: \"kubernetes.io/projected/ccdb1451-54db-4ac4-ad11-0deb92293513-kube-api-access-w4kpd\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:14:29.918322 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:29.918245    2579 generic.go:356] "Generic (PLEG): container finished" podID="ccdb1451-54db-4ac4-ad11-0deb92293513" containerID="7bb91744a021ecc5e31b4b8c4f762f9d518a922597b9cac62b0f885631004258" exitCode=0
Sep 09 06:14:29.918322 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:29.918305    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7cf6be/auth-test-model-7cf6be-kserve-579d696f59-hhqwt"
Sep 09 06:14:29.918478 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:29.918328    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7cf6be/auth-test-model-7cf6be-kserve-579d696f59-hhqwt" event={"ID":"ccdb1451-54db-4ac4-ad11-0deb92293513","Type":"ContainerDied","Data":"7bb91744a021ecc5e31b4b8c4f762f9d518a922597b9cac62b0f885631004258"}
Sep 09 06:14:29.918478 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:29.918377    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7cf6be/auth-test-model-7cf6be-kserve-579d696f59-hhqwt" event={"ID":"ccdb1451-54db-4ac4-ad11-0deb92293513","Type":"ContainerDied","Data":"14742f206b55ae0579de0a5b0f9bf212a8bfb57e7560f71cf225d6d2628a6ce2"}
Sep 09 06:14:29.918478 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:29.918393    2579 scope.go:117] "RemoveContainer" containerID="7bb91744a021ecc5e31b4b8c4f762f9d518a922597b9cac62b0f885631004258"
Sep 09 06:14:29.927584 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:29.927566    2579 scope.go:117] "RemoveContainer" containerID="7bb91744a021ecc5e31b4b8c4f762f9d518a922597b9cac62b0f885631004258"
Sep 09 06:14:29.927863 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:14:29.927845    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bb91744a021ecc5e31b4b8c4f762f9d518a922597b9cac62b0f885631004258\": container with ID starting with 7bb91744a021ecc5e31b4b8c4f762f9d518a922597b9cac62b0f885631004258 not found: ID does not exist" containerID="7bb91744a021ecc5e31b4b8c4f762f9d518a922597b9cac62b0f885631004258"
Sep 09 06:14:29.927909 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:29.927871    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bb91744a021ecc5e31b4b8c4f762f9d518a922597b9cac62b0f885631004258"} err="failed to get container status \"7bb91744a021ecc5e31b4b8c4f762f9d518a922597b9cac62b0f885631004258\": rpc error: code = NotFound desc = could not find container \"7bb91744a021ecc5e31b4b8c4f762f9d518a922597b9cac62b0f885631004258\": container with ID starting with 7bb91744a021ecc5e31b4b8c4f762f9d518a922597b9cac62b0f885631004258 not found: ID does not exist"
Sep 09 06:14:29.939823 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:29.939800    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7cf6be/auth-test-model-7cf6be-kserve-579d696f59-hhqwt"]
Sep 09 06:14:29.945140 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:29.945121    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7cf6be/auth-test-model-7cf6be-kserve-579d696f59-hhqwt"]
Sep 09 06:14:30.205328 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:30.205253    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-6b5d89/sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7"
Sep 09 06:14:30.205328 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:30.205298    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-6b5d89/sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7"
Sep 09 06:14:30.217602 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:30.217580    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-6b5d89/sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7"
Sep 09 06:14:30.936351 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:30.936324    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-6b5d89/sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7"
Sep 09 06:14:31.148064 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:31.148029    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccdb1451-54db-4ac4-ad11-0deb92293513" path="/var/lib/kubelet/pods/ccdb1451-54db-4ac4-ad11-0deb92293513/volumes"
Sep 09 06:14:35.392202 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:35.392167    2579 scope.go:117] "RemoveContainer" containerID="f794c748e93da30e6c83fe8f2acb2a334db3608a3f83fc6321bb64e14f330332"
Sep 09 06:14:35.400283 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:35.400267    2579 scope.go:117] "RemoveContainer" containerID="214ecd98386aac7c21a85ac2cc97fc6849de794afaa5490321548dff45c93045"
Sep 09 06:14:35.407875 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:35.407861    2579 scope.go:117] "RemoveContainer" containerID="c984e7bb3cdcf054b01903cede1cc6cc06c1d462089b53413c7b9e94200fb2be"
Sep 09 06:14:35.415256 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:35.415242    2579 scope.go:117] "RemoveContainer" containerID="bbe6d90e91eac90908b18d88125501397a57f27b2061a05aa9f4ba5f4e0e35e3"
Sep 09 06:14:35.422269 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:35.422254    2579 scope.go:117] "RemoveContainer" containerID="152d24b981de79f230157e40eaee9ea630f710fb983409ed2ef7cf937e2ccc81"
Sep 09 06:14:58.510757 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:14:58.510662    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:15:00.133659 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:00.133606    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5d89-29815575-mx82j"]
Sep 09 06:15:00.134037 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:00.133941    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccdb1451-54db-4ac4-ad11-0deb92293513" containerName="main"
Sep 09 06:15:00.134037 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:00.133953    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccdb1451-54db-4ac4-ad11-0deb92293513" containerName="main"
Sep 09 06:15:00.134037 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:00.134013    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccdb1451-54db-4ac4-ad11-0deb92293513" containerName="main"
Sep 09 06:15:00.139640 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:00.139596    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29815575-hg6mt"]
Sep 09 06:15:00.139777 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:00.139719    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5d89-29815575-mx82j"
Sep 09 06:15:00.141851 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:00.141830    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ksjk6\""
Sep 09 06:15:00.142946 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:00.142926    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815575-hg6mt"
Sep 09 06:15:00.145452 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:00.145432    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5d89-29815575-mx82j"]
Sep 09 06:15:00.149152 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:00.148923    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29815575-hg6mt"]
Sep 09 06:15:00.153282 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:00.153265    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5d89-29815575-mx82j"
Sep 09 06:15:00.161757 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:00.161735    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815575-hg6mt"
Sep 09 06:15:00.430613 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:00.430456    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5d89-29815575-mx82j"]
Sep 09 06:15:00.433588 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:15:00.433560    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a514b3e_a43e_4dc1_bf63_277e9c94a07a.slice/crio-bea4e7dc0cad1f813db9842af0a3633b81b9d6c3dfaa53a97cec3ef23b32554f WatchSource:0}: Error finding container bea4e7dc0cad1f813db9842af0a3633b81b9d6c3dfaa53a97cec3ef23b32554f: Status 404 returned error can't find the container with id bea4e7dc0cad1f813db9842af0a3633b81b9d6c3dfaa53a97cec3ef23b32554f
Sep 09 06:15:00.437960 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:00.437940    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29815575-hg6mt"]
Sep 09 06:15:00.439383 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:15:00.439360    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf712432c_2d21_474e_b0e7_ef649879284f.slice/crio-e081c839d19d31796c19679dda622a5250da89378b1f4616023415951dac334a WatchSource:0}: Error finding container e081c839d19d31796c19679dda622a5250da89378b1f4616023415951dac334a: Status 404 returned error can't find the container with id e081c839d19d31796c19679dda622a5250da89378b1f4616023415951dac334a
Sep 09 06:15:01.070877 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:01.070844    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815575-hg6mt" event={"ID":"f712432c-2d21-474e-b0e7-ef649879284f","Type":"ContainerStarted","Data":"47717ac225983fdb270cbe0f77a0eaa020940589853ccf97742a6db62cde472b"}
Sep 09 06:15:01.071080 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:01.070904    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815575-hg6mt" event={"ID":"f712432c-2d21-474e-b0e7-ef649879284f","Type":"ContainerStarted","Data":"e081c839d19d31796c19679dda622a5250da89378b1f4616023415951dac334a"}
Sep 09 06:15:01.072356 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:01.072335    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5d89-29815575-mx82j" event={"ID":"3a514b3e-a43e-4dc1-bf63-277e9c94a07a","Type":"ContainerStarted","Data":"5a59a95e96585ab977b7eb88276d773bab6a13399b714065ed84a844637f1082"}
Sep 09 06:15:01.072457 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:01.072372    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5d89-29815575-mx82j" event={"ID":"3a514b3e-a43e-4dc1-bf63-277e9c94a07a","Type":"ContainerStarted","Data":"bea4e7dc0cad1f813db9842af0a3633b81b9d6c3dfaa53a97cec3ef23b32554f"}
Sep 09 06:15:01.084135 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:01.084093    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815575-hg6mt" podStartSLOduration=1.084079658 podStartE2EDuration="1.084079658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:15:01.083667547 +0000 UTC m=+1526.424155461" watchObservedRunningTime="2026-09-09 06:15:01.084079658 +0000 UTC m=+1526.424567565"
Sep 09 06:15:01.097215 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:01.097176    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5d89-29815575-mx82j" podStartSLOduration=1.097164573 podStartE2EDuration="1.097164573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:15:01.096143511 +0000 UTC m=+1526.436631422" watchObservedRunningTime="2026-09-09 06:15:01.097164573 +0000 UTC m=+1526.437652518"
Sep 09 06:15:02.078888 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:02.078856    2579 generic.go:356] "Generic (PLEG): container finished" podID="f712432c-2d21-474e-b0e7-ef649879284f" containerID="47717ac225983fdb270cbe0f77a0eaa020940589853ccf97742a6db62cde472b" exitCode=0
Sep 09 06:15:02.079284 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:02.078942    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815575-hg6mt" event={"ID":"f712432c-2d21-474e-b0e7-ef649879284f","Type":"ContainerDied","Data":"47717ac225983fdb270cbe0f77a0eaa020940589853ccf97742a6db62cde472b"}
Sep 09 06:15:02.080398 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:02.080376    2579 generic.go:356] "Generic (PLEG): container finished" podID="3a514b3e-a43e-4dc1-bf63-277e9c94a07a" containerID="5a59a95e96585ab977b7eb88276d773bab6a13399b714065ed84a844637f1082" exitCode=0
Sep 09 06:15:02.080497 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:02.080435    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5d89-29815575-mx82j" event={"ID":"3a514b3e-a43e-4dc1-bf63-277e9c94a07a","Type":"ContainerDied","Data":"5a59a95e96585ab977b7eb88276d773bab6a13399b714065ed84a844637f1082"}
Sep 09 06:15:03.243411 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:03.243388    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5d89-29815575-mx82j"
Sep 09 06:15:03.246536 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:03.246519    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815575-hg6mt"
Sep 09 06:15:04.091298 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:04.091270    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5d89-29815575-mx82j"
Sep 09 06:15:04.091491 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:04.091315    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5d89-29815575-mx82j" event={"ID":"3a514b3e-a43e-4dc1-bf63-277e9c94a07a","Type":"ContainerDied","Data":"bea4e7dc0cad1f813db9842af0a3633b81b9d6c3dfaa53a97cec3ef23b32554f"}
Sep 09 06:15:04.091491 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:04.091359    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bea4e7dc0cad1f813db9842af0a3633b81b9d6c3dfaa53a97cec3ef23b32554f"
Sep 09 06:15:04.092555 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:04.092528    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815575-hg6mt"
Sep 09 06:15:04.092691 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:04.092523    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815575-hg6mt" event={"ID":"f712432c-2d21-474e-b0e7-ef649879284f","Type":"ContainerDied","Data":"e081c839d19d31796c19679dda622a5250da89378b1f4616023415951dac334a"}
Sep 09 06:15:04.092691 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:04.092611    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e081c839d19d31796c19679dda622a5250da89378b1f4616023415951dac334a"
Sep 09 06:15:12.419670 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:12.419619    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:15:22.859853 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:22.859822    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7982-e0dcd5"]
Sep 09 06:15:22.860239 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:22.860099    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f712432c-2d21-474e-b0e7-ef649879284f" containerName="cleanup"
Sep 09 06:15:22.860239 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:22.860111    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f712432c-2d21-474e-b0e7-ef649879284f" containerName="cleanup"
Sep 09 06:15:22.860239 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:22.860133    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a514b3e-a43e-4dc1-bf63-277e9c94a07a" containerName="cleanup"
Sep 09 06:15:22.860239 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:22.860138    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a514b3e-a43e-4dc1-bf63-277e9c94a07a" containerName="cleanup"
Sep 09 06:15:22.860239 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:22.860189    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a514b3e-a43e-4dc1-bf63-277e9c94a07a" containerName="cleanup"
Sep 09 06:15:22.860239 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:22.860198    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f712432c-2d21-474e-b0e7-ef649879284f" containerName="cleanup"
Sep 09 06:15:22.863108 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:22.863088    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7982-e0dcd5"
Sep 09 06:15:22.866852 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:22.866830    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vljc5\""
Sep 09 06:15:22.873184 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:22.873164    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7982-e0dcd5"]
Sep 09 06:15:23.011566 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:23.011524    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cc6r\" (UniqueName: \"kubernetes.io/projected/d91631d5-1af8-47fa-88b9-1da2d72fe14f-kube-api-access-9cc6r\") pod \"mt-curl-7982-e0dcd5\" (UID: \"d91631d5-1af8-47fa-88b9-1da2d72fe14f\") " pod="opendatahub/mt-curl-7982-e0dcd5"
Sep 09 06:15:23.112390 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:23.112322    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cc6r\" (UniqueName: \"kubernetes.io/projected/d91631d5-1af8-47fa-88b9-1da2d72fe14f-kube-api-access-9cc6r\") pod \"mt-curl-7982-e0dcd5\" (UID: \"d91631d5-1af8-47fa-88b9-1da2d72fe14f\") " pod="opendatahub/mt-curl-7982-e0dcd5"
Sep 09 06:15:23.126782 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:23.126762    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cc6r\" (UniqueName: \"kubernetes.io/projected/d91631d5-1af8-47fa-88b9-1da2d72fe14f-kube-api-access-9cc6r\") pod \"mt-curl-7982-e0dcd5\" (UID: \"d91631d5-1af8-47fa-88b9-1da2d72fe14f\") " pod="opendatahub/mt-curl-7982-e0dcd5"
Sep 09 06:15:23.172923 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:23.172905    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7982-e0dcd5"
Sep 09 06:15:23.448532 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:23.444202    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7982-e0dcd5"]
Sep 09 06:15:23.450687 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:15:23.450655    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd91631d5_1af8_47fa_88b9_1da2d72fe14f.slice/crio-1c7355b8a0f304a7542d7e5af320c7ff57b0aa6d606a15ebef678f0b98f3dfff WatchSource:0}: Error finding container 1c7355b8a0f304a7542d7e5af320c7ff57b0aa6d606a15ebef678f0b98f3dfff: Status 404 returned error can't find the container with id 1c7355b8a0f304a7542d7e5af320c7ff57b0aa6d606a15ebef678f0b98f3dfff
Sep 09 06:15:24.185730 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:24.185701    2579 generic.go:356] "Generic (PLEG): container finished" podID="d91631d5-1af8-47fa-88b9-1da2d72fe14f" containerID="162b7ca18138bee051a50733d2e28390a349fd9f405f049d584d85c5cc57e55e" exitCode=0
Sep 09 06:15:24.186047 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:24.185772    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7982-e0dcd5" event={"ID":"d91631d5-1af8-47fa-88b9-1da2d72fe14f","Type":"ContainerDied","Data":"162b7ca18138bee051a50733d2e28390a349fd9f405f049d584d85c5cc57e55e"}
Sep 09 06:15:24.186047 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:24.185796    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7982-e0dcd5" event={"ID":"d91631d5-1af8-47fa-88b9-1da2d72fe14f","Type":"ContainerStarted","Data":"1c7355b8a0f304a7542d7e5af320c7ff57b0aa6d606a15ebef678f0b98f3dfff"}
Sep 09 06:15:25.314512 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:25.314491    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7982-e0dcd5"
Sep 09 06:15:25.342784 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:25.342764    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7982-e0dcd5_d91631d5-1af8-47fa-88b9-1da2d72fe14f/mt-curl-7982-e0dcd5/0.log"
Sep 09 06:15:25.366734 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:25.366712    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7982-e0dcd5"]
Sep 09 06:15:25.369183 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:25.369165    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7982-e0dcd5"]
Sep 09 06:15:25.430708 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:25.430686    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cc6r\" (UniqueName: \"kubernetes.io/projected/d91631d5-1af8-47fa-88b9-1da2d72fe14f-kube-api-access-9cc6r\") pod \"d91631d5-1af8-47fa-88b9-1da2d72fe14f\" (UID: \"d91631d5-1af8-47fa-88b9-1da2d72fe14f\") "
Sep 09 06:15:25.432803 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:25.432784    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d91631d5-1af8-47fa-88b9-1da2d72fe14f-kube-api-access-9cc6r" (OuterVolumeSpecName: "kube-api-access-9cc6r") pod "d91631d5-1af8-47fa-88b9-1da2d72fe14f" (UID: "d91631d5-1af8-47fa-88b9-1da2d72fe14f"). InnerVolumeSpecName "kube-api-access-9cc6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:15:25.491039 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:25.490983    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7982-86c815"]
Sep 09 06:15:25.491352 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:25.491337    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d91631d5-1af8-47fa-88b9-1da2d72fe14f" containerName="mt-curl-7982-e0dcd5"
Sep 09 06:15:25.491352 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:25.491352    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d91631d5-1af8-47fa-88b9-1da2d72fe14f" containerName="mt-curl-7982-e0dcd5"
Sep 09 06:15:25.491479 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:25.491430    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d91631d5-1af8-47fa-88b9-1da2d72fe14f" containerName="mt-curl-7982-e0dcd5"
Sep 09 06:15:25.494471 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:25.494454    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7982-86c815"
Sep 09 06:15:25.500811 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:25.500790    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7982-86c815"]
Sep 09 06:15:25.531973 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:25.531950    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cc6r\" (UniqueName: \"kubernetes.io/projected/d91631d5-1af8-47fa-88b9-1da2d72fe14f-kube-api-access-9cc6r\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:15:25.632316 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:25.632290    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w82n7\" (UniqueName: \"kubernetes.io/projected/cc1f4c1d-f69b-467d-bb24-87c1371cff3a-kube-api-access-w82n7\") pod \"mt-curl-7982-86c815\" (UID: \"cc1f4c1d-f69b-467d-bb24-87c1371cff3a\") " pod="opendatahub/mt-curl-7982-86c815"
Sep 09 06:15:25.733181 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:25.733155    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w82n7\" (UniqueName: \"kubernetes.io/projected/cc1f4c1d-f69b-467d-bb24-87c1371cff3a-kube-api-access-w82n7\") pod \"mt-curl-7982-86c815\" (UID: \"cc1f4c1d-f69b-467d-bb24-87c1371cff3a\") " pod="opendatahub/mt-curl-7982-86c815"
Sep 09 06:15:25.742806 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:25.742749    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w82n7\" (UniqueName: \"kubernetes.io/projected/cc1f4c1d-f69b-467d-bb24-87c1371cff3a-kube-api-access-w82n7\") pod \"mt-curl-7982-86c815\" (UID: \"cc1f4c1d-f69b-467d-bb24-87c1371cff3a\") " pod="opendatahub/mt-curl-7982-86c815"
Sep 09 06:15:25.804661 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:25.804638    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7982-86c815"
Sep 09 06:15:26.077927 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:26.077898    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7982-86c815"]
Sep 09 06:15:26.080762 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:15:26.080737    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc1f4c1d_f69b_467d_bb24_87c1371cff3a.slice/crio-f6210c9e3aa50443c80a4b108173d5176cb7d797f607387a1e0b4dd3aa205b94 WatchSource:0}: Error finding container f6210c9e3aa50443c80a4b108173d5176cb7d797f607387a1e0b4dd3aa205b94: Status 404 returned error can't find the container with id f6210c9e3aa50443c80a4b108173d5176cb7d797f607387a1e0b4dd3aa205b94
Sep 09 06:15:26.196810 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:26.196783    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7982-86c815" event={"ID":"cc1f4c1d-f69b-467d-bb24-87c1371cff3a","Type":"ContainerStarted","Data":"f6210c9e3aa50443c80a4b108173d5176cb7d797f607387a1e0b4dd3aa205b94"}
Sep 09 06:15:26.198057 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:26.198041    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7982-e0dcd5"
Sep 09 06:15:26.198147 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:26.198067    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c7355b8a0f304a7542d7e5af320c7ff57b0aa6d606a15ebef678f0b98f3dfff"
Sep 09 06:15:26.208031 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:15:26.208009    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7982-e0dcd5\" is forbidden: User \"system:node:ip-10-0-131-142.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-142.ec2.internal' and this object" podUID="d91631d5-1af8-47fa-88b9-1da2d72fe14f" pod="opendatahub/mt-curl-7982-e0dcd5"
Sep 09 06:15:27.148723 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:27.148692    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d91631d5-1af8-47fa-88b9-1da2d72fe14f" path="/var/lib/kubelet/pods/d91631d5-1af8-47fa-88b9-1da2d72fe14f/volumes"
Sep 09 06:15:27.204250 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:27.204225    2579 generic.go:356] "Generic (PLEG): container finished" podID="cc1f4c1d-f69b-467d-bb24-87c1371cff3a" containerID="113d5cef10068238525c11616080d849281d6d40444eba4390c493774189cfcf" exitCode=0
Sep 09 06:15:27.204410 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:27.204300    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7982-86c815" event={"ID":"cc1f4c1d-f69b-467d-bb24-87c1371cff3a","Type":"ContainerDied","Data":"113d5cef10068238525c11616080d849281d6d40444eba4390c493774189cfcf"}
Sep 09 06:15:27.712005 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:27.711975    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:15:28.336980 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:28.336959    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7982-86c815"
Sep 09 06:15:28.361816 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:28.361785    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7982-86c815_cc1f4c1d-f69b-467d-bb24-87c1371cff3a/mt-curl-7982-86c815/0.log"
Sep 09 06:15:28.387015 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:28.386991    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7982-86c815"]
Sep 09 06:15:28.388866 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:28.388848    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7982-86c815"]
Sep 09 06:15:28.458424 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:28.458401    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w82n7\" (UniqueName: \"kubernetes.io/projected/cc1f4c1d-f69b-467d-bb24-87c1371cff3a-kube-api-access-w82n7\") pod \"cc1f4c1d-f69b-467d-bb24-87c1371cff3a\" (UID: \"cc1f4c1d-f69b-467d-bb24-87c1371cff3a\") "
Sep 09 06:15:28.460556 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:28.460504    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc1f4c1d-f69b-467d-bb24-87c1371cff3a-kube-api-access-w82n7" (OuterVolumeSpecName: "kube-api-access-w82n7") pod "cc1f4c1d-f69b-467d-bb24-87c1371cff3a" (UID: "cc1f4c1d-f69b-467d-bb24-87c1371cff3a"). InnerVolumeSpecName "kube-api-access-w82n7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:15:28.559736 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:28.559714    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w82n7\" (UniqueName: \"kubernetes.io/projected/cc1f4c1d-f69b-467d-bb24-87c1371cff3a-kube-api-access-w82n7\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:15:29.147314 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:29.147278    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc1f4c1d-f69b-467d-bb24-87c1371cff3a" path="/var/lib/kubelet/pods/cc1f4c1d-f69b-467d-bb24-87c1371cff3a/volumes"
Sep 09 06:15:29.214581 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:29.214534    2579 scope.go:117] "RemoveContainer" containerID="113d5cef10068238525c11616080d849281d6d40444eba4390c493774189cfcf"
Sep 09 06:15:29.214718 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:29.214560    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7982-86c815"
Sep 09 06:15:30.431074 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:30.431043    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6b5d89/sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7"]
Sep 09 06:15:37.701183 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:37.701145    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:15:45.450919 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:45.450875    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-6b5d89/sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7" podUID="17b3a247-4c4b-4378-b5f2-f04e9740aa77" containerName="main" containerID="cri-o://7dffa7a746a1a15a6a9c62a252a2f288b338327a045a680c98ce66f5a0626793" gracePeriod=45
Sep 09 06:15:45.689236 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:45.689214    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6b5d89/sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7"
Sep 09 06:15:45.791452 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:45.791383    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17b3a247-4c4b-4378-b5f2-f04e9740aa77-tls-certs\") pod \"17b3a247-4c4b-4378-b5f2-f04e9740aa77\" (UID: \"17b3a247-4c4b-4378-b5f2-f04e9740aa77\") "
Sep 09 06:15:45.791602 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:45.791453    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17b3a247-4c4b-4378-b5f2-f04e9740aa77-model-cache\") pod \"17b3a247-4c4b-4378-b5f2-f04e9740aa77\" (UID: \"17b3a247-4c4b-4378-b5f2-f04e9740aa77\") "
Sep 09 06:15:45.791703 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:45.791607    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvz27\" (UniqueName: \"kubernetes.io/projected/17b3a247-4c4b-4378-b5f2-f04e9740aa77-kube-api-access-fvz27\") pod \"17b3a247-4c4b-4378-b5f2-f04e9740aa77\" (UID: \"17b3a247-4c4b-4378-b5f2-f04e9740aa77\") "
Sep 09 06:15:45.791703 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:45.791696    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17b3a247-4c4b-4378-b5f2-f04e9740aa77-tmp-dir\") pod \"17b3a247-4c4b-4378-b5f2-f04e9740aa77\" (UID: \"17b3a247-4c4b-4378-b5f2-f04e9740aa77\") "
Sep 09 06:15:45.791820 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:45.791727    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b3a247-4c4b-4378-b5f2-f04e9740aa77-model-cache" (OuterVolumeSpecName: "model-cache") pod "17b3a247-4c4b-4378-b5f2-f04e9740aa77" (UID: "17b3a247-4c4b-4378-b5f2-f04e9740aa77"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:15:45.791820 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:45.791737    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17b3a247-4c4b-4378-b5f2-f04e9740aa77-home\") pod \"17b3a247-4c4b-4378-b5f2-f04e9740aa77\" (UID: \"17b3a247-4c4b-4378-b5f2-f04e9740aa77\") "
Sep 09 06:15:45.791820 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:45.791804    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17b3a247-4c4b-4378-b5f2-f04e9740aa77-dshm\") pod \"17b3a247-4c4b-4378-b5f2-f04e9740aa77\" (UID: \"17b3a247-4c4b-4378-b5f2-f04e9740aa77\") "
Sep 09 06:15:45.791972 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:45.791946    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b3a247-4c4b-4378-b5f2-f04e9740aa77-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "17b3a247-4c4b-4378-b5f2-f04e9740aa77" (UID: "17b3a247-4c4b-4378-b5f2-f04e9740aa77"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:15:45.791972 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:45.791955    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b3a247-4c4b-4378-b5f2-f04e9740aa77-home" (OuterVolumeSpecName: "home") pod "17b3a247-4c4b-4378-b5f2-f04e9740aa77" (UID: "17b3a247-4c4b-4378-b5f2-f04e9740aa77"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:15:45.792156 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:45.792132    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17b3a247-4c4b-4378-b5f2-f04e9740aa77-tmp-dir\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:15:45.792232 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:45.792159    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17b3a247-4c4b-4378-b5f2-f04e9740aa77-home\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:15:45.792294 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:45.792231    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17b3a247-4c4b-4378-b5f2-f04e9740aa77-model-cache\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:15:45.793812 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:45.793788    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17b3a247-4c4b-4378-b5f2-f04e9740aa77-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "17b3a247-4c4b-4378-b5f2-f04e9740aa77" (UID: "17b3a247-4c4b-4378-b5f2-f04e9740aa77"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:15:45.793901 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:45.793825    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b3a247-4c4b-4378-b5f2-f04e9740aa77-dshm" (OuterVolumeSpecName: "dshm") pod "17b3a247-4c4b-4378-b5f2-f04e9740aa77" (UID: "17b3a247-4c4b-4378-b5f2-f04e9740aa77"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:15:45.793901 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:45.793845    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17b3a247-4c4b-4378-b5f2-f04e9740aa77-kube-api-access-fvz27" (OuterVolumeSpecName: "kube-api-access-fvz27") pod "17b3a247-4c4b-4378-b5f2-f04e9740aa77" (UID: "17b3a247-4c4b-4378-b5f2-f04e9740aa77"). InnerVolumeSpecName "kube-api-access-fvz27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:15:45.893275 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:45.893240    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvz27\" (UniqueName: \"kubernetes.io/projected/17b3a247-4c4b-4378-b5f2-f04e9740aa77-kube-api-access-fvz27\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:15:45.893275 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:45.893273    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17b3a247-4c4b-4378-b5f2-f04e9740aa77-dshm\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:15:45.893275 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:45.893284    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17b3a247-4c4b-4378-b5f2-f04e9740aa77-tls-certs\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:15:46.297956 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:46.297926    2579 generic.go:356] "Generic (PLEG): container finished" podID="17b3a247-4c4b-4378-b5f2-f04e9740aa77" containerID="7dffa7a746a1a15a6a9c62a252a2f288b338327a045a680c98ce66f5a0626793" exitCode=0
Sep 09 06:15:46.298146 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:46.297971    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6b5d89/sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7"
Sep 09 06:15:46.298146 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:46.298012    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6b5d89/sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7" event={"ID":"17b3a247-4c4b-4378-b5f2-f04e9740aa77","Type":"ContainerDied","Data":"7dffa7a746a1a15a6a9c62a252a2f288b338327a045a680c98ce66f5a0626793"}
Sep 09 06:15:46.298146 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:46.298054    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6b5d89/sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7" event={"ID":"17b3a247-4c4b-4378-b5f2-f04e9740aa77","Type":"ContainerDied","Data":"59d1a7173d61e619fc638a5a5bec28c7de7cd2c980c2f28c838717199e3d1306"}
Sep 09 06:15:46.298146 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:46.298071    2579 scope.go:117] "RemoveContainer" containerID="7dffa7a746a1a15a6a9c62a252a2f288b338327a045a680c98ce66f5a0626793"
Sep 09 06:15:46.307398 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:46.307383    2579 scope.go:117] "RemoveContainer" containerID="7dffa7a746a1a15a6a9c62a252a2f288b338327a045a680c98ce66f5a0626793"
Sep 09 06:15:46.307679 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:15:46.307661    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dffa7a746a1a15a6a9c62a252a2f288b338327a045a680c98ce66f5a0626793\": container with ID starting with 7dffa7a746a1a15a6a9c62a252a2f288b338327a045a680c98ce66f5a0626793 not found: ID does not exist" containerID="7dffa7a746a1a15a6a9c62a252a2f288b338327a045a680c98ce66f5a0626793"
Sep 09 06:15:46.307762 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:46.307686    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dffa7a746a1a15a6a9c62a252a2f288b338327a045a680c98ce66f5a0626793"} err="failed to get container status \"7dffa7a746a1a15a6a9c62a252a2f288b338327a045a680c98ce66f5a0626793\": rpc error: code = NotFound desc = could not find container \"7dffa7a746a1a15a6a9c62a252a2f288b338327a045a680c98ce66f5a0626793\": container with ID starting with 7dffa7a746a1a15a6a9c62a252a2f288b338327a045a680c98ce66f5a0626793 not found: ID does not exist"
Sep 09 06:15:46.318598 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:46.318573    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6b5d89/sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7"]
Sep 09 06:15:46.322235 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:46.322216    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6b5d89/sub-test-model-6b5d89-kserve-57ffc8fd7f-ffqt7"]
Sep 09 06:15:47.155068 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:47.155038    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17b3a247-4c4b-4378-b5f2-f04e9740aa77" path="/var/lib/kubelet/pods/17b3a247-4c4b-4378-b5f2-f04e9740aa77/volumes"
Sep 09 06:15:47.529044 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:47.528963    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-6b5d89/rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56"]
Sep 09 06:15:47.529341 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:47.529323    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc1f4c1d-f69b-467d-bb24-87c1371cff3a" containerName="mt-curl-7982-86c815"
Sep 09 06:15:47.529429 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:47.529345    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc1f4c1d-f69b-467d-bb24-87c1371cff3a" containerName="mt-curl-7982-86c815"
Sep 09 06:15:47.529429 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:47.529371    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17b3a247-4c4b-4378-b5f2-f04e9740aa77" containerName="main"
Sep 09 06:15:47.529429 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:47.529380    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="17b3a247-4c4b-4378-b5f2-f04e9740aa77" containerName="main"
Sep 09 06:15:47.529586 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:47.529460    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="17b3a247-4c4b-4378-b5f2-f04e9740aa77" containerName="main"
Sep 09 06:15:47.529586 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:47.529474    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc1f4c1d-f69b-467d-bb24-87c1371cff3a" containerName="mt-curl-7982-86c815"
Sep 09 06:15:47.533712 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:47.533692    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6b5d89/rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56"
Sep 09 06:15:47.536835 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:47.536800    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-6b5d89\"/\"kube-root-ca.crt\""
Sep 09 06:15:47.536835 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:47.536835    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-6b5d89\"/\"default-dockercfg-cjhg2\""
Sep 09 06:15:47.536998 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:47.536840    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-6b5d89\"/\"openshift-service-ca.crt\""
Sep 09 06:15:47.536998 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:47.536895    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-6b5d89\"/\"rate-test-model-6b5d89-kserve-self-signed-certs\""
Sep 09 06:15:47.541166 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:47.541145    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6b5d89/rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56"]
Sep 09 06:15:47.604311 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:47.604281    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31bac816-f890-4811-a1e0-51a721fee215-dshm\") pod \"rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56\" (UID: \"31bac816-f890-4811-a1e0-51a721fee215\") " pod="ai-tenant-e2e-shared-a-w3-6b5d89/rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56"
Sep 09 06:15:47.604432 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:47.604341    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31bac816-f890-4811-a1e0-51a721fee215-tmp-dir\") pod \"rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56\" (UID: \"31bac816-f890-4811-a1e0-51a721fee215\") " pod="ai-tenant-e2e-shared-a-w3-6b5d89/rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56"
Sep 09 06:15:47.604432 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:47.604380    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31bac816-f890-4811-a1e0-51a721fee215-home\") pod \"rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56\" (UID: \"31bac816-f890-4811-a1e0-51a721fee215\") " pod="ai-tenant-e2e-shared-a-w3-6b5d89/rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56"
Sep 09 06:15:47.604543 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:47.604431    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31bac816-f890-4811-a1e0-51a721fee215-tls-certs\") pod \"rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56\" (UID: \"31bac816-f890-4811-a1e0-51a721fee215\") " pod="ai-tenant-e2e-shared-a-w3-6b5d89/rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56"
Sep 09 06:15:47.604543 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:47.604471    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65g6v\" (UniqueName: \"kubernetes.io/projected/31bac816-f890-4811-a1e0-51a721fee215-kube-api-access-65g6v\") pod \"rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56\" (UID: \"31bac816-f890-4811-a1e0-51a721fee215\") " pod="ai-tenant-e2e-shared-a-w3-6b5d89/rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56"
Sep 09 06:15:47.604543 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:47.604491    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31bac816-f890-4811-a1e0-51a721fee215-model-cache\") pod \"rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56\" (UID: \"31bac816-f890-4811-a1e0-51a721fee215\") " pod="ai-tenant-e2e-shared-a-w3-6b5d89/rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56"
Sep 09 06:15:47.704912 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:47.704883    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65g6v\" (UniqueName: \"kubernetes.io/projected/31bac816-f890-4811-a1e0-51a721fee215-kube-api-access-65g6v\") pod \"rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56\" (UID: \"31bac816-f890-4811-a1e0-51a721fee215\") " pod="ai-tenant-e2e-shared-a-w3-6b5d89/rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56"
Sep 09 06:15:47.705059 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:47.704917    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31bac816-f890-4811-a1e0-51a721fee215-model-cache\") pod \"rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56\" (UID: \"31bac816-f890-4811-a1e0-51a721fee215\") " pod="ai-tenant-e2e-shared-a-w3-6b5d89/rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56"
Sep 09 06:15:47.705059 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:47.704938    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31bac816-f890-4811-a1e0-51a721fee215-dshm\") pod \"rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56\" (UID: \"31bac816-f890-4811-a1e0-51a721fee215\") " pod="ai-tenant-e2e-shared-a-w3-6b5d89/rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56"
Sep 09 06:15:47.705059 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:47.704968    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31bac816-f890-4811-a1e0-51a721fee215-tmp-dir\") pod \"rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56\" (UID: \"31bac816-f890-4811-a1e0-51a721fee215\") " pod="ai-tenant-e2e-shared-a-w3-6b5d89/rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56"
Sep 09 06:15:47.705213 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:47.705073    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31bac816-f890-4811-a1e0-51a721fee215-home\") pod \"rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56\" (UID: \"31bac816-f890-4811-a1e0-51a721fee215\") " pod="ai-tenant-e2e-shared-a-w3-6b5d89/rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56"
Sep 09 06:15:47.705213 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:47.705130    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31bac816-f890-4811-a1e0-51a721fee215-tls-certs\") pod \"rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56\" (UID: \"31bac816-f890-4811-a1e0-51a721fee215\") " pod="ai-tenant-e2e-shared-a-w3-6b5d89/rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56"
Sep 09 06:15:47.705422 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:47.705401    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31bac816-f890-4811-a1e0-51a721fee215-tmp-dir\") pod \"rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56\" (UID: \"31bac816-f890-4811-a1e0-51a721fee215\") " pod="ai-tenant-e2e-shared-a-w3-6b5d89/rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56"
Sep 09 06:15:47.705482 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:47.705414    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31bac816-f890-4811-a1e0-51a721fee215-model-cache\") pod \"rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56\" (UID: \"31bac816-f890-4811-a1e0-51a721fee215\") " pod="ai-tenant-e2e-shared-a-w3-6b5d89/rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56"
Sep 09 06:15:47.705482 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:47.705442    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31bac816-f890-4811-a1e0-51a721fee215-home\") pod \"rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56\" (UID: \"31bac816-f890-4811-a1e0-51a721fee215\") " pod="ai-tenant-e2e-shared-a-w3-6b5d89/rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56"
Sep 09 06:15:47.707185 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:47.707166    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31bac816-f890-4811-a1e0-51a721fee215-dshm\") pod \"rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56\" (UID: \"31bac816-f890-4811-a1e0-51a721fee215\") " pod="ai-tenant-e2e-shared-a-w3-6b5d89/rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56"
Sep 09 06:15:47.707498 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:47.707482    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31bac816-f890-4811-a1e0-51a721fee215-tls-certs\") pod \"rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56\" (UID: \"31bac816-f890-4811-a1e0-51a721fee215\") " pod="ai-tenant-e2e-shared-a-w3-6b5d89/rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56"
Sep 09 06:15:47.711888 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:47.711866    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65g6v\" (UniqueName: \"kubernetes.io/projected/31bac816-f890-4811-a1e0-51a721fee215-kube-api-access-65g6v\") pod \"rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56\" (UID: \"31bac816-f890-4811-a1e0-51a721fee215\") " pod="ai-tenant-e2e-shared-a-w3-6b5d89/rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56"
Sep 09 06:15:47.844948 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:47.844925    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6b5d89/rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56"
Sep 09 06:15:48.113117 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:48.113055    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6b5d89/rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56"]
Sep 09 06:15:48.115484 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:15:48.115455    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31bac816_f890_4811_a1e0_51a721fee215.slice/crio-f0ab08d4f976838885b1bf5e6428a01cafbf61ac901eeea6e151670a10ce6878 WatchSource:0}: Error finding container f0ab08d4f976838885b1bf5e6428a01cafbf61ac901eeea6e151670a10ce6878: Status 404 returned error can't find the container with id f0ab08d4f976838885b1bf5e6428a01cafbf61ac901eeea6e151670a10ce6878
Sep 09 06:15:48.310170 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:48.310138    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6b5d89/rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56" event={"ID":"31bac816-f890-4811-a1e0-51a721fee215","Type":"ContainerStarted","Data":"85823b58253c69834f3e57d65cd0c8edb9436c9fb27c49a56c1c43b52aaec728"}
Sep 09 06:15:48.310615 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:48.310187    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6b5d89/rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56" event={"ID":"31bac816-f890-4811-a1e0-51a721fee215","Type":"ContainerStarted","Data":"f0ab08d4f976838885b1bf5e6428a01cafbf61ac901eeea6e151670a10ce6878"}
Sep 09 06:15:48.327053 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:48.327008    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-6b5d89/rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56" podStartSLOduration=1.326992296 podStartE2EDuration="1.326992296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:15:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:15:48.326230506 +0000 UTC m=+1573.666718414" watchObservedRunningTime="2026-09-09 06:15:48.326992296 +0000 UTC m=+1573.667480205"
Sep 09 06:15:50.604695 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:50.604657    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:15:57.845331 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:57.845300    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-6b5d89/rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56"
Sep 09 06:15:57.845753 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:57.845368    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-6b5d89/rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56"
Sep 09 06:15:57.857514 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:57.857491    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-6b5d89/rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56"
Sep 09 06:15:58.369561 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:15:58.369534    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-6b5d89/rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56"
Sep 09 06:16:13.093956 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:16:13.093920    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:16:29.295024 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:16:29.294956    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:16:35.493705 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:16:35.493678    2579 scope.go:117] "RemoveContainer" containerID="cf6f1ae0dc04e945c85c1d00407c0a375e2d190994a722c3cf00da002bf4b132"
Sep 09 06:16:40.001272 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:16:40.001237    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:16:45.065739 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:16:45.065704    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6b5d89/rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56"]
Sep 09 06:16:51.890968 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:16:51.890932    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8a6cf7-bdf474ddb-85l7f"]
Sep 09 06:16:51.894289 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:16:51.894274    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a6cf7-bdf474ddb-85l7f"
Sep 09 06:16:51.899199 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:16:51.899169    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8a6cf7-dockercfg-2hx6j\""
Sep 09 06:16:51.899333 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:16:51.899231    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8a6cf7\""
Sep 09 06:16:51.911925 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:16:51.911900    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8a6cf7-bdf474ddb-85l7f"]
Sep 09 06:16:51.920264 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:16:51.920241    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxpfv\" (UniqueName: \"kubernetes.io/projected/112458ce-0a56-4371-8b93-c98a3693f25c-kube-api-access-wxpfv\") pod \"payload-pre-processing-e2e-ipp-a-8a6cf7-bdf474ddb-85l7f\" (UID: \"112458ce-0a56-4371-8b93-c98a3693f25c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a6cf7-bdf474ddb-85l7f"
Sep 09 06:16:51.920356 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:16:51.920297    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/112458ce-0a56-4371-8b93-c98a3693f25c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8a6cf7-bdf474ddb-85l7f\" (UID: \"112458ce-0a56-4371-8b93-c98a3693f25c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a6cf7-bdf474ddb-85l7f"
Sep 09 06:16:52.021248 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:16:52.021215    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/112458ce-0a56-4371-8b93-c98a3693f25c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8a6cf7-bdf474ddb-85l7f\" (UID: \"112458ce-0a56-4371-8b93-c98a3693f25c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a6cf7-bdf474ddb-85l7f"
Sep 09 06:16:52.021398 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:16:52.021358    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxpfv\" (UniqueName: \"kubernetes.io/projected/112458ce-0a56-4371-8b93-c98a3693f25c-kube-api-access-wxpfv\") pod \"payload-pre-processing-e2e-ipp-a-8a6cf7-bdf474ddb-85l7f\" (UID: \"112458ce-0a56-4371-8b93-c98a3693f25c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a6cf7-bdf474ddb-85l7f"
Sep 09 06:16:52.021895 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:16:52.021879    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/112458ce-0a56-4371-8b93-c98a3693f25c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8a6cf7-bdf474ddb-85l7f\" (UID: \"112458ce-0a56-4371-8b93-c98a3693f25c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a6cf7-bdf474ddb-85l7f"
Sep 09 06:16:52.028919 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:16:52.028894    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxpfv\" (UniqueName: \"kubernetes.io/projected/112458ce-0a56-4371-8b93-c98a3693f25c-kube-api-access-wxpfv\") pod \"payload-pre-processing-e2e-ipp-a-8a6cf7-bdf474ddb-85l7f\" (UID: \"112458ce-0a56-4371-8b93-c98a3693f25c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a6cf7-bdf474ddb-85l7f"
Sep 09 06:16:52.206163 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:16:52.206101    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a6cf7-bdf474ddb-85l7f"
Sep 09 06:16:52.476370 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:16:52.476347    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8a6cf7-bdf474ddb-85l7f"]
Sep 09 06:16:52.478861 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:16:52.478833    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod112458ce_0a56_4371_8b93_c98a3693f25c.slice/crio-40b8cc682418ce34d568d4a3af160a2fab8f211d13557d7d3056bfd0d3d0d31f WatchSource:0}: Error finding container 40b8cc682418ce34d568d4a3af160a2fab8f211d13557d7d3056bfd0d3d0d31f: Status 404 returned error can't find the container with id 40b8cc682418ce34d568d4a3af160a2fab8f211d13557d7d3056bfd0d3d0d31f
Sep 09 06:16:52.606110 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:16:52.606084    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a6cf7-bdf474ddb-85l7f" event={"ID":"112458ce-0a56-4371-8b93-c98a3693f25c","Type":"ContainerStarted","Data":"2cacd68ef473ceb24ceb12ec9c604816e80a00aeca4f6a9fb111883625f30d12"}
Sep 09 06:16:52.606254 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:16:52.606129    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a6cf7-bdf474ddb-85l7f" event={"ID":"112458ce-0a56-4371-8b93-c98a3693f25c","Type":"ContainerStarted","Data":"40b8cc682418ce34d568d4a3af160a2fab8f211d13557d7d3056bfd0d3d0d31f"}
Sep 09 06:16:52.606254 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:16:52.606177    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a6cf7-bdf474ddb-85l7f"
Sep 09 06:16:52.623787 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:16:52.623742    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a6cf7-bdf474ddb-85l7f" podStartSLOduration=1.623729103 podStartE2EDuration="1.623729103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:16:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:16:52.620877496 +0000 UTC m=+1637.961365406" watchObservedRunningTime="2026-09-09 06:16:52.623729103 +0000 UTC m=+1637.964217045"
Sep 09 06:16:54.199914 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:16:54.199873    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:17:00.085491 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:00.085452    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-6b5d89/rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56" podUID="31bac816-f890-4811-a1e0-51a721fee215" containerName="main" containerID="cri-o://85823b58253c69834f3e57d65cd0c8edb9436c9fb27c49a56c1c43b52aaec728" gracePeriod=45
Sep 09 06:17:00.335601 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:00.335546    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6b5d89/rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56"
Sep 09 06:17:00.379297 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:00.379267    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31bac816-f890-4811-a1e0-51a721fee215-model-cache\") pod \"31bac816-f890-4811-a1e0-51a721fee215\" (UID: \"31bac816-f890-4811-a1e0-51a721fee215\") "
Sep 09 06:17:00.379444 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:00.379308    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31bac816-f890-4811-a1e0-51a721fee215-tmp-dir\") pod \"31bac816-f890-4811-a1e0-51a721fee215\" (UID: \"31bac816-f890-4811-a1e0-51a721fee215\") "
Sep 09 06:17:00.379444 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:00.379342    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31bac816-f890-4811-a1e0-51a721fee215-tls-certs\") pod \"31bac816-f890-4811-a1e0-51a721fee215\" (UID: \"31bac816-f890-4811-a1e0-51a721fee215\") "
Sep 09 06:17:00.379444 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:00.379377    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65g6v\" (UniqueName: \"kubernetes.io/projected/31bac816-f890-4811-a1e0-51a721fee215-kube-api-access-65g6v\") pod \"31bac816-f890-4811-a1e0-51a721fee215\" (UID: \"31bac816-f890-4811-a1e0-51a721fee215\") "
Sep 09 06:17:00.379620 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:00.379450    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31bac816-f890-4811-a1e0-51a721fee215-dshm\") pod \"31bac816-f890-4811-a1e0-51a721fee215\" (UID: \"31bac816-f890-4811-a1e0-51a721fee215\") "
Sep 09 06:17:00.379620 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:00.379472    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31bac816-f890-4811-a1e0-51a721fee215-home\") pod \"31bac816-f890-4811-a1e0-51a721fee215\" (UID: \"31bac816-f890-4811-a1e0-51a721fee215\") "
Sep 09 06:17:00.379620 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:00.379571    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31bac816-f890-4811-a1e0-51a721fee215-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "31bac816-f890-4811-a1e0-51a721fee215" (UID: "31bac816-f890-4811-a1e0-51a721fee215"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:17:00.379620 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:00.379583    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31bac816-f890-4811-a1e0-51a721fee215-model-cache" (OuterVolumeSpecName: "model-cache") pod "31bac816-f890-4811-a1e0-51a721fee215" (UID: "31bac816-f890-4811-a1e0-51a721fee215"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:17:00.379853 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:00.379817    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31bac816-f890-4811-a1e0-51a721fee215-model-cache\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:17:00.379853 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:00.379829    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31bac816-f890-4811-a1e0-51a721fee215-home" (OuterVolumeSpecName: "home") pod "31bac816-f890-4811-a1e0-51a721fee215" (UID: "31bac816-f890-4811-a1e0-51a721fee215"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:17:00.379853 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:00.379837    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31bac816-f890-4811-a1e0-51a721fee215-tmp-dir\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:17:00.381781 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:00.381751    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31bac816-f890-4811-a1e0-51a721fee215-kube-api-access-65g6v" (OuterVolumeSpecName: "kube-api-access-65g6v") pod "31bac816-f890-4811-a1e0-51a721fee215" (UID: "31bac816-f890-4811-a1e0-51a721fee215"). InnerVolumeSpecName "kube-api-access-65g6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:17:00.382189 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:00.382159    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31bac816-f890-4811-a1e0-51a721fee215-dshm" (OuterVolumeSpecName: "dshm") pod "31bac816-f890-4811-a1e0-51a721fee215" (UID: "31bac816-f890-4811-a1e0-51a721fee215"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:17:00.382189 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:00.382168    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31bac816-f890-4811-a1e0-51a721fee215-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "31bac816-f890-4811-a1e0-51a721fee215" (UID: "31bac816-f890-4811-a1e0-51a721fee215"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:17:00.480535 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:00.480511    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31bac816-f890-4811-a1e0-51a721fee215-tls-certs\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:17:00.480535 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:00.480531    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65g6v\" (UniqueName: \"kubernetes.io/projected/31bac816-f890-4811-a1e0-51a721fee215-kube-api-access-65g6v\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:17:00.480712 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:00.480540    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31bac816-f890-4811-a1e0-51a721fee215-dshm\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:17:00.480712 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:00.480548    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31bac816-f890-4811-a1e0-51a721fee215-home\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:17:00.643954 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:00.643887    2579 generic.go:356] "Generic (PLEG): container finished" podID="31bac816-f890-4811-a1e0-51a721fee215" containerID="85823b58253c69834f3e57d65cd0c8edb9436c9fb27c49a56c1c43b52aaec728" exitCode=0
Sep 09 06:17:00.643954 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:00.643929    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6b5d89/rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56"
Sep 09 06:17:00.644096 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:00.643979    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6b5d89/rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56" event={"ID":"31bac816-f890-4811-a1e0-51a721fee215","Type":"ContainerDied","Data":"85823b58253c69834f3e57d65cd0c8edb9436c9fb27c49a56c1c43b52aaec728"}
Sep 09 06:17:00.644096 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:00.644019    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6b5d89/rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56" event={"ID":"31bac816-f890-4811-a1e0-51a721fee215","Type":"ContainerDied","Data":"f0ab08d4f976838885b1bf5e6428a01cafbf61ac901eeea6e151670a10ce6878"}
Sep 09 06:17:00.644096 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:00.644039    2579 scope.go:117] "RemoveContainer" containerID="85823b58253c69834f3e57d65cd0c8edb9436c9fb27c49a56c1c43b52aaec728"
Sep 09 06:17:00.652851 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:00.652834    2579 scope.go:117] "RemoveContainer" containerID="85823b58253c69834f3e57d65cd0c8edb9436c9fb27c49a56c1c43b52aaec728"
Sep 09 06:17:00.653130 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:17:00.653111    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85823b58253c69834f3e57d65cd0c8edb9436c9fb27c49a56c1c43b52aaec728\": container with ID starting with 85823b58253c69834f3e57d65cd0c8edb9436c9fb27c49a56c1c43b52aaec728 not found: ID does not exist" containerID="85823b58253c69834f3e57d65cd0c8edb9436c9fb27c49a56c1c43b52aaec728"
Sep 09 06:17:00.653190 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:00.653139    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85823b58253c69834f3e57d65cd0c8edb9436c9fb27c49a56c1c43b52aaec728"} err="failed to get container status \"85823b58253c69834f3e57d65cd0c8edb9436c9fb27c49a56c1c43b52aaec728\": rpc error: code = NotFound desc = could not find container \"85823b58253c69834f3e57d65cd0c8edb9436c9fb27c49a56c1c43b52aaec728\": container with ID starting with 85823b58253c69834f3e57d65cd0c8edb9436c9fb27c49a56c1c43b52aaec728 not found: ID does not exist"
Sep 09 06:17:00.663347 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:00.663326    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6b5d89/rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56"]
Sep 09 06:17:00.666645 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:00.666610    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6b5d89/rate-test-model-6b5d89-kserve-6d86fbffbc-7ls56"]
Sep 09 06:17:01.147943 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:01.147910    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31bac816-f890-4811-a1e0-51a721fee215" path="/var/lib/kubelet/pods/31bac816-f890-4811-a1e0-51a721fee215/volumes"
Sep 09 06:17:03.612243 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:03.612219    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a6cf7-bdf474ddb-85l7f"
Sep 09 06:17:15.862002 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:15.861968    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"]
Sep 09 06:17:15.862440 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:15.862259    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31bac816-f890-4811-a1e0-51a721fee215" containerName="main"
Sep 09 06:17:15.862440 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:15.862269    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="31bac816-f890-4811-a1e0-51a721fee215" containerName="main"
Sep 09 06:17:15.862440 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:15.862330    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="31bac816-f890-4811-a1e0-51a721fee215" containerName="main"
Sep 09 06:17:15.865173 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:15.865155    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"
Sep 09 06:17:15.867534 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:15.867514    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-32ea4d-openshift-default-dockercfg-8rfvj\""
Sep 09 06:17:15.875592 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:15.875572    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"]
Sep 09 06:17:15.892097 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:15.892074    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ec6702-7142-4ff6-bba5-cf1969ac2a46-workload-socket\") pod \"e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq\" (UID: \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\") " pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"
Sep 09 06:17:15.892192 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:15.892141    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34ec6702-7142-4ff6-bba5-cf1969ac2a46-workload-certs\") pod \"e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq\" (UID: \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\") " pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"
Sep 09 06:17:15.892192 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:15.892175    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ec6702-7142-4ff6-bba5-cf1969ac2a46-credential-socket\") pod \"e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq\" (UID: \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\") " pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"
Sep 09 06:17:15.892279 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:15.892203    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34ec6702-7142-4ff6-bba5-cf1969ac2a46-istio-data\") pod \"e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq\" (UID: \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\") " pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"
Sep 09 06:17:15.892324 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:15.892286    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjjqx\" (UniqueName: \"kubernetes.io/projected/34ec6702-7142-4ff6-bba5-cf1969ac2a46-kube-api-access-rjjqx\") pod \"e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq\" (UID: \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\") " pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"
Sep 09 06:17:15.892382 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:15.892326    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34ec6702-7142-4ff6-bba5-cf1969ac2a46-istio-token\") pod \"e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq\" (UID: \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\") " pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"
Sep 09 06:17:15.892461 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:15.892438    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34ec6702-7142-4ff6-bba5-cf1969ac2a46-istiod-ca-cert\") pod \"e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq\" (UID: \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\") " pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"
Sep 09 06:17:15.892527 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:15.892511    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34ec6702-7142-4ff6-bba5-cf1969ac2a46-istio-envoy\") pod \"e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq\" (UID: \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\") " pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"
Sep 09 06:17:15.892597 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:15.892582    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34ec6702-7142-4ff6-bba5-cf1969ac2a46-istio-podinfo\") pod \"e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq\" (UID: \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\") " pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"
Sep 09 06:17:15.993556 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:15.993526    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34ec6702-7142-4ff6-bba5-cf1969ac2a46-istio-data\") pod \"e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq\" (UID: \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\") " pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"
Sep 09 06:17:15.993718 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:15.993570    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjjqx\" (UniqueName: \"kubernetes.io/projected/34ec6702-7142-4ff6-bba5-cf1969ac2a46-kube-api-access-rjjqx\") pod \"e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq\" (UID: \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\") " pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"
Sep 09 06:17:15.993718 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:15.993588    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34ec6702-7142-4ff6-bba5-cf1969ac2a46-istio-token\") pod \"e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq\" (UID: \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\") " pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"
Sep 09 06:17:15.993718 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:15.993620    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34ec6702-7142-4ff6-bba5-cf1969ac2a46-istiod-ca-cert\") pod \"e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq\" (UID: \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\") " pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"
Sep 09 06:17:15.993718 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:15.993665    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34ec6702-7142-4ff6-bba5-cf1969ac2a46-istio-envoy\") pod \"e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq\" (UID: \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\") " pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"
Sep 09 06:17:15.993718 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:15.993702    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34ec6702-7142-4ff6-bba5-cf1969ac2a46-istio-podinfo\") pod \"e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq\" (UID: \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\") " pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"
Sep 09 06:17:15.994007 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:15.993844    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ec6702-7142-4ff6-bba5-cf1969ac2a46-workload-socket\") pod \"e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq\" (UID: \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\") " pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"
Sep 09 06:17:15.994007 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:15.993906    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34ec6702-7142-4ff6-bba5-cf1969ac2a46-workload-certs\") pod \"e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq\" (UID: \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\") " pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"
Sep 09 06:17:15.994007 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:15.993948    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ec6702-7142-4ff6-bba5-cf1969ac2a46-credential-socket\") pod \"e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq\" (UID: \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\") " pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"
Sep 09 06:17:15.994170 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:15.994008    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34ec6702-7142-4ff6-bba5-cf1969ac2a46-istio-data\") pod \"e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq\" (UID: \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\") " pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"
Sep 09 06:17:15.994231 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:15.994210    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ec6702-7142-4ff6-bba5-cf1969ac2a46-workload-socket\") pod \"e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq\" (UID: \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\") " pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"
Sep 09 06:17:15.994231 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:15.994221    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34ec6702-7142-4ff6-bba5-cf1969ac2a46-workload-certs\") pod \"e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq\" (UID: \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\") " pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"
Sep 09 06:17:15.994348 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:15.994253    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ec6702-7142-4ff6-bba5-cf1969ac2a46-credential-socket\") pod \"e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq\" (UID: \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\") " pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"
Sep 09 06:17:15.994407 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:15.994392    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34ec6702-7142-4ff6-bba5-cf1969ac2a46-istiod-ca-cert\") pod \"e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq\" (UID: \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\") " pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"
Sep 09 06:17:15.996050 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:15.996029    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34ec6702-7142-4ff6-bba5-cf1969ac2a46-istio-envoy\") pod \"e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq\" (UID: \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\") " pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"
Sep 09 06:17:15.996365 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:15.996346    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34ec6702-7142-4ff6-bba5-cf1969ac2a46-istio-podinfo\") pod \"e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq\" (UID: \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\") " pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"
Sep 09 06:17:16.001255 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:16.001226    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34ec6702-7142-4ff6-bba5-cf1969ac2a46-istio-token\") pod \"e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq\" (UID: \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\") " pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"
Sep 09 06:17:16.001414 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:16.001400    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjjqx\" (UniqueName: \"kubernetes.io/projected/34ec6702-7142-4ff6-bba5-cf1969ac2a46-kube-api-access-rjjqx\") pod \"e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq\" (UID: \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\") " pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"
Sep 09 06:17:16.176752 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:16.176677    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"
Sep 09 06:17:16.451833 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:16.451812    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"]
Sep 09 06:17:16.453879 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:17:16.453848    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34ec6702_7142_4ff6_bba5_cf1969ac2a46.slice/crio-e48d774c581814411d1cd3a54cd02edf3296a2929d610a475f0be27655d097ff WatchSource:0}: Error finding container e48d774c581814411d1cd3a54cd02edf3296a2929d610a475f0be27655d097ff: Status 404 returned error can't find the container with id e48d774c581814411d1cd3a54cd02edf3296a2929d610a475f0be27655d097ff
Sep 09 06:17:16.456300 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:16.456265    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Sep 09 06:17:16.456372 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:16.456333    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Sep 09 06:17:16.456372 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:16.456360    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Sep 09 06:17:16.728388 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:16.728305    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq" event={"ID":"34ec6702-7142-4ff6-bba5-cf1969ac2a46","Type":"ContainerStarted","Data":"4dad64755eeb62c362b115b033143321f147ac76498f6e8e856d9e5e79a542d6"}
Sep 09 06:17:16.728388 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:16.728355    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq" event={"ID":"34ec6702-7142-4ff6-bba5-cf1969ac2a46","Type":"ContainerStarted","Data":"e48d774c581814411d1cd3a54cd02edf3296a2929d610a475f0be27655d097ff"}
Sep 09 06:17:16.749789 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:16.749744    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq" podStartSLOduration=1.749729221 podStartE2EDuration="1.749729221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:17:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:17:16.747650945 +0000 UTC m=+1662.088138858" watchObservedRunningTime="2026-09-09 06:17:16.749729221 +0000 UTC m=+1662.090217129"
Sep 09 06:17:17.177325 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:17.177293    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"
Sep 09 06:17:17.182658 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:17.182621    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"
Sep 09 06:17:17.733338 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:17.733310    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"
Sep 09 06:17:17.734194 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:17.734178    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"
Sep 09 06:17:20.374373 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:20.374337    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-32ea4d-7cc4848ff4-j4fwv"]
Sep 09 06:17:20.377816 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:20.377799    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32ea4d-7cc4848ff4-j4fwv"
Sep 09 06:17:20.380462 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:20.380438    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-32ea4d-dockercfg-cpcjh\""
Sep 09 06:17:20.380462 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:20.380457    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-32ea4d\""
Sep 09 06:17:20.388346 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:20.388325    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-32ea4d-7cc4848ff4-j4fwv"]
Sep 09 06:17:20.433310 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:20.433284    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/911c3161-466c-48f9-af08-0e07138953f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-32ea4d-7cc4848ff4-j4fwv\" (UID: \"911c3161-466c-48f9-af08-0e07138953f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32ea4d-7cc4848ff4-j4fwv"
Sep 09 06:17:20.433425 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:20.433316    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8xn7\" (UniqueName: \"kubernetes.io/projected/911c3161-466c-48f9-af08-0e07138953f1-kube-api-access-v8xn7\") pod \"payload-pre-processing-e2e-ipp-b-32ea4d-7cc4848ff4-j4fwv\" (UID: \"911c3161-466c-48f9-af08-0e07138953f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32ea4d-7cc4848ff4-j4fwv"
Sep 09 06:17:20.533922 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:20.533890    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8xn7\" (UniqueName: \"kubernetes.io/projected/911c3161-466c-48f9-af08-0e07138953f1-kube-api-access-v8xn7\") pod \"payload-pre-processing-e2e-ipp-b-32ea4d-7cc4848ff4-j4fwv\" (UID: \"911c3161-466c-48f9-af08-0e07138953f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32ea4d-7cc4848ff4-j4fwv"
Sep 09 06:17:20.534088 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:20.533995    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/911c3161-466c-48f9-af08-0e07138953f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-32ea4d-7cc4848ff4-j4fwv\" (UID: \"911c3161-466c-48f9-af08-0e07138953f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32ea4d-7cc4848ff4-j4fwv"
Sep 09 06:17:20.534701 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:20.534683    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/911c3161-466c-48f9-af08-0e07138953f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-32ea4d-7cc4848ff4-j4fwv\" (UID: \"911c3161-466c-48f9-af08-0e07138953f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32ea4d-7cc4848ff4-j4fwv"
Sep 09 06:17:20.541331 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:20.541301    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8xn7\" (UniqueName: \"kubernetes.io/projected/911c3161-466c-48f9-af08-0e07138953f1-kube-api-access-v8xn7\") pod \"payload-pre-processing-e2e-ipp-b-32ea4d-7cc4848ff4-j4fwv\" (UID: \"911c3161-466c-48f9-af08-0e07138953f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32ea4d-7cc4848ff4-j4fwv"
Sep 09 06:17:20.689504 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:20.689437    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32ea4d-7cc4848ff4-j4fwv"
Sep 09 06:17:20.961155 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:20.961135    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-32ea4d-7cc4848ff4-j4fwv"]
Sep 09 06:17:20.963250 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:17:20.963222    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod911c3161_466c_48f9_af08_0e07138953f1.slice/crio-028b0a7771c620c87d692af2079333e88ae8f4523a0bf87c5c04f887370defcc WatchSource:0}: Error finding container 028b0a7771c620c87d692af2079333e88ae8f4523a0bf87c5c04f887370defcc: Status 404 returned error can't find the container with id 028b0a7771c620c87d692af2079333e88ae8f4523a0bf87c5c04f887370defcc
Sep 09 06:17:21.754932 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:21.754900    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32ea4d-7cc4848ff4-j4fwv" event={"ID":"911c3161-466c-48f9-af08-0e07138953f1","Type":"ContainerStarted","Data":"b345d2761e62ed0165a86d5d077ec2f394f2da0f637020ce51333edcd48fee40"}
Sep 09 06:17:21.755337 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:21.755048    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32ea4d-7cc4848ff4-j4fwv"
Sep 09 06:17:21.755337 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:21.755080    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32ea4d-7cc4848ff4-j4fwv" event={"ID":"911c3161-466c-48f9-af08-0e07138953f1","Type":"ContainerStarted","Data":"028b0a7771c620c87d692af2079333e88ae8f4523a0bf87c5c04f887370defcc"}
Sep 09 06:17:21.769479 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:21.769309    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32ea4d-7cc4848ff4-j4fwv" podStartSLOduration=1.76929225 podStartE2EDuration="1.76929225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:17:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:17:21.769015693 +0000 UTC m=+1667.109503593" watchObservedRunningTime="2026-09-09 06:17:21.76929225 +0000 UTC m=+1667.109780159"
Sep 09 06:17:23.700125 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:23.700087    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:17:32.762497 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:32.762464    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32ea4d-7cc4848ff4-j4fwv"
Sep 09 06:17:33.775787 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:33.775747    2579 patch_prober.go:28] interesting pod/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 06:17:33.776227 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:33.775824    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq" podUID="34ec6702-7142-4ff6-bba5-cf1969ac2a46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 06:17:34.821525 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:34.821491    2579 generic.go:356] "Generic (PLEG): container finished" podID="34ec6702-7142-4ff6-bba5-cf1969ac2a46" containerID="4dad64755eeb62c362b115b033143321f147ac76498f6e8e856d9e5e79a542d6" exitCode=0
Sep 09 06:17:34.821968 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:34.821563    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq" event={"ID":"34ec6702-7142-4ff6-bba5-cf1969ac2a46","Type":"ContainerDied","Data":"4dad64755eeb62c362b115b033143321f147ac76498f6e8e856d9e5e79a542d6"}
Sep 09 06:17:34.821968 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:34.821926    2579 scope.go:117] "RemoveContainer" containerID="4dad64755eeb62c362b115b033143321f147ac76498f6e8e856d9e5e79a542d6"
Sep 09 06:17:34.822986 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:34.822962    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Sep 09 06:17:34.823041 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:34.823028    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Sep 09 06:17:34.823082 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:34.823058    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Sep 09 06:17:35.832710 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:35.832669    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq" event={"ID":"34ec6702-7142-4ff6-bba5-cf1969ac2a46","Type":"ContainerStarted","Data":"08780f5d4e945de992eecb63e1999e3498984567e1f735c66d767ec039fa7202"}
Sep 09 06:17:36.177296 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:36.177208    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"
Sep 09 06:17:36.177443 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:36.177401    2579 patch_prober.go:28] interesting pod/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Sep 09 06:17:36.177488 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:36.177445    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq" podUID="34ec6702-7142-4ff6-bba5-cf1969ac2a46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Sep 09 06:17:37.178066 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:37.178022    2579 patch_prober.go:28] interesting pod/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Sep 09 06:17:37.178543 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:37.178085    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq" podUID="34ec6702-7142-4ff6-bba5-cf1969ac2a46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Sep 09 06:17:38.177512 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:38.177471    2579 patch_prober.go:28] interesting pod/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Sep 09 06:17:38.177735 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:38.177535    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq" podUID="34ec6702-7142-4ff6-bba5-cf1969ac2a46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Sep 09 06:17:39.177789 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:39.177756    2579 patch_prober.go:28] interesting pod/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Sep 09 06:17:39.178179 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:39.177807    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq" podUID="34ec6702-7142-4ff6-bba5-cf1969ac2a46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Sep 09 06:17:40.177830 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:40.177795    2579 patch_prober.go:28] interesting pod/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Sep 09 06:17:40.178208 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:40.177851    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq" podUID="34ec6702-7142-4ff6-bba5-cf1969ac2a46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Sep 09 06:17:41.181098 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:41.181069    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"
Sep 09 06:17:41.181560 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:41.181345    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"
Sep 09 06:17:41.182175 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:41.182158    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"
Sep 09 06:17:57.433154 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:57.433073    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-8a6cf7/ipp-route-8a6cf7-kserve-56b86956c5-lnhfq"]
Sep 09 06:17:57.436849 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:57.436827    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8a6cf7/ipp-route-8a6cf7-kserve-56b86956c5-lnhfq"
Sep 09 06:17:57.440188 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:57.440159    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-8a6cf7\"/\"default-dockercfg-7nbzx\""
Sep 09 06:17:57.440188 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:57.440173    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-8a6cf7\"/\"openshift-service-ca.crt\""
Sep 09 06:17:57.440365 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:57.440166    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-8a6cf7\"/\"ipp-route-8a6cf7-kserve-self-signed-certs\""
Sep 09 06:17:57.440365 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:57.440243    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-8a6cf7\"/\"kube-root-ca.crt\""
Sep 09 06:17:57.447234 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:57.447208    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-8a6cf7/ipp-route-8a6cf7-kserve-56b86956c5-lnhfq"]
Sep 09 06:17:57.468768 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:57.468744    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05f94a7b-65e7-4157-8fc0-1da60261ba70-model-cache\") pod \"ipp-route-8a6cf7-kserve-56b86956c5-lnhfq\" (UID: \"05f94a7b-65e7-4157-8fc0-1da60261ba70\") " pod="ai-tenant-e2e-ipp-a-8a6cf7/ipp-route-8a6cf7-kserve-56b86956c5-lnhfq"
Sep 09 06:17:57.468859 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:57.468792    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05f94a7b-65e7-4157-8fc0-1da60261ba70-home\") pod \"ipp-route-8a6cf7-kserve-56b86956c5-lnhfq\" (UID: \"05f94a7b-65e7-4157-8fc0-1da60261ba70\") " pod="ai-tenant-e2e-ipp-a-8a6cf7/ipp-route-8a6cf7-kserve-56b86956c5-lnhfq"
Sep 09 06:17:57.468859 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:57.468808    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05f94a7b-65e7-4157-8fc0-1da60261ba70-dshm\") pod \"ipp-route-8a6cf7-kserve-56b86956c5-lnhfq\" (UID: \"05f94a7b-65e7-4157-8fc0-1da60261ba70\") " pod="ai-tenant-e2e-ipp-a-8a6cf7/ipp-route-8a6cf7-kserve-56b86956c5-lnhfq"
Sep 09 06:17:57.468859 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:57.468823    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdbwp\" (UniqueName: \"kubernetes.io/projected/05f94a7b-65e7-4157-8fc0-1da60261ba70-kube-api-access-rdbwp\") pod \"ipp-route-8a6cf7-kserve-56b86956c5-lnhfq\" (UID: \"05f94a7b-65e7-4157-8fc0-1da60261ba70\") " pod="ai-tenant-e2e-ipp-a-8a6cf7/ipp-route-8a6cf7-kserve-56b86956c5-lnhfq"
Sep 09 06:17:57.468967 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:57.468873    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05f94a7b-65e7-4157-8fc0-1da60261ba70-tmp-dir\") pod \"ipp-route-8a6cf7-kserve-56b86956c5-lnhfq\" (UID: \"05f94a7b-65e7-4157-8fc0-1da60261ba70\") " pod="ai-tenant-e2e-ipp-a-8a6cf7/ipp-route-8a6cf7-kserve-56b86956c5-lnhfq"
Sep 09 06:17:57.468967 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:57.468913    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05f94a7b-65e7-4157-8fc0-1da60261ba70-tls-certs\") pod \"ipp-route-8a6cf7-kserve-56b86956c5-lnhfq\" (UID: \"05f94a7b-65e7-4157-8fc0-1da60261ba70\") " pod="ai-tenant-e2e-ipp-a-8a6cf7/ipp-route-8a6cf7-kserve-56b86956c5-lnhfq"
Sep 09 06:17:57.569982 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:57.569954    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05f94a7b-65e7-4157-8fc0-1da60261ba70-model-cache\") pod \"ipp-route-8a6cf7-kserve-56b86956c5-lnhfq\" (UID: \"05f94a7b-65e7-4157-8fc0-1da60261ba70\") " pod="ai-tenant-e2e-ipp-a-8a6cf7/ipp-route-8a6cf7-kserve-56b86956c5-lnhfq"
Sep 09 06:17:57.570115 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:57.570007    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05f94a7b-65e7-4157-8fc0-1da60261ba70-home\") pod \"ipp-route-8a6cf7-kserve-56b86956c5-lnhfq\" (UID: \"05f94a7b-65e7-4157-8fc0-1da60261ba70\") " pod="ai-tenant-e2e-ipp-a-8a6cf7/ipp-route-8a6cf7-kserve-56b86956c5-lnhfq"
Sep 09 06:17:57.570115 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:57.570024    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05f94a7b-65e7-4157-8fc0-1da60261ba70-dshm\") pod \"ipp-route-8a6cf7-kserve-56b86956c5-lnhfq\" (UID: \"05f94a7b-65e7-4157-8fc0-1da60261ba70\") " pod="ai-tenant-e2e-ipp-a-8a6cf7/ipp-route-8a6cf7-kserve-56b86956c5-lnhfq"
Sep 09 06:17:57.570214 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:57.570133    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdbwp\" (UniqueName: \"kubernetes.io/projected/05f94a7b-65e7-4157-8fc0-1da60261ba70-kube-api-access-rdbwp\") pod \"ipp-route-8a6cf7-kserve-56b86956c5-lnhfq\" (UID: \"05f94a7b-65e7-4157-8fc0-1da60261ba70\") " pod="ai-tenant-e2e-ipp-a-8a6cf7/ipp-route-8a6cf7-kserve-56b86956c5-lnhfq"
Sep 09 06:17:57.570214 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:57.570169    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05f94a7b-65e7-4157-8fc0-1da60261ba70-tmp-dir\") pod \"ipp-route-8a6cf7-kserve-56b86956c5-lnhfq\" (UID: \"05f94a7b-65e7-4157-8fc0-1da60261ba70\") " pod="ai-tenant-e2e-ipp-a-8a6cf7/ipp-route-8a6cf7-kserve-56b86956c5-lnhfq"
Sep 09 06:17:57.570319 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:57.570222    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05f94a7b-65e7-4157-8fc0-1da60261ba70-tls-certs\") pod \"ipp-route-8a6cf7-kserve-56b86956c5-lnhfq\" (UID: \"05f94a7b-65e7-4157-8fc0-1da60261ba70\") " pod="ai-tenant-e2e-ipp-a-8a6cf7/ipp-route-8a6cf7-kserve-56b86956c5-lnhfq"
Sep 09 06:17:57.570383 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:57.570323    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05f94a7b-65e7-4157-8fc0-1da60261ba70-home\") pod \"ipp-route-8a6cf7-kserve-56b86956c5-lnhfq\" (UID: \"05f94a7b-65e7-4157-8fc0-1da60261ba70\") " pod="ai-tenant-e2e-ipp-a-8a6cf7/ipp-route-8a6cf7-kserve-56b86956c5-lnhfq"
Sep 09 06:17:57.570464 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:57.570443    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05f94a7b-65e7-4157-8fc0-1da60261ba70-model-cache\") pod \"ipp-route-8a6cf7-kserve-56b86956c5-lnhfq\" (UID: \"05f94a7b-65e7-4157-8fc0-1da60261ba70\") " pod="ai-tenant-e2e-ipp-a-8a6cf7/ipp-route-8a6cf7-kserve-56b86956c5-lnhfq"
Sep 09 06:17:57.570528 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:57.570512    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05f94a7b-65e7-4157-8fc0-1da60261ba70-tmp-dir\") pod \"ipp-route-8a6cf7-kserve-56b86956c5-lnhfq\" (UID: \"05f94a7b-65e7-4157-8fc0-1da60261ba70\") " pod="ai-tenant-e2e-ipp-a-8a6cf7/ipp-route-8a6cf7-kserve-56b86956c5-lnhfq"
Sep 09 06:17:57.572421 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:57.572402    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05f94a7b-65e7-4157-8fc0-1da60261ba70-dshm\") pod \"ipp-route-8a6cf7-kserve-56b86956c5-lnhfq\" (UID: \"05f94a7b-65e7-4157-8fc0-1da60261ba70\") " pod="ai-tenant-e2e-ipp-a-8a6cf7/ipp-route-8a6cf7-kserve-56b86956c5-lnhfq"
Sep 09 06:17:57.572616 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:57.572600    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05f94a7b-65e7-4157-8fc0-1da60261ba70-tls-certs\") pod \"ipp-route-8a6cf7-kserve-56b86956c5-lnhfq\" (UID: \"05f94a7b-65e7-4157-8fc0-1da60261ba70\") " pod="ai-tenant-e2e-ipp-a-8a6cf7/ipp-route-8a6cf7-kserve-56b86956c5-lnhfq"
Sep 09 06:17:57.578081 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:57.578033    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdbwp\" (UniqueName: \"kubernetes.io/projected/05f94a7b-65e7-4157-8fc0-1da60261ba70-kube-api-access-rdbwp\") pod \"ipp-route-8a6cf7-kserve-56b86956c5-lnhfq\" (UID: \"05f94a7b-65e7-4157-8fc0-1da60261ba70\") " pod="ai-tenant-e2e-ipp-a-8a6cf7/ipp-route-8a6cf7-kserve-56b86956c5-lnhfq"
Sep 09 06:17:57.747768 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:57.747680    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8a6cf7/ipp-route-8a6cf7-kserve-56b86956c5-lnhfq"
Sep 09 06:17:58.018122 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:58.018098    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-8a6cf7/ipp-route-8a6cf7-kserve-56b86956c5-lnhfq"]
Sep 09 06:17:58.020109 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:17:58.020078    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05f94a7b_65e7_4157_8fc0_1da60261ba70.slice/crio-af10cdd4fb8d8c10084ac4d59c33d1df9b67f7e0046904fcb4e3711490259141 WatchSource:0}: Error finding container af10cdd4fb8d8c10084ac4d59c33d1df9b67f7e0046904fcb4e3711490259141: Status 404 returned error can't find the container with id af10cdd4fb8d8c10084ac4d59c33d1df9b67f7e0046904fcb4e3711490259141
Sep 09 06:17:58.945978 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:58.945947    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8a6cf7/ipp-route-8a6cf7-kserve-56b86956c5-lnhfq" event={"ID":"05f94a7b-65e7-4157-8fc0-1da60261ba70","Type":"ContainerStarted","Data":"c5324edc9d17828c56cd4637870e0dff2f35b3085ec705338209231c1d510bc5"}
Sep 09 06:17:58.946360 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:58.945985    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8a6cf7/ipp-route-8a6cf7-kserve-56b86956c5-lnhfq" event={"ID":"05f94a7b-65e7-4157-8fc0-1da60261ba70","Type":"ContainerStarted","Data":"af10cdd4fb8d8c10084ac4d59c33d1df9b67f7e0046904fcb4e3711490259141"}
Sep 09 06:17:58.963223 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:17:58.963178    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-8a6cf7/ipp-route-8a6cf7-kserve-56b86956c5-lnhfq" podStartSLOduration=1.9631632159999999 podStartE2EDuration="1.963163216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:17:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:17:58.961310607 +0000 UTC m=+1704.301798517" watchObservedRunningTime="2026-09-09 06:17:58.963163216 +0000 UTC m=+1704.303651124"
Sep 09 06:18:01.398570 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:01.398534    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:18:07.748669 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:07.748617    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-8a6cf7/ipp-route-8a6cf7-kserve-56b86956c5-lnhfq"
Sep 09 06:18:07.749165 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:07.748689    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-8a6cf7/ipp-route-8a6cf7-kserve-56b86956c5-lnhfq"
Sep 09 06:18:07.761380 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:07.761355    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-8a6cf7/ipp-route-8a6cf7-kserve-56b86956c5-lnhfq"
Sep 09 06:18:08.008150 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:08.008064    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-8a6cf7/ipp-route-8a6cf7-kserve-56b86956c5-lnhfq"
Sep 09 06:18:18.416167 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:18.416137    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:18:25.224399 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:25.224364    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"]
Sep 09 06:18:25.227789 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:25.227772    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"
Sep 09 06:18:25.231960 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:25.231938    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-2e28ff-openshift-default-dockercfg-tdb9d\""
Sep 09 06:18:25.250023 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:25.250001    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"]
Sep 09 06:18:25.316751 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:25.316721    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb4411cf-23fb-4ed7-86eb-ad1855e31364-istio-envoy\") pod \"e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz\" (UID: \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\") " pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"
Sep 09 06:18:25.316913 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:25.316759    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb4411cf-23fb-4ed7-86eb-ad1855e31364-credential-socket\") pod \"e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz\" (UID: \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\") " pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"
Sep 09 06:18:25.316913 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:25.316782    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxjww\" (UniqueName: \"kubernetes.io/projected/cb4411cf-23fb-4ed7-86eb-ad1855e31364-kube-api-access-zxjww\") pod \"e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz\" (UID: \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\") " pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"
Sep 09 06:18:25.316913 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:25.316837    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb4411cf-23fb-4ed7-86eb-ad1855e31364-istio-token\") pod \"e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz\" (UID: \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\") " pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"
Sep 09 06:18:25.316913 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:25.316906    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb4411cf-23fb-4ed7-86eb-ad1855e31364-istiod-ca-cert\") pod \"e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz\" (UID: \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\") " pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"
Sep 09 06:18:25.317088 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:25.316933    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb4411cf-23fb-4ed7-86eb-ad1855e31364-istio-podinfo\") pod \"e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz\" (UID: \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\") " pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"
Sep 09 06:18:25.317088 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:25.316961    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb4411cf-23fb-4ed7-86eb-ad1855e31364-istio-data\") pod \"e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz\" (UID: \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\") " pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"
Sep 09 06:18:25.317088 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:25.316992    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb4411cf-23fb-4ed7-86eb-ad1855e31364-workload-certs\") pod \"e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz\" (UID: \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\") " pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"
Sep 09 06:18:25.317088 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:25.317020    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb4411cf-23fb-4ed7-86eb-ad1855e31364-workload-socket\") pod \"e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz\" (UID: \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\") " pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"
Sep 09 06:18:25.417692 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:25.417620    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb4411cf-23fb-4ed7-86eb-ad1855e31364-istio-podinfo\") pod \"e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz\" (UID: \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\") " pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"
Sep 09 06:18:25.417863 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:25.417726    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb4411cf-23fb-4ed7-86eb-ad1855e31364-istio-data\") pod \"e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz\" (UID: \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\") " pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"
Sep 09 06:18:25.417863 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:25.417773    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb4411cf-23fb-4ed7-86eb-ad1855e31364-workload-certs\") pod \"e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz\" (UID: \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\") " pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"
Sep 09 06:18:25.417863 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:25.417800    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb4411cf-23fb-4ed7-86eb-ad1855e31364-workload-socket\") pod \"e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz\" (UID: \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\") " pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"
Sep 09 06:18:25.418025 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:25.417946    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb4411cf-23fb-4ed7-86eb-ad1855e31364-istio-envoy\") pod \"e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz\" (UID: \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\") " pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"
Sep 09 06:18:25.418025 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:25.418004    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb4411cf-23fb-4ed7-86eb-ad1855e31364-credential-socket\") pod \"e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz\" (UID: \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\") " pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"
Sep 09 06:18:25.418131 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:25.418042    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxjww\" (UniqueName: \"kubernetes.io/projected/cb4411cf-23fb-4ed7-86eb-ad1855e31364-kube-api-access-zxjww\") pod \"e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz\" (UID: \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\") " pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"
Sep 09 06:18:25.418131 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:25.418076    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb4411cf-23fb-4ed7-86eb-ad1855e31364-istio-token\") pod \"e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz\" (UID: \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\") " pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"
Sep 09 06:18:25.418240 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:25.418147    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb4411cf-23fb-4ed7-86eb-ad1855e31364-istiod-ca-cert\") pod \"e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz\" (UID: \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\") " pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"
Sep 09 06:18:25.418240 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:25.418150    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb4411cf-23fb-4ed7-86eb-ad1855e31364-istio-data\") pod \"e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz\" (UID: \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\") " pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"
Sep 09 06:18:25.418240 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:25.418189    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb4411cf-23fb-4ed7-86eb-ad1855e31364-workload-socket\") pod \"e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz\" (UID: \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\") " pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"
Sep 09 06:18:25.418373 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:25.418245    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb4411cf-23fb-4ed7-86eb-ad1855e31364-workload-certs\") pod \"e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz\" (UID: \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\") " pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"
Sep 09 06:18:25.418465 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:25.418439    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb4411cf-23fb-4ed7-86eb-ad1855e31364-credential-socket\") pod \"e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz\" (UID: \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\") " pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"
Sep 09 06:18:25.418752 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:25.418734    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb4411cf-23fb-4ed7-86eb-ad1855e31364-istiod-ca-cert\") pod \"e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz\" (UID: \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\") " pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"
Sep 09 06:18:25.420205 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:25.420185    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb4411cf-23fb-4ed7-86eb-ad1855e31364-istio-envoy\") pod \"e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz\" (UID: \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\") " pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"
Sep 09 06:18:25.420443 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:25.420425    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb4411cf-23fb-4ed7-86eb-ad1855e31364-istio-podinfo\") pod \"e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz\" (UID: \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\") " pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"
Sep 09 06:18:25.427151 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:25.427123    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb4411cf-23fb-4ed7-86eb-ad1855e31364-istio-token\") pod \"e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz\" (UID: \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\") " pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"
Sep 09 06:18:25.427260 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:25.427207    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxjww\" (UniqueName: \"kubernetes.io/projected/cb4411cf-23fb-4ed7-86eb-ad1855e31364-kube-api-access-zxjww\") pod \"e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz\" (UID: \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\") " pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"
Sep 09 06:18:25.537646 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:25.537545    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"
Sep 09 06:18:25.813892 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:25.813865    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"]
Sep 09 06:18:25.815377 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:18:25.815336    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb4411cf_23fb_4ed7_86eb_ad1855e31364.slice/crio-248d5c799f97cb260284b9f8d9f95af7fcc8c0cfc894a22859decbcfd48b8dab WatchSource:0}: Error finding container 248d5c799f97cb260284b9f8d9f95af7fcc8c0cfc894a22859decbcfd48b8dab: Status 404 returned error can't find the container with id 248d5c799f97cb260284b9f8d9f95af7fcc8c0cfc894a22859decbcfd48b8dab
Sep 09 06:18:25.817788 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:25.817750    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Sep 09 06:18:25.817870 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:25.817831    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Sep 09 06:18:25.817910 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:25.817872    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Sep 09 06:18:26.089854 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:26.089761    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz" event={"ID":"cb4411cf-23fb-4ed7-86eb-ad1855e31364","Type":"ContainerStarted","Data":"235b0a53920a60e06d730a7cd0a5d53e5073245cfa8ec8c953c9a6aa42f87279"}
Sep 09 06:18:26.089854 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:26.089821    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz" event={"ID":"cb4411cf-23fb-4ed7-86eb-ad1855e31364","Type":"ContainerStarted","Data":"248d5c799f97cb260284b9f8d9f95af7fcc8c0cfc894a22859decbcfd48b8dab"}
Sep 09 06:18:26.111860 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:26.111814    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz" podStartSLOduration=1.111797646 podStartE2EDuration="1.111797646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:18:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:18:26.110231892 +0000 UTC m=+1731.450719802" watchObservedRunningTime="2026-09-09 06:18:26.111797646 +0000 UTC m=+1731.452285554"
Sep 09 06:18:26.538717 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:26.538615    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"
Sep 09 06:18:26.543614 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:26.543592    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"
Sep 09 06:18:27.095127 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:27.095088    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"
Sep 09 06:18:27.096184 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:27.096165    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"
Sep 09 06:18:38.302255 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:38.302214    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:18:48.200158 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:48.200125    2579 generic.go:356] "Generic (PLEG): container finished" podID="cb4411cf-23fb-4ed7-86eb-ad1855e31364" containerID="235b0a53920a60e06d730a7cd0a5d53e5073245cfa8ec8c953c9a6aa42f87279" exitCode=0
Sep 09 06:18:48.200545 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:48.200199    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz" event={"ID":"cb4411cf-23fb-4ed7-86eb-ad1855e31364","Type":"ContainerDied","Data":"235b0a53920a60e06d730a7cd0a5d53e5073245cfa8ec8c953c9a6aa42f87279"}
Sep 09 06:18:48.200598 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:48.200555    2579 scope.go:117] "RemoveContainer" containerID="235b0a53920a60e06d730a7cd0a5d53e5073245cfa8ec8c953c9a6aa42f87279"
Sep 09 06:18:48.201718 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:48.201686    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Sep 09 06:18:48.201768 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:48.201757    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Sep 09 06:18:48.201804 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:48.201787    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Sep 09 06:18:49.206616 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:49.206581    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz" event={"ID":"cb4411cf-23fb-4ed7-86eb-ad1855e31364","Type":"ContainerStarted","Data":"972d905da2e28993c8292972a970832cb66d7e36080b31e4906c3837f0255d98"}
Sep 09 06:18:49.539088 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:49.538998    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"
Sep 09 06:18:49.539986 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:49.539401    2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Sep 09 06:18:49.540216 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:49.540191    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz" podUID="cb4411cf-23fb-4ed7-86eb-ad1855e31364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Sep 09 06:18:50.538098 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:50.538061    2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Sep 09 06:18:50.538700 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:50.538669    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz" podUID="cb4411cf-23fb-4ed7-86eb-ad1855e31364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Sep 09 06:18:51.539403 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:51.538964    2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Sep 09 06:18:51.539403 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:51.539037    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz" podUID="cb4411cf-23fb-4ed7-86eb-ad1855e31364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Sep 09 06:18:52.538712 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:52.538672    2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Sep 09 06:18:52.538919 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:52.538752    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz" podUID="cb4411cf-23fb-4ed7-86eb-ad1855e31364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Sep 09 06:18:53.538931 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:53.538887    2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Sep 09 06:18:53.539438 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:53.538953    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz" podUID="cb4411cf-23fb-4ed7-86eb-ad1855e31364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Sep 09 06:18:53.609026 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:53.608999    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e28ff-k444w"]
Sep 09 06:18:53.612361 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:53.612343    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e28ff-k444w"
Sep 09 06:18:53.614898 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:53.614878    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 06:18:53.614962 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:53.614878    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ksjk6\""
Sep 09 06:18:53.620181 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:53.620161    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e28ff-k444w"]
Sep 09 06:18:53.677866 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:53.677836    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e404e6e7-dc13-47d6-b3b3-e36242205f7f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-2e28ff-k444w\" (UID: \"e404e6e7-dc13-47d6-b3b3-e36242205f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e28ff-k444w"
Sep 09 06:18:53.779070 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:53.779032    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e404e6e7-dc13-47d6-b3b3-e36242205f7f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-2e28ff-k444w\" (UID: \"e404e6e7-dc13-47d6-b3b3-e36242205f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e28ff-k444w"
Sep 09 06:18:53.779684 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:53.779667    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e404e6e7-dc13-47d6-b3b3-e36242205f7f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-2e28ff-k444w\" (UID: \"e404e6e7-dc13-47d6-b3b3-e36242205f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e28ff-k444w"
Sep 09 06:18:53.924912 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:53.924869    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e28ff-k444w"
Sep 09 06:18:54.205370 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:54.205294    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e28ff-k444w"]
Sep 09 06:18:54.209815 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:18:54.209776    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode404e6e7_dc13_47d6_b3b3_e36242205f7f.slice/crio-42753b3bac3e45456dd66d5d18fa8dc9ac0b6840231d867091a92ed2dfc404f1 WatchSource:0}: Error finding container 42753b3bac3e45456dd66d5d18fa8dc9ac0b6840231d867091a92ed2dfc404f1: Status 404 returned error can't find the container with id 42753b3bac3e45456dd66d5d18fa8dc9ac0b6840231d867091a92ed2dfc404f1
Sep 09 06:18:54.242651 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:54.241652    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e28ff-k444w" event={"ID":"e404e6e7-dc13-47d6-b3b3-e36242205f7f","Type":"ContainerStarted","Data":"42753b3bac3e45456dd66d5d18fa8dc9ac0b6840231d867091a92ed2dfc404f1"}
Sep 09 06:18:54.613436 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:54.613391    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"
Sep 09 06:18:54.613914 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:54.613806    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"
Sep 09 06:18:54.614744 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:54.614723    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"
Sep 09 06:18:55.248741 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:55.248659    2579 generic.go:356] "Generic (PLEG): container finished" podID="e404e6e7-dc13-47d6-b3b3-e36242205f7f" containerID="d3e78c86b9b47ee1738cf74c1d108cd4d00c30bf04c97fce24bcf5bbe3b5bec3" exitCode=0
Sep 09 06:18:55.248894 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:55.248755    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e28ff-k444w" event={"ID":"e404e6e7-dc13-47d6-b3b3-e36242205f7f","Type":"ContainerDied","Data":"d3e78c86b9b47ee1738cf74c1d108cd4d00c30bf04c97fce24bcf5bbe3b5bec3"}
Sep 09 06:18:56.380959 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:56.380936    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e28ff-k444w"
Sep 09 06:18:56.505231 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:56.505160    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e404e6e7-dc13-47d6-b3b3-e36242205f7f-maas-api-service-ca\") pod \"e404e6e7-dc13-47d6-b3b3-e36242205f7f\" (UID: \"e404e6e7-dc13-47d6-b3b3-e36242205f7f\") "
Sep 09 06:18:56.505456 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:56.505433    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e404e6e7-dc13-47d6-b3b3-e36242205f7f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e404e6e7-dc13-47d6-b3b3-e36242205f7f" (UID: "e404e6e7-dc13-47d6-b3b3-e36242205f7f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:18:56.606641 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:56.606600    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e404e6e7-dc13-47d6-b3b3-e36242205f7f-maas-api-service-ca\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:18:57.259526 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:57.259498    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e28ff-k444w"
Sep 09 06:18:57.259685 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:57.259543    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e28ff-k444w" event={"ID":"e404e6e7-dc13-47d6-b3b3-e36242205f7f","Type":"ContainerDied","Data":"42753b3bac3e45456dd66d5d18fa8dc9ac0b6840231d867091a92ed2dfc404f1"}
Sep 09 06:18:57.259685 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:57.259587    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42753b3bac3e45456dd66d5d18fa8dc9ac0b6840231d867091a92ed2dfc404f1"
Sep 09 06:18:59.965015 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:59.964984    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e28ff-k444w"]
Sep 09 06:18:59.968554 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:18:59.968532    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e28ff-k444w"]
Sep 09 06:19:01.148406 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:01.148375    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e404e6e7-dc13-47d6-b3b3-e36242205f7f" path="/var/lib/kubelet/pods/e404e6e7-dc13-47d6-b3b3-e36242205f7f/volumes"
Sep 09 06:19:03.894124 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:03.894094    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:19:06.452734 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:06.452703    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"]
Sep 09 06:19:06.453144 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:06.452925    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz" podUID="cb4411cf-23fb-4ed7-86eb-ad1855e31364" containerName="istio-proxy" containerID="cri-o://972d905da2e28993c8292972a970832cb66d7e36080b31e4906c3837f0255d98" gracePeriod=30
Sep 09 06:19:09.615877 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:09.615844    2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 06:19:09.616333 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:09.615915    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz" podUID="cb4411cf-23fb-4ed7-86eb-ad1855e31364" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 06:19:09.998482 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:09.998404    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8a6cf7-q56tk"]
Sep 09 06:19:09.998813 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:09.998797    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e404e6e7-dc13-47d6-b3b3-e36242205f7f" containerName="revoke-keys"
Sep 09 06:19:09.998879 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:09.998815    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e404e6e7-dc13-47d6-b3b3-e36242205f7f" containerName="revoke-keys"
Sep 09 06:19:09.998916 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:09.998878    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e404e6e7-dc13-47d6-b3b3-e36242205f7f" containerName="revoke-keys"
Sep 09 06:19:10.001603 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:10.001587    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8a6cf7-q56tk"
Sep 09 06:19:10.004010 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:10.003989    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 06:19:10.004153 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:10.004021    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ksjk6\""
Sep 09 06:19:10.009367 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:10.009342    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8a6cf7-q56tk"]
Sep 09 06:19:10.118790 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:10.118760    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e78cc84d-57d1-4a40-8397-8108fbc1be35-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8a6cf7-q56tk\" (UID: \"e78cc84d-57d1-4a40-8397-8108fbc1be35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8a6cf7-q56tk"
Sep 09 06:19:10.219843 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:10.219815    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e78cc84d-57d1-4a40-8397-8108fbc1be35-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8a6cf7-q56tk\" (UID: \"e78cc84d-57d1-4a40-8397-8108fbc1be35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8a6cf7-q56tk"
Sep 09 06:19:10.220455 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:10.220437    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e78cc84d-57d1-4a40-8397-8108fbc1be35-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8a6cf7-q56tk\" (UID: \"e78cc84d-57d1-4a40-8397-8108fbc1be35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8a6cf7-q56tk"
Sep 09 06:19:10.312945 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:10.312921    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8a6cf7-q56tk"
Sep 09 06:19:10.587263 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:10.587186    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8a6cf7-q56tk"]
Sep 09 06:19:10.590093 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:19:10.590063    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode78cc84d_57d1_4a40_8397_8108fbc1be35.slice/crio-d6b5e82a2a17d4b54e69f074dcb0339b03de1f18fb01442c122004df1427b02a WatchSource:0}: Error finding container d6b5e82a2a17d4b54e69f074dcb0339b03de1f18fb01442c122004df1427b02a: Status 404 returned error can't find the container with id d6b5e82a2a17d4b54e69f074dcb0339b03de1f18fb01442c122004df1427b02a
Sep 09 06:19:11.327313 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:11.327275    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8a6cf7-q56tk" event={"ID":"e78cc84d-57d1-4a40-8397-8108fbc1be35","Type":"ContainerStarted","Data":"c08d94a72c488d01aa02884ef4086d75b52699fcfd38e0121087828bf0e5f869"}
Sep 09 06:19:11.327688 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:11.327374    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8a6cf7-q56tk" event={"ID":"e78cc84d-57d1-4a40-8397-8108fbc1be35","Type":"ContainerStarted","Data":"d6b5e82a2a17d4b54e69f074dcb0339b03de1f18fb01442c122004df1427b02a"}
Sep 09 06:19:11.343212 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:11.343170    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8a6cf7-q56tk" podStartSLOduration=2.343156211 podStartE2EDuration="2.343156211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:19:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:19:11.341463237 +0000 UTC m=+1776.681951146" watchObservedRunningTime="2026-09-09 06:19:11.343156211 +0000 UTC m=+1776.683644120"
Sep 09 06:19:11.703544 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:11.703520    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"
Sep 09 06:19:11.832648 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:11.832597    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb4411cf-23fb-4ed7-86eb-ad1855e31364-workload-certs\") pod \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\" (UID: \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\") "
Sep 09 06:19:11.832648 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:11.832654    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb4411cf-23fb-4ed7-86eb-ad1855e31364-workload-socket\") pod \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\" (UID: \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\") "
Sep 09 06:19:11.832897 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:11.832696    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb4411cf-23fb-4ed7-86eb-ad1855e31364-istiod-ca-cert\") pod \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\" (UID: \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\") "
Sep 09 06:19:11.832897 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:11.832725    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxjww\" (UniqueName: \"kubernetes.io/projected/cb4411cf-23fb-4ed7-86eb-ad1855e31364-kube-api-access-zxjww\") pod \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\" (UID: \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\") "
Sep 09 06:19:11.832897 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:11.832741    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb4411cf-23fb-4ed7-86eb-ad1855e31364-istio-token\") pod \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\" (UID: \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\") "
Sep 09 06:19:11.832897 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:11.832769    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb4411cf-23fb-4ed7-86eb-ad1855e31364-istio-envoy\") pod \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\" (UID: \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\") "
Sep 09 06:19:11.832897 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:11.832810    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb4411cf-23fb-4ed7-86eb-ad1855e31364-istio-data\") pod \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\" (UID: \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\") "
Sep 09 06:19:11.832897 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:11.832830    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb4411cf-23fb-4ed7-86eb-ad1855e31364-istio-podinfo\") pod \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\" (UID: \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\") "
Sep 09 06:19:11.832897 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:11.832880    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb4411cf-23fb-4ed7-86eb-ad1855e31364-credential-socket\") pod \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\" (UID: \"cb4411cf-23fb-4ed7-86eb-ad1855e31364\") "
Sep 09 06:19:11.833265 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:11.832925    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb4411cf-23fb-4ed7-86eb-ad1855e31364-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cb4411cf-23fb-4ed7-86eb-ad1855e31364" (UID: "cb4411cf-23fb-4ed7-86eb-ad1855e31364"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:19:11.833265 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:11.833000    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb4411cf-23fb-4ed7-86eb-ad1855e31364-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cb4411cf-23fb-4ed7-86eb-ad1855e31364" (UID: "cb4411cf-23fb-4ed7-86eb-ad1855e31364"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:19:11.833265 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:11.833170    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb4411cf-23fb-4ed7-86eb-ad1855e31364-istio-data" (OuterVolumeSpecName: "istio-data") pod "cb4411cf-23fb-4ed7-86eb-ad1855e31364" (UID: "cb4411cf-23fb-4ed7-86eb-ad1855e31364"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:19:11.833265 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:11.833192    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb4411cf-23fb-4ed7-86eb-ad1855e31364-workload-certs\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:19:11.833265 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:11.833193    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb4411cf-23fb-4ed7-86eb-ad1855e31364-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cb4411cf-23fb-4ed7-86eb-ad1855e31364" (UID: "cb4411cf-23fb-4ed7-86eb-ad1855e31364"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:19:11.833265 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:11.833210    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb4411cf-23fb-4ed7-86eb-ad1855e31364-workload-socket\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:19:11.833265 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:11.833221    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb4411cf-23fb-4ed7-86eb-ad1855e31364-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cb4411cf-23fb-4ed7-86eb-ad1855e31364" (UID: "cb4411cf-23fb-4ed7-86eb-ad1855e31364"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:19:11.835079 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:11.835050    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb4411cf-23fb-4ed7-86eb-ad1855e31364-kube-api-access-zxjww" (OuterVolumeSpecName: "kube-api-access-zxjww") pod "cb4411cf-23fb-4ed7-86eb-ad1855e31364" (UID: "cb4411cf-23fb-4ed7-86eb-ad1855e31364"). InnerVolumeSpecName "kube-api-access-zxjww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:19:11.835193 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:11.835074    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb4411cf-23fb-4ed7-86eb-ad1855e31364-istio-token" (OuterVolumeSpecName: "istio-token") pod "cb4411cf-23fb-4ed7-86eb-ad1855e31364" (UID: "cb4411cf-23fb-4ed7-86eb-ad1855e31364"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:19:11.835193 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:11.835125    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb4411cf-23fb-4ed7-86eb-ad1855e31364-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cb4411cf-23fb-4ed7-86eb-ad1855e31364" (UID: "cb4411cf-23fb-4ed7-86eb-ad1855e31364"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:19:11.835275 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:11.835214    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb4411cf-23fb-4ed7-86eb-ad1855e31364-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cb4411cf-23fb-4ed7-86eb-ad1855e31364" (UID: "cb4411cf-23fb-4ed7-86eb-ad1855e31364"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 06:19:11.933968 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:11.933909    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb4411cf-23fb-4ed7-86eb-ad1855e31364-credential-socket\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:19:11.933968 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:11.933929    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb4411cf-23fb-4ed7-86eb-ad1855e31364-istiod-ca-cert\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:19:11.933968 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:11.933940    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxjww\" (UniqueName: \"kubernetes.io/projected/cb4411cf-23fb-4ed7-86eb-ad1855e31364-kube-api-access-zxjww\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:19:11.933968 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:11.933948    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb4411cf-23fb-4ed7-86eb-ad1855e31364-istio-token\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:19:11.933968 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:11.933956    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb4411cf-23fb-4ed7-86eb-ad1855e31364-istio-envoy\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:19:11.933968 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:11.933963    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb4411cf-23fb-4ed7-86eb-ad1855e31364-istio-data\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:19:11.933968 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:11.933972    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb4411cf-23fb-4ed7-86eb-ad1855e31364-istio-podinfo\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:19:12.333367 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:12.333337    2579 generic.go:356] "Generic (PLEG): container finished" podID="e78cc84d-57d1-4a40-8397-8108fbc1be35" containerID="c08d94a72c488d01aa02884ef4086d75b52699fcfd38e0121087828bf0e5f869" exitCode=0
Sep 09 06:19:12.333763 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:12.333398    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8a6cf7-q56tk" event={"ID":"e78cc84d-57d1-4a40-8397-8108fbc1be35","Type":"ContainerDied","Data":"c08d94a72c488d01aa02884ef4086d75b52699fcfd38e0121087828bf0e5f869"}
Sep 09 06:19:12.335261 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:12.335241    2579 generic.go:356] "Generic (PLEG): container finished" podID="cb4411cf-23fb-4ed7-86eb-ad1855e31364" containerID="972d905da2e28993c8292972a970832cb66d7e36080b31e4906c3837f0255d98" exitCode=0
Sep 09 06:19:12.335343 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:12.335288    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"
Sep 09 06:19:12.335383 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:12.335289    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz" event={"ID":"cb4411cf-23fb-4ed7-86eb-ad1855e31364","Type":"ContainerDied","Data":"972d905da2e28993c8292972a970832cb66d7e36080b31e4906c3837f0255d98"}
Sep 09 06:19:12.335431 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:12.335417    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz" event={"ID":"cb4411cf-23fb-4ed7-86eb-ad1855e31364","Type":"ContainerDied","Data":"248d5c799f97cb260284b9f8d9f95af7fcc8c0cfc894a22859decbcfd48b8dab"}
Sep 09 06:19:12.335471 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:12.335450    2579 scope.go:117] "RemoveContainer" containerID="972d905da2e28993c8292972a970832cb66d7e36080b31e4906c3837f0255d98"
Sep 09 06:19:12.344619 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:12.344599    2579 scope.go:117] "RemoveContainer" containerID="235b0a53920a60e06d730a7cd0a5d53e5073245cfa8ec8c953c9a6aa42f87279"
Sep 09 06:19:12.353265 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:12.353249    2579 scope.go:117] "RemoveContainer" containerID="972d905da2e28993c8292972a970832cb66d7e36080b31e4906c3837f0255d98"
Sep 09 06:19:12.353545 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:19:12.353521    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"972d905da2e28993c8292972a970832cb66d7e36080b31e4906c3837f0255d98\": container with ID starting with 972d905da2e28993c8292972a970832cb66d7e36080b31e4906c3837f0255d98 not found: ID does not exist" containerID="972d905da2e28993c8292972a970832cb66d7e36080b31e4906c3837f0255d98"
Sep 09 06:19:12.353602 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:12.353559    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"972d905da2e28993c8292972a970832cb66d7e36080b31e4906c3837f0255d98"} err="failed to get container status \"972d905da2e28993c8292972a970832cb66d7e36080b31e4906c3837f0255d98\": rpc error: code = NotFound desc = could not find container \"972d905da2e28993c8292972a970832cb66d7e36080b31e4906c3837f0255d98\": container with ID starting with 972d905da2e28993c8292972a970832cb66d7e36080b31e4906c3837f0255d98 not found: ID does not exist"
Sep 09 06:19:12.353602 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:12.353582    2579 scope.go:117] "RemoveContainer" containerID="235b0a53920a60e06d730a7cd0a5d53e5073245cfa8ec8c953c9a6aa42f87279"
Sep 09 06:19:12.353878 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:19:12.353862    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"235b0a53920a60e06d730a7cd0a5d53e5073245cfa8ec8c953c9a6aa42f87279\": container with ID starting with 235b0a53920a60e06d730a7cd0a5d53e5073245cfa8ec8c953c9a6aa42f87279 not found: ID does not exist" containerID="235b0a53920a60e06d730a7cd0a5d53e5073245cfa8ec8c953c9a6aa42f87279"
Sep 09 06:19:12.353931 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:12.353885    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"235b0a53920a60e06d730a7cd0a5d53e5073245cfa8ec8c953c9a6aa42f87279"} err="failed to get container status \"235b0a53920a60e06d730a7cd0a5d53e5073245cfa8ec8c953c9a6aa42f87279\": rpc error: code = NotFound desc = could not find container \"235b0a53920a60e06d730a7cd0a5d53e5073245cfa8ec8c953c9a6aa42f87279\": container with ID starting with 235b0a53920a60e06d730a7cd0a5d53e5073245cfa8ec8c953c9a6aa42f87279 not found: ID does not exist"
Sep 09 06:19:12.361207 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:12.361187    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"]
Sep 09 06:19:12.367371 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:12.367352    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-2e28ff-openshift-default-7cb9675d5d-g5vtz"]
Sep 09 06:19:13.149180 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:13.149151    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb4411cf-23fb-4ed7-86eb-ad1855e31364" path="/var/lib/kubelet/pods/cb4411cf-23fb-4ed7-86eb-ad1855e31364/volumes"
Sep 09 06:19:13.473793 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:13.473753    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8a6cf7-q56tk"
Sep 09 06:19:13.647879 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:13.647852    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e78cc84d-57d1-4a40-8397-8108fbc1be35-maas-api-service-ca\") pod \"e78cc84d-57d1-4a40-8397-8108fbc1be35\" (UID: \"e78cc84d-57d1-4a40-8397-8108fbc1be35\") "
Sep 09 06:19:13.648192 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:13.648168    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e78cc84d-57d1-4a40-8397-8108fbc1be35-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e78cc84d-57d1-4a40-8397-8108fbc1be35" (UID: "e78cc84d-57d1-4a40-8397-8108fbc1be35"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:19:13.749482 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:13.749410    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e78cc84d-57d1-4a40-8397-8108fbc1be35-maas-api-service-ca\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:19:14.346396 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:14.346360    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8a6cf7-q56tk" event={"ID":"e78cc84d-57d1-4a40-8397-8108fbc1be35","Type":"ContainerDied","Data":"d6b5e82a2a17d4b54e69f074dcb0339b03de1f18fb01442c122004df1427b02a"}
Sep 09 06:19:14.346396 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:14.346383    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8a6cf7-q56tk"
Sep 09 06:19:14.346597 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:14.346416    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6b5e82a2a17d4b54e69f074dcb0339b03de1f18fb01442c122004df1427b02a"
Sep 09 06:19:23.406015 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:23.405983    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:19:25.595860 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:25.595830    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8a6cf7-bdf474ddb-85l7f"]
Sep 09 06:19:25.596276 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:25.596063    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a6cf7-bdf474ddb-85l7f" podUID="112458ce-0a56-4371-8b93-c98a3693f25c" containerName="payload-pre-processing" containerID="cri-o://2cacd68ef473ceb24ceb12ec9c604816e80a00aeca4f6a9fb111883625f30d12" gracePeriod=30
Sep 09 06:19:25.774598 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:25.774563    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8a6cf7-q56tk"]
Sep 09 06:19:25.778038 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:25.778015    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8a6cf7-q56tk"]
Sep 09 06:19:25.841952 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:25.841933    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a6cf7-bdf474ddb-85l7f"
Sep 09 06:19:25.843929 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:25.843914    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxpfv\" (UniqueName: \"kubernetes.io/projected/112458ce-0a56-4371-8b93-c98a3693f25c-kube-api-access-wxpfv\") pod \"112458ce-0a56-4371-8b93-c98a3693f25c\" (UID: \"112458ce-0a56-4371-8b93-c98a3693f25c\") "
Sep 09 06:19:25.844052 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:25.843949    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/112458ce-0a56-4371-8b93-c98a3693f25c-plugins-config-volume\") pod \"112458ce-0a56-4371-8b93-c98a3693f25c\" (UID: \"112458ce-0a56-4371-8b93-c98a3693f25c\") "
Sep 09 06:19:25.844316 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:25.844299    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/112458ce-0a56-4371-8b93-c98a3693f25c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "112458ce-0a56-4371-8b93-c98a3693f25c" (UID: "112458ce-0a56-4371-8b93-c98a3693f25c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:19:25.846032 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:25.845978    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/112458ce-0a56-4371-8b93-c98a3693f25c-kube-api-access-wxpfv" (OuterVolumeSpecName: "kube-api-access-wxpfv") pod "112458ce-0a56-4371-8b93-c98a3693f25c" (UID: "112458ce-0a56-4371-8b93-c98a3693f25c"). InnerVolumeSpecName "kube-api-access-wxpfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:19:25.944706 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:25.944676    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxpfv\" (UniqueName: \"kubernetes.io/projected/112458ce-0a56-4371-8b93-c98a3693f25c-kube-api-access-wxpfv\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:19:25.944706 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:25.944705    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/112458ce-0a56-4371-8b93-c98a3693f25c-plugins-config-volume\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:19:26.406649 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:26.406551    2579 generic.go:356] "Generic (PLEG): container finished" podID="112458ce-0a56-4371-8b93-c98a3693f25c" containerID="2cacd68ef473ceb24ceb12ec9c604816e80a00aeca4f6a9fb111883625f30d12" exitCode=0
Sep 09 06:19:26.406649 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:26.406600    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a6cf7-bdf474ddb-85l7f"
Sep 09 06:19:26.406649 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:26.406617    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a6cf7-bdf474ddb-85l7f" event={"ID":"112458ce-0a56-4371-8b93-c98a3693f25c","Type":"ContainerDied","Data":"2cacd68ef473ceb24ceb12ec9c604816e80a00aeca4f6a9fb111883625f30d12"}
Sep 09 06:19:26.406877 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:26.406693    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a6cf7-bdf474ddb-85l7f" event={"ID":"112458ce-0a56-4371-8b93-c98a3693f25c","Type":"ContainerDied","Data":"40b8cc682418ce34d568d4a3af160a2fab8f211d13557d7d3056bfd0d3d0d31f"}
Sep 09 06:19:26.406877 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:26.406724    2579 scope.go:117] "RemoveContainer" containerID="2cacd68ef473ceb24ceb12ec9c604816e80a00aeca4f6a9fb111883625f30d12"
Sep 09 06:19:26.416028 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:26.416004    2579 scope.go:117] "RemoveContainer" containerID="2cacd68ef473ceb24ceb12ec9c604816e80a00aeca4f6a9fb111883625f30d12"
Sep 09 06:19:26.416286 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:19:26.416268    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cacd68ef473ceb24ceb12ec9c604816e80a00aeca4f6a9fb111883625f30d12\": container with ID starting with 2cacd68ef473ceb24ceb12ec9c604816e80a00aeca4f6a9fb111883625f30d12 not found: ID does not exist" containerID="2cacd68ef473ceb24ceb12ec9c604816e80a00aeca4f6a9fb111883625f30d12"
Sep 09 06:19:26.416343 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:26.416294    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cacd68ef473ceb24ceb12ec9c604816e80a00aeca4f6a9fb111883625f30d12"} err="failed to get container status \"2cacd68ef473ceb24ceb12ec9c604816e80a00aeca4f6a9fb111883625f30d12\": rpc error: code = NotFound desc = could not find container \"2cacd68ef473ceb24ceb12ec9c604816e80a00aeca4f6a9fb111883625f30d12\": container with ID starting with 2cacd68ef473ceb24ceb12ec9c604816e80a00aeca4f6a9fb111883625f30d12 not found: ID does not exist"
Sep 09 06:19:26.427805 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:26.427777    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8a6cf7-bdf474ddb-85l7f"]
Sep 09 06:19:26.432479 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:26.432459    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8a6cf7-bdf474ddb-85l7f"]
Sep 09 06:19:27.148331 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:27.148300    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="112458ce-0a56-4371-8b93-c98a3693f25c" path="/var/lib/kubelet/pods/112458ce-0a56-4371-8b93-c98a3693f25c/volumes"
Sep 09 06:19:27.148716 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:27.148619    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e78cc84d-57d1-4a40-8397-8108fbc1be35" path="/var/lib/kubelet/pods/e78cc84d-57d1-4a40-8397-8108fbc1be35/volumes"
Sep 09 06:19:31.110773 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:31.110737    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-8a6cf7/ipp-route-8a6cf7-kserve-56b86956c5-lnhfq"]
Sep 09 06:19:35.558257 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:35.558225    2579 scope.go:117] "RemoveContainer" containerID="d63c588d1180e2c634a32465aaedb1213606d6275bed98ab02c0bb38d73d3901"
Sep 09 06:19:35.566772 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:35.566752    2579 scope.go:117] "RemoveContainer" containerID="8e156decab8b63888831ac002675e1dbf908aaa693c6a64d325578f3524cad70"
Sep 09 06:19:35.574505 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:35.574485    2579 scope.go:117] "RemoveContainer" containerID="494aa76251c124b1cf8ab6d9c3770c5170b8606430c1820ab3099337a5084b59"
Sep 09 06:19:35.702189 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:35.702163    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:19:46.131734 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:46.131668    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-8a6cf7/ipp-route-8a6cf7-kserve-56b86956c5-lnhfq" podUID="05f94a7b-65e7-4157-8fc0-1da60261ba70" containerName="main" containerID="cri-o://c5324edc9d17828c56cd4637870e0dff2f35b3085ec705338209231c1d510bc5" gracePeriod=45
Sep 09 06:19:46.153476 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:19:46.153450    2579 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05f94a7b_65e7_4157_8fc0_1da60261ba70.slice/crio-c5324edc9d17828c56cd4637870e0dff2f35b3085ec705338209231c1d510bc5.scope\": RecentStats: unable to find data in memory cache]"
Sep 09 06:19:46.153563 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:19:46.153469    2579 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05f94a7b_65e7_4157_8fc0_1da60261ba70.slice/crio-c5324edc9d17828c56cd4637870e0dff2f35b3085ec705338209231c1d510bc5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05f94a7b_65e7_4157_8fc0_1da60261ba70.slice/crio-conmon-c5324edc9d17828c56cd4637870e0dff2f35b3085ec705338209231c1d510bc5.scope\": RecentStats: unable to find data in memory cache]"
Sep 09 06:19:46.382093 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:46.382035    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8a6cf7/ipp-route-8a6cf7-kserve-56b86956c5-lnhfq"
Sep 09 06:19:46.494106 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:46.494080    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05f94a7b-65e7-4157-8fc0-1da60261ba70-home\") pod \"05f94a7b-65e7-4157-8fc0-1da60261ba70\" (UID: \"05f94a7b-65e7-4157-8fc0-1da60261ba70\") "
Sep 09 06:19:46.494287 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:46.494147    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05f94a7b-65e7-4157-8fc0-1da60261ba70-dshm\") pod \"05f94a7b-65e7-4157-8fc0-1da60261ba70\" (UID: \"05f94a7b-65e7-4157-8fc0-1da60261ba70\") "
Sep 09 06:19:46.494287 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:46.494180    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdbwp\" (UniqueName: \"kubernetes.io/projected/05f94a7b-65e7-4157-8fc0-1da60261ba70-kube-api-access-rdbwp\") pod \"05f94a7b-65e7-4157-8fc0-1da60261ba70\" (UID: \"05f94a7b-65e7-4157-8fc0-1da60261ba70\") "
Sep 09 06:19:46.494287 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:46.494205    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05f94a7b-65e7-4157-8fc0-1da60261ba70-tmp-dir\") pod \"05f94a7b-65e7-4157-8fc0-1da60261ba70\" (UID: \"05f94a7b-65e7-4157-8fc0-1da60261ba70\") "
Sep 09 06:19:46.494446 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:46.494294    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05f94a7b-65e7-4157-8fc0-1da60261ba70-model-cache\") pod \"05f94a7b-65e7-4157-8fc0-1da60261ba70\" (UID: \"05f94a7b-65e7-4157-8fc0-1da60261ba70\") "
Sep 09 06:19:46.494446 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:46.494319    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05f94a7b-65e7-4157-8fc0-1da60261ba70-tls-certs\") pod \"05f94a7b-65e7-4157-8fc0-1da60261ba70\" (UID: \"05f94a7b-65e7-4157-8fc0-1da60261ba70\") "
Sep 09 06:19:46.494446 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:46.494358    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f94a7b-65e7-4157-8fc0-1da60261ba70-home" (OuterVolumeSpecName: "home") pod "05f94a7b-65e7-4157-8fc0-1da60261ba70" (UID: "05f94a7b-65e7-4157-8fc0-1da60261ba70"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:19:46.494600 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:46.494483    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f94a7b-65e7-4157-8fc0-1da60261ba70-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "05f94a7b-65e7-4157-8fc0-1da60261ba70" (UID: "05f94a7b-65e7-4157-8fc0-1da60261ba70"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:19:46.494600 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:46.494504    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f94a7b-65e7-4157-8fc0-1da60261ba70-model-cache" (OuterVolumeSpecName: "model-cache") pod "05f94a7b-65e7-4157-8fc0-1da60261ba70" (UID: "05f94a7b-65e7-4157-8fc0-1da60261ba70"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:19:46.494730 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:46.494604    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05f94a7b-65e7-4157-8fc0-1da60261ba70-model-cache\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:19:46.494730 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:46.494617    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05f94a7b-65e7-4157-8fc0-1da60261ba70-home\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:19:46.494730 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:46.494654    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05f94a7b-65e7-4157-8fc0-1da60261ba70-tmp-dir\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:19:46.496406 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:46.496380    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f94a7b-65e7-4157-8fc0-1da60261ba70-dshm" (OuterVolumeSpecName: "dshm") pod "05f94a7b-65e7-4157-8fc0-1da60261ba70" (UID: "05f94a7b-65e7-4157-8fc0-1da60261ba70"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:19:46.496506 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:46.496405    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05f94a7b-65e7-4157-8fc0-1da60261ba70-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "05f94a7b-65e7-4157-8fc0-1da60261ba70" (UID: "05f94a7b-65e7-4157-8fc0-1da60261ba70"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:19:46.496506 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:46.496436    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f94a7b-65e7-4157-8fc0-1da60261ba70-kube-api-access-rdbwp" (OuterVolumeSpecName: "kube-api-access-rdbwp") pod "05f94a7b-65e7-4157-8fc0-1da60261ba70" (UID: "05f94a7b-65e7-4157-8fc0-1da60261ba70"). InnerVolumeSpecName "kube-api-access-rdbwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:19:46.507975 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:46.507949    2579 generic.go:356] "Generic (PLEG): container finished" podID="05f94a7b-65e7-4157-8fc0-1da60261ba70" containerID="c5324edc9d17828c56cd4637870e0dff2f35b3085ec705338209231c1d510bc5" exitCode=0
Sep 09 06:19:46.508087 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:46.507995    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8a6cf7/ipp-route-8a6cf7-kserve-56b86956c5-lnhfq"
Sep 09 06:19:46.508087 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:46.508013    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8a6cf7/ipp-route-8a6cf7-kserve-56b86956c5-lnhfq" event={"ID":"05f94a7b-65e7-4157-8fc0-1da60261ba70","Type":"ContainerDied","Data":"c5324edc9d17828c56cd4637870e0dff2f35b3085ec705338209231c1d510bc5"}
Sep 09 06:19:46.508087 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:46.508046    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8a6cf7/ipp-route-8a6cf7-kserve-56b86956c5-lnhfq" event={"ID":"05f94a7b-65e7-4157-8fc0-1da60261ba70","Type":"ContainerDied","Data":"af10cdd4fb8d8c10084ac4d59c33d1df9b67f7e0046904fcb4e3711490259141"}
Sep 09 06:19:46.508087 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:46.508062    2579 scope.go:117] "RemoveContainer" containerID="c5324edc9d17828c56cd4637870e0dff2f35b3085ec705338209231c1d510bc5"
Sep 09 06:19:46.516708 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:46.516692    2579 scope.go:117] "RemoveContainer" containerID="c5324edc9d17828c56cd4637870e0dff2f35b3085ec705338209231c1d510bc5"
Sep 09 06:19:46.516972 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:19:46.516951    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5324edc9d17828c56cd4637870e0dff2f35b3085ec705338209231c1d510bc5\": container with ID starting with c5324edc9d17828c56cd4637870e0dff2f35b3085ec705338209231c1d510bc5 not found: ID does not exist" containerID="c5324edc9d17828c56cd4637870e0dff2f35b3085ec705338209231c1d510bc5"
Sep 09 06:19:46.517016 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:46.516982    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5324edc9d17828c56cd4637870e0dff2f35b3085ec705338209231c1d510bc5"} err="failed to get container status \"c5324edc9d17828c56cd4637870e0dff2f35b3085ec705338209231c1d510bc5\": rpc error: code = NotFound desc = could not find container \"c5324edc9d17828c56cd4637870e0dff2f35b3085ec705338209231c1d510bc5\": container with ID starting with c5324edc9d17828c56cd4637870e0dff2f35b3085ec705338209231c1d510bc5 not found: ID does not exist"
Sep 09 06:19:46.528476 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:46.528456    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-8a6cf7/ipp-route-8a6cf7-kserve-56b86956c5-lnhfq"]
Sep 09 06:19:46.534081 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:46.534063    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-8a6cf7/ipp-route-8a6cf7-kserve-56b86956c5-lnhfq"]
Sep 09 06:19:46.595208 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:46.595187    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05f94a7b-65e7-4157-8fc0-1da60261ba70-dshm\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:19:46.595208 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:46.595206    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdbwp\" (UniqueName: \"kubernetes.io/projected/05f94a7b-65e7-4157-8fc0-1da60261ba70-kube-api-access-rdbwp\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:19:46.595325 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:46.595219    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05f94a7b-65e7-4157-8fc0-1da60261ba70-tls-certs\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:19:47.148126 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:19:47.148093    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05f94a7b-65e7-4157-8fc0-1da60261ba70" path="/var/lib/kubelet/pods/05f94a7b-65e7-4157-8fc0-1da60261ba70/volumes"
Sep 09 06:20:14.326706 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:14.326670    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-32ea4d-cvkpk"]
Sep 09 06:20:14.327163 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:14.326982    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="112458ce-0a56-4371-8b93-c98a3693f25c" containerName="payload-pre-processing"
Sep 09 06:20:14.327163 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:14.326993    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="112458ce-0a56-4371-8b93-c98a3693f25c" containerName="payload-pre-processing"
Sep 09 06:20:14.327163 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:14.327009    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e78cc84d-57d1-4a40-8397-8108fbc1be35" containerName="revoke-keys"
Sep 09 06:20:14.327163 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:14.327014    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e78cc84d-57d1-4a40-8397-8108fbc1be35" containerName="revoke-keys"
Sep 09 06:20:14.327163 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:14.327022    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05f94a7b-65e7-4157-8fc0-1da60261ba70" containerName="main"
Sep 09 06:20:14.327163 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:14.327028    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="05f94a7b-65e7-4157-8fc0-1da60261ba70" containerName="main"
Sep 09 06:20:14.327163 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:14.327034    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb4411cf-23fb-4ed7-86eb-ad1855e31364" containerName="istio-proxy"
Sep 09 06:20:14.327163 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:14.327039    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb4411cf-23fb-4ed7-86eb-ad1855e31364" containerName="istio-proxy"
Sep 09 06:20:14.327163 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:14.327052    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb4411cf-23fb-4ed7-86eb-ad1855e31364" containerName="istio-proxy"
Sep 09 06:20:14.327163 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:14.327057    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb4411cf-23fb-4ed7-86eb-ad1855e31364" containerName="istio-proxy"
Sep 09 06:20:14.327163 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:14.327103    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e78cc84d-57d1-4a40-8397-8108fbc1be35" containerName="revoke-keys"
Sep 09 06:20:14.327163 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:14.327112    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="112458ce-0a56-4371-8b93-c98a3693f25c" containerName="payload-pre-processing"
Sep 09 06:20:14.327163 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:14.327117    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb4411cf-23fb-4ed7-86eb-ad1855e31364" containerName="istio-proxy"
Sep 09 06:20:14.327163 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:14.327124    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="05f94a7b-65e7-4157-8fc0-1da60261ba70" containerName="main"
Sep 09 06:20:14.327163 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:14.327128    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb4411cf-23fb-4ed7-86eb-ad1855e31364" containerName="istio-proxy"
Sep 09 06:20:14.329875 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:14.329858    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-32ea4d-cvkpk"
Sep 09 06:20:14.331993 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:14.331963    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 06:20:14.332112 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:14.332018    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ksjk6\""
Sep 09 06:20:14.336552 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:14.336517    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-32ea4d-cvkpk"]
Sep 09 06:20:14.426088 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:14.426052    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50ab755a-caed-4aab-82ff-7e1d9202cbfc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-32ea4d-cvkpk\" (UID: \"50ab755a-caed-4aab-82ff-7e1d9202cbfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-32ea4d-cvkpk"
Sep 09 06:20:14.527507 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:14.527467    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50ab755a-caed-4aab-82ff-7e1d9202cbfc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-32ea4d-cvkpk\" (UID: \"50ab755a-caed-4aab-82ff-7e1d9202cbfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-32ea4d-cvkpk"
Sep 09 06:20:14.528156 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:14.528135    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50ab755a-caed-4aab-82ff-7e1d9202cbfc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-32ea4d-cvkpk\" (UID: \"50ab755a-caed-4aab-82ff-7e1d9202cbfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-32ea4d-cvkpk"
Sep 09 06:20:14.641796 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:14.641716    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-32ea4d-cvkpk"
Sep 09 06:20:14.911434 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:14.911407    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-32ea4d-cvkpk"]
Sep 09 06:20:14.914144 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:20:14.914106    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50ab755a_caed_4aab_82ff_7e1d9202cbfc.slice/crio-a911b1e17eb27b7f9d6625976780a4820ce9080edcd19c2976a8fb0c38c8aab3 WatchSource:0}: Error finding container a911b1e17eb27b7f9d6625976780a4820ce9080edcd19c2976a8fb0c38c8aab3: Status 404 returned error can't find the container with id a911b1e17eb27b7f9d6625976780a4820ce9080edcd19c2976a8fb0c38c8aab3
Sep 09 06:20:15.645680 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:15.645618    2579 generic.go:356] "Generic (PLEG): container finished" podID="50ab755a-caed-4aab-82ff-7e1d9202cbfc" containerID="9a94fae37230ef4010712a28fc88aef46385fe2bf65e7aaa6584cbbe3230f43d" exitCode=0
Sep 09 06:20:15.645991 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:15.645658    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-32ea4d-cvkpk" event={"ID":"50ab755a-caed-4aab-82ff-7e1d9202cbfc","Type":"ContainerDied","Data":"9a94fae37230ef4010712a28fc88aef46385fe2bf65e7aaa6584cbbe3230f43d"}
Sep 09 06:20:15.645991 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:15.645777    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-32ea4d-cvkpk" event={"ID":"50ab755a-caed-4aab-82ff-7e1d9202cbfc","Type":"ContainerStarted","Data":"a911b1e17eb27b7f9d6625976780a4820ce9080edcd19c2976a8fb0c38c8aab3"}
Sep 09 06:20:16.774412 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:16.774385    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-32ea4d-cvkpk"
Sep 09 06:20:16.949729 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:16.949614    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50ab755a-caed-4aab-82ff-7e1d9202cbfc-maas-api-service-ca\") pod \"50ab755a-caed-4aab-82ff-7e1d9202cbfc\" (UID: \"50ab755a-caed-4aab-82ff-7e1d9202cbfc\") "
Sep 09 06:20:16.950008 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:16.949987    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50ab755a-caed-4aab-82ff-7e1d9202cbfc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "50ab755a-caed-4aab-82ff-7e1d9202cbfc" (UID: "50ab755a-caed-4aab-82ff-7e1d9202cbfc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:20:17.050582 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:17.050542    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50ab755a-caed-4aab-82ff-7e1d9202cbfc-maas-api-service-ca\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:20:17.656865 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:17.656833    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-32ea4d-cvkpk"
Sep 09 06:20:17.656865 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:17.656851    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-32ea4d-cvkpk" event={"ID":"50ab755a-caed-4aab-82ff-7e1d9202cbfc","Type":"ContainerDied","Data":"a911b1e17eb27b7f9d6625976780a4820ce9080edcd19c2976a8fb0c38c8aab3"}
Sep 09 06:20:17.657082 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:17.656899    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a911b1e17eb27b7f9d6625976780a4820ce9080edcd19c2976a8fb0c38c8aab3"
Sep 09 06:20:25.036829 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:25.036796    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-32ea4d-7cc4848ff4-j4fwv"]
Sep 09 06:20:25.037261 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:25.037031    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32ea4d-7cc4848ff4-j4fwv" podUID="911c3161-466c-48f9-af08-0e07138953f1" containerName="payload-pre-processing" containerID="cri-o://b345d2761e62ed0165a86d5d077ec2f394f2da0f637020ce51333edcd48fee40" gracePeriod=30
Sep 09 06:20:25.212706 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:25.212644    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-32ea4d-cvkpk"]
Sep 09 06:20:25.215129 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:25.215105    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-32ea4d-cvkpk"]
Sep 09 06:20:25.286912 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:25.286850    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32ea4d-7cc4848ff4-j4fwv"
Sep 09 06:20:25.425825 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:25.425794    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8xn7\" (UniqueName: \"kubernetes.io/projected/911c3161-466c-48f9-af08-0e07138953f1-kube-api-access-v8xn7\") pod \"911c3161-466c-48f9-af08-0e07138953f1\" (UID: \"911c3161-466c-48f9-af08-0e07138953f1\") "
Sep 09 06:20:25.425994 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:25.425881    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/911c3161-466c-48f9-af08-0e07138953f1-plugins-config-volume\") pod \"911c3161-466c-48f9-af08-0e07138953f1\" (UID: \"911c3161-466c-48f9-af08-0e07138953f1\") "
Sep 09 06:20:25.426198 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:25.426177    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/911c3161-466c-48f9-af08-0e07138953f1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "911c3161-466c-48f9-af08-0e07138953f1" (UID: "911c3161-466c-48f9-af08-0e07138953f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:20:25.428091 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:25.428065    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/911c3161-466c-48f9-af08-0e07138953f1-kube-api-access-v8xn7" (OuterVolumeSpecName: "kube-api-access-v8xn7") pod "911c3161-466c-48f9-af08-0e07138953f1" (UID: "911c3161-466c-48f9-af08-0e07138953f1"). InnerVolumeSpecName "kube-api-access-v8xn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:20:25.526385 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:25.526353    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8xn7\" (UniqueName: \"kubernetes.io/projected/911c3161-466c-48f9-af08-0e07138953f1-kube-api-access-v8xn7\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:20:25.526385 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:25.526383    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/911c3161-466c-48f9-af08-0e07138953f1-plugins-config-volume\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:20:25.695525 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:25.695493    2579 generic.go:356] "Generic (PLEG): container finished" podID="911c3161-466c-48f9-af08-0e07138953f1" containerID="b345d2761e62ed0165a86d5d077ec2f394f2da0f637020ce51333edcd48fee40" exitCode=0
Sep 09 06:20:25.695724 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:25.695533    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32ea4d-7cc4848ff4-j4fwv"
Sep 09 06:20:25.695724 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:25.695588    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32ea4d-7cc4848ff4-j4fwv" event={"ID":"911c3161-466c-48f9-af08-0e07138953f1","Type":"ContainerDied","Data":"b345d2761e62ed0165a86d5d077ec2f394f2da0f637020ce51333edcd48fee40"}
Sep 09 06:20:25.695724 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:25.695654    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32ea4d-7cc4848ff4-j4fwv" event={"ID":"911c3161-466c-48f9-af08-0e07138953f1","Type":"ContainerDied","Data":"028b0a7771c620c87d692af2079333e88ae8f4523a0bf87c5c04f887370defcc"}
Sep 09 06:20:25.695724 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:25.695679    2579 scope.go:117] "RemoveContainer" containerID="b345d2761e62ed0165a86d5d077ec2f394f2da0f637020ce51333edcd48fee40"
Sep 09 06:20:25.704507 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:25.704491    2579 scope.go:117] "RemoveContainer" containerID="b345d2761e62ed0165a86d5d077ec2f394f2da0f637020ce51333edcd48fee40"
Sep 09 06:20:25.704796 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:20:25.704778    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b345d2761e62ed0165a86d5d077ec2f394f2da0f637020ce51333edcd48fee40\": container with ID starting with b345d2761e62ed0165a86d5d077ec2f394f2da0f637020ce51333edcd48fee40 not found: ID does not exist" containerID="b345d2761e62ed0165a86d5d077ec2f394f2da0f637020ce51333edcd48fee40"
Sep 09 06:20:25.704853 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:25.704806    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b345d2761e62ed0165a86d5d077ec2f394f2da0f637020ce51333edcd48fee40"} err="failed to get container status \"b345d2761e62ed0165a86d5d077ec2f394f2da0f637020ce51333edcd48fee40\": rpc error: code = NotFound desc = could not find container \"b345d2761e62ed0165a86d5d077ec2f394f2da0f637020ce51333edcd48fee40\": container with ID starting with b345d2761e62ed0165a86d5d077ec2f394f2da0f637020ce51333edcd48fee40 not found: ID does not exist"
Sep 09 06:20:25.717582 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:25.717560    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-32ea4d-7cc4848ff4-j4fwv"]
Sep 09 06:20:25.724467 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:25.724450    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-32ea4d-7cc4848ff4-j4fwv"]
Sep 09 06:20:27.148088 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:27.148056    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50ab755a-caed-4aab-82ff-7e1d9202cbfc" path="/var/lib/kubelet/pods/50ab755a-caed-4aab-82ff-7e1d9202cbfc/volumes"
Sep 09 06:20:27.148463 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:27.148409    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="911c3161-466c-48f9-af08-0e07138953f1" path="/var/lib/kubelet/pods/911c3161-466c-48f9-af08-0e07138953f1/volumes"
Sep 09 06:20:28.103121 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:28.103086    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:20:31.680359 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:31.680324    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"]
Sep 09 06:20:31.680845 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:31.680521    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq" podUID="34ec6702-7142-4ff6-bba5-cf1969ac2a46" containerName="istio-proxy" containerID="cri-o://08780f5d4e945de992eecb63e1999e3498984567e1f735c66d767ec039fa7202" gracePeriod=30
Sep 09 06:20:32.093678 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:32.093644    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6b5d89-hk5nh"]
Sep 09 06:20:32.093957 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:32.093945    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="911c3161-466c-48f9-af08-0e07138953f1" containerName="payload-pre-processing"
Sep 09 06:20:32.094004 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:32.093959    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="911c3161-466c-48f9-af08-0e07138953f1" containerName="payload-pre-processing"
Sep 09 06:20:32.094004 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:32.093970    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50ab755a-caed-4aab-82ff-7e1d9202cbfc" containerName="revoke-keys"
Sep 09 06:20:32.094004 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:32.093976    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="50ab755a-caed-4aab-82ff-7e1d9202cbfc" containerName="revoke-keys"
Sep 09 06:20:32.094095 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:32.094034    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="911c3161-466c-48f9-af08-0e07138953f1" containerName="payload-pre-processing"
Sep 09 06:20:32.094095 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:32.094043    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="50ab755a-caed-4aab-82ff-7e1d9202cbfc" containerName="revoke-keys"
Sep 09 06:20:32.098186 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:32.098170    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6b5d89-hk5nh"
Sep 09 06:20:32.100378 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:32.100357    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 06:20:32.100471 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:32.100413    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ksjk6\""
Sep 09 06:20:32.104058 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:32.104038    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6b5d89-hk5nh"]
Sep 09 06:20:32.176084 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:32.176048    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/466a84a3-8dea-4754-8df8-654181b60e3d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-6b5d89-hk5nh\" (UID: \"466a84a3-8dea-4754-8df8-654181b60e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6b5d89-hk5nh"
Sep 09 06:20:32.276994 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:32.276962    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/466a84a3-8dea-4754-8df8-654181b60e3d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-6b5d89-hk5nh\" (UID: \"466a84a3-8dea-4754-8df8-654181b60e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6b5d89-hk5nh"
Sep 09 06:20:32.277566 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:32.277545    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/466a84a3-8dea-4754-8df8-654181b60e3d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-6b5d89-hk5nh\" (UID: \"466a84a3-8dea-4754-8df8-654181b60e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6b5d89-hk5nh"
Sep 09 06:20:32.409610 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:32.409538    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6b5d89-hk5nh"
Sep 09 06:20:32.678855 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:32.678836    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6b5d89-hk5nh"]
Sep 09 06:20:32.681174 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:20:32.681147    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod466a84a3_8dea_4754_8df8_654181b60e3d.slice/crio-69e11b1d0bebcbb8ba53805687b3ad26935cf813404831b80326348327529c7d WatchSource:0}: Error finding container 69e11b1d0bebcbb8ba53805687b3ad26935cf813404831b80326348327529c7d: Status 404 returned error can't find the container with id 69e11b1d0bebcbb8ba53805687b3ad26935cf813404831b80326348327529c7d
Sep 09 06:20:32.730671 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:32.730617    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6b5d89-hk5nh" event={"ID":"466a84a3-8dea-4754-8df8-654181b60e3d","Type":"ContainerStarted","Data":"69e11b1d0bebcbb8ba53805687b3ad26935cf813404831b80326348327529c7d"}
Sep 09 06:20:33.737452 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:33.737417    2579 generic.go:356] "Generic (PLEG): container finished" podID="466a84a3-8dea-4754-8df8-654181b60e3d" containerID="3ad1e7d4e96e8cb3a5dafd3ae703058dff60a8f8ed0700ad3209e8299956560d" exitCode=0
Sep 09 06:20:33.737876 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:33.737471    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6b5d89-hk5nh" event={"ID":"466a84a3-8dea-4754-8df8-654181b60e3d","Type":"ContainerDied","Data":"3ad1e7d4e96e8cb3a5dafd3ae703058dff60a8f8ed0700ad3209e8299956560d"}
Sep 09 06:20:34.867942 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:34.867920    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6b5d89-hk5nh"
Sep 09 06:20:34.900335 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:34.900309    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/466a84a3-8dea-4754-8df8-654181b60e3d-maas-api-service-ca\") pod \"466a84a3-8dea-4754-8df8-654181b60e3d\" (UID: \"466a84a3-8dea-4754-8df8-654181b60e3d\") "
Sep 09 06:20:34.900671 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:34.900648    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/466a84a3-8dea-4754-8df8-654181b60e3d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "466a84a3-8dea-4754-8df8-654181b60e3d" (UID: "466a84a3-8dea-4754-8df8-654181b60e3d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:20:35.001981 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:35.001918    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/466a84a3-8dea-4754-8df8-654181b60e3d-maas-api-service-ca\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:20:35.606254 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:35.606227    2579 scope.go:117] "RemoveContainer" containerID="99a7388760d6060d3cfe05af701bd687828c0059e77fc38d6f78e1b12557e1ba"
Sep 09 06:20:35.614085 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:35.614066    2579 scope.go:117] "RemoveContainer" containerID="64edcf84c041b79d00b7ebfdb44f1a37d8dcecdd7b7d7ea93da343137512516b"
Sep 09 06:20:35.621304 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:35.621285    2579 scope.go:117] "RemoveContainer" containerID="042e6f6724b97df117dabde2bff052fa082ed08c25d09a630ec46446a0631a20"
Sep 09 06:20:35.747639 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:35.747604    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6b5d89-hk5nh"
Sep 09 06:20:35.747740 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:35.747658    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6b5d89-hk5nh" event={"ID":"466a84a3-8dea-4754-8df8-654181b60e3d","Type":"ContainerDied","Data":"69e11b1d0bebcbb8ba53805687b3ad26935cf813404831b80326348327529c7d"}
Sep 09 06:20:35.747740 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:35.747705    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69e11b1d0bebcbb8ba53805687b3ad26935cf813404831b80326348327529c7d"
Sep 09 06:20:36.934686 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:36.934658    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"
Sep 09 06:20:37.018258 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:37.018200    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34ec6702-7142-4ff6-bba5-cf1969ac2a46-istio-envoy\") pod \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\" (UID: \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\") "
Sep 09 06:20:37.018258 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:37.018238    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34ec6702-7142-4ff6-bba5-cf1969ac2a46-istio-data\") pod \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\" (UID: \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\") "
Sep 09 06:20:37.018440 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:37.018263    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ec6702-7142-4ff6-bba5-cf1969ac2a46-credential-socket\") pod \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\" (UID: \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\") "
Sep 09 06:20:37.018440 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:37.018289    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjjqx\" (UniqueName: \"kubernetes.io/projected/34ec6702-7142-4ff6-bba5-cf1969ac2a46-kube-api-access-rjjqx\") pod \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\" (UID: \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\") "
Sep 09 06:20:37.018440 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:37.018309    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34ec6702-7142-4ff6-bba5-cf1969ac2a46-istiod-ca-cert\") pod \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\" (UID: \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\") "
Sep 09 06:20:37.018440 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:37.018333    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34ec6702-7142-4ff6-bba5-cf1969ac2a46-istio-podinfo\") pod \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\" (UID: \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\") "
Sep 09 06:20:37.018440 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:37.018368    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ec6702-7142-4ff6-bba5-cf1969ac2a46-workload-socket\") pod \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\" (UID: \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\") "
Sep 09 06:20:37.018440 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:37.018423    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34ec6702-7142-4ff6-bba5-cf1969ac2a46-workload-certs\") pod \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\" (UID: \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\") "
Sep 09 06:20:37.018803 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:37.018476    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34ec6702-7142-4ff6-bba5-cf1969ac2a46-istio-token\") pod \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\" (UID: \"34ec6702-7142-4ff6-bba5-cf1969ac2a46\") "
Sep 09 06:20:37.018803 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:37.018479    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34ec6702-7142-4ff6-bba5-cf1969ac2a46-istio-data" (OuterVolumeSpecName: "istio-data") pod "34ec6702-7142-4ff6-bba5-cf1969ac2a46" (UID: "34ec6702-7142-4ff6-bba5-cf1969ac2a46"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:20:37.018803 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:37.018699    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34ec6702-7142-4ff6-bba5-cf1969ac2a46-istio-data\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:20:37.018803 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:37.018669    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34ec6702-7142-4ff6-bba5-cf1969ac2a46-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "34ec6702-7142-4ff6-bba5-cf1969ac2a46" (UID: "34ec6702-7142-4ff6-bba5-cf1969ac2a46"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:20:37.018803 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:37.018735    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34ec6702-7142-4ff6-bba5-cf1969ac2a46-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "34ec6702-7142-4ff6-bba5-cf1969ac2a46" (UID: "34ec6702-7142-4ff6-bba5-cf1969ac2a46"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:20:37.018803 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:37.018748    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34ec6702-7142-4ff6-bba5-cf1969ac2a46-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "34ec6702-7142-4ff6-bba5-cf1969ac2a46" (UID: "34ec6702-7142-4ff6-bba5-cf1969ac2a46"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:20:37.018803 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:37.018755    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34ec6702-7142-4ff6-bba5-cf1969ac2a46-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "34ec6702-7142-4ff6-bba5-cf1969ac2a46" (UID: "34ec6702-7142-4ff6-bba5-cf1969ac2a46"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:20:37.020469 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:37.020444    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34ec6702-7142-4ff6-bba5-cf1969ac2a46-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "34ec6702-7142-4ff6-bba5-cf1969ac2a46" (UID: "34ec6702-7142-4ff6-bba5-cf1969ac2a46"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:20:37.020709 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:37.020680    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34ec6702-7142-4ff6-bba5-cf1969ac2a46-kube-api-access-rjjqx" (OuterVolumeSpecName: "kube-api-access-rjjqx") pod "34ec6702-7142-4ff6-bba5-cf1969ac2a46" (UID: "34ec6702-7142-4ff6-bba5-cf1969ac2a46"). InnerVolumeSpecName "kube-api-access-rjjqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:20:37.020808 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:37.020735    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34ec6702-7142-4ff6-bba5-cf1969ac2a46-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "34ec6702-7142-4ff6-bba5-cf1969ac2a46" (UID: "34ec6702-7142-4ff6-bba5-cf1969ac2a46"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 06:20:37.020808 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:37.020780    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34ec6702-7142-4ff6-bba5-cf1969ac2a46-istio-token" (OuterVolumeSpecName: "istio-token") pod "34ec6702-7142-4ff6-bba5-cf1969ac2a46" (UID: "34ec6702-7142-4ff6-bba5-cf1969ac2a46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:20:37.119912 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:37.119883    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ec6702-7142-4ff6-bba5-cf1969ac2a46-credential-socket\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:20:37.119912 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:37.119908    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjjqx\" (UniqueName: \"kubernetes.io/projected/34ec6702-7142-4ff6-bba5-cf1969ac2a46-kube-api-access-rjjqx\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:20:37.120057 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:37.119922    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34ec6702-7142-4ff6-bba5-cf1969ac2a46-istiod-ca-cert\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:20:37.120057 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:37.119935    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34ec6702-7142-4ff6-bba5-cf1969ac2a46-istio-podinfo\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:20:37.120057 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:37.119946    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ec6702-7142-4ff6-bba5-cf1969ac2a46-workload-socket\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:20:37.120057 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:37.119959    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34ec6702-7142-4ff6-bba5-cf1969ac2a46-workload-certs\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:20:37.120057 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:37.119971    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34ec6702-7142-4ff6-bba5-cf1969ac2a46-istio-token\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:20:37.120057 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:37.119982    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34ec6702-7142-4ff6-bba5-cf1969ac2a46-istio-envoy\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:20:37.757720 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:37.757693    2579 generic.go:356] "Generic (PLEG): container finished" podID="34ec6702-7142-4ff6-bba5-cf1969ac2a46" containerID="08780f5d4e945de992eecb63e1999e3498984567e1f735c66d767ec039fa7202" exitCode=0
Sep 09 06:20:37.757894 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:37.757737    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"
Sep 09 06:20:37.757894 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:37.757781    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq" event={"ID":"34ec6702-7142-4ff6-bba5-cf1969ac2a46","Type":"ContainerDied","Data":"08780f5d4e945de992eecb63e1999e3498984567e1f735c66d767ec039fa7202"}
Sep 09 06:20:37.757894 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:37.757840    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq" event={"ID":"34ec6702-7142-4ff6-bba5-cf1969ac2a46","Type":"ContainerDied","Data":"e48d774c581814411d1cd3a54cd02edf3296a2929d610a475f0be27655d097ff"}
Sep 09 06:20:37.757894 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:37.757869    2579 scope.go:117] "RemoveContainer" containerID="08780f5d4e945de992eecb63e1999e3498984567e1f735c66d767ec039fa7202"
Sep 09 06:20:37.766220 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:37.766204    2579 scope.go:117] "RemoveContainer" containerID="4dad64755eeb62c362b115b033143321f147ac76498f6e8e856d9e5e79a542d6"
Sep 09 06:20:37.774209 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:37.774185    2579 scope.go:117] "RemoveContainer" containerID="08780f5d4e945de992eecb63e1999e3498984567e1f735c66d767ec039fa7202"
Sep 09 06:20:37.774476 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:20:37.774458    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08780f5d4e945de992eecb63e1999e3498984567e1f735c66d767ec039fa7202\": container with ID starting with 08780f5d4e945de992eecb63e1999e3498984567e1f735c66d767ec039fa7202 not found: ID does not exist" containerID="08780f5d4e945de992eecb63e1999e3498984567e1f735c66d767ec039fa7202"
Sep 09 06:20:37.774552 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:37.774482    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08780f5d4e945de992eecb63e1999e3498984567e1f735c66d767ec039fa7202"} err="failed to get container status \"08780f5d4e945de992eecb63e1999e3498984567e1f735c66d767ec039fa7202\": rpc error: code = NotFound desc = could not find container \"08780f5d4e945de992eecb63e1999e3498984567e1f735c66d767ec039fa7202\": container with ID starting with 08780f5d4e945de992eecb63e1999e3498984567e1f735c66d767ec039fa7202 not found: ID does not exist"
Sep 09 06:20:37.774552 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:37.774500    2579 scope.go:117] "RemoveContainer" containerID="4dad64755eeb62c362b115b033143321f147ac76498f6e8e856d9e5e79a542d6"
Sep 09 06:20:37.774766 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:20:37.774749    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dad64755eeb62c362b115b033143321f147ac76498f6e8e856d9e5e79a542d6\": container with ID starting with 4dad64755eeb62c362b115b033143321f147ac76498f6e8e856d9e5e79a542d6 not found: ID does not exist" containerID="4dad64755eeb62c362b115b033143321f147ac76498f6e8e856d9e5e79a542d6"
Sep 09 06:20:37.774830 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:37.774772    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dad64755eeb62c362b115b033143321f147ac76498f6e8e856d9e5e79a542d6"} err="failed to get container status \"4dad64755eeb62c362b115b033143321f147ac76498f6e8e856d9e5e79a542d6\": rpc error: code = NotFound desc = could not find container \"4dad64755eeb62c362b115b033143321f147ac76498f6e8e856d9e5e79a542d6\": container with ID starting with 4dad64755eeb62c362b115b033143321f147ac76498f6e8e856d9e5e79a542d6 not found: ID does not exist"
Sep 09 06:20:37.775878 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:37.775857    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"]
Sep 09 06:20:37.779791 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:37.779768    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-32ea4d-openshift-default-5cdf75fdb6-cwmwq"]
Sep 09 06:20:39.147353 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:39.147318    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34ec6702-7142-4ff6-bba5-cf1969ac2a46" path="/var/lib/kubelet/pods/34ec6702-7142-4ff6-bba5-cf1969ac2a46/volumes"
Sep 09 06:20:42.611177 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:42.611149    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5d89-29815575-mx82j"]
Sep 09 06:20:42.617023 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:42.617001    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5d89-29815575-mx82j"]
Sep 09 06:20:42.827180 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:42.827152    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6b5d89-hk5nh"]
Sep 09 06:20:42.830073 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:42.830046    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6b5d89-hk5nh"]
Sep 09 06:20:43.147689 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:43.147654    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a514b3e-a43e-4dc1-bf63-277e9c94a07a" path="/var/lib/kubelet/pods/3a514b3e-a43e-4dc1-bf63-277e9c94a07a/volumes"
Sep 09 06:20:43.147972 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:43.147960    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="466a84a3-8dea-4754-8df8-654181b60e3d" path="/var/lib/kubelet/pods/466a84a3-8dea-4754-8df8-654181b60e3d/volumes"
Sep 09 06:20:45.400012 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:45.399981    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:20:49.325066 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:49.325037    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"]
Sep 09 06:20:49.325601 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:49.325224    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4" podUID="0a027fe8-30f7-424a-9634-5341f728d4b0" containerName="istio-proxy" containerID="cri-o://a648646b8865fafc9345de206318e04436a65c8874e756a5b26b794101950bb9" gracePeriod=30
Sep 09 06:20:49.696314 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:49.696232    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7cf6be-6r9fd"]
Sep 09 06:20:49.696616 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:49.696598    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34ec6702-7142-4ff6-bba5-cf1969ac2a46" containerName="istio-proxy"
Sep 09 06:20:49.696726 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:49.696620    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="34ec6702-7142-4ff6-bba5-cf1969ac2a46" containerName="istio-proxy"
Sep 09 06:20:49.696726 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:49.696662    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="466a84a3-8dea-4754-8df8-654181b60e3d" containerName="revoke-keys"
Sep 09 06:20:49.696726 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:49.696671    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="466a84a3-8dea-4754-8df8-654181b60e3d" containerName="revoke-keys"
Sep 09 06:20:49.696888 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:49.696765    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="466a84a3-8dea-4754-8df8-654181b60e3d" containerName="revoke-keys"
Sep 09 06:20:49.696888 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:49.696778    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="34ec6702-7142-4ff6-bba5-cf1969ac2a46" containerName="istio-proxy"
Sep 09 06:20:49.696888 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:49.696786    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="34ec6702-7142-4ff6-bba5-cf1969ac2a46" containerName="istio-proxy"
Sep 09 06:20:49.701313 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:49.701290    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7cf6be-6r9fd"
Sep 09 06:20:49.703489 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:49.703468    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 06:20:49.703594 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:49.703468    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ksjk6\""
Sep 09 06:20:49.707230 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:49.707208    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7cf6be-6r9fd"]
Sep 09 06:20:49.810986 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:49.810950    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73e75e1c-d716-49f7-913f-ab9979ca5d6e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7cf6be-6r9fd\" (UID: \"73e75e1c-d716-49f7-913f-ab9979ca5d6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7cf6be-6r9fd"
Sep 09 06:20:49.912347 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:49.912319    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73e75e1c-d716-49f7-913f-ab9979ca5d6e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7cf6be-6r9fd\" (UID: \"73e75e1c-d716-49f7-913f-ab9979ca5d6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7cf6be-6r9fd"
Sep 09 06:20:49.912986 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:49.912961    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73e75e1c-d716-49f7-913f-ab9979ca5d6e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7cf6be-6r9fd\" (UID: \"73e75e1c-d716-49f7-913f-ab9979ca5d6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7cf6be-6r9fd"
Sep 09 06:20:50.013393 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:50.013324    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7cf6be-6r9fd"
Sep 09 06:20:50.281945 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:50.281912    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7cf6be-6r9fd"]
Sep 09 06:20:50.284197 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:20:50.284173    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73e75e1c_d716_49f7_913f_ab9979ca5d6e.slice/crio-dcab2bc7b5c13e6b9fd2793d0adb3a617c0941390e0e6dcdb007181b563454f1 WatchSource:0}: Error finding container dcab2bc7b5c13e6b9fd2793d0adb3a617c0941390e0e6dcdb007181b563454f1: Status 404 returned error can't find the container with id dcab2bc7b5c13e6b9fd2793d0adb3a617c0941390e0e6dcdb007181b563454f1
Sep 09 06:20:50.819734 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:50.819693    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7cf6be-6r9fd" event={"ID":"73e75e1c-d716-49f7-913f-ab9979ca5d6e","Type":"ContainerStarted","Data":"9d04a7f151940f8992871a9818690f90953251b6c3a5c1d737aa4f5d27aa024e"}
Sep 09 06:20:50.820155 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:50.819759    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7cf6be-6r9fd" event={"ID":"73e75e1c-d716-49f7-913f-ab9979ca5d6e","Type":"ContainerStarted","Data":"dcab2bc7b5c13e6b9fd2793d0adb3a617c0941390e0e6dcdb007181b563454f1"}
Sep 09 06:20:50.838992 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:50.838950    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7cf6be-6r9fd" podStartSLOduration=1.838935135 podStartE2EDuration="1.838935135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:20:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:20:50.838038564 +0000 UTC m=+1876.178526474" watchObservedRunningTime="2026-09-09 06:20:50.838935135 +0000 UTC m=+1876.179423075"
Sep 09 06:20:51.825902 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:51.825866    2579 generic.go:356] "Generic (PLEG): container finished" podID="73e75e1c-d716-49f7-913f-ab9979ca5d6e" containerID="9d04a7f151940f8992871a9818690f90953251b6c3a5c1d737aa4f5d27aa024e" exitCode=0
Sep 09 06:20:51.826286 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:51.825936    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7cf6be-6r9fd" event={"ID":"73e75e1c-d716-49f7-913f-ab9979ca5d6e","Type":"ContainerDied","Data":"9d04a7f151940f8992871a9818690f90953251b6c3a5c1d737aa4f5d27aa024e"}
Sep 09 06:20:52.951204 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:52.951181    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7cf6be-6r9fd"
Sep 09 06:20:53.036791 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:53.036761    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73e75e1c-d716-49f7-913f-ab9979ca5d6e-maas-api-service-ca\") pod \"73e75e1c-d716-49f7-913f-ab9979ca5d6e\" (UID: \"73e75e1c-d716-49f7-913f-ab9979ca5d6e\") "
Sep 09 06:20:53.037084 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:53.037062    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73e75e1c-d716-49f7-913f-ab9979ca5d6e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "73e75e1c-d716-49f7-913f-ab9979ca5d6e" (UID: "73e75e1c-d716-49f7-913f-ab9979ca5d6e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:20:53.137847 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:53.137790    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73e75e1c-d716-49f7-913f-ab9979ca5d6e-maas-api-service-ca\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:20:53.835717 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:53.835683    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7cf6be-6r9fd"
Sep 09 06:20:53.835897 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:53.835721    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7cf6be-6r9fd" event={"ID":"73e75e1c-d716-49f7-913f-ab9979ca5d6e","Type":"ContainerDied","Data":"dcab2bc7b5c13e6b9fd2793d0adb3a617c0941390e0e6dcdb007181b563454f1"}
Sep 09 06:20:53.835897 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:53.835772    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcab2bc7b5c13e6b9fd2793d0adb3a617c0941390e0e6dcdb007181b563454f1"
Sep 09 06:20:54.568450 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:54.568428    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"
Sep 09 06:20:54.648106 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:54.648042    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a027fe8-30f7-424a-9634-5341f728d4b0-workload-socket\") pod \"0a027fe8-30f7-424a-9634-5341f728d4b0\" (UID: \"0a027fe8-30f7-424a-9634-5341f728d4b0\") "
Sep 09 06:20:54.648229 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:54.648105    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcx49\" (UniqueName: \"kubernetes.io/projected/0a027fe8-30f7-424a-9634-5341f728d4b0-kube-api-access-jcx49\") pod \"0a027fe8-30f7-424a-9634-5341f728d4b0\" (UID: \"0a027fe8-30f7-424a-9634-5341f728d4b0\") "
Sep 09 06:20:54.648229 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:54.648129    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a027fe8-30f7-424a-9634-5341f728d4b0-istio-podinfo\") pod \"0a027fe8-30f7-424a-9634-5341f728d4b0\" (UID: \"0a027fe8-30f7-424a-9634-5341f728d4b0\") "
Sep 09 06:20:54.648229 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:54.648157    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a027fe8-30f7-424a-9634-5341f728d4b0-istio-envoy\") pod \"0a027fe8-30f7-424a-9634-5341f728d4b0\" (UID: \"0a027fe8-30f7-424a-9634-5341f728d4b0\") "
Sep 09 06:20:54.648229 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:54.648178    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a027fe8-30f7-424a-9634-5341f728d4b0-istiod-ca-cert\") pod \"0a027fe8-30f7-424a-9634-5341f728d4b0\" (UID: \"0a027fe8-30f7-424a-9634-5341f728d4b0\") "
Sep 09 06:20:54.648229 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:54.648196    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a027fe8-30f7-424a-9634-5341f728d4b0-credential-socket\") pod \"0a027fe8-30f7-424a-9634-5341f728d4b0\" (UID: \"0a027fe8-30f7-424a-9634-5341f728d4b0\") "
Sep 09 06:20:54.648486 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:54.648308    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a027fe8-30f7-424a-9634-5341f728d4b0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0a027fe8-30f7-424a-9634-5341f728d4b0" (UID: "0a027fe8-30f7-424a-9634-5341f728d4b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:20:54.648486 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:54.648318    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a027fe8-30f7-424a-9634-5341f728d4b0-workload-certs\") pod \"0a027fe8-30f7-424a-9634-5341f728d4b0\" (UID: \"0a027fe8-30f7-424a-9634-5341f728d4b0\") "
Sep 09 06:20:54.648486 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:54.648365    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a027fe8-30f7-424a-9634-5341f728d4b0-istio-data\") pod \"0a027fe8-30f7-424a-9634-5341f728d4b0\" (UID: \"0a027fe8-30f7-424a-9634-5341f728d4b0\") "
Sep 09 06:20:54.648486 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:54.648400    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a027fe8-30f7-424a-9634-5341f728d4b0-istio-token\") pod \"0a027fe8-30f7-424a-9634-5341f728d4b0\" (UID: \"0a027fe8-30f7-424a-9634-5341f728d4b0\") "
Sep 09 06:20:54.648738 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:54.648665    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a027fe8-30f7-424a-9634-5341f728d4b0-istio-data" (OuterVolumeSpecName: "istio-data") pod "0a027fe8-30f7-424a-9634-5341f728d4b0" (UID: "0a027fe8-30f7-424a-9634-5341f728d4b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:20:54.648738 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:54.648699    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a027fe8-30f7-424a-9634-5341f728d4b0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0a027fe8-30f7-424a-9634-5341f728d4b0" (UID: "0a027fe8-30f7-424a-9634-5341f728d4b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:20:54.648832 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:54.648750    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a027fe8-30f7-424a-9634-5341f728d4b0-istio-data\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:20:54.648832 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:54.648769    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a027fe8-30f7-424a-9634-5341f728d4b0-workload-socket\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:20:54.648910 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:54.648588    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a027fe8-30f7-424a-9634-5341f728d4b0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0a027fe8-30f7-424a-9634-5341f728d4b0" (UID: "0a027fe8-30f7-424a-9634-5341f728d4b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:20:54.649170 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:54.649148    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a027fe8-30f7-424a-9634-5341f728d4b0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0a027fe8-30f7-424a-9634-5341f728d4b0" (UID: "0a027fe8-30f7-424a-9634-5341f728d4b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:20:54.650517 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:54.650489    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a027fe8-30f7-424a-9634-5341f728d4b0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0a027fe8-30f7-424a-9634-5341f728d4b0" (UID: "0a027fe8-30f7-424a-9634-5341f728d4b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 06:20:54.650517 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:54.650506    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a027fe8-30f7-424a-9634-5341f728d4b0-kube-api-access-jcx49" (OuterVolumeSpecName: "kube-api-access-jcx49") pod "0a027fe8-30f7-424a-9634-5341f728d4b0" (UID: "0a027fe8-30f7-424a-9634-5341f728d4b0"). InnerVolumeSpecName "kube-api-access-jcx49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:20:54.651077 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:54.651057    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a027fe8-30f7-424a-9634-5341f728d4b0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0a027fe8-30f7-424a-9634-5341f728d4b0" (UID: "0a027fe8-30f7-424a-9634-5341f728d4b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:20:54.651146 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:54.651111    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a027fe8-30f7-424a-9634-5341f728d4b0-istio-token" (OuterVolumeSpecName: "istio-token") pod "0a027fe8-30f7-424a-9634-5341f728d4b0" (UID: "0a027fe8-30f7-424a-9634-5341f728d4b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:20:54.749801 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:54.749773    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcx49\" (UniqueName: \"kubernetes.io/projected/0a027fe8-30f7-424a-9634-5341f728d4b0-kube-api-access-jcx49\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:20:54.749801 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:54.749799    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a027fe8-30f7-424a-9634-5341f728d4b0-istio-podinfo\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:20:54.749951 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:54.749812    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a027fe8-30f7-424a-9634-5341f728d4b0-istio-envoy\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:20:54.749951 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:54.749825    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a027fe8-30f7-424a-9634-5341f728d4b0-istiod-ca-cert\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:20:54.749951 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:54.749836    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a027fe8-30f7-424a-9634-5341f728d4b0-credential-socket\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:20:54.749951 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:54.749848    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a027fe8-30f7-424a-9634-5341f728d4b0-workload-certs\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:20:54.749951 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:54.749860    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a027fe8-30f7-424a-9634-5341f728d4b0-istio-token\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:20:54.842073 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:54.842045    2579 generic.go:356] "Generic (PLEG): container finished" podID="0a027fe8-30f7-424a-9634-5341f728d4b0" containerID="a648646b8865fafc9345de206318e04436a65c8874e756a5b26b794101950bb9" exitCode=0
Sep 09 06:20:54.842150 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:54.842086    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"
Sep 09 06:20:54.842150 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:54.842132    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4" event={"ID":"0a027fe8-30f7-424a-9634-5341f728d4b0","Type":"ContainerDied","Data":"a648646b8865fafc9345de206318e04436a65c8874e756a5b26b794101950bb9"}
Sep 09 06:20:54.842226 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:54.842179    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4" event={"ID":"0a027fe8-30f7-424a-9634-5341f728d4b0","Type":"ContainerDied","Data":"9fe6774d8c37ebc97716ae79a87347ffb0d45c986a4ad630b6af79d6d8ee6bde"}
Sep 09 06:20:54.842226 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:54.842200    2579 scope.go:117] "RemoveContainer" containerID="a648646b8865fafc9345de206318e04436a65c8874e756a5b26b794101950bb9"
Sep 09 06:20:54.853010 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:54.852995    2579 scope.go:117] "RemoveContainer" containerID="18aac7bddf92ff25354888ff84c7661a007064c924a765d44c8dc802cf6e56d8"
Sep 09 06:20:54.860767 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:54.860751    2579 scope.go:117] "RemoveContainer" containerID="a648646b8865fafc9345de206318e04436a65c8874e756a5b26b794101950bb9"
Sep 09 06:20:54.861012 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:20:54.860996    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a648646b8865fafc9345de206318e04436a65c8874e756a5b26b794101950bb9\": container with ID starting with a648646b8865fafc9345de206318e04436a65c8874e756a5b26b794101950bb9 not found: ID does not exist" containerID="a648646b8865fafc9345de206318e04436a65c8874e756a5b26b794101950bb9"
Sep 09 06:20:54.861048 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:54.861020    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a648646b8865fafc9345de206318e04436a65c8874e756a5b26b794101950bb9"} err="failed to get container status \"a648646b8865fafc9345de206318e04436a65c8874e756a5b26b794101950bb9\": rpc error: code = NotFound desc = could not find container \"a648646b8865fafc9345de206318e04436a65c8874e756a5b26b794101950bb9\": container with ID starting with a648646b8865fafc9345de206318e04436a65c8874e756a5b26b794101950bb9 not found: ID does not exist"
Sep 09 06:20:54.861048 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:54.861035    2579 scope.go:117] "RemoveContainer" containerID="18aac7bddf92ff25354888ff84c7661a007064c924a765d44c8dc802cf6e56d8"
Sep 09 06:20:54.861244 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:20:54.861230    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18aac7bddf92ff25354888ff84c7661a007064c924a765d44c8dc802cf6e56d8\": container with ID starting with 18aac7bddf92ff25354888ff84c7661a007064c924a765d44c8dc802cf6e56d8 not found: ID does not exist" containerID="18aac7bddf92ff25354888ff84c7661a007064c924a765d44c8dc802cf6e56d8"
Sep 09 06:20:54.861283 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:54.861249    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18aac7bddf92ff25354888ff84c7661a007064c924a765d44c8dc802cf6e56d8"} err="failed to get container status \"18aac7bddf92ff25354888ff84c7661a007064c924a765d44c8dc802cf6e56d8\": rpc error: code = NotFound desc = could not find container \"18aac7bddf92ff25354888ff84c7661a007064c924a765d44c8dc802cf6e56d8\": container with ID starting with 18aac7bddf92ff25354888ff84c7661a007064c924a765d44c8dc802cf6e56d8 not found: ID does not exist"
Sep 09 06:20:54.864822 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:54.864798    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"]
Sep 09 06:20:54.868309 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:54.868289    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-6b5d89-openshift-default-c6c54d956-nwww4"]
Sep 09 06:20:55.147703 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:20:55.147676    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a027fe8-30f7-424a-9634-5341f728d4b0" path="/var/lib/kubelet/pods/0a027fe8-30f7-424a-9634-5341f728d4b0/volumes"
Sep 09 06:21:00.167577 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:00.167497    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7cf6be-59fdf7cbf5-fv8wn"]
Sep 09 06:21:00.168026 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:00.167788    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7cf6be-59fdf7cbf5-fv8wn" podUID="3f1bc250-9dcd-437f-ae35-af778ebdad1e" containerName="payload-processing" containerID="cri-o://0e35f676b177b456c654ba7b00855caebd6952bf69a83b3608bba6d229828958" gracePeriod=30
Sep 09 06:21:00.173109 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:00.173076    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cf6be-7f5595974-mqgd8"]
Sep 09 06:21:00.173380 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:00.173329    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cf6be-7f5595974-mqgd8" podUID="2666d09c-dba9-4a67-8b69-df034c55ce9a" containerName="payload-pre-processing" containerID="cri-o://a4cc44ad704a3ecc9b85bd3b591e87edd2335505a12c04e6a1c318b9c7a29e00" gracePeriod=30
Sep 09 06:21:00.329304 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:00.329197    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7cf6be-6r9fd"]
Sep 09 06:21:00.331027 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:00.331000    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7cf6be-6r9fd"]
Sep 09 06:21:00.411199 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:00.411174    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7cf6be-59fdf7cbf5-fv8wn"
Sep 09 06:21:00.444544 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:00.444523    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cf6be-7f5595974-mqgd8"
Sep 09 06:21:00.493324 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:00.493293    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7pmr\" (UniqueName: \"kubernetes.io/projected/2666d09c-dba9-4a67-8b69-df034c55ce9a-kube-api-access-s7pmr\") pod \"2666d09c-dba9-4a67-8b69-df034c55ce9a\" (UID: \"2666d09c-dba9-4a67-8b69-df034c55ce9a\") "
Sep 09 06:21:00.493449 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:00.493396    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6qr4\" (UniqueName: \"kubernetes.io/projected/3f1bc250-9dcd-437f-ae35-af778ebdad1e-kube-api-access-c6qr4\") pod \"3f1bc250-9dcd-437f-ae35-af778ebdad1e\" (UID: \"3f1bc250-9dcd-437f-ae35-af778ebdad1e\") "
Sep 09 06:21:00.493492 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:00.493448    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f1bc250-9dcd-437f-ae35-af778ebdad1e-plugins-config-volume\") pod \"3f1bc250-9dcd-437f-ae35-af778ebdad1e\" (UID: \"3f1bc250-9dcd-437f-ae35-af778ebdad1e\") "
Sep 09 06:21:00.493492 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:00.493477    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2666d09c-dba9-4a67-8b69-df034c55ce9a-plugins-config-volume\") pod \"2666d09c-dba9-4a67-8b69-df034c55ce9a\" (UID: \"2666d09c-dba9-4a67-8b69-df034c55ce9a\") "
Sep 09 06:21:00.493879 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:00.493853    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f1bc250-9dcd-437f-ae35-af778ebdad1e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3f1bc250-9dcd-437f-ae35-af778ebdad1e" (UID: "3f1bc250-9dcd-437f-ae35-af778ebdad1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:21:00.493986 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:00.493883    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2666d09c-dba9-4a67-8b69-df034c55ce9a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2666d09c-dba9-4a67-8b69-df034c55ce9a" (UID: "2666d09c-dba9-4a67-8b69-df034c55ce9a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:21:00.495586 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:00.495571    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2666d09c-dba9-4a67-8b69-df034c55ce9a-kube-api-access-s7pmr" (OuterVolumeSpecName: "kube-api-access-s7pmr") pod "2666d09c-dba9-4a67-8b69-df034c55ce9a" (UID: "2666d09c-dba9-4a67-8b69-df034c55ce9a"). InnerVolumeSpecName "kube-api-access-s7pmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:21:00.495664 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:00.495614    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f1bc250-9dcd-437f-ae35-af778ebdad1e-kube-api-access-c6qr4" (OuterVolumeSpecName: "kube-api-access-c6qr4") pod "3f1bc250-9dcd-437f-ae35-af778ebdad1e" (UID: "3f1bc250-9dcd-437f-ae35-af778ebdad1e"). InnerVolumeSpecName "kube-api-access-c6qr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:21:00.594229 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:00.594210    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f1bc250-9dcd-437f-ae35-af778ebdad1e-plugins-config-volume\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:21:00.594229 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:00.594230    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2666d09c-dba9-4a67-8b69-df034c55ce9a-plugins-config-volume\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:21:00.594347 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:00.594239    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7pmr\" (UniqueName: \"kubernetes.io/projected/2666d09c-dba9-4a67-8b69-df034c55ce9a-kube-api-access-s7pmr\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:21:00.594347 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:00.594248    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6qr4\" (UniqueName: \"kubernetes.io/projected/3f1bc250-9dcd-437f-ae35-af778ebdad1e-kube-api-access-c6qr4\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:21:00.871102 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:00.871072    2579 generic.go:356] "Generic (PLEG): container finished" podID="2666d09c-dba9-4a67-8b69-df034c55ce9a" containerID="a4cc44ad704a3ecc9b85bd3b591e87edd2335505a12c04e6a1c318b9c7a29e00" exitCode=0
Sep 09 06:21:00.871234 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:00.871113    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cf6be-7f5595974-mqgd8"
Sep 09 06:21:00.871234 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:00.871158    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cf6be-7f5595974-mqgd8" event={"ID":"2666d09c-dba9-4a67-8b69-df034c55ce9a","Type":"ContainerDied","Data":"a4cc44ad704a3ecc9b85bd3b591e87edd2335505a12c04e6a1c318b9c7a29e00"}
Sep 09 06:21:00.871234 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:00.871205    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cf6be-7f5595974-mqgd8" event={"ID":"2666d09c-dba9-4a67-8b69-df034c55ce9a","Type":"ContainerDied","Data":"c74404be11a2ef7562348ada500e8b1cec6a49ee51c7837118be82556cfe2241"}
Sep 09 06:21:00.871234 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:00.871228    2579 scope.go:117] "RemoveContainer" containerID="a4cc44ad704a3ecc9b85bd3b591e87edd2335505a12c04e6a1c318b9c7a29e00"
Sep 09 06:21:00.872682 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:00.872655    2579 generic.go:356] "Generic (PLEG): container finished" podID="3f1bc250-9dcd-437f-ae35-af778ebdad1e" containerID="0e35f676b177b456c654ba7b00855caebd6952bf69a83b3608bba6d229828958" exitCode=0
Sep 09 06:21:00.872769 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:00.872682    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7cf6be-59fdf7cbf5-fv8wn"
Sep 09 06:21:00.872769 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:00.872696    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7cf6be-59fdf7cbf5-fv8wn" event={"ID":"3f1bc250-9dcd-437f-ae35-af778ebdad1e","Type":"ContainerDied","Data":"0e35f676b177b456c654ba7b00855caebd6952bf69a83b3608bba6d229828958"}
Sep 09 06:21:00.872769 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:00.872736    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7cf6be-59fdf7cbf5-fv8wn" event={"ID":"3f1bc250-9dcd-437f-ae35-af778ebdad1e","Type":"ContainerDied","Data":"80dac74ac731dbb0296a89067e517ddda269aafe18f33cfcf41e3c2ca64580d1"}
Sep 09 06:21:00.880535 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:00.880522    2579 scope.go:117] "RemoveContainer" containerID="a4cc44ad704a3ecc9b85bd3b591e87edd2335505a12c04e6a1c318b9c7a29e00"
Sep 09 06:21:00.880828 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:21:00.880807    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4cc44ad704a3ecc9b85bd3b591e87edd2335505a12c04e6a1c318b9c7a29e00\": container with ID starting with a4cc44ad704a3ecc9b85bd3b591e87edd2335505a12c04e6a1c318b9c7a29e00 not found: ID does not exist" containerID="a4cc44ad704a3ecc9b85bd3b591e87edd2335505a12c04e6a1c318b9c7a29e00"
Sep 09 06:21:00.880913 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:00.880834    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4cc44ad704a3ecc9b85bd3b591e87edd2335505a12c04e6a1c318b9c7a29e00"} err="failed to get container status \"a4cc44ad704a3ecc9b85bd3b591e87edd2335505a12c04e6a1c318b9c7a29e00\": rpc error: code = NotFound desc = could not find container \"a4cc44ad704a3ecc9b85bd3b591e87edd2335505a12c04e6a1c318b9c7a29e00\": container with ID starting with a4cc44ad704a3ecc9b85bd3b591e87edd2335505a12c04e6a1c318b9c7a29e00 not found: ID does not exist"
Sep 09 06:21:00.880913 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:00.880848    2579 scope.go:117] "RemoveContainer" containerID="0e35f676b177b456c654ba7b00855caebd6952bf69a83b3608bba6d229828958"
Sep 09 06:21:00.888097 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:00.888075    2579 scope.go:117] "RemoveContainer" containerID="0e35f676b177b456c654ba7b00855caebd6952bf69a83b3608bba6d229828958"
Sep 09 06:21:00.888343 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:21:00.888327    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e35f676b177b456c654ba7b00855caebd6952bf69a83b3608bba6d229828958\": container with ID starting with 0e35f676b177b456c654ba7b00855caebd6952bf69a83b3608bba6d229828958 not found: ID does not exist" containerID="0e35f676b177b456c654ba7b00855caebd6952bf69a83b3608bba6d229828958"
Sep 09 06:21:00.888389 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:00.888353    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e35f676b177b456c654ba7b00855caebd6952bf69a83b3608bba6d229828958"} err="failed to get container status \"0e35f676b177b456c654ba7b00855caebd6952bf69a83b3608bba6d229828958\": rpc error: code = NotFound desc = could not find container \"0e35f676b177b456c654ba7b00855caebd6952bf69a83b3608bba6d229828958\": container with ID starting with 0e35f676b177b456c654ba7b00855caebd6952bf69a83b3608bba6d229828958 not found: ID does not exist"
Sep 09 06:21:00.892310 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:00.892291    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cf6be-7f5595974-mqgd8"]
Sep 09 06:21:00.895508 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:00.895487    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cf6be-7f5595974-mqgd8"]
Sep 09 06:21:00.904739 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:00.904720    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7cf6be-59fdf7cbf5-fv8wn"]
Sep 09 06:21:00.908769 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:00.908752    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7cf6be-59fdf7cbf5-fv8wn"]
Sep 09 06:21:01.148776 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:01.148705    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2666d09c-dba9-4a67-8b69-df034c55ce9a" path="/var/lib/kubelet/pods/2666d09c-dba9-4a67-8b69-df034c55ce9a/volumes"
Sep 09 06:21:01.149055 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:01.149043    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f1bc250-9dcd-437f-ae35-af778ebdad1e" path="/var/lib/kubelet/pods/3f1bc250-9dcd-437f-ae35-af778ebdad1e/volumes"
Sep 09 06:21:01.149318 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:01.149307    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73e75e1c-d716-49f7-913f-ab9979ca5d6e" path="/var/lib/kubelet/pods/73e75e1c-d716-49f7-913f-ab9979ca5d6e/volumes"
Sep 09 06:21:02.889994 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:02.889962    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:21:11.597862 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:11.597823    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:21:35.652812 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:35.652782    2579 scope.go:117] "RemoveContainer" containerID="162b7ca18138bee051a50733d2e28390a349fd9f405f049d584d85c5cc57e55e"
Sep 09 06:21:35.660437 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:21:35.660422    2579 scope.go:117] "RemoveContainer" containerID="5a59a95e96585ab977b7eb88276d773bab6a13399b714065ed84a844637f1082"
Sep 09 06:22:33.299512 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:22:33.299479    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:22:59.102763 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:22:59.102730    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:23:06.992133 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:23:06.992099    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:23:13.998457 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:23:13.998421    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:23:21.496062 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:23:21.496030    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:23:35.589667 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:23:35.589617    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:23:43.896518 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:23:43.896483    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:25:13.887481 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:13.887449    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:25:22.038091 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:22.038057    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:25:28.206520 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:28.206444    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:25:31.123341 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:31.123305    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t"]
Sep 09 06:25:31.123734 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:31.123607    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34ec6702-7142-4ff6-bba5-cf1969ac2a46" containerName="istio-proxy"
Sep 09 06:25:31.123734 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:31.123618    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="34ec6702-7142-4ff6-bba5-cf1969ac2a46" containerName="istio-proxy"
Sep 09 06:25:31.123734 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:31.123646    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a027fe8-30f7-424a-9634-5341f728d4b0" containerName="istio-proxy"
Sep 09 06:25:31.123734 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:31.123655    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a027fe8-30f7-424a-9634-5341f728d4b0" containerName="istio-proxy"
Sep 09 06:25:31.123734 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:31.123662    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2666d09c-dba9-4a67-8b69-df034c55ce9a" containerName="payload-pre-processing"
Sep 09 06:25:31.123734 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:31.123668    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2666d09c-dba9-4a67-8b69-df034c55ce9a" containerName="payload-pre-processing"
Sep 09 06:25:31.123734 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:31.123682    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f1bc250-9dcd-437f-ae35-af778ebdad1e" containerName="payload-processing"
Sep 09 06:25:31.123734 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:31.123688    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f1bc250-9dcd-437f-ae35-af778ebdad1e" containerName="payload-processing"
Sep 09 06:25:31.123734 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:31.123700    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73e75e1c-d716-49f7-913f-ab9979ca5d6e" containerName="revoke-keys"
Sep 09 06:25:31.123734 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:31.123705    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="73e75e1c-d716-49f7-913f-ab9979ca5d6e" containerName="revoke-keys"
Sep 09 06:25:31.124043 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:31.123757    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2666d09c-dba9-4a67-8b69-df034c55ce9a" containerName="payload-pre-processing"
Sep 09 06:25:31.124043 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:31.123765    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="73e75e1c-d716-49f7-913f-ab9979ca5d6e" containerName="revoke-keys"
Sep 09 06:25:31.124043 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:31.123769    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a027fe8-30f7-424a-9634-5341f728d4b0" containerName="istio-proxy"
Sep 09 06:25:31.124043 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:31.123777    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f1bc250-9dcd-437f-ae35-af778ebdad1e" containerName="payload-processing"
Sep 09 06:25:31.126591 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:31.126576    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t"
Sep 09 06:25:31.129849 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:31.129832    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-27bbc785-kserve-self-signed-certs\""
Sep 09 06:25:31.148672 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:31.148652    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t"]
Sep 09 06:25:31.173957 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:31.173931    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d4bc669-2e6b-432e-9285-698ef81acbce-dshm\") pod \"e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t\" (UID: \"4d4bc669-2e6b-432e-9285-698ef81acbce\") " pod="llm/e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t"
Sep 09 06:25:31.174048 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:31.173963    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d4bc669-2e6b-432e-9285-698ef81acbce-model-cache\") pod \"e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t\" (UID: \"4d4bc669-2e6b-432e-9285-698ef81acbce\") " pod="llm/e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t"
Sep 09 06:25:31.174048 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:31.173990    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8rd4\" (UniqueName: \"kubernetes.io/projected/4d4bc669-2e6b-432e-9285-698ef81acbce-kube-api-access-p8rd4\") pod \"e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t\" (UID: \"4d4bc669-2e6b-432e-9285-698ef81acbce\") " pod="llm/e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t"
Sep 09 06:25:31.174121 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:31.174052    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d4bc669-2e6b-432e-9285-698ef81acbce-home\") pod \"e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t\" (UID: \"4d4bc669-2e6b-432e-9285-698ef81acbce\") " pod="llm/e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t"
Sep 09 06:25:31.174121 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:31.174103    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d4bc669-2e6b-432e-9285-698ef81acbce-tmp-dir\") pod \"e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t\" (UID: \"4d4bc669-2e6b-432e-9285-698ef81acbce\") " pod="llm/e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t"
Sep 09 06:25:31.174190 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:31.174128    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d4bc669-2e6b-432e-9285-698ef81acbce-tls-certs\") pod \"e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t\" (UID: \"4d4bc669-2e6b-432e-9285-698ef81acbce\") " pod="llm/e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t"
Sep 09 06:25:31.275104 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:31.275075    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d4bc669-2e6b-432e-9285-698ef81acbce-home\") pod \"e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t\" (UID: \"4d4bc669-2e6b-432e-9285-698ef81acbce\") " pod="llm/e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t"
Sep 09 06:25:31.275228 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:31.275117    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d4bc669-2e6b-432e-9285-698ef81acbce-tmp-dir\") pod \"e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t\" (UID: \"4d4bc669-2e6b-432e-9285-698ef81acbce\") " pod="llm/e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t"
Sep 09 06:25:31.275228 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:31.275143    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d4bc669-2e6b-432e-9285-698ef81acbce-tls-certs\") pod \"e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t\" (UID: \"4d4bc669-2e6b-432e-9285-698ef81acbce\") " pod="llm/e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t"
Sep 09 06:25:31.275343 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:31.275226    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d4bc669-2e6b-432e-9285-698ef81acbce-dshm\") pod \"e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t\" (UID: \"4d4bc669-2e6b-432e-9285-698ef81acbce\") " pod="llm/e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t"
Sep 09 06:25:31.275343 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:31.275260    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d4bc669-2e6b-432e-9285-698ef81acbce-model-cache\") pod \"e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t\" (UID: \"4d4bc669-2e6b-432e-9285-698ef81acbce\") " pod="llm/e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t"
Sep 09 06:25:31.275343 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:31.275288    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8rd4\" (UniqueName: \"kubernetes.io/projected/4d4bc669-2e6b-432e-9285-698ef81acbce-kube-api-access-p8rd4\") pod \"e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t\" (UID: \"4d4bc669-2e6b-432e-9285-698ef81acbce\") " pod="llm/e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t"
Sep 09 06:25:31.275534 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:31.275504    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d4bc669-2e6b-432e-9285-698ef81acbce-home\") pod \"e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t\" (UID: \"4d4bc669-2e6b-432e-9285-698ef81acbce\") " pod="llm/e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t"
Sep 09 06:25:31.275676 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:31.275517    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d4bc669-2e6b-432e-9285-698ef81acbce-tmp-dir\") pod \"e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t\" (UID: \"4d4bc669-2e6b-432e-9285-698ef81acbce\") " pod="llm/e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t"
Sep 09 06:25:31.275760 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:31.275741    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d4bc669-2e6b-432e-9285-698ef81acbce-model-cache\") pod \"e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t\" (UID: \"4d4bc669-2e6b-432e-9285-698ef81acbce\") " pod="llm/e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t"
Sep 09 06:25:31.277597 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:31.277569    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d4bc669-2e6b-432e-9285-698ef81acbce-dshm\") pod \"e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t\" (UID: \"4d4bc669-2e6b-432e-9285-698ef81acbce\") " pod="llm/e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t"
Sep 09 06:25:31.277763 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:31.277745    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d4bc669-2e6b-432e-9285-698ef81acbce-tls-certs\") pod \"e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t\" (UID: \"4d4bc669-2e6b-432e-9285-698ef81acbce\") " pod="llm/e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t"
Sep 09 06:25:31.294440 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:31.294421    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8rd4\" (UniqueName: \"kubernetes.io/projected/4d4bc669-2e6b-432e-9285-698ef81acbce-kube-api-access-p8rd4\") pod \"e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t\" (UID: \"4d4bc669-2e6b-432e-9285-698ef81acbce\") " pod="llm/e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t"
Sep 09 06:25:31.435915 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:31.435845    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t"
Sep 09 06:25:31.710697 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:31.710674    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t"]
Sep 09 06:25:31.712793 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:25:31.712771    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d4bc669_2e6b_432e_9285_698ef81acbce.slice/crio-be0c8bbbc84c3ed8c1da1034ce48a9c1fddfffe14971aececb0eaf82683c53c7 WatchSource:0}: Error finding container be0c8bbbc84c3ed8c1da1034ce48a9c1fddfffe14971aececb0eaf82683c53c7: Status 404 returned error can't find the container with id be0c8bbbc84c3ed8c1da1034ce48a9c1fddfffe14971aececb0eaf82683c53c7
Sep 09 06:25:32.040465 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:32.040376    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t" event={"ID":"4d4bc669-2e6b-432e-9285-698ef81acbce","Type":"ContainerStarted","Data":"b9ebc58ca6214b5267b7f92a7b3a437301600319d22d54ea673e7eece066af6b"}
Sep 09 06:25:32.040465 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:32.040416    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t" event={"ID":"4d4bc669-2e6b-432e-9285-698ef81acbce","Type":"ContainerStarted","Data":"be0c8bbbc84c3ed8c1da1034ce48a9c1fddfffe14971aececb0eaf82683c53c7"}
Sep 09 06:25:32.059986 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:32.059944    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t" podStartSLOduration=1.059928503 podStartE2EDuration="1.059928503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:25:32.059172619 +0000 UTC m=+2157.399660529" watchObservedRunningTime="2026-09-09 06:25:32.059928503 +0000 UTC m=+2157.400416411"
Sep 09 06:25:34.001316 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:34.001272    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:25:34.217686 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:34.217654    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6"]
Sep 09 06:25:34.218002 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:34.217989    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a027fe8-30f7-424a-9634-5341f728d4b0" containerName="istio-proxy"
Sep 09 06:25:34.218048 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:34.218006    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a027fe8-30f7-424a-9634-5341f728d4b0" containerName="istio-proxy"
Sep 09 06:25:34.218084 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:34.218069    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a027fe8-30f7-424a-9634-5341f728d4b0" containerName="istio-proxy"
Sep 09 06:25:34.221314 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:34.221290    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6"
Sep 09 06:25:34.226300 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:34.226280    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-27bbc785-kserve-self-signed-certs\""
Sep 09 06:25:34.237351 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:34.237324    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6"]
Sep 09 06:25:34.301936 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:34.301858    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc6dc48c-ebb8-4afe-affb-c6771248c618-dshm\") pod \"e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6\" (UID: \"dc6dc48c-ebb8-4afe-affb-c6771248c618\") " pod="llm/e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6"
Sep 09 06:25:34.301936 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:34.301919    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc6dc48c-ebb8-4afe-affb-c6771248c618-model-cache\") pod \"e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6\" (UID: \"dc6dc48c-ebb8-4afe-affb-c6771248c618\") " pod="llm/e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6"
Sep 09 06:25:34.302124 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:34.301947    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc6dc48c-ebb8-4afe-affb-c6771248c618-tls-certs\") pod \"e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6\" (UID: \"dc6dc48c-ebb8-4afe-affb-c6771248c618\") " pod="llm/e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6"
Sep 09 06:25:34.302124 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:34.301979    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc6dc48c-ebb8-4afe-affb-c6771248c618-home\") pod \"e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6\" (UID: \"dc6dc48c-ebb8-4afe-affb-c6771248c618\") " pod="llm/e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6"
Sep 09 06:25:34.302124 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:34.302007    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc6dc48c-ebb8-4afe-affb-c6771248c618-tmp-dir\") pod \"e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6\" (UID: \"dc6dc48c-ebb8-4afe-affb-c6771248c618\") " pod="llm/e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6"
Sep 09 06:25:34.302124 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:34.302070    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwd7j\" (UniqueName: \"kubernetes.io/projected/dc6dc48c-ebb8-4afe-affb-c6771248c618-kube-api-access-nwd7j\") pod \"e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6\" (UID: \"dc6dc48c-ebb8-4afe-affb-c6771248c618\") " pod="llm/e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6"
Sep 09 06:25:34.402663 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:34.402608    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc6dc48c-ebb8-4afe-affb-c6771248c618-dshm\") pod \"e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6\" (UID: \"dc6dc48c-ebb8-4afe-affb-c6771248c618\") " pod="llm/e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6"
Sep 09 06:25:34.402815 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:34.402702    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc6dc48c-ebb8-4afe-affb-c6771248c618-model-cache\") pod \"e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6\" (UID: \"dc6dc48c-ebb8-4afe-affb-c6771248c618\") " pod="llm/e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6"
Sep 09 06:25:34.402815 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:34.402727    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc6dc48c-ebb8-4afe-affb-c6771248c618-tls-certs\") pod \"e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6\" (UID: \"dc6dc48c-ebb8-4afe-affb-c6771248c618\") " pod="llm/e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6"
Sep 09 06:25:34.402815 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:34.402762    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc6dc48c-ebb8-4afe-affb-c6771248c618-home\") pod \"e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6\" (UID: \"dc6dc48c-ebb8-4afe-affb-c6771248c618\") " pod="llm/e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6"
Sep 09 06:25:34.402815 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:34.402791    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc6dc48c-ebb8-4afe-affb-c6771248c618-tmp-dir\") pod \"e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6\" (UID: \"dc6dc48c-ebb8-4afe-affb-c6771248c618\") " pod="llm/e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6"
Sep 09 06:25:34.403010 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:34.402836    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwd7j\" (UniqueName: \"kubernetes.io/projected/dc6dc48c-ebb8-4afe-affb-c6771248c618-kube-api-access-nwd7j\") pod \"e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6\" (UID: \"dc6dc48c-ebb8-4afe-affb-c6771248c618\") " pod="llm/e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6"
Sep 09 06:25:34.403155 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:34.403129    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc6dc48c-ebb8-4afe-affb-c6771248c618-model-cache\") pod \"e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6\" (UID: \"dc6dc48c-ebb8-4afe-affb-c6771248c618\") " pod="llm/e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6"
Sep 09 06:25:34.403155 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:34.403148    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc6dc48c-ebb8-4afe-affb-c6771248c618-tmp-dir\") pod \"e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6\" (UID: \"dc6dc48c-ebb8-4afe-affb-c6771248c618\") " pod="llm/e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6"
Sep 09 06:25:34.403283 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:34.403182    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc6dc48c-ebb8-4afe-affb-c6771248c618-home\") pod \"e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6\" (UID: \"dc6dc48c-ebb8-4afe-affb-c6771248c618\") " pod="llm/e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6"
Sep 09 06:25:34.404982 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:34.404960    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc6dc48c-ebb8-4afe-affb-c6771248c618-dshm\") pod \"e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6\" (UID: \"dc6dc48c-ebb8-4afe-affb-c6771248c618\") " pod="llm/e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6"
Sep 09 06:25:34.405291 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:34.405269    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc6dc48c-ebb8-4afe-affb-c6771248c618-tls-certs\") pod \"e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6\" (UID: \"dc6dc48c-ebb8-4afe-affb-c6771248c618\") " pod="llm/e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6"
Sep 09 06:25:34.411766 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:34.411745    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwd7j\" (UniqueName: \"kubernetes.io/projected/dc6dc48c-ebb8-4afe-affb-c6771248c618-kube-api-access-nwd7j\") pod \"e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6\" (UID: \"dc6dc48c-ebb8-4afe-affb-c6771248c618\") " pod="llm/e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6"
Sep 09 06:25:34.531935 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:34.531901    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6"
Sep 09 06:25:34.819897 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:34.819872    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6"]
Sep 09 06:25:34.822447 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:25:34.822421    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc6dc48c_ebb8_4afe_affb_c6771248c618.slice/crio-9ed6c96e5e256aa7ac404e7f60736ab83472a465a69740154f99588982c2ef83 WatchSource:0}: Error finding container 9ed6c96e5e256aa7ac404e7f60736ab83472a465a69740154f99588982c2ef83: Status 404 returned error can't find the container with id 9ed6c96e5e256aa7ac404e7f60736ab83472a465a69740154f99588982c2ef83
Sep 09 06:25:35.055812 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:35.055781    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6" event={"ID":"dc6dc48c-ebb8-4afe-affb-c6771248c618","Type":"ContainerStarted","Data":"1245cec130f4ce17577ad4e683d35c4523a84dbe0e832bad8a01bfa3f5d49da8"}
Sep 09 06:25:35.055812 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:35.055819    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6" event={"ID":"dc6dc48c-ebb8-4afe-affb-c6771248c618","Type":"ContainerStarted","Data":"9ed6c96e5e256aa7ac404e7f60736ab83472a465a69740154f99588982c2ef83"}
Sep 09 06:25:35.084950 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:35.084902    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6" podStartSLOduration=1.084886246 podStartE2EDuration="1.084886246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:25:35.083658594 +0000 UTC m=+2160.424146506" watchObservedRunningTime="2026-09-09 06:25:35.084886246 +0000 UTC m=+2160.425374155"
Sep 09 06:25:35.712720 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:35.712692    2579 scope.go:117] "RemoveContainer" containerID="c08d94a72c488d01aa02884ef4086d75b52699fcfd38e0121087828bf0e5f869"
Sep 09 06:25:35.720383 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:35.720368    2579 scope.go:117] "RemoveContainer" containerID="d3e78c86b9b47ee1738cf74c1d108cd4d00c30bf04c97fce24bcf5bbe3b5bec3"
Sep 09 06:25:41.436080 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:41.436044    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t"
Sep 09 06:25:41.436080 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:41.436093    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t"
Sep 09 06:25:41.453969 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:41.453942    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t"
Sep 09 06:25:42.095804 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:42.095775    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t"
Sep 09 06:25:42.400395 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:42.400311    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:25:44.532321 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:44.532281    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6"
Sep 09 06:25:44.532321 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:44.532337    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6"
Sep 09 06:25:44.544511 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:44.544488    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6"
Sep 09 06:25:45.109357 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:45.109331    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6"
Sep 09 06:25:59.728921 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:25:59.728883    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t"]
Sep 09 06:26:01.561941 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:01.561904    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6"]
Sep 09 06:26:01.907557 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:01.907477    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:26:10.695736 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:10.695702    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:26:14.749643 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:14.749565    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t" podUID="4d4bc669-2e6b-432e-9285-698ef81acbce" containerName="main" containerID="cri-o://b9ebc58ca6214b5267b7f92a7b3a437301600319d22d54ea673e7eece066af6b" gracePeriod=45
Sep 09 06:26:14.987215 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:14.987192    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t"
Sep 09 06:26:15.015330 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:15.015257    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d4bc669-2e6b-432e-9285-698ef81acbce-tls-certs\") pod \"4d4bc669-2e6b-432e-9285-698ef81acbce\" (UID: \"4d4bc669-2e6b-432e-9285-698ef81acbce\") "
Sep 09 06:26:15.015330 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:15.015294    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d4bc669-2e6b-432e-9285-698ef81acbce-dshm\") pod \"4d4bc669-2e6b-432e-9285-698ef81acbce\" (UID: \"4d4bc669-2e6b-432e-9285-698ef81acbce\") "
Sep 09 06:26:15.015330 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:15.015312    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d4bc669-2e6b-432e-9285-698ef81acbce-tmp-dir\") pod \"4d4bc669-2e6b-432e-9285-698ef81acbce\" (UID: \"4d4bc669-2e6b-432e-9285-698ef81acbce\") "
Sep 09 06:26:15.015598 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:15.015342    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d4bc669-2e6b-432e-9285-698ef81acbce-model-cache\") pod \"4d4bc669-2e6b-432e-9285-698ef81acbce\" (UID: \"4d4bc669-2e6b-432e-9285-698ef81acbce\") "
Sep 09 06:26:15.015598 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:15.015405    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d4bc669-2e6b-432e-9285-698ef81acbce-home\") pod \"4d4bc669-2e6b-432e-9285-698ef81acbce\" (UID: \"4d4bc669-2e6b-432e-9285-698ef81acbce\") "
Sep 09 06:26:15.015598 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:15.015439    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8rd4\" (UniqueName: \"kubernetes.io/projected/4d4bc669-2e6b-432e-9285-698ef81acbce-kube-api-access-p8rd4\") pod \"4d4bc669-2e6b-432e-9285-698ef81acbce\" (UID: \"4d4bc669-2e6b-432e-9285-698ef81acbce\") "
Sep 09 06:26:15.015793 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:15.015598    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d4bc669-2e6b-432e-9285-698ef81acbce-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4d4bc669-2e6b-432e-9285-698ef81acbce" (UID: "4d4bc669-2e6b-432e-9285-698ef81acbce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:26:15.015793 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:15.015727    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d4bc669-2e6b-432e-9285-698ef81acbce-model-cache" (OuterVolumeSpecName: "model-cache") pod "4d4bc669-2e6b-432e-9285-698ef81acbce" (UID: "4d4bc669-2e6b-432e-9285-698ef81acbce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:26:15.015793 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:15.015756    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d4bc669-2e6b-432e-9285-698ef81acbce-tmp-dir\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:26:15.015952 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:15.015907    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d4bc669-2e6b-432e-9285-698ef81acbce-home" (OuterVolumeSpecName: "home") pod "4d4bc669-2e6b-432e-9285-698ef81acbce" (UID: "4d4bc669-2e6b-432e-9285-698ef81acbce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:26:15.017739 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:15.017712    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d4bc669-2e6b-432e-9285-698ef81acbce-dshm" (OuterVolumeSpecName: "dshm") pod "4d4bc669-2e6b-432e-9285-698ef81acbce" (UID: "4d4bc669-2e6b-432e-9285-698ef81acbce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:26:15.017845 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:15.017770    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d4bc669-2e6b-432e-9285-698ef81acbce-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4d4bc669-2e6b-432e-9285-698ef81acbce" (UID: "4d4bc669-2e6b-432e-9285-698ef81acbce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:26:15.017845 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:15.017781    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d4bc669-2e6b-432e-9285-698ef81acbce-kube-api-access-p8rd4" (OuterVolumeSpecName: "kube-api-access-p8rd4") pod "4d4bc669-2e6b-432e-9285-698ef81acbce" (UID: "4d4bc669-2e6b-432e-9285-698ef81acbce"). InnerVolumeSpecName "kube-api-access-p8rd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:26:15.116898 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:15.116867    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d4bc669-2e6b-432e-9285-698ef81acbce-tls-certs\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:26:15.116898 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:15.116891    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d4bc669-2e6b-432e-9285-698ef81acbce-dshm\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:26:15.116898 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:15.116899    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d4bc669-2e6b-432e-9285-698ef81acbce-model-cache\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:26:15.116898 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:15.116907    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d4bc669-2e6b-432e-9285-698ef81acbce-home\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:26:15.117122 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:15.116915    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8rd4\" (UniqueName: \"kubernetes.io/projected/4d4bc669-2e6b-432e-9285-698ef81acbce-kube-api-access-p8rd4\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:26:15.227814 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:15.227782    2579 generic.go:356] "Generic (PLEG): container finished" podID="4d4bc669-2e6b-432e-9285-698ef81acbce" containerID="b9ebc58ca6214b5267b7f92a7b3a437301600319d22d54ea673e7eece066af6b" exitCode=0
Sep 09 06:26:15.227988 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:15.227864    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t" event={"ID":"4d4bc669-2e6b-432e-9285-698ef81acbce","Type":"ContainerDied","Data":"b9ebc58ca6214b5267b7f92a7b3a437301600319d22d54ea673e7eece066af6b"}
Sep 09 06:26:15.227988 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:15.227899    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t" event={"ID":"4d4bc669-2e6b-432e-9285-698ef81acbce","Type":"ContainerDied","Data":"be0c8bbbc84c3ed8c1da1034ce48a9c1fddfffe14971aececb0eaf82683c53c7"}
Sep 09 06:26:15.227988 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:15.227912    2579 scope.go:117] "RemoveContainer" containerID="b9ebc58ca6214b5267b7f92a7b3a437301600319d22d54ea673e7eece066af6b"
Sep 09 06:26:15.227988 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:15.227896    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t"
Sep 09 06:26:15.236182 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:15.236164    2579 scope.go:117] "RemoveContainer" containerID="b9ebc58ca6214b5267b7f92a7b3a437301600319d22d54ea673e7eece066af6b"
Sep 09 06:26:15.236445 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:26:15.236420    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9ebc58ca6214b5267b7f92a7b3a437301600319d22d54ea673e7eece066af6b\": container with ID starting with b9ebc58ca6214b5267b7f92a7b3a437301600319d22d54ea673e7eece066af6b not found: ID does not exist" containerID="b9ebc58ca6214b5267b7f92a7b3a437301600319d22d54ea673e7eece066af6b"
Sep 09 06:26:15.236551 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:15.236452    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9ebc58ca6214b5267b7f92a7b3a437301600319d22d54ea673e7eece066af6b"} err="failed to get container status \"b9ebc58ca6214b5267b7f92a7b3a437301600319d22d54ea673e7eece066af6b\": rpc error: code = NotFound desc = could not find container \"b9ebc58ca6214b5267b7f92a7b3a437301600319d22d54ea673e7eece066af6b\": container with ID starting with b9ebc58ca6214b5267b7f92a7b3a437301600319d22d54ea673e7eece066af6b not found: ID does not exist"
Sep 09 06:26:15.243381 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:15.243360    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t"]
Sep 09 06:26:15.250322 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:15.250302    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-27bbc785-kserve-7968665695-j7x2t"]
Sep 09 06:26:16.582769 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:16.582723    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6" podUID="dc6dc48c-ebb8-4afe-affb-c6771248c618" containerName="main" containerID="cri-o://1245cec130f4ce17577ad4e683d35c4523a84dbe0e832bad8a01bfa3f5d49da8" gracePeriod=45
Sep 09 06:26:16.815031 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:16.815009    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6"
Sep 09 06:26:16.830534 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:16.830510    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwd7j\" (UniqueName: \"kubernetes.io/projected/dc6dc48c-ebb8-4afe-affb-c6771248c618-kube-api-access-nwd7j\") pod \"dc6dc48c-ebb8-4afe-affb-c6771248c618\" (UID: \"dc6dc48c-ebb8-4afe-affb-c6771248c618\") "
Sep 09 06:26:16.830678 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:16.830549    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc6dc48c-ebb8-4afe-affb-c6771248c618-tmp-dir\") pod \"dc6dc48c-ebb8-4afe-affb-c6771248c618\" (UID: \"dc6dc48c-ebb8-4afe-affb-c6771248c618\") "
Sep 09 06:26:16.830678 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:16.830580    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc6dc48c-ebb8-4afe-affb-c6771248c618-model-cache\") pod \"dc6dc48c-ebb8-4afe-affb-c6771248c618\" (UID: \"dc6dc48c-ebb8-4afe-affb-c6771248c618\") "
Sep 09 06:26:16.830678 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:16.830656    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc6dc48c-ebb8-4afe-affb-c6771248c618-home\") pod \"dc6dc48c-ebb8-4afe-affb-c6771248c618\" (UID: \"dc6dc48c-ebb8-4afe-affb-c6771248c618\") "
Sep 09 06:26:16.830848 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:16.830683    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc6dc48c-ebb8-4afe-affb-c6771248c618-tls-certs\") pod \"dc6dc48c-ebb8-4afe-affb-c6771248c618\" (UID: \"dc6dc48c-ebb8-4afe-affb-c6771248c618\") "
Sep 09 06:26:16.830848 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:16.830724    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc6dc48c-ebb8-4afe-affb-c6771248c618-dshm\") pod \"dc6dc48c-ebb8-4afe-affb-c6771248c618\" (UID: \"dc6dc48c-ebb8-4afe-affb-c6771248c618\") "
Sep 09 06:26:16.831671 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:16.831604    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc6dc48c-ebb8-4afe-affb-c6771248c618-model-cache" (OuterVolumeSpecName: "model-cache") pod "dc6dc48c-ebb8-4afe-affb-c6771248c618" (UID: "dc6dc48c-ebb8-4afe-affb-c6771248c618"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:26:16.831671 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:16.831619    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc6dc48c-ebb8-4afe-affb-c6771248c618-home" (OuterVolumeSpecName: "home") pod "dc6dc48c-ebb8-4afe-affb-c6771248c618" (UID: "dc6dc48c-ebb8-4afe-affb-c6771248c618"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:26:16.831897 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:16.831847    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc6dc48c-ebb8-4afe-affb-c6771248c618-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dc6dc48c-ebb8-4afe-affb-c6771248c618" (UID: "dc6dc48c-ebb8-4afe-affb-c6771248c618"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:26:16.833559 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:16.833489    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc6dc48c-ebb8-4afe-affb-c6771248c618-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dc6dc48c-ebb8-4afe-affb-c6771248c618" (UID: "dc6dc48c-ebb8-4afe-affb-c6771248c618"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:26:16.834237 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:16.834211    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc6dc48c-ebb8-4afe-affb-c6771248c618-kube-api-access-nwd7j" (OuterVolumeSpecName: "kube-api-access-nwd7j") pod "dc6dc48c-ebb8-4afe-affb-c6771248c618" (UID: "dc6dc48c-ebb8-4afe-affb-c6771248c618"). InnerVolumeSpecName "kube-api-access-nwd7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:26:16.834237 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:16.834213    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc6dc48c-ebb8-4afe-affb-c6771248c618-dshm" (OuterVolumeSpecName: "dshm") pod "dc6dc48c-ebb8-4afe-affb-c6771248c618" (UID: "dc6dc48c-ebb8-4afe-affb-c6771248c618"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:26:16.931774 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:16.931742    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc6dc48c-ebb8-4afe-affb-c6771248c618-home\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:26:16.931774 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:16.931770    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc6dc48c-ebb8-4afe-affb-c6771248c618-tls-certs\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:26:16.931774 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:16.931781    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc6dc48c-ebb8-4afe-affb-c6771248c618-dshm\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:26:16.932009 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:16.931789    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwd7j\" (UniqueName: \"kubernetes.io/projected/dc6dc48c-ebb8-4afe-affb-c6771248c618-kube-api-access-nwd7j\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:26:16.932009 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:16.931796    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc6dc48c-ebb8-4afe-affb-c6771248c618-tmp-dir\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:26:16.932009 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:16.931805    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc6dc48c-ebb8-4afe-affb-c6771248c618-model-cache\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:26:17.147501 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:17.147421    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d4bc669-2e6b-432e-9285-698ef81acbce" path="/var/lib/kubelet/pods/4d4bc669-2e6b-432e-9285-698ef81acbce/volumes"
Sep 09 06:26:17.238261 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:17.238233    2579 generic.go:356] "Generic (PLEG): container finished" podID="dc6dc48c-ebb8-4afe-affb-c6771248c618" containerID="1245cec130f4ce17577ad4e683d35c4523a84dbe0e832bad8a01bfa3f5d49da8" exitCode=0
Sep 09 06:26:17.238439 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:17.238276    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6"
Sep 09 06:26:17.238439 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:17.238321    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6" event={"ID":"dc6dc48c-ebb8-4afe-affb-c6771248c618","Type":"ContainerDied","Data":"1245cec130f4ce17577ad4e683d35c4523a84dbe0e832bad8a01bfa3f5d49da8"}
Sep 09 06:26:17.238439 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:17.238366    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6" event={"ID":"dc6dc48c-ebb8-4afe-affb-c6771248c618","Type":"ContainerDied","Data":"9ed6c96e5e256aa7ac404e7f60736ab83472a465a69740154f99588982c2ef83"}
Sep 09 06:26:17.238439 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:17.238383    2579 scope.go:117] "RemoveContainer" containerID="1245cec130f4ce17577ad4e683d35c4523a84dbe0e832bad8a01bfa3f5d49da8"
Sep 09 06:26:17.246806 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:17.246774    2579 scope.go:117] "RemoveContainer" containerID="1245cec130f4ce17577ad4e683d35c4523a84dbe0e832bad8a01bfa3f5d49da8"
Sep 09 06:26:17.247046 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:26:17.247028    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1245cec130f4ce17577ad4e683d35c4523a84dbe0e832bad8a01bfa3f5d49da8\": container with ID starting with 1245cec130f4ce17577ad4e683d35c4523a84dbe0e832bad8a01bfa3f5d49da8 not found: ID does not exist" containerID="1245cec130f4ce17577ad4e683d35c4523a84dbe0e832bad8a01bfa3f5d49da8"
Sep 09 06:26:17.247097 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:17.247060    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1245cec130f4ce17577ad4e683d35c4523a84dbe0e832bad8a01bfa3f5d49da8"} err="failed to get container status \"1245cec130f4ce17577ad4e683d35c4523a84dbe0e832bad8a01bfa3f5d49da8\": rpc error: code = NotFound desc = could not find container \"1245cec130f4ce17577ad4e683d35c4523a84dbe0e832bad8a01bfa3f5d49da8\": container with ID starting with 1245cec130f4ce17577ad4e683d35c4523a84dbe0e832bad8a01bfa3f5d49da8 not found: ID does not exist"
Sep 09 06:26:17.253354 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:17.253333    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6"]
Sep 09 06:26:17.256858 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:17.256839    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-27bbc785-kserve-9f5cf7d4c-wffw6"]
Sep 09 06:26:17.896549 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:17.896515    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:26:19.147457 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:19.147425    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc6dc48c-ebb8-4afe-affb-c6771248c618" path="/var/lib/kubelet/pods/dc6dc48c-ebb8-4afe-affb-c6771248c618/volumes"
Sep 09 06:26:24.486655 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:24.486609    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:26:24.913826 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:24.913794    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-75c5f7dfb6-gdnnk"]
Sep 09 06:26:24.914081 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:24.914070    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d4bc669-2e6b-432e-9285-698ef81acbce" containerName="main"
Sep 09 06:26:24.914132 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:24.914083    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d4bc669-2e6b-432e-9285-698ef81acbce" containerName="main"
Sep 09 06:26:24.914132 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:24.914093    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc6dc48c-ebb8-4afe-affb-c6771248c618" containerName="main"
Sep 09 06:26:24.914132 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:24.914099    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc6dc48c-ebb8-4afe-affb-c6771248c618" containerName="main"
Sep 09 06:26:24.914229 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:24.914152    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc6dc48c-ebb8-4afe-affb-c6771248c618" containerName="main"
Sep 09 06:26:24.914229 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:24.914160    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d4bc669-2e6b-432e-9285-698ef81acbce" containerName="main"
Sep 09 06:26:24.918354 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:24.918332    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-75c5f7dfb6-gdnnk"
Sep 09 06:26:24.921656 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:24.921619    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 06:26:24.921816 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:24.921757    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 09 06:26:24.921816 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:24.921775    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 09 06:26:24.921816 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:24.921787    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 06:26:24.922013 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:24.921789    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-927tl\""
Sep 09 06:26:24.927699 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:24.927675    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-75c5f7dfb6-gdnnk"]
Sep 09 06:26:24.995283 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:24.995251    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2501282-f21c-4954-9082-4cf55830f5f9-maas-api-tls\") pod \"maas-api-75c5f7dfb6-gdnnk\" (UID: \"c2501282-f21c-4954-9082-4cf55830f5f9\") " pod="odh-ai-gateway-infra/maas-api-75c5f7dfb6-gdnnk"
Sep 09 06:26:24.995283 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:24.995293    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2501282-f21c-4954-9082-4cf55830f5f9-openshift-service-ca\") pod \"maas-api-75c5f7dfb6-gdnnk\" (UID: \"c2501282-f21c-4954-9082-4cf55830f5f9\") " pod="odh-ai-gateway-infra/maas-api-75c5f7dfb6-gdnnk"
Sep 09 06:26:24.995505 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:24.995365    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzlqx\" (UniqueName: \"kubernetes.io/projected/c2501282-f21c-4954-9082-4cf55830f5f9-kube-api-access-lzlqx\") pod \"maas-api-75c5f7dfb6-gdnnk\" (UID: \"c2501282-f21c-4954-9082-4cf55830f5f9\") " pod="odh-ai-gateway-infra/maas-api-75c5f7dfb6-gdnnk"
Sep 09 06:26:24.995505 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:24.995396    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2501282-f21c-4954-9082-4cf55830f5f9-metrics-certs\") pod \"maas-api-75c5f7dfb6-gdnnk\" (UID: \"c2501282-f21c-4954-9082-4cf55830f5f9\") " pod="odh-ai-gateway-infra/maas-api-75c5f7dfb6-gdnnk"
Sep 09 06:26:25.096513 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:25.096466    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2501282-f21c-4954-9082-4cf55830f5f9-openshift-service-ca\") pod \"maas-api-75c5f7dfb6-gdnnk\" (UID: \"c2501282-f21c-4954-9082-4cf55830f5f9\") " pod="odh-ai-gateway-infra/maas-api-75c5f7dfb6-gdnnk"
Sep 09 06:26:25.096731 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:25.096530    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzlqx\" (UniqueName: \"kubernetes.io/projected/c2501282-f21c-4954-9082-4cf55830f5f9-kube-api-access-lzlqx\") pod \"maas-api-75c5f7dfb6-gdnnk\" (UID: \"c2501282-f21c-4954-9082-4cf55830f5f9\") " pod="odh-ai-gateway-infra/maas-api-75c5f7dfb6-gdnnk"
Sep 09 06:26:25.096731 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:25.096569    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2501282-f21c-4954-9082-4cf55830f5f9-metrics-certs\") pod \"maas-api-75c5f7dfb6-gdnnk\" (UID: \"c2501282-f21c-4954-9082-4cf55830f5f9\") " pod="odh-ai-gateway-infra/maas-api-75c5f7dfb6-gdnnk"
Sep 09 06:26:25.096731 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:25.096611    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2501282-f21c-4954-9082-4cf55830f5f9-maas-api-tls\") pod \"maas-api-75c5f7dfb6-gdnnk\" (UID: \"c2501282-f21c-4954-9082-4cf55830f5f9\") " pod="odh-ai-gateway-infra/maas-api-75c5f7dfb6-gdnnk"
Sep 09 06:26:25.097139 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:25.097115    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2501282-f21c-4954-9082-4cf55830f5f9-openshift-service-ca\") pod \"maas-api-75c5f7dfb6-gdnnk\" (UID: \"c2501282-f21c-4954-9082-4cf55830f5f9\") " pod="odh-ai-gateway-infra/maas-api-75c5f7dfb6-gdnnk"
Sep 09 06:26:25.099152 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:25.099129    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2501282-f21c-4954-9082-4cf55830f5f9-maas-api-tls\") pod \"maas-api-75c5f7dfb6-gdnnk\" (UID: \"c2501282-f21c-4954-9082-4cf55830f5f9\") " pod="odh-ai-gateway-infra/maas-api-75c5f7dfb6-gdnnk"
Sep 09 06:26:25.099324 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:25.099302    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2501282-f21c-4954-9082-4cf55830f5f9-metrics-certs\") pod \"maas-api-75c5f7dfb6-gdnnk\" (UID: \"c2501282-f21c-4954-9082-4cf55830f5f9\") " pod="odh-ai-gateway-infra/maas-api-75c5f7dfb6-gdnnk"
Sep 09 06:26:25.104304 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:25.104281    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzlqx\" (UniqueName: \"kubernetes.io/projected/c2501282-f21c-4954-9082-4cf55830f5f9-kube-api-access-lzlqx\") pod \"maas-api-75c5f7dfb6-gdnnk\" (UID: \"c2501282-f21c-4954-9082-4cf55830f5f9\") " pod="odh-ai-gateway-infra/maas-api-75c5f7dfb6-gdnnk"
Sep 09 06:26:25.229732 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:25.229658    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-75c5f7dfb6-gdnnk"
Sep 09 06:26:25.506896 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:25.506862    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-75c5f7dfb6-gdnnk"]
Sep 09 06:26:25.509743 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:26:25.509716    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2501282_f21c_4954_9082_4cf55830f5f9.slice/crio-c107c21c9c93a710922ead3428879951dbf1e41ddd6ee7cc00fc5be0c5ca0685 WatchSource:0}: Error finding container c107c21c9c93a710922ead3428879951dbf1e41ddd6ee7cc00fc5be0c5ca0685: Status 404 returned error can't find the container with id c107c21c9c93a710922ead3428879951dbf1e41ddd6ee7cc00fc5be0c5ca0685
Sep 09 06:26:25.511473 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:25.511456    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 06:26:26.279361 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:26.279329    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-75c5f7dfb6-gdnnk" event={"ID":"c2501282-f21c-4954-9082-4cf55830f5f9","Type":"ContainerStarted","Data":"67eabd56c35b8dbe7bb8f65d7ab6da4aae10a7ad70295c1b4e726466735f8f3c"}
Sep 09 06:26:26.279527 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:26.279381    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-75c5f7dfb6-gdnnk" event={"ID":"c2501282-f21c-4954-9082-4cf55830f5f9","Type":"ContainerStarted","Data":"c107c21c9c93a710922ead3428879951dbf1e41ddd6ee7cc00fc5be0c5ca0685"}
Sep 09 06:26:26.279527 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:26.279444    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-75c5f7dfb6-gdnnk"
Sep 09 06:26:26.298060 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:26.298009    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-75c5f7dfb6-gdnnk" podStartSLOduration=1.6969702070000001 podStartE2EDuration="2.297982303s" totalImagesPullingTime="601.012096ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:26:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:26:26.296593976 +0000 UTC m=+2211.637081887" watchObservedRunningTime="2026-09-09 06:26:26.297982303 +0000 UTC m=+2211.638470211"
Sep 09 06:26:32.195705 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:32.195671    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:26:32.288647 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:32.288602    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-75c5f7dfb6-gdnnk"
Sep 09 06:26:35.739810 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:35.739770    2579 scope.go:117] "RemoveContainer" containerID="3ad1e7d4e96e8cb3a5dafd3ae703058dff60a8f8ed0700ad3209e8299956560d"
Sep 09 06:26:35.747672 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:35.747653    2579 scope.go:117] "RemoveContainer" containerID="9a94fae37230ef4010712a28fc88aef46385fe2bf65e7aaa6584cbbe3230f43d"
Sep 09 06:26:39.198156 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:39.198128    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:26:40.724161 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:40.724128    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-75c5f7dfb6-gdnnk"]
Sep 09 06:26:40.724640 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:40.724360    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-75c5f7dfb6-gdnnk" podUID="c2501282-f21c-4954-9082-4cf55830f5f9" containerName="maas-api" containerID="cri-o://67eabd56c35b8dbe7bb8f65d7ab6da4aae10a7ad70295c1b4e726466735f8f3c" gracePeriod=30
Sep 09 06:26:41.960705 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:41.960683    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-75c5f7dfb6-gdnnk"
Sep 09 06:26:42.035700 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:42.035672    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2501282-f21c-4954-9082-4cf55830f5f9-metrics-certs\") pod \"c2501282-f21c-4954-9082-4cf55830f5f9\" (UID: \"c2501282-f21c-4954-9082-4cf55830f5f9\") "
Sep 09 06:26:42.035831 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:42.035747    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2501282-f21c-4954-9082-4cf55830f5f9-openshift-service-ca\") pod \"c2501282-f21c-4954-9082-4cf55830f5f9\" (UID: \"c2501282-f21c-4954-9082-4cf55830f5f9\") "
Sep 09 06:26:42.035886 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:42.035834    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2501282-f21c-4954-9082-4cf55830f5f9-maas-api-tls\") pod \"c2501282-f21c-4954-9082-4cf55830f5f9\" (UID: \"c2501282-f21c-4954-9082-4cf55830f5f9\") "
Sep 09 06:26:42.035942 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:42.035883    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzlqx\" (UniqueName: \"kubernetes.io/projected/c2501282-f21c-4954-9082-4cf55830f5f9-kube-api-access-lzlqx\") pod \"c2501282-f21c-4954-9082-4cf55830f5f9\" (UID: \"c2501282-f21c-4954-9082-4cf55830f5f9\") "
Sep 09 06:26:42.036013 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:42.035993    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2501282-f21c-4954-9082-4cf55830f5f9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c2501282-f21c-4954-9082-4cf55830f5f9" (UID: "c2501282-f21c-4954-9082-4cf55830f5f9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:26:42.036125 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:42.036104    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2501282-f21c-4954-9082-4cf55830f5f9-openshift-service-ca\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:26:42.037838 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:42.037815    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2501282-f21c-4954-9082-4cf55830f5f9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c2501282-f21c-4954-9082-4cf55830f5f9" (UID: "c2501282-f21c-4954-9082-4cf55830f5f9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:26:42.037935 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:42.037912    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2501282-f21c-4954-9082-4cf55830f5f9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c2501282-f21c-4954-9082-4cf55830f5f9" (UID: "c2501282-f21c-4954-9082-4cf55830f5f9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:26:42.038009 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:42.037989    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2501282-f21c-4954-9082-4cf55830f5f9-kube-api-access-lzlqx" (OuterVolumeSpecName: "kube-api-access-lzlqx") pod "c2501282-f21c-4954-9082-4cf55830f5f9" (UID: "c2501282-f21c-4954-9082-4cf55830f5f9"). InnerVolumeSpecName "kube-api-access-lzlqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:26:42.137013 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:42.136984    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2501282-f21c-4954-9082-4cf55830f5f9-maas-api-tls\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:26:42.137013 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:42.137013    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzlqx\" (UniqueName: \"kubernetes.io/projected/c2501282-f21c-4954-9082-4cf55830f5f9-kube-api-access-lzlqx\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:26:42.137144 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:42.137023    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2501282-f21c-4954-9082-4cf55830f5f9-metrics-certs\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 09 06:26:42.353057 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:42.353025    2579 generic.go:356] "Generic (PLEG): container finished" podID="c2501282-f21c-4954-9082-4cf55830f5f9" containerID="67eabd56c35b8dbe7bb8f65d7ab6da4aae10a7ad70295c1b4e726466735f8f3c" exitCode=0
Sep 09 06:26:42.353228 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:42.353064    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-75c5f7dfb6-gdnnk" event={"ID":"c2501282-f21c-4954-9082-4cf55830f5f9","Type":"ContainerDied","Data":"67eabd56c35b8dbe7bb8f65d7ab6da4aae10a7ad70295c1b4e726466735f8f3c"}
Sep 09 06:26:42.353228 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:42.353083    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-75c5f7dfb6-gdnnk"
Sep 09 06:26:42.353228 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:42.353128    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-75c5f7dfb6-gdnnk" event={"ID":"c2501282-f21c-4954-9082-4cf55830f5f9","Type":"ContainerDied","Data":"c107c21c9c93a710922ead3428879951dbf1e41ddd6ee7cc00fc5be0c5ca0685"}
Sep 09 06:26:42.353228 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:42.353149    2579 scope.go:117] "RemoveContainer" containerID="67eabd56c35b8dbe7bb8f65d7ab6da4aae10a7ad70295c1b4e726466735f8f3c"
Sep 09 06:26:42.362364 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:42.362340    2579 scope.go:117] "RemoveContainer" containerID="67eabd56c35b8dbe7bb8f65d7ab6da4aae10a7ad70295c1b4e726466735f8f3c"
Sep 09 06:26:42.362676 ip-10-0-131-142 kubenswrapper[2579]: E0909 06:26:42.362656    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67eabd56c35b8dbe7bb8f65d7ab6da4aae10a7ad70295c1b4e726466735f8f3c\": container with ID starting with 67eabd56c35b8dbe7bb8f65d7ab6da4aae10a7ad70295c1b4e726466735f8f3c not found: ID does not exist" containerID="67eabd56c35b8dbe7bb8f65d7ab6da4aae10a7ad70295c1b4e726466735f8f3c"
Sep 09 06:26:42.362780 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:42.362685    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67eabd56c35b8dbe7bb8f65d7ab6da4aae10a7ad70295c1b4e726466735f8f3c"} err="failed to get container status \"67eabd56c35b8dbe7bb8f65d7ab6da4aae10a7ad70295c1b4e726466735f8f3c\": rpc error: code = NotFound desc = could not find container \"67eabd56c35b8dbe7bb8f65d7ab6da4aae10a7ad70295c1b4e726466735f8f3c\": container with ID starting with 67eabd56c35b8dbe7bb8f65d7ab6da4aae10a7ad70295c1b4e726466735f8f3c not found: ID does not exist"
Sep 09 06:26:42.373916 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:42.373894    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-75c5f7dfb6-gdnnk"]
Sep 09 06:26:42.376714 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:42.376691    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-75c5f7dfb6-gdnnk"]
Sep 09 06:26:43.147108 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:43.147077    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2501282-f21c-4954-9082-4cf55830f5f9" path="/var/lib/kubelet/pods/c2501282-f21c-4954-9082-4cf55830f5f9/volumes"
Sep 09 06:26:45.398782 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:45.398753    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:26:51.102780 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:51.102747    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f5j8g"]
Sep 09 06:26:54.474299 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:26:54.474263    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-788587d9b8-42kfs_7456999c-8987-428f-ac8d-4065a6fb5f76/authorino/0.log"
Sep 09 06:27:00.341881 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:00.341812    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76b47fccc7-f4lfj_284a6006-17ba-4ed4-ab73-173bf23c0eae/copy-manifests/0.log"
Sep 09 06:27:00.354268 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:00.354243    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76b47fccc7-f4lfj_284a6006-17ba-4ed4-ab73-173bf23c0eae/manager/0.log"
Sep 09 06:27:00.913502 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:00.913473    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29815560-pmw8d_a8bc8589-e0c9-4be7-a38d-e3ec3f9bbbf6/cleanup/0.log"
Sep 09 06:27:01.017176 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:01.017150    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29815575-hg6mt_f712432c-2d21-474e-b0e7-ef649879284f/cleanup/0.log"
Sep 09 06:27:01.872276 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:01.872244    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-788587d9b8-42kfs_7456999c-8987-428f-ac8d-4065a6fb5f76/authorino/0.log"
Sep 09 06:27:02.202302 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:02.202219    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-54mm9_d8bd638b-da4c-4e1d-bdaf-3b7978363e51/kuadrant-console-plugin/0.log"
Sep 09 06:27:02.460018 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:02.459931    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-f5j8g_1919f47d-0880-4d98-ad97-1e8dd6d34408/limitador/0.log"
Sep 09 06:27:03.125694 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:03.125666    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f4d448d7f-58zzf_bf900674-4feb-4a9d-aed9-ed685de58c8f/kube-auth-proxy/0.log"
Sep 09 06:27:04.567775 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:04.567735    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-vzp9x_68d348a3-d2ba-4526-ac49-64c657279cd0/main/0.log"
Sep 09 06:27:04.679072 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:04.679046    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-bwxdw_277bc490-a853-416c-a048-f06558f49f85/main/0.log"
Sep 09 06:27:09.689247 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:09.689218    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kn2d7/must-gather-w59zd"]
Sep 09 06:27:09.689613 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:09.689484    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2501282-f21c-4954-9082-4cf55830f5f9" containerName="maas-api"
Sep 09 06:27:09.689613 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:09.689494    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2501282-f21c-4954-9082-4cf55830f5f9" containerName="maas-api"
Sep 09 06:27:09.689613 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:09.689550    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2501282-f21c-4954-9082-4cf55830f5f9" containerName="maas-api"
Sep 09 06:27:09.693695 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:09.693678    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kn2d7/must-gather-w59zd"
Sep 09 06:27:09.695891 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:09.695872    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kn2d7\"/\"kube-root-ca.crt\""
Sep 09 06:27:09.696732 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:09.696716    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kn2d7\"/\"openshift-service-ca.crt\""
Sep 09 06:27:09.696835 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:09.696716    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kn2d7\"/\"default-dockercfg-z6lpm\""
Sep 09 06:27:09.699966 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:09.699947    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kn2d7/must-gather-w59zd"]
Sep 09 06:27:09.843019 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:09.842997    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/65cb0d96-ca8b-42cc-acb7-ba6021e00c51-must-gather-output\") pod \"must-gather-w59zd\" (UID: \"65cb0d96-ca8b-42cc-acb7-ba6021e00c51\") " pod="openshift-must-gather-kn2d7/must-gather-w59zd"
Sep 09 06:27:09.843168 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:09.843036    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqs4w\" (UniqueName: \"kubernetes.io/projected/65cb0d96-ca8b-42cc-acb7-ba6021e00c51-kube-api-access-rqs4w\") pod \"must-gather-w59zd\" (UID: \"65cb0d96-ca8b-42cc-acb7-ba6021e00c51\") " pod="openshift-must-gather-kn2d7/must-gather-w59zd"
Sep 09 06:27:09.943955 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:09.943892    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqs4w\" (UniqueName: \"kubernetes.io/projected/65cb0d96-ca8b-42cc-acb7-ba6021e00c51-kube-api-access-rqs4w\") pod \"must-gather-w59zd\" (UID: \"65cb0d96-ca8b-42cc-acb7-ba6021e00c51\") " pod="openshift-must-gather-kn2d7/must-gather-w59zd"
Sep 09 06:27:09.944126 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:09.943958    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/65cb0d96-ca8b-42cc-acb7-ba6021e00c51-must-gather-output\") pod \"must-gather-w59zd\" (UID: \"65cb0d96-ca8b-42cc-acb7-ba6021e00c51\") " pod="openshift-must-gather-kn2d7/must-gather-w59zd"
Sep 09 06:27:09.944207 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:09.944193    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/65cb0d96-ca8b-42cc-acb7-ba6021e00c51-must-gather-output\") pod \"must-gather-w59zd\" (UID: \"65cb0d96-ca8b-42cc-acb7-ba6021e00c51\") " pod="openshift-must-gather-kn2d7/must-gather-w59zd"
Sep 09 06:27:09.951940 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:09.951918    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqs4w\" (UniqueName: \"kubernetes.io/projected/65cb0d96-ca8b-42cc-acb7-ba6021e00c51-kube-api-access-rqs4w\") pod \"must-gather-w59zd\" (UID: \"65cb0d96-ca8b-42cc-acb7-ba6021e00c51\") " pod="openshift-must-gather-kn2d7/must-gather-w59zd"
Sep 09 06:27:10.003558 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:10.003527    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kn2d7/must-gather-w59zd"
Sep 09 06:27:10.278004 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:10.277864    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kn2d7/must-gather-w59zd"]
Sep 09 06:27:10.280589 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:27:10.280563    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65cb0d96_ca8b_42cc_acb7_ba6021e00c51.slice/crio-7440ade5283c36d2883ce711d5cf744c90d5af786cbd653bef62fe16707704c6 WatchSource:0}: Error finding container 7440ade5283c36d2883ce711d5cf744c90d5af786cbd653bef62fe16707704c6: Status 404 returned error can't find the container with id 7440ade5283c36d2883ce711d5cf744c90d5af786cbd653bef62fe16707704c6
Sep 09 06:27:10.472986 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:10.472950    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kn2d7/must-gather-w59zd" event={"ID":"65cb0d96-ca8b-42cc-acb7-ba6021e00c51","Type":"ContainerStarted","Data":"7440ade5283c36d2883ce711d5cf744c90d5af786cbd653bef62fe16707704c6"}
Sep 09 06:27:11.481613 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:11.480836    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kn2d7/must-gather-w59zd" event={"ID":"65cb0d96-ca8b-42cc-acb7-ba6021e00c51","Type":"ContainerStarted","Data":"60778ed681612640a82f676c93330b316f8d6b353c104ea137b161f4685b5d2e"}
Sep 09 06:27:11.481613 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:11.480887    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kn2d7/must-gather-w59zd" event={"ID":"65cb0d96-ca8b-42cc-acb7-ba6021e00c51","Type":"ContainerStarted","Data":"87948f0b7ce6f6945ba5b5925d36bf06c266b5dbadb65eaa77b5661ff1405363"}
Sep 09 06:27:11.496876 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:11.496811    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kn2d7/must-gather-w59zd" podStartSLOduration=1.657159716 podStartE2EDuration="2.496789299s" totalImagesPullingTime="839.629583ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:27:11.49610829 +0000 UTC m=+2256.836596237" watchObservedRunningTime="2026-09-09 06:27:11.496789299 +0000 UTC m=+2256.837277209"
Sep 09 06:27:12.631187 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:12.631156    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fs4n6_26cac444-abfa-4060-8f52-4e5756866f3f/global-pull-secret-syncer/0.log"
Sep 09 06:27:12.784822 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:12.784787    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-fv99k_b5f162ab-38ab-4515-81a3-f46c8f2c0e4c/konnectivity-agent/0.log"
Sep 09 06:27:12.833947 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:12.833921    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-142.ec2.internal_58007dd99525496ea22ff71e4a80cd63/haproxy/0.log"
Sep 09 06:27:17.371552 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:17.371517    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-788587d9b8-42kfs_7456999c-8987-428f-ac8d-4065a6fb5f76/authorino/0.log"
Sep 09 06:27:17.473225 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:17.473192    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-54mm9_d8bd638b-da4c-4e1d-bdaf-3b7978363e51/kuadrant-console-plugin/0.log"
Sep 09 06:27:17.542995 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:17.542970    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-f5j8g_1919f47d-0880-4d98-ad97-1e8dd6d34408/limitador/0.log"
Sep 09 06:27:19.275774 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:19.275748    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-649f6cfb6f-8c5s5_32c4b08b-c3e9-4c12-b35a-2b2ca5e59757/metrics-server/0.log"
Sep 09 06:27:19.470680 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:19.470645    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lrzrl_fbb14eb0-b938-4d9a-a16e-9c9857c978ef/node-exporter/0.log"
Sep 09 06:27:19.494876 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:19.494766    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lrzrl_fbb14eb0-b938-4d9a-a16e-9c9857c978ef/kube-rbac-proxy/0.log"
Sep 09 06:27:19.514165 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:19.514143    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lrzrl_fbb14eb0-b938-4d9a-a16e-9c9857c978ef/init-textfile/0.log"
Sep 09 06:27:19.942405 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:19.942378    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f6bf9cf8-zcbh7_1ea34ee7-a545-437f-ab74-68f893f758ff/telemeter-client/0.log"
Sep 09 06:27:19.965134 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:19.965078    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f6bf9cf8-zcbh7_1ea34ee7-a545-437f-ab74-68f893f758ff/reload/0.log"
Sep 09 06:27:19.985254 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:19.985214    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f6bf9cf8-zcbh7_1ea34ee7-a545-437f-ab74-68f893f758ff/kube-rbac-proxy/0.log"
Sep 09 06:27:22.828523 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:22.828477    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kn2d7/perf-node-gather-daemonset-ddjwz"]
Sep 09 06:27:22.836398 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:22.836366    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kn2d7/perf-node-gather-daemonset-ddjwz"
Sep 09 06:27:22.841947 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:22.841921    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kn2d7/perf-node-gather-daemonset-ddjwz"]
Sep 09 06:27:22.864920 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:22.864883    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d9e191f-f2c8-4c39-95b3-7d1ff33c638b-sys\") pod \"perf-node-gather-daemonset-ddjwz\" (UID: \"3d9e191f-f2c8-4c39-95b3-7d1ff33c638b\") " pod="openshift-must-gather-kn2d7/perf-node-gather-daemonset-ddjwz"
Sep 09 06:27:22.865082 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:22.864934    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3d9e191f-f2c8-4c39-95b3-7d1ff33c638b-proc\") pod \"perf-node-gather-daemonset-ddjwz\" (UID: \"3d9e191f-f2c8-4c39-95b3-7d1ff33c638b\") " pod="openshift-must-gather-kn2d7/perf-node-gather-daemonset-ddjwz"
Sep 09 06:27:22.865082 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:22.864970    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfxnt\" (UniqueName: \"kubernetes.io/projected/3d9e191f-f2c8-4c39-95b3-7d1ff33c638b-kube-api-access-pfxnt\") pod \"perf-node-gather-daemonset-ddjwz\" (UID: \"3d9e191f-f2c8-4c39-95b3-7d1ff33c638b\") " pod="openshift-must-gather-kn2d7/perf-node-gather-daemonset-ddjwz"
Sep 09 06:27:22.865082 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:22.865028    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3d9e191f-f2c8-4c39-95b3-7d1ff33c638b-podres\") pod \"perf-node-gather-daemonset-ddjwz\" (UID: \"3d9e191f-f2c8-4c39-95b3-7d1ff33c638b\") " pod="openshift-must-gather-kn2d7/perf-node-gather-daemonset-ddjwz"
Sep 09 06:27:22.865082 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:22.865054    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d9e191f-f2c8-4c39-95b3-7d1ff33c638b-lib-modules\") pod \"perf-node-gather-daemonset-ddjwz\" (UID: \"3d9e191f-f2c8-4c39-95b3-7d1ff33c638b\") " pod="openshift-must-gather-kn2d7/perf-node-gather-daemonset-ddjwz"
Sep 09 06:27:22.966028 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:22.965991    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3d9e191f-f2c8-4c39-95b3-7d1ff33c638b-proc\") pod \"perf-node-gather-daemonset-ddjwz\" (UID: \"3d9e191f-f2c8-4c39-95b3-7d1ff33c638b\") " pod="openshift-must-gather-kn2d7/perf-node-gather-daemonset-ddjwz"
Sep 09 06:27:22.966296 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:22.966272    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfxnt\" (UniqueName: \"kubernetes.io/projected/3d9e191f-f2c8-4c39-95b3-7d1ff33c638b-kube-api-access-pfxnt\") pod \"perf-node-gather-daemonset-ddjwz\" (UID: \"3d9e191f-f2c8-4c39-95b3-7d1ff33c638b\") " pod="openshift-must-gather-kn2d7/perf-node-gather-daemonset-ddjwz"
Sep 09 06:27:22.966485 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:22.966468    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3d9e191f-f2c8-4c39-95b3-7d1ff33c638b-podres\") pod \"perf-node-gather-daemonset-ddjwz\" (UID: \"3d9e191f-f2c8-4c39-95b3-7d1ff33c638b\") " pod="openshift-must-gather-kn2d7/perf-node-gather-daemonset-ddjwz"
Sep 09 06:27:22.966606 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:22.966591    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d9e191f-f2c8-4c39-95b3-7d1ff33c638b-lib-modules\") pod \"perf-node-gather-daemonset-ddjwz\" (UID: \"3d9e191f-f2c8-4c39-95b3-7d1ff33c638b\") " pod="openshift-must-gather-kn2d7/perf-node-gather-daemonset-ddjwz"
Sep 09 06:27:22.966820 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:22.966800    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d9e191f-f2c8-4c39-95b3-7d1ff33c638b-sys\") pod \"perf-node-gather-daemonset-ddjwz\" (UID: \"3d9e191f-f2c8-4c39-95b3-7d1ff33c638b\") " pod="openshift-must-gather-kn2d7/perf-node-gather-daemonset-ddjwz"
Sep 09 06:27:22.967038 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:22.967021    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d9e191f-f2c8-4c39-95b3-7d1ff33c638b-sys\") pod \"perf-node-gather-daemonset-ddjwz\" (UID: \"3d9e191f-f2c8-4c39-95b3-7d1ff33c638b\") " pod="openshift-must-gather-kn2d7/perf-node-gather-daemonset-ddjwz"
Sep 09 06:27:22.967267 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:22.967248    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3d9e191f-f2c8-4c39-95b3-7d1ff33c638b-podres\") pod \"perf-node-gather-daemonset-ddjwz\" (UID: \"3d9e191f-f2c8-4c39-95b3-7d1ff33c638b\") " pod="openshift-must-gather-kn2d7/perf-node-gather-daemonset-ddjwz"
Sep 09 06:27:22.967468 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:22.967448    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d9e191f-f2c8-4c39-95b3-7d1ff33c638b-lib-modules\") pod \"perf-node-gather-daemonset-ddjwz\" (UID: \"3d9e191f-f2c8-4c39-95b3-7d1ff33c638b\") " pod="openshift-must-gather-kn2d7/perf-node-gather-daemonset-ddjwz"
Sep 09 06:27:22.967572 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:22.967488    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3d9e191f-f2c8-4c39-95b3-7d1ff33c638b-proc\") pod \"perf-node-gather-daemonset-ddjwz\" (UID: \"3d9e191f-f2c8-4c39-95b3-7d1ff33c638b\") " pod="openshift-must-gather-kn2d7/perf-node-gather-daemonset-ddjwz"
Sep 09 06:27:22.975970 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:22.975940    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfxnt\" (UniqueName: \"kubernetes.io/projected/3d9e191f-f2c8-4c39-95b3-7d1ff33c638b-kube-api-access-pfxnt\") pod \"perf-node-gather-daemonset-ddjwz\" (UID: \"3d9e191f-f2c8-4c39-95b3-7d1ff33c638b\") " pod="openshift-must-gather-kn2d7/perf-node-gather-daemonset-ddjwz"
Sep 09 06:27:23.153804 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:23.153729    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kn2d7/perf-node-gather-daemonset-ddjwz"
Sep 09 06:27:23.449760 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:23.449685    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kn2d7/perf-node-gather-daemonset-ddjwz"]
Sep 09 06:27:23.455355 ip-10-0-131-142 kubenswrapper[2579]: W0909 06:27:23.455321    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3d9e191f_f2c8_4c39_95b3_7d1ff33c638b.slice/crio-0e3544f37142584786dd84b30d5c44d02d7b6ddac6755e7a7125257c3bcf6944 WatchSource:0}: Error finding container 0e3544f37142584786dd84b30d5c44d02d7b6ddac6755e7a7125257c3bcf6944: Status 404 returned error can't find the container with id 0e3544f37142584786dd84b30d5c44d02d7b6ddac6755e7a7125257c3bcf6944
Sep 09 06:27:23.555817 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:23.554960    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kn2d7/perf-node-gather-daemonset-ddjwz" event={"ID":"3d9e191f-f2c8-4c39-95b3-7d1ff33c638b","Type":"ContainerStarted","Data":"551d53fd0d9bf303987cc90b926144528e1553bd0faa94169e2b03cd85ed17f9"}
Sep 09 06:27:23.555817 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:23.555027    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kn2d7/perf-node-gather-daemonset-ddjwz" event={"ID":"3d9e191f-f2c8-4c39-95b3-7d1ff33c638b","Type":"ContainerStarted","Data":"0e3544f37142584786dd84b30d5c44d02d7b6ddac6755e7a7125257c3bcf6944"}
Sep 09 06:27:23.555817 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:23.555749    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kn2d7/perf-node-gather-daemonset-ddjwz"
Sep 09 06:27:23.571468 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:23.571420    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kn2d7/perf-node-gather-daemonset-ddjwz" podStartSLOduration=1.571402493 podStartE2EDuration="1.571402493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 06:27:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:27:23.57044394 +0000 UTC m=+2268.910931848" watchObservedRunningTime="2026-09-09 06:27:23.571402493 +0000 UTC m=+2268.911890403"
Sep 09 06:27:24.798745 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:24.798714    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sddl4_bc357870-f940-491c-be5f-3fbe8626631b/dns/0.log"
Sep 09 06:27:24.817942 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:24.817916    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sddl4_bc357870-f940-491c-be5f-3fbe8626631b/kube-rbac-proxy/0.log"
Sep 09 06:27:24.839004 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:24.838983    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-b9lbm_d6d0d7b7-703b-439d-9aae-064c217b875a/dns-node-resolver/0.log"
Sep 09 06:27:25.517060 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:25.517036    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-brqzd_cd282f88-a442-4ad3-931a-18e86703ece9/node-ca/0.log"
Sep 09 06:27:26.719794 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:26.719760    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f4d448d7f-58zzf_bf900674-4feb-4a9d-aed9-ed685de58c8f/kube-auth-proxy/0.log"
Sep 09 06:27:27.765649 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:27.765588    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-v95w5_cde197f1-1feb-4110-970e-53389ad986ae/serve-healthcheck-canary/0.log"
Sep 09 06:27:28.211091 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:28.211057    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5vkbv_f15c4b09-8d4c-4662-b9df-ef40e5ccc281/kube-rbac-proxy/0.log"
Sep 09 06:27:28.231562 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:28.231538    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5vkbv_f15c4b09-8d4c-4662-b9df-ef40e5ccc281/exporter/0.log"
Sep 09 06:27:28.252523 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:28.252487    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5vkbv_f15c4b09-8d4c-4662-b9df-ef40e5ccc281/extractor/0.log"
Sep 09 06:27:30.576393 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:30.576364    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kn2d7/perf-node-gather-daemonset-ddjwz"
Sep 09 06:27:30.978037 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:30.977952    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76b47fccc7-f4lfj_284a6006-17ba-4ed4-ab73-173bf23c0eae/manager/0.log"
Sep 09 06:27:30.994236 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:30.994211    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76b47fccc7-f4lfj_284a6006-17ba-4ed4-ab73-173bf23c0eae/copy-manifests/0.log"
Sep 09 06:27:32.377322 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:32.377286    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-58b7d76c47-c8lp2_26d3cd25-48db-43be-8b1e-b3ace484daf4/manager/0.log"
Sep 09 06:27:33.215058 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:33.215018    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f4d448d7f-58zzf_bf900674-4feb-4a9d-aed9-ed685de58c8f/kube-auth-proxy/0.log"
Sep 09 06:27:35.080470 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:35.080433    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f4d448d7f-58zzf_bf900674-4feb-4a9d-aed9-ed685de58c8f/kube-auth-proxy/0.log"
Sep 09 06:27:35.772637 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:35.772595    2579 scope.go:117] "RemoveContainer" containerID="9d04a7f151940f8992871a9818690f90953251b6c3a5c1d737aa4f5d27aa024e"
Sep 09 06:27:38.918797 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:38.918766    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-6z9q7_bbab69e9-3123-418a-92e7-88e2fb389fb5/kube-multus/0.log"
Sep 09 06:27:39.114871 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:39.114844    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q4pdb_08607c27-ea7f-4487-9e8c-d84f000464d7/kube-multus-additional-cni-plugins/0.log"
Sep 09 06:27:39.134729 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:39.134697    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q4pdb_08607c27-ea7f-4487-9e8c-d84f000464d7/egress-router-binary-copy/0.log"
Sep 09 06:27:39.156042 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:39.155992    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q4pdb_08607c27-ea7f-4487-9e8c-d84f000464d7/cni-plugins/0.log"
Sep 09 06:27:39.178488 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:39.178417    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q4pdb_08607c27-ea7f-4487-9e8c-d84f000464d7/bond-cni-plugin/0.log"
Sep 09 06:27:39.199181 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:39.199162    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q4pdb_08607c27-ea7f-4487-9e8c-d84f000464d7/routeoverride-cni/0.log"
Sep 09 06:27:39.219551 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:39.219525    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q4pdb_08607c27-ea7f-4487-9e8c-d84f000464d7/whereabouts-cni-bincopy/0.log"
Sep 09 06:27:39.239296 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:39.239276    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q4pdb_08607c27-ea7f-4487-9e8c-d84f000464d7/whereabouts-cni/0.log"
Sep 09 06:27:39.527081 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:39.527007    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-s7r5d_b179b936-0cca-4a9e-86ce-775441316808/network-metrics-daemon/0.log"
Sep 09 06:27:39.548975 ip-10-0-131-142 kubenswrapper[2579]: I0909 06:27:39.548949    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-s7r5d_b179b936-0cca-4a9e-86ce-775441316808/kube-rbac-proxy/0.log"