Sep 14 04:46:41.709024 ip-10-0-133-99 systemd[1]: Starting Kubernetes Kubelet...
Sep 14 04:46:42.138500 ip-10-0-133-99 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 14 04:46:42.138500 ip-10-0-133-99 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 14 04:46:42.138500 ip-10-0-133-99 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 14 04:46:42.138500 ip-10-0-133-99 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 14 04:46:42.138500 ip-10-0-133-99 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 14 04:46:42.140620 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.140541    2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 14 04:46:42.144550 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144535    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 04:46:42.144550 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144550    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 04:46:42.144616 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144554    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 04:46:42.144616 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144557    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 04:46:42.144616 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144560    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 04:46:42.144616 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144564    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 04:46:42.144616 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144567    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 04:46:42.144616 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144570    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 04:46:42.144616 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144572    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 04:46:42.144616 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144575    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 04:46:42.144616 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144578    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 04:46:42.144616 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144581    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 04:46:42.144616 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144583    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 04:46:42.144616 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144586    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 04:46:42.144616 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144594    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 04:46:42.144616 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144597    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 04:46:42.144616 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144599    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 04:46:42.144616 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144602    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 04:46:42.144616 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144604    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 04:46:42.144616 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144607    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 04:46:42.144616 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144610    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 04:46:42.145079 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144612    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 04:46:42.145079 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144615    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 04:46:42.145079 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144617    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 04:46:42.145079 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144620    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 04:46:42.145079 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144623    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 04:46:42.145079 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144626    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 04:46:42.145079 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144629    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 04:46:42.145079 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144632    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 04:46:42.145079 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144634    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 04:46:42.145079 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144637    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 04:46:42.145079 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144640    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 04:46:42.145079 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144642    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 04:46:42.145079 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144645    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 04:46:42.145079 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144647    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 04:46:42.145079 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144651    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 04:46:42.145079 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144653    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 04:46:42.145079 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144656    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 04:46:42.145079 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144658    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 04:46:42.145079 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144662    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 04:46:42.145079 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144664    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 04:46:42.145583 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144667    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 04:46:42.145583 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144669    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 04:46:42.145583 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144672    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 04:46:42.145583 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144674    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 04:46:42.145583 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144677    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 04:46:42.145583 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144680    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 04:46:42.145583 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144682    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 04:46:42.145583 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144685    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 04:46:42.145583 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144687    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 04:46:42.145583 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144690    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 04:46:42.145583 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144692    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 04:46:42.145583 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144695    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 04:46:42.145583 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144698    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 04:46:42.145583 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144700    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 04:46:42.145583 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144703    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 04:46:42.145583 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144705    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 04:46:42.145583 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144708    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 04:46:42.145583 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144711    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 04:46:42.145583 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144713    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 04:46:42.145583 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144716    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 04:46:42.146235 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144719    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 04:46:42.146235 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144721    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 04:46:42.146235 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144724    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 04:46:42.146235 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144726    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 04:46:42.146235 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144729    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 04:46:42.146235 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144732    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 04:46:42.146235 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144735    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 04:46:42.146235 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144739    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 04:46:42.146235 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144742    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 04:46:42.146235 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144745    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 04:46:42.146235 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144749    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 04:46:42.146235 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144764    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 04:46:42.146235 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144767    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 04:46:42.146235 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144770    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 04:46:42.146235 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144773    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 04:46:42.146235 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144776    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 04:46:42.146235 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144779    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 04:46:42.146235 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144782    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 04:46:42.146235 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144785    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 04:46:42.146875 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144788    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 04:46:42.146875 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144791    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 04:46:42.146875 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144794    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 04:46:42.146875 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144796    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 04:46:42.146875 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144799    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 04:46:42.146875 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144802    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 04:46:42.146875 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144805    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 04:46:42.146875 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144808    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 04:46:42.146875 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144812    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 04:46:42.146875 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144815    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 04:46:42.146875 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144819    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 04:46:42.146875 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144822    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 04:46:42.146875 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144825    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 04:46:42.146875 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144827    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 04:46:42.146875 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144831    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 04:46:42.146875 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144833    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 04:46:42.146875 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144836    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 04:46:42.146875 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144838    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 04:46:42.146875 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.144841    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 04:46:42.146875 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145500    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 04:46:42.147593 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145506    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 04:46:42.147593 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145509    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 04:46:42.147593 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145513    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 04:46:42.147593 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145518    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 04:46:42.147593 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145523    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 04:46:42.147593 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145526    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 04:46:42.147593 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145529    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 04:46:42.147593 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145532    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 04:46:42.147593 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145535    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 04:46:42.147593 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145538    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 04:46:42.147593 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145541    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 04:46:42.147593 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145544    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 04:46:42.147593 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145546    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 04:46:42.147593 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145549    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 04:46:42.147593 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145552    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 04:46:42.147593 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145554    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 04:46:42.147593 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145557    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 04:46:42.147593 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145560    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 04:46:42.147593 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145563    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 04:46:42.148146 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145565    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 04:46:42.148146 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145568    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 04:46:42.148146 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145571    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 04:46:42.148146 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145573    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 04:46:42.148146 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145576    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 04:46:42.148146 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145578    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 04:46:42.148146 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145581    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 04:46:42.148146 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145583    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 04:46:42.148146 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145586    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 04:46:42.148146 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145589    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 04:46:42.148146 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145591    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 04:46:42.148146 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145594    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 04:46:42.148146 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145597    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 04:46:42.148146 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145599    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 04:46:42.148146 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145602    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 04:46:42.148146 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145604    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 04:46:42.148146 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145607    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 04:46:42.148146 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145609    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 04:46:42.148146 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145612    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 04:46:42.148146 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145614    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 04:46:42.148640 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145617    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 04:46:42.148640 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145620    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 04:46:42.148640 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145624    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 04:46:42.148640 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145626    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 04:46:42.148640 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145629    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 04:46:42.148640 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145632    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 04:46:42.148640 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145634    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 04:46:42.148640 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145637    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 04:46:42.148640 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145639    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 04:46:42.148640 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145642    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 04:46:42.148640 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145644    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 04:46:42.148640 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145646    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 04:46:42.148640 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145649    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 04:46:42.148640 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145651    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 04:46:42.148640 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145654    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 04:46:42.148640 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145656    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 04:46:42.148640 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145659    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 04:46:42.148640 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145662    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 04:46:42.148640 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145664    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 04:46:42.148640 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145667    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 04:46:42.149145 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145669    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 04:46:42.149145 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145672    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 04:46:42.149145 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145674    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 04:46:42.149145 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145676    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 04:46:42.149145 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145679    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 04:46:42.149145 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145681    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 04:46:42.149145 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145684    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 04:46:42.149145 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145687    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 04:46:42.149145 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145691    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 04:46:42.149145 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145694    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 04:46:42.149145 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145697    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 04:46:42.149145 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145700    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 04:46:42.149145 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145732    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 04:46:42.149145 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145782    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 04:46:42.149145 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145790    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 04:46:42.149145 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145795    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 04:46:42.149145 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145800    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 04:46:42.149145 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145805    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 04:46:42.149145 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145809    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 04:46:42.149145 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145814    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 04:46:42.149626 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145819    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 04:46:42.149626 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145823    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 04:46:42.149626 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145828    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 04:46:42.149626 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145832    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 04:46:42.149626 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145837    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 04:46:42.149626 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145846    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 04:46:42.149626 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.145851    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 04:46:42.149626 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147144    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 04:46:42.149626 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147325    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 04:46:42.149626 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147342    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 04:46:42.149626 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147347    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 04:46:42.149626 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147351    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 04:46:42.149626 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147354    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 04:46:42.149626 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147358    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 04:46:42.149626 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147361    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 04:46:42.149626 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147364    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 04:46:42.149626 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147367    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 04:46:42.149626 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147369    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 04:46:42.149626 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147372    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 04:46:42.150107 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147779    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 04:46:42.150107 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147785    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 04:46:42.150107 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147788    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 04:46:42.150107 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147791    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 04:46:42.150107 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147794    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 04:46:42.150107 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147796    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 04:46:42.150107 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147799    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 04:46:42.150107 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147802    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 04:46:42.150107 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147805    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 04:46:42.150107 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147808    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 04:46:42.150107 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147811    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 04:46:42.150107 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147813    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 04:46:42.150107 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147816    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 04:46:42.150107 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147818    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 04:46:42.150107 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147821    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 04:46:42.150107 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147824    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 04:46:42.150107 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147826    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 04:46:42.150107 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147829    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 04:46:42.150107 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147831    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 04:46:42.150107 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147834    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 04:46:42.150580 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147837    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 04:46:42.150580 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147846    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 04:46:42.150580 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147849    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 04:46:42.150580 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147852    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 04:46:42.150580 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147855    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 04:46:42.150580 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147858    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 04:46:42.150580 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147860    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 04:46:42.150580 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147863    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 04:46:42.150580 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147868    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 04:46:42.150580 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147871    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 04:46:42.150580 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147874    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 04:46:42.150580 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147878    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 04:46:42.150580 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147880    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 04:46:42.150580 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147883    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 04:46:42.150580 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147886    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 04:46:42.150580 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147888    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 04:46:42.150580 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147891    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 04:46:42.150580 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147893    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 04:46:42.150580 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147896    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 04:46:42.151055 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147899    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 04:46:42.151055 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147901    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 04:46:42.151055 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147904    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 04:46:42.151055 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147906    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 04:46:42.151055 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147909    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 04:46:42.151055 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147912    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 04:46:42.151055 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147915    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 04:46:42.151055 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147917    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 04:46:42.151055 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147920    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 04:46:42.151055 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147922    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 04:46:42.151055 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147925    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 04:46:42.151055 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147927    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 04:46:42.151055 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147930    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 04:46:42.151055 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147933    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 04:46:42.151055 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147935    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 04:46:42.151055 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147938    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 04:46:42.151055 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147941    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 04:46:42.151055 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147943    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 04:46:42.151055 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147946    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 04:46:42.151504 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147948    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 04:46:42.151504 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147951    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 04:46:42.151504 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147953    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 04:46:42.151504 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147956    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 04:46:42.151504 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147960    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 04:46:42.151504 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147964    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 04:46:42.151504 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147967    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 04:46:42.151504 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147970    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 04:46:42.151504 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147973    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 04:46:42.151504 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147976    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 04:46:42.151504 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147978    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 04:46:42.151504 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147981    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 04:46:42.151504 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147984    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 04:46:42.151504 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147987    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 04:46:42.151504 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147990    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 04:46:42.151504 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147992    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 04:46:42.151504 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.147997    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 04:46:42.151504 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148000    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 04:46:42.151504 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148003    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 04:46:42.151977 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148005    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 04:46:42.151977 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148008    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 04:46:42.151977 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148010    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 04:46:42.151977 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148013    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 04:46:42.151977 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148016    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 04:46:42.151977 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148018    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 04:46:42.151977 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148021    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 04:46:42.151977 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148023    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 04:46:42.151977 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148026    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 04:46:42.151977 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148029    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 04:46:42.151977 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148031    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 04:46:42.151977 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148034    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 04:46:42.151977 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148037    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 04:46:42.151977 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148039    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 04:46:42.151977 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148042    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 04:46:42.151977 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148045    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 04:46:42.151977 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148047    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 04:46:42.151977 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148050    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 04:46:42.151977 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148052    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 04:46:42.151977 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148055    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 04:46:42.152456 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148058    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 04:46:42.152456 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148063    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 04:46:42.152456 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148133    2565 flags.go:64] FLAG: --address="0.0.0.0"
Sep 14 04:46:42.152456 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148170    2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 14 04:46:42.152456 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148176    2565 flags.go:64] FLAG: --anonymous-auth="true"
Sep 14 04:46:42.152456 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148180    2565 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 14 04:46:42.152456 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148184    2565 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 14 04:46:42.152456 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148187    2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 14 04:46:42.152456 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148191    2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 14 04:46:42.152456 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148195    2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 14 04:46:42.152456 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148198    2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 14 04:46:42.152456 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148201    2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 14 04:46:42.152456 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148205    2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 14 04:46:42.152456 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148208    2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 14 04:46:42.152456 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148211    2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 14 04:46:42.152456 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148214    2565 flags.go:64] FLAG: --cgroup-root=""
Sep 14 04:46:42.152456 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148217    2565 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 14 04:46:42.152456 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148220    2565 flags.go:64] FLAG: --client-ca-file=""
Sep 14 04:46:42.152456 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148223    2565 flags.go:64] FLAG: --cloud-provider="external"
Sep 14 04:46:42.152456 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148226    2565 flags.go:64] FLAG: --cluster-dns="[]"
Sep 14 04:46:42.152456 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148230    2565 flags.go:64] FLAG: --cluster-domain=""
Sep 14 04:46:42.152456 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148233    2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 14 04:46:42.152456 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148237    2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 14 04:46:42.152456 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148240    2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 14 04:46:42.153062 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148244    2565 flags.go:64] FLAG: --container-log-max-files="5"
Sep 14 04:46:42.153062 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148248    2565 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 14 04:46:42.153062 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148251    2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 14 04:46:42.153062 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148253    2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 14 04:46:42.153062 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148257    2565 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 14 04:46:42.153062 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148260    2565 flags.go:64] FLAG: --contention-profiling="false"
Sep 14 04:46:42.153062 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148263    2565 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 14 04:46:42.153062 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148266    2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 14 04:46:42.153062 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148269    2565 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 14 04:46:42.153062 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148273    2565 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 14 04:46:42.153062 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148277    2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 14 04:46:42.153062 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148280    2565 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 14 04:46:42.153062 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148283    2565 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 14 04:46:42.153062 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148286    2565 flags.go:64] FLAG: --enable-load-reader="false"
Sep 14 04:46:42.153062 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148289    2565 flags.go:64] FLAG: --enable-server="true"
Sep 14 04:46:42.153062 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148292    2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 14 04:46:42.153062 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148296    2565 flags.go:64] FLAG: --event-burst="100"
Sep 14 04:46:42.153062 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148299    2565 flags.go:64] FLAG: --event-qps="50"
Sep 14 04:46:42.153062 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148302    2565 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 14 04:46:42.153062 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148305    2565 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 14 04:46:42.153062 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148308    2565 flags.go:64] FLAG: --eviction-hard=""
Sep 14 04:46:42.153062 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148311    2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 14 04:46:42.153062 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148314    2565 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 14 04:46:42.153062 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148317    2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 14 04:46:42.153062 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148320    2565 flags.go:64] FLAG: --eviction-soft=""
Sep 14 04:46:42.153831 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148323    2565 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 14 04:46:42.153831 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148326    2565 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 14 04:46:42.153831 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148329    2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 14 04:46:42.153831 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148332    2565 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 14 04:46:42.153831 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148335    2565 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 14 04:46:42.153831 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148338    2565 flags.go:64] FLAG: --fail-swap-on="true"
Sep 14 04:46:42.153831 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148341    2565 flags.go:64] FLAG: --feature-gates=""
Sep 14 04:46:42.153831 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148349    2565 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 14 04:46:42.153831 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148353    2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 14 04:46:42.153831 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148356    2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 14 04:46:42.153831 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148359    2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 14 04:46:42.153831 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148362    2565 flags.go:64] FLAG: --healthz-port="10248"
Sep 14 04:46:42.153831 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148365    2565 flags.go:64] FLAG: --help="false"
Sep 14 04:46:42.153831 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148368    2565 flags.go:64] FLAG: --hostname-override="ip-10-0-133-99.ec2.internal"
Sep 14 04:46:42.153831 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148371    2565 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 14 04:46:42.153831 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148374    2565 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 14 04:46:42.153831 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148378    2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 14 04:46:42.153831 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148381    2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 14 04:46:42.153831 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148384    2565 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 14 04:46:42.153831 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148387    2565 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 14 04:46:42.153831 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148390    2565 flags.go:64] FLAG: --image-service-endpoint=""
Sep 14 04:46:42.153831 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148393    2565 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 14 04:46:42.153831 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148396    2565 flags.go:64] FLAG: --kube-api-burst="100"
Sep 14 04:46:42.154437 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148399    2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 14 04:46:42.154437 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148402    2565 flags.go:64] FLAG: --kube-api-qps="50"
Sep 14 04:46:42.154437 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148405    2565 flags.go:64] FLAG: --kube-reserved=""
Sep 14 04:46:42.154437 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148408    2565 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 14 04:46:42.154437 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148411    2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 14 04:46:42.154437 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148414    2565 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 14 04:46:42.154437 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148417    2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 14 04:46:42.154437 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148420    2565 flags.go:64] FLAG: --lock-file=""
Sep 14 04:46:42.154437 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148422    2565 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 14 04:46:42.154437 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148425    2565 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 14 04:46:42.154437 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148428    2565 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 14 04:46:42.154437 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148438    2565 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 14 04:46:42.154437 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148442    2565 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 14 04:46:42.154437 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148445    2565 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 14 04:46:42.154437 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148448    2565 flags.go:64] FLAG: --logging-format="text"
Sep 14 04:46:42.154437 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148451    2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 14 04:46:42.154437 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148454    2565 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 14 04:46:42.154437 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148457    2565 flags.go:64] FLAG: --manifest-url=""
Sep 14 04:46:42.154437 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148460    2565 flags.go:64] FLAG: --manifest-url-header=""
Sep 14 04:46:42.154437 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148465    2565 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 14 04:46:42.154437 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148468    2565 flags.go:64] FLAG: --max-open-files="1000000"
Sep 14 04:46:42.154437 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148471    2565 flags.go:64] FLAG: --max-pods="110"
Sep 14 04:46:42.154437 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148474    2565 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 14 04:46:42.154437 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148477    2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 14 04:46:42.154437 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148480    2565 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 14 04:46:42.155068 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148484    2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 14 04:46:42.155068 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148487    2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 14 04:46:42.155068 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148490    2565 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 14 04:46:42.155068 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148494    2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 14 04:46:42.155068 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148500    2565 flags.go:64] FLAG: --node-status-max-images="50"
Sep 14 04:46:42.155068 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148503    2565 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 14 04:46:42.155068 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148506    2565 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 14 04:46:42.155068 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148509    2565 flags.go:64] FLAG: --pod-cidr=""
Sep 14 04:46:42.155068 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148512    2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 14 04:46:42.155068 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148517    2565 flags.go:64] FLAG: --pod-manifest-path=""
Sep 14 04:46:42.155068 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148520    2565 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 14 04:46:42.155068 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148523    2565 flags.go:64] FLAG: --pods-per-core="0"
Sep 14 04:46:42.155068 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148526    2565 flags.go:64] FLAG: --port="10250"
Sep 14 04:46:42.155068 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148529    2565 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 14 04:46:42.155068 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148532    2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05e389e88b519bd04"
Sep 14 04:46:42.155068 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148536    2565 flags.go:64] FLAG: --qos-reserved=""
Sep 14 04:46:42.155068 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148538    2565 flags.go:64] FLAG: --read-only-port="10255"
Sep 14 04:46:42.155068 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148541    2565 flags.go:64] FLAG: --register-node="true"
Sep 14 04:46:42.155068 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148544    2565 flags.go:64] FLAG: --register-with-taints=""
Sep 14 04:46:42.155068 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148548    2565 flags.go:64] FLAG: --registry-burst="10"
Sep 14 04:46:42.155068 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148552    2565 flags.go:64] FLAG: --registry-qps="5"
Sep 14 04:46:42.155068 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148555    2565 flags.go:64] FLAG: --reserved-cpus=""
Sep 14 04:46:42.155068 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148557    2565 flags.go:64] FLAG: --reserved-memory=""
Sep 14 04:46:42.155068 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148561    2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 14 04:46:42.155068 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148564    2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 14 04:46:42.155668 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148567    2565 flags.go:64] FLAG: --rotate-certificates="false"
Sep 14 04:46:42.155668 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148571    2565 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 14 04:46:42.155668 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148573    2565 flags.go:64] FLAG: --runonce="false"
Sep 14 04:46:42.155668 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148576    2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 14 04:46:42.155668 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148579    2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 14 04:46:42.155668 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148582    2565 flags.go:64] FLAG: --seccomp-default="false"
Sep 14 04:46:42.155668 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148585    2565 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 14 04:46:42.155668 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148589    2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 14 04:46:42.155668 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148592    2565 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 14 04:46:42.155668 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148595    2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 14 04:46:42.155668 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148598    2565 flags.go:64] FLAG: --storage-driver-password="root"
Sep 14 04:46:42.155668 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148601    2565 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 14 04:46:42.155668 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148604    2565 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 14 04:46:42.155668 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148607    2565 flags.go:64] FLAG: --storage-driver-user="root"
Sep 14 04:46:42.155668 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148610    2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 14 04:46:42.155668 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148613    2565 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 14 04:46:42.155668 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148615    2565 flags.go:64] FLAG: --system-cgroups=""
Sep 14 04:46:42.155668 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148618    2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 14 04:46:42.155668 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148624    2565 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 14 04:46:42.155668 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148626    2565 flags.go:64] FLAG: --tls-cert-file=""
Sep 14 04:46:42.155668 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148629    2565 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 14 04:46:42.155668 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148633    2565 flags.go:64] FLAG: --tls-min-version=""
Sep 14 04:46:42.155668 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148636    2565 flags.go:64] FLAG: --tls-private-key-file=""
Sep 14 04:46:42.155668 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148639    2565 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 14 04:46:42.155668 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148642    2565 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 14 04:46:42.156298 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148645    2565 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 14 04:46:42.156298 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148647    2565 flags.go:64] FLAG: --v="2"
Sep 14 04:46:42.156298 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148652    2565 flags.go:64] FLAG: --version="false"
Sep 14 04:46:42.156298 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148656    2565 flags.go:64] FLAG: --vmodule=""
Sep 14 04:46:42.156298 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148660    2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 14 04:46:42.156298 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.148663    2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 14 04:46:42.156298 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148779    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 04:46:42.156298 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148783    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 04:46:42.156298 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148786    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 04:46:42.156298 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148789    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 04:46:42.156298 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148792    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 04:46:42.156298 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148795    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 04:46:42.156298 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148798    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 04:46:42.156298 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148800    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 04:46:42.156298 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148805    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 04:46:42.156298 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148808    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 04:46:42.156298 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148811    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 04:46:42.156298 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148814    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 04:46:42.156298 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148816    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 04:46:42.156298 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148819    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 04:46:42.156298 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148822    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 04:46:42.156817 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148824    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 04:46:42.156817 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148827    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 04:46:42.156817 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148830    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 04:46:42.156817 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148834    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 04:46:42.156817 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148838    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 04:46:42.156817 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148841    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 04:46:42.156817 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148844    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 04:46:42.156817 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148846    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 04:46:42.156817 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148849    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 04:46:42.156817 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148852    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 04:46:42.156817 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148855    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 04:46:42.156817 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148857    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 04:46:42.156817 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148860    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 04:46:42.156817 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148864    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 04:46:42.156817 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148866    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 04:46:42.156817 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148869    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 04:46:42.156817 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148873    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 04:46:42.156817 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148876    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 04:46:42.157281 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148880    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 04:46:42.157281 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148883    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 04:46:42.157281 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148886    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 04:46:42.157281 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148893    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 04:46:42.157281 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148896    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 04:46:42.157281 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148899    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 04:46:42.157281 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148902    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 04:46:42.157281 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148906    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 04:46:42.157281 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148908    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 04:46:42.157281 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148911    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 04:46:42.157281 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148914    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 04:46:42.157281 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148916    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 04:46:42.157281 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148919    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 04:46:42.157281 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148922    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 04:46:42.157281 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148924    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 04:46:42.157281 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148927    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 04:46:42.157281 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148929    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 04:46:42.157281 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148932    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 04:46:42.157281 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148935    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 04:46:42.157281 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148937    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 04:46:42.157777 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148940    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 04:46:42.157777 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148943    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 04:46:42.157777 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148946    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 04:46:42.157777 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148948    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 04:46:42.157777 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148951    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 04:46:42.157777 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148954    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 04:46:42.157777 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148956    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 04:46:42.157777 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148960    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 04:46:42.157777 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148963    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 04:46:42.157777 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148966    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 04:46:42.157777 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148968    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 04:46:42.157777 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148971    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 04:46:42.157777 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148973    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 04:46:42.157777 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148976    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 04:46:42.157777 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148979    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 04:46:42.157777 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148981    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 04:46:42.157777 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148984    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 04:46:42.157777 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148987    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 04:46:42.157777 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148990    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 04:46:42.157777 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148994    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 04:46:42.158278 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148997    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 04:46:42.158278 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.148999    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 04:46:42.158278 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.149002    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 04:46:42.158278 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.149005    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 04:46:42.158278 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.149007    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 04:46:42.158278 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.149010    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 04:46:42.158278 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.149013    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 04:46:42.158278 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.149015    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 04:46:42.158278 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.149018    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 04:46:42.158278 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.149020    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 04:46:42.158278 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.149023    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 04:46:42.158278 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.149026    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 04:46:42.158278 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.149028    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 04:46:42.158278 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.149031    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 04:46:42.158278 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.149034    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 04:46:42.158278 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.149036    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 04:46:42.158278 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.149039    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 04:46:42.158278 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.149041    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 04:46:42.158278 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.149044    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 04:46:42.158278 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.149050    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 04:46:42.158779 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.149052    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 04:46:42.158779 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.149055    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 04:46:42.158779 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.149058    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 04:46:42.158779 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.149060    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 04:46:42.158779 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.149063    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 04:46:42.158779 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.149066    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 04:46:42.158779 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.149738    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 14 04:46:42.158779 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.157138    2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 14 04:46:42.158779 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.157152    2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 14 04:46:42.158779 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.158061    2565 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 14 04:46:42.158779 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.158072    2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 14 04:46:42.158779 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158135    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 04:46:42.158779 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158140    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 04:46:42.158779 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158144    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 04:46:42.159134 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158147    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 04:46:42.159134 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158151    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 04:46:42.159134 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158154    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 04:46:42.159134 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158157    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 04:46:42.159134 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158160    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 04:46:42.159134 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158162    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 04:46:42.159134 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158165    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 04:46:42.159134 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158167    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 04:46:42.159134 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158170    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 04:46:42.159134 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158173    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 04:46:42.159134 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158176    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 04:46:42.159134 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158178    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 04:46:42.159134 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158181    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 04:46:42.159134 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158184    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 04:46:42.159134 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158187    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 04:46:42.159134 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158189    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 04:46:42.159134 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158192    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 04:46:42.159134 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158195    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 04:46:42.159134 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158198    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 04:46:42.159134 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158201    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 04:46:42.159613 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158203    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 04:46:42.159613 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158206    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 04:46:42.159613 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158208    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 04:46:42.159613 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158211    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 04:46:42.159613 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158213    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 04:46:42.159613 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158216    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 04:46:42.159613 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158219    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 04:46:42.159613 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158222    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 04:46:42.159613 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158224    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 04:46:42.159613 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158227    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 04:46:42.159613 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158229    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 04:46:42.159613 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158232    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 04:46:42.159613 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158235    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 04:46:42.159613 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158237    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 04:46:42.159613 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158240    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 04:46:42.159613 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158242    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 04:46:42.159613 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158245    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 04:46:42.159613 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158247    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 04:46:42.159613 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158250    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 04:46:42.159613 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158252    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 04:46:42.160170 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158255    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 04:46:42.160170 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158258    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 04:46:42.160170 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158261    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 04:46:42.160170 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158265    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 04:46:42.160170 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158268    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 04:46:42.160170 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158271    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 04:46:42.160170 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158274    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 04:46:42.160170 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158276    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 04:46:42.160170 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158279    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 04:46:42.160170 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158282    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 04:46:42.160170 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158286    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 04:46:42.160170 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158290    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 04:46:42.160170 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158293    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 04:46:42.160170 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158295    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 04:46:42.160170 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158298    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 04:46:42.160170 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158300    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 04:46:42.160170 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158303    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 04:46:42.160170 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158306    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 04:46:42.160170 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158309    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 04:46:42.160659 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158311    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 04:46:42.160659 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158314    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 04:46:42.160659 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158317    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 04:46:42.160659 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158319    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 04:46:42.160659 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158322    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 04:46:42.160659 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158324    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 04:46:42.160659 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158327    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 04:46:42.160659 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158329    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 04:46:42.160659 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158332    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 04:46:42.160659 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158334    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 04:46:42.160659 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158336    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 04:46:42.160659 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158339    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 04:46:42.160659 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158343    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 04:46:42.160659 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158346    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 04:46:42.160659 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158349    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 04:46:42.160659 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158352    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 04:46:42.160659 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158354    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 04:46:42.160659 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158357    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 04:46:42.160659 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158360    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 04:46:42.160659 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158362    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 04:46:42.161156 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158365    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 04:46:42.161156 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158368    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 04:46:42.161156 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158370    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 04:46:42.161156 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158373    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 04:46:42.161156 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158376    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 04:46:42.161156 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158378    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 04:46:42.161156 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158381    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 04:46:42.161156 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158383    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 04:46:42.161156 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158386    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 04:46:42.161156 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158388    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 04:46:42.161156 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158391    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 04:46:42.161156 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158393    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 04:46:42.161156 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158396    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 04:46:42.161156 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158399    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 04:46:42.161156 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158402    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 04:46:42.161156 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158404    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 04:46:42.161156 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158407    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 04:46:42.161571 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.158412    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 14 04:46:42.161571 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158537    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 04:46:42.161571 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158542    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 04:46:42.161571 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158545    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 04:46:42.161571 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158548    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 04:46:42.161571 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158553    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 04:46:42.161571 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158556    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 04:46:42.161571 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158558    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 04:46:42.161571 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158561    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 04:46:42.161571 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158564    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 04:46:42.161571 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158567    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 04:46:42.161571 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158569    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 04:46:42.161571 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158572    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 04:46:42.161571 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158574    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 04:46:42.161571 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158577    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 04:46:42.161937 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158579    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 04:46:42.161937 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158582    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 04:46:42.161937 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158585    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 04:46:42.161937 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158587    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 04:46:42.161937 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158590    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 04:46:42.161937 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158592    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 04:46:42.161937 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158595    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 04:46:42.161937 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158598    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 04:46:42.161937 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158600    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 04:46:42.161937 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158603    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 04:46:42.161937 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158605    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 04:46:42.161937 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158608    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 04:46:42.161937 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158611    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 04:46:42.161937 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158613    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 04:46:42.161937 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158617    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 04:46:42.161937 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158619    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 04:46:42.161937 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158622    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 04:46:42.161937 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158625    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 04:46:42.161937 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158628    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 04:46:42.161937 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158630    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 04:46:42.162420 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158633    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 04:46:42.162420 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158635    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 04:46:42.162420 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158638    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 04:46:42.162420 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158640    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 04:46:42.162420 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158644    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 04:46:42.162420 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158648    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 04:46:42.162420 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158652    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 04:46:42.162420 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158655    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 04:46:42.162420 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158658    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 04:46:42.162420 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158661    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 04:46:42.162420 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158665    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 04:46:42.162420 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158668    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 04:46:42.162420 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158671    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 04:46:42.162420 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158674    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 04:46:42.162420 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158676    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 04:46:42.162420 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158679    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 04:46:42.162420 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158682    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 04:46:42.162420 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158684    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 04:46:42.162420 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158687    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 04:46:42.162887 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158690    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 04:46:42.162887 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158692    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 04:46:42.162887 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158695    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 04:46:42.162887 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158697    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 04:46:42.162887 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158700    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 04:46:42.162887 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158703    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 04:46:42.162887 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158705    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 04:46:42.162887 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158708    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 04:46:42.162887 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158711    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 04:46:42.162887 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158714    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 04:46:42.162887 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158716    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 04:46:42.162887 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158719    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 04:46:42.162887 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158721    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 04:46:42.162887 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158724    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 04:46:42.162887 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158727    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 04:46:42.162887 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158729    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 04:46:42.162887 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158732    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 04:46:42.162887 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158734    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 04:46:42.162887 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158737    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 04:46:42.163337 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158740    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 04:46:42.163337 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158742    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 04:46:42.163337 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158745    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 04:46:42.163337 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158747    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 04:46:42.163337 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158750    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 04:46:42.163337 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158764    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 04:46:42.163337 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158767    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 04:46:42.163337 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158769    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 04:46:42.163337 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158772    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 04:46:42.163337 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158774    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 04:46:42.163337 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158777    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 04:46:42.163337 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158780    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 04:46:42.163337 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158782    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 04:46:42.163337 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158785    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 04:46:42.163337 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158788    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 04:46:42.163337 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158791    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 04:46:42.163337 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158793    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 04:46:42.163337 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158796    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 04:46:42.163337 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158799    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 04:46:42.163337 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158801    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 04:46:42.163817 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158804    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 04:46:42.163817 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158806    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 04:46:42.163817 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158809    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 04:46:42.163817 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158812    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 04:46:42.163817 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158814    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 04:46:42.163817 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158817    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 04:46:42.163817 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:42.158819    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 04:46:42.163817 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.158824    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 14 04:46:42.163817 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.158920    2565 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 14 04:46:42.163817 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.161293    2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 14 04:46:42.163817 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.162343    2565 server.go:1019] "Starting client certificate rotation"
Sep 14 04:46:42.163817 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.162444    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 14 04:46:42.163817 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.162473    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 14 04:46:42.165249 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.165237    2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 14 04:46:42.168457 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.168443    2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 14 04:46:42.178051 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.178036    2565 log.go:25] "Validated CRI v1 runtime API"
Sep 14 04:46:42.181889 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.181876    2565 log.go:25] "Validated CRI v1 image API"
Sep 14 04:46:42.183420 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.183401    2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 14 04:46:42.186785 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.186748    2565 fs.go:135] Filesystem UUIDs: map[28bff568-db8c-415b-816e-fb1f3e2cd58f:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 993746a3-39bf-40f0-8a0b-f128460973fc:/dev/nvme0n1p4]
Sep 14 04:46:42.186838 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.186785    2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 14 04:46:42.192962 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.192818    2565 manager.go:216] Machine: {Timestamp:2026-09-14 04:46:42.191394631 +0000 UTC m=+0.367166183 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105478 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec284c439c861017b6a8cbcb5aa1d588 SystemUUID:ec284c43-9c86-1017-b6a8-cbcb5aa1d588 BootID:c983cb5c-d71a-4060-ac27-5da3eb3a4a24 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e9:d3:6c:bf Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e9:d3:6c:bf Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:14:cb:2f:51:0e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 14 04:46:42.193052 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.192961    2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 14 04:46:42.193085 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.193048    2565 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 14 04:46:42.196745 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.196724    2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 14 04:46:42.196884 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.196748    2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-99.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 14 04:46:42.198240 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.198227    2565 topology_manager.go:138] "Creating topology manager with none policy"
Sep 14 04:46:42.198240 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.198242    2565 container_manager_linux.go:309] "Creating device plugin manager"
Sep 14 04:46:42.198332 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.198251    2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 14 04:46:42.198891 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.198881    2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 14 04:46:42.200062 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.200053    2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 14 04:46:42.200522 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.200505    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 14 04:46:42.204387 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.204374    2565 state_mem.go:36] "Initialized new in-memory state store"
Sep 14 04:46:42.204508 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.204499    2565 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 14 04:46:42.204538 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.204534    2565 kubelet.go:486] "Attempting to sync node with API server"
Sep 14 04:46:42.204573 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.204556    2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 14 04:46:42.204573 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.204571    2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 14 04:46:42.204628 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.204580    2565 kubelet.go:398] "Adding apiserver pod source"
Sep 14 04:46:42.204628 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.204589    2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 14 04:46:42.204991 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.204980    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 14 04:46:42.205029 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.204999    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 14 04:46:42.207652 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.207627    2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 14 04:46:42.210740 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.210727    2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 14 04:46:42.210798 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.210771    2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 14 04:46:42.212712 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.212699    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 14 04:46:42.212785 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.212717    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 14 04:46:42.212785 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.212723    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 14 04:46:42.212785 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.212729    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 14 04:46:42.212785 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.212738    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 14 04:46:42.212785 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.212743    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 14 04:46:42.212785 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.212772    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 14 04:46:42.212785 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.212781    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 14 04:46:42.212785 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.212789    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 14 04:46:42.212996 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.212795    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 14 04:46:42.212996 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.212804    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 14 04:46:42.212996 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.212809    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 14 04:46:42.214566 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.214554    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 14 04:46:42.214566 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.214565    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 14 04:46:42.217666 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:42.217642    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 14 04:46:42.217743 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:42.217642    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 14 04:46:42.217872 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.217856    2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 14 04:46:42.218239 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.218228    2565 server.go:1268] "Started kubelet"
Sep 14 04:46:42.218336 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.218303    2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 14 04:46:42.218412 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.218320    2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 14 04:46:42.218412 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.218384    2565 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 14 04:46:42.219113 ip-10-0-133-99 systemd[1]: Started Kubernetes Kubelet.
Sep 14 04:46:42.219829 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.219816    2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 14 04:46:42.220956 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.220944    2565 server.go:310] "Adding debug handlers to kubelet server"
Sep 14 04:46:42.225160 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:42.225132    2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 14 04:46:42.229405 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.229386    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 14 04:46:42.230095 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.230070    2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 14 04:46:42.230172 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.230118    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-67szt"
Sep 14 04:46:42.230501 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:42.230464    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-99.ec2.internal\" not found"
Sep 14 04:46:42.230893 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.230876    2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 14 04:46:42.230961 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.230878    2565 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 14 04:46:42.230961 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.230914    2565 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 14 04:46:42.230961 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.230894    2565 factory.go:55] Registering systemd factory
Sep 14 04:46:42.231051 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.230975    2565 factory.go:223] Registration of the systemd container factory successfully
Sep 14 04:46:42.231051 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.231000    2565 reconstruct.go:97] "Volume reconstruction finished"
Sep 14 04:46:42.231051 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.231011    2565 reconciler.go:29] "Reconciler: start to sync state"
Sep 14 04:46:42.231200 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.231186    2565 factory.go:153] Registering CRI-O factory
Sep 14 04:46:42.231257 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.231204    2565 factory.go:223] Registration of the crio container factory successfully
Sep 14 04:46:42.231319 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.231305    2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 14 04:46:42.231377 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.231329    2565 factory.go:103] Registering Raw factory
Sep 14 04:46:42.231377 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.231346    2565 manager.go:1196] Started watching for new ooms in manager
Sep 14 04:46:42.231729 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.231718    2565 manager.go:349] Starting recovery of all containers
Sep 14 04:46:42.233971 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:42.233942    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 14 04:46:42.234677 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:42.234652    2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 14 04:46:42.235294 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:42.234037    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-99.ec2.internal.18d516887114f182  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-99.ec2.internal,UID:ip-10-0-133-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-99.ec2.internal,},FirstTimestamp:2026-09-14 04:46:42.218201474 +0000 UTC m=+0.393973024,LastTimestamp:2026-09-14 04:46:42.218201474 +0000 UTC m=+0.393973024,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-99.ec2.internal,}"
Sep 14 04:46:42.237254 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.237231    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-67szt"
Sep 14 04:46:42.237370 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.237334    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 14 04:46:42.240843 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.240827    2565 manager.go:354] Recovery completed
Sep 14 04:46:42.246990 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.246977    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 04:46:42.249584 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.248696    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-99.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 04:46:42.249584 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.248721    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-99.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 04:46:42.249584 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.248736    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-99.ec2.internal" event="NodeHasSufficientPID"
Sep 14 04:46:42.249584 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.249341    2565 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 14 04:46:42.249584 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.249350    2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 14 04:46:42.249584 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.249363    2565 state_mem.go:36] "Initialized new in-memory state store"
Sep 14 04:46:42.249976 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.249797    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 04:46:42.250540 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.250525    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-99.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 04:46:42.250649 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.250549    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-99.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 04:46:42.250649 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.250559    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-99.ec2.internal" event="NodeHasSufficientPID"
Sep 14 04:46:42.251435 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.251419    2565 policy_none.go:49] "None policy: Start"
Sep 14 04:46:42.251495 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.251439    2565 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 14 04:46:42.251495 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.251454    2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 14 04:46:42.251731 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:42.251651    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-99.ec2.internal.18d5168872e672a2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-99.ec2.internal,UID:ip-10-0-133-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-99.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-99.ec2.internal,},FirstTimestamp:2026-09-14 04:46:42.24870877 +0000 UTC m=+0.424480325,LastTimestamp:2026-09-14 04:46:42.24870877 +0000 UTC m=+0.424480325,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-99.ec2.internal,}"
Sep 14 04:46:42.253329 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.253317    2565 policy_none.go:47] "Start"
Sep 14 04:46:42.278518 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.278383    2565 manager.go:337] "Starting Device Plugin manager"
Sep 14 04:46:42.278601 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:42.278540    2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 14 04:46:42.278601 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.278552    2565 server.go:87] "Starting device plugin registration server"
Sep 14 04:46:42.279157 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.278779    2565 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 14 04:46:42.279157 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.278790    2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 14 04:46:42.279157 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.279089    2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 14 04:46:42.279688 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.279213    2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 14 04:46:42.279688 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.279221    2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 14 04:46:42.279915 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:42.279895    2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 14 04:46:42.279981 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:42.279945    2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-99.ec2.internal\" not found"
Sep 14 04:46:42.327816 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.327788    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 14 04:46:42.327910 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.327841    2565 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 14 04:46:42.327910 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.327865    2565 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 14 04:46:42.328004 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:42.327938    2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 14 04:46:42.331644 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.331626    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 14 04:46:42.379782 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.379761    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 04:46:42.382113 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.381337    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-99.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 04:46:42.382113 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.381370    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-99.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 04:46:42.382113 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.381384    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-99.ec2.internal" event="NodeHasSufficientPID"
Sep 14 04:46:42.382113 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.381412    2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-99.ec2.internal"
Sep 14 04:46:42.392637 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.392572    2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-99.ec2.internal"
Sep 14 04:46:42.392637 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:42.392593    2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-99.ec2.internal\": node \"ip-10-0-133-99.ec2.internal\" not found"
Sep 14 04:46:42.407597 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:42.407578    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-99.ec2.internal\" not found"
Sep 14 04:46:42.428272 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.428245    2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-99.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-99.ec2.internal"]
Sep 14 04:46:42.428375 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.428313    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 04:46:42.429141 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.429128    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-99.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 04:46:42.429201 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.429153    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-99.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 04:46:42.429201 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.429165    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-99.ec2.internal" event="NodeHasSufficientPID"
Sep 14 04:46:42.431222 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.431210    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 04:46:42.431372 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.431357    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-99.ec2.internal"
Sep 14 04:46:42.431443 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.431391    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 04:46:42.431908 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.431891    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-99.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 04:46:42.431968 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.431921    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-99.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 04:46:42.431968 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.431932    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-99.ec2.internal" event="NodeHasSufficientPID"
Sep 14 04:46:42.432043 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.431977    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-99.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 04:46:42.432043 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.431995    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-99.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 04:46:42.432043 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.432005    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-99.ec2.internal" event="NodeHasSufficientPID"
Sep 14 04:46:42.434025 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.434012    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-99.ec2.internal"
Sep 14 04:46:42.434062 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.434042    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 04:46:42.434766 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.434743    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-99.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 04:46:42.434818 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.434774    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-99.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 04:46:42.434818 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.434785    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-99.ec2.internal" event="NodeHasSufficientPID"
Sep 14 04:46:42.447329 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:42.447312    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-99.ec2.internal\" not found" node="ip-10-0-133-99.ec2.internal"
Sep 14 04:46:42.450722 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:42.450709    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-99.ec2.internal\" not found" node="ip-10-0-133-99.ec2.internal"
Sep 14 04:46:42.507851 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:42.507828    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-99.ec2.internal\" not found"
Sep 14 04:46:42.531877 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.531844    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/53c18eb6ba51406b052ec7b9ec1cf464-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-99.ec2.internal\" (UID: \"53c18eb6ba51406b052ec7b9ec1cf464\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-99.ec2.internal"
Sep 14 04:46:42.531877 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.531881    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53c18eb6ba51406b052ec7b9ec1cf464-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-99.ec2.internal\" (UID: \"53c18eb6ba51406b052ec7b9ec1cf464\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-99.ec2.internal"
Sep 14 04:46:42.531998 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.531910    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d5acb9fe33c38810a5100cf8c524f161-config\") pod \"kube-apiserver-proxy-ip-10-0-133-99.ec2.internal\" (UID: \"d5acb9fe33c38810a5100cf8c524f161\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-99.ec2.internal"
Sep 14 04:46:42.608394 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:42.608365    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-99.ec2.internal\" not found"
Sep 14 04:46:42.632263 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.632240    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/53c18eb6ba51406b052ec7b9ec1cf464-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-99.ec2.internal\" (UID: \"53c18eb6ba51406b052ec7b9ec1cf464\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-99.ec2.internal"
Sep 14 04:46:42.632364 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.632276    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53c18eb6ba51406b052ec7b9ec1cf464-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-99.ec2.internal\" (UID: \"53c18eb6ba51406b052ec7b9ec1cf464\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-99.ec2.internal"
Sep 14 04:46:42.632364 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.632289    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/53c18eb6ba51406b052ec7b9ec1cf464-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-99.ec2.internal\" (UID: \"53c18eb6ba51406b052ec7b9ec1cf464\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-99.ec2.internal"
Sep 14 04:46:42.632364 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.632296    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d5acb9fe33c38810a5100cf8c524f161-config\") pod \"kube-apiserver-proxy-ip-10-0-133-99.ec2.internal\" (UID: \"d5acb9fe33c38810a5100cf8c524f161\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-99.ec2.internal"
Sep 14 04:46:42.632364 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.632342    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53c18eb6ba51406b052ec7b9ec1cf464-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-99.ec2.internal\" (UID: \"53c18eb6ba51406b052ec7b9ec1cf464\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-99.ec2.internal"
Sep 14 04:46:42.632497 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.632373    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d5acb9fe33c38810a5100cf8c524f161-config\") pod \"kube-apiserver-proxy-ip-10-0-133-99.ec2.internal\" (UID: \"d5acb9fe33c38810a5100cf8c524f161\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-99.ec2.internal"
Sep 14 04:46:42.708708 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:42.708650    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-99.ec2.internal\" not found"
Sep 14 04:46:42.750135 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.750111    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-99.ec2.internal"
Sep 14 04:46:42.753165 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:42.753144    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-99.ec2.internal"
Sep 14 04:46:42.809561 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:42.809534    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-99.ec2.internal\" not found"
Sep 14 04:46:42.910662 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:42.910635    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-99.ec2.internal\" not found"
Sep 14 04:46:43.011396 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:43.011300    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-99.ec2.internal\" not found"
Sep 14 04:46:43.112299 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:43.112266    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-99.ec2.internal\" not found"
Sep 14 04:46:43.162715 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.162667    2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 14 04:46:43.163273 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.162876    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 14 04:46:43.182875 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.182841    2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 14 04:46:43.205060 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.205027    2565 apiserver.go:52] "Watching apiserver"
Sep 14 04:46:43.216447 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.216420    2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 14 04:46:43.216796 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.216760    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-whcnp","openshift-cluster-node-tuning-operator/tuned-ptbx9","openshift-dns/node-resolver-rqw8q","openshift-multus/multus-6g897","openshift-network-diagnostics/network-check-target-6sw5h","openshift-ovn-kubernetes/ovnkube-node-kndxq","kube-system/konnectivity-agent-zv7p8","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvr2","openshift-image-registry/node-ca-pfl42","openshift-multus/multus-additional-cni-plugins-z9dck","openshift-multus/network-metrics-daemon-2gwsl","openshift-network-operator/iptables-alerter-xkjrf"]
Sep 14 04:46:43.219502 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.219484    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-whcnp"
Sep 14 04:46:43.219575 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:43.219558    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-whcnp" podUID="ca784040-efa0-4d69-8c25-2b284f3bbe65"
Sep 14 04:46:43.221933 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.221907    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ptbx9"
Sep 14 04:46:43.224225 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.224207    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rqw8q"
Sep 14 04:46:43.226655 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.226636    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.227137 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.227119    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 14 04:46:43.227137 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.227134    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qdzj5\""
Sep 14 04:46:43.227302 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.227119    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 14 04:46:43.228099 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.228066    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 14 04:46:43.228422 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.228399    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 14 04:46:43.228532 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.228454    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7rqxz\""
Sep 14 04:46:43.230318 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.229626    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 14 04:46:43.230318 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.229747    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-xbbzg\""
Sep 14 04:46:43.230318 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.229862    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 14 04:46:43.230318 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.230162    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 14 04:46:43.230318 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.230229    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 14 04:46:43.230318 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.230298    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-99.ec2.internal"
Sep 14 04:46:43.230680 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.230654    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6sw5h"
Sep 14 04:46:43.230959 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:43.230875    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6sw5h" podUID="6b6a2ac6-52be-4747-b544-a61782e29691"
Sep 14 04:46:43.231472 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.231440    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 14 04:46:43.233878 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.233856    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.235590 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.235564    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-host-run-ovn-kubernetes\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.235666 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.235605    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.235666 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.235631    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0cfe37ed-e18d-4e25-84d6-cba59e2b2d81-hosts-file\") pod \"node-resolver-rqw8q\" (UID: \"0cfe37ed-e18d-4e25-84d6-cba59e2b2d81\") " pod="openshift-dns/node-resolver-rqw8q"
Sep 14 04:46:43.235666 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.235655    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-host-run-multus-certs\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.235837 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.235679    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-etc-kubernetes\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.235837 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.235702    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/67d2dcbd-5882-4609-a9cd-527e8067af4b-etc-modprobe-d\") pod \"tuned-ptbx9\" (UID: \"67d2dcbd-5882-4609-a9cd-527e8067af4b\") " pod="openshift-cluster-node-tuning-operator/tuned-ptbx9"
Sep 14 04:46:43.235837 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.235740    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67d2dcbd-5882-4609-a9cd-527e8067af4b-host\") pod \"tuned-ptbx9\" (UID: \"67d2dcbd-5882-4609-a9cd-527e8067af4b\") " pod="openshift-cluster-node-tuning-operator/tuned-ptbx9"
Sep 14 04:46:43.235837 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.235777    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lc5b\" (UniqueName: \"kubernetes.io/projected/0cfe37ed-e18d-4e25-84d6-cba59e2b2d81-kube-api-access-4lc5b\") pod \"node-resolver-rqw8q\" (UID: \"0cfe37ed-e18d-4e25-84d6-cba59e2b2d81\") " pod="openshift-dns/node-resolver-rqw8q"
Sep 14 04:46:43.235837 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.235797    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8zlx\" (UniqueName: \"kubernetes.io/projected/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-kube-api-access-q8zlx\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.235837 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.235817    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67d2dcbd-5882-4609-a9cd-527e8067af4b-etc-kubernetes\") pod \"tuned-ptbx9\" (UID: \"67d2dcbd-5882-4609-a9cd-527e8067af4b\") " pod="openshift-cluster-node-tuning-operator/tuned-ptbx9"
Sep 14 04:46:43.235837 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.235832    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/67d2dcbd-5882-4609-a9cd-527e8067af4b-sys\") pod \"tuned-ptbx9\" (UID: \"67d2dcbd-5882-4609-a9cd-527e8067af4b\") " pod="openshift-cluster-node-tuning-operator/tuned-ptbx9"
Sep 14 04:46:43.236112 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.235848    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/67d2dcbd-5882-4609-a9cd-527e8067af4b-lib-modules\") pod \"tuned-ptbx9\" (UID: \"67d2dcbd-5882-4609-a9cd-527e8067af4b\") " pod="openshift-cluster-node-tuning-operator/tuned-ptbx9"
Sep 14 04:46:43.236112 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.235866    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-host-run-netns\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.236112 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.235917    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-etc-openvswitch\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.236112 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.235955    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbd7g\" (UniqueName: \"kubernetes.io/projected/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-kube-api-access-xbd7g\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.236112 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.235975    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ca784040-efa0-4d69-8c25-2b284f3bbe65-original-pull-secret\") pod \"global-pull-secret-syncer-whcnp\" (UID: \"ca784040-efa0-4d69-8c25-2b284f3bbe65\") " pod="kube-system/global-pull-secret-syncer-whcnp"
Sep 14 04:46:43.236112 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.236018    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-cnibin\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.236112 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.236059    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-run-openvswitch\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.236112 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.236087    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-node-log\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.236112 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.236112    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-log-socket\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.236658 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.236135    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/67d2dcbd-5882-4609-a9cd-527e8067af4b-etc-systemd\") pod \"tuned-ptbx9\" (UID: \"67d2dcbd-5882-4609-a9cd-527e8067af4b\") " pod="openshift-cluster-node-tuning-operator/tuned-ptbx9"
Sep 14 04:46:43.236658 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.236160    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cfe37ed-e18d-4e25-84d6-cba59e2b2d81-tmp-dir\") pod \"node-resolver-rqw8q\" (UID: \"0cfe37ed-e18d-4e25-84d6-cba59e2b2d81\") " pod="openshift-dns/node-resolver-rqw8q"
Sep 14 04:46:43.236658 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.236185    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-systemd-units\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.236658 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.236216    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-hostroot\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.236658 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.236249    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-multus-cni-dir\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.236658 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.236279    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-ovnkube-config\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.236658 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.236303    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/67d2dcbd-5882-4609-a9cd-527e8067af4b-etc-sysctl-conf\") pod \"tuned-ptbx9\" (UID: \"67d2dcbd-5882-4609-a9cd-527e8067af4b\") " pod="openshift-cluster-node-tuning-operator/tuned-ptbx9"
Sep 14 04:46:43.236658 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.236327    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg9mv\" (UniqueName: \"kubernetes.io/projected/67d2dcbd-5882-4609-a9cd-527e8067af4b-kube-api-access-jg9mv\") pod \"tuned-ptbx9\" (UID: \"67d2dcbd-5882-4609-a9cd-527e8067af4b\") " pod="openshift-cluster-node-tuning-operator/tuned-ptbx9"
Sep 14 04:46:43.236658 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.236369    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-host-cni-bin\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.236658 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.236414    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-ovnkube-script-lib\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.236658 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.236446    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/67d2dcbd-5882-4609-a9cd-527e8067af4b-etc-sysctl-d\") pod \"tuned-ptbx9\" (UID: \"67d2dcbd-5882-4609-a9cd-527e8067af4b\") " pod="openshift-cluster-node-tuning-operator/tuned-ptbx9"
Sep 14 04:46:43.236658 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.236471    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67d2dcbd-5882-4609-a9cd-527e8067af4b-tmp\") pod \"tuned-ptbx9\" (UID: \"67d2dcbd-5882-4609-a9cd-527e8067af4b\") " pod="openshift-cluster-node-tuning-operator/tuned-ptbx9"
Sep 14 04:46:43.236658 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.236499    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ca784040-efa0-4d69-8c25-2b284f3bbe65-dbus\") pod \"global-pull-secret-syncer-whcnp\" (UID: \"ca784040-efa0-4d69-8c25-2b284f3bbe65\") " pod="kube-system/global-pull-secret-syncer-whcnp"
Sep 14 04:46:43.236658 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.236522    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/67d2dcbd-5882-4609-a9cd-527e8067af4b-etc-sysconfig\") pod \"tuned-ptbx9\" (UID: \"67d2dcbd-5882-4609-a9cd-527e8067af4b\") " pod="openshift-cluster-node-tuning-operator/tuned-ptbx9"
Sep 14 04:46:43.236658 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.236545    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/67d2dcbd-5882-4609-a9cd-527e8067af4b-run\") pod \"tuned-ptbx9\" (UID: \"67d2dcbd-5882-4609-a9cd-527e8067af4b\") " pod="openshift-cluster-node-tuning-operator/tuned-ptbx9"
Sep 14 04:46:43.236658 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.236573    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/67d2dcbd-5882-4609-a9cd-527e8067af4b-etc-tuned\") pod \"tuned-ptbx9\" (UID: \"67d2dcbd-5882-4609-a9cd-527e8067af4b\") " pod="openshift-cluster-node-tuning-operator/tuned-ptbx9"
Sep 14 04:46:43.236658 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.236593    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zv7p8"
Sep 14 04:46:43.238008 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.236598    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ca784040-efa0-4d69-8c25-2b284f3bbe65-kubelet-config\") pod \"global-pull-secret-syncer-whcnp\" (UID: \"ca784040-efa0-4d69-8c25-2b284f3bbe65\") " pod="kube-system/global-pull-secret-syncer-whcnp"
Sep 14 04:46:43.238008 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.236625    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-multus-socket-dir-parent\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.238008 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.236651    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-multus-daemon-config\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.238008 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.236677    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-ovn-node-metrics-cert\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.238008 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.236708    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67d2dcbd-5882-4609-a9cd-527e8067af4b-var-lib-kubelet\") pod \"tuned-ptbx9\" (UID: \"67d2dcbd-5882-4609-a9cd-527e8067af4b\") " pod="openshift-cluster-node-tuning-operator/tuned-ptbx9"
Sep 14 04:46:43.238008 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.236733    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-cni-binary-copy\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.238008 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.236758    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-host-var-lib-cni-multus\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.238008 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.236826    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-host-var-lib-cni-bin\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.238008 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.236863    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-host-var-lib-kubelet\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.238008 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.236917    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-multus-conf-dir\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.238008 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.236956    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh4l8\" (UniqueName: \"kubernetes.io/projected/6b6a2ac6-52be-4747-b544-a61782e29691-kube-api-access-wh4l8\") pod \"network-check-target-6sw5h\" (UID: \"6b6a2ac6-52be-4747-b544-a61782e29691\") " pod="openshift-network-diagnostics/network-check-target-6sw5h"
Sep 14 04:46:43.238008 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.236981    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-host-kubelet\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.238008 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.237008    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-host-slash\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.238008 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.237019    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 14 04:46:43.238008 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.237057    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-host-cni-netd\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.238008 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.237093    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-host-run-k8s-cni-cncf-io\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.238008 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.237121    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-host-run-netns\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.238543 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.237205    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-run-systemd\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.238543 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.237249    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-var-lib-openvswitch\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.238543 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.237287    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-run-ovn\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.238543 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.237361    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-env-overrides\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.238543 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.237374    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 14 04:46:43.238543 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.237401    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-system-cni-dir\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.238543 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.237453    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 14 04:46:43.238543 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.237527    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 14 04:46:43.238543 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.237668    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 14 04:46:43.238543 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.237673    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 14 04:46:43.238543 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.237837    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-894jg\""
Sep 14 04:46:43.239098 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.239083    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvr2"
Sep 14 04:46:43.239692 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.239676    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 14 04:46:43.240268 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.240249    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2cmpc\""
Sep 14 04:46:43.240403 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.240388    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 14 04:46:43.241754 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.241723    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pfl42"
Sep 14 04:46:43.241959 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.241895    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 14 04:46:43.242480 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.242462    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 14 04:46:43.242570 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.242556    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 14 04:46:43.242653 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.242601    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7g4rv\""
Sep 14 04:46:43.243070 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.243029    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-13 04:41:42 +0000 UTC" deadline="2028-05-04 08:34:08.279941052 +0000 UTC"
Sep 14 04:46:43.243141 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.243070    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14355h47m25.036875517s"
Sep 14 04:46:43.244213 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.244196    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 14 04:46:43.244408 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.244391    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 14 04:46:43.244493 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.244396    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2scrm\""
Sep 14 04:46:43.244800 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.244785    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-z9dck"
Sep 14 04:46:43.245072 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.244864    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 14 04:46:43.247180 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.247162    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 14 04:46:43.247292 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.247238    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gltrn\""
Sep 14 04:46:43.247490 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.247476    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 14 04:46:43.247578 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.247549    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2gwsl"
Sep 14 04:46:43.247640 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:43.247614    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2gwsl" podUID="ee8a9555-8b90-4ad8-a399-22d414487421"
Sep 14 04:46:43.250298 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.250280    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xkjrf"
Sep 14 04:46:43.255121 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.255095    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xch56\""
Sep 14 04:46:43.255226 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.255165    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 14 04:46:43.255274 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.255232    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-99.ec2.internal"]
Sep 14 04:46:43.255328 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.255301    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 14 04:46:43.255406 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.255392    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 14 04:46:43.255441 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.255394    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 14 04:46:43.255485 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.255445    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-99.ec2.internal"
Sep 14 04:46:43.258705 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.258688    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 14 04:46:43.266309 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.266266    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-99.ec2.internal"]
Sep 14 04:46:43.267082 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.267064    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 14 04:46:43.281837 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.281811    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-q6pj4"
Sep 14 04:46:43.291188 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.290820    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-q6pj4"
Sep 14 04:46:43.331994 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.331966    2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 14 04:46:43.338088 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.338059    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-systemd-units\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.338205 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.338102    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-hostroot\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.338205 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.338134    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-multus-cni-dir\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.338205 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.338157    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-ovnkube-config\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.338205 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.338180    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/67d2dcbd-5882-4609-a9cd-527e8067af4b-etc-sysctl-conf\") pod \"tuned-ptbx9\" (UID: \"67d2dcbd-5882-4609-a9cd-527e8067af4b\") " pod="openshift-cluster-node-tuning-operator/tuned-ptbx9"
Sep 14 04:46:43.338205 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.338205    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg9mv\" (UniqueName: \"kubernetes.io/projected/67d2dcbd-5882-4609-a9cd-527e8067af4b-kube-api-access-jg9mv\") pod \"tuned-ptbx9\" (UID: \"67d2dcbd-5882-4609-a9cd-527e8067af4b\") " pod="openshift-cluster-node-tuning-operator/tuned-ptbx9"
Sep 14 04:46:43.338458 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.338230    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-host-cni-bin\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.338458 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.338252    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-ovnkube-script-lib\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.338458 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.338275    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/67d2dcbd-5882-4609-a9cd-527e8067af4b-etc-sysctl-d\") pod \"tuned-ptbx9\" (UID: \"67d2dcbd-5882-4609-a9cd-527e8067af4b\") " pod="openshift-cluster-node-tuning-operator/tuned-ptbx9"
Sep 14 04:46:43.338458 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.338298    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67d2dcbd-5882-4609-a9cd-527e8067af4b-tmp\") pod \"tuned-ptbx9\" (UID: \"67d2dcbd-5882-4609-a9cd-527e8067af4b\") " pod="openshift-cluster-node-tuning-operator/tuned-ptbx9"
Sep 14 04:46:43.338458 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.338365    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-systemd-units\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.338458 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.338421    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6ac6666-f468-4757-8748-38b7e4fa90a9-system-cni-dir\") pod \"multus-additional-cni-plugins-z9dck\" (UID: \"f6ac6666-f468-4757-8748-38b7e4fa90a9\") " pod="openshift-multus/multus-additional-cni-plugins-z9dck"
Sep 14 04:46:43.338809 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.338462    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-host-cni-bin\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.338809 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.338506    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-multus-cni-dir\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.338809 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.338557    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjpp8\" (UniqueName: \"kubernetes.io/projected/842b0bc4-80e3-4936-a363-328c75f66265-kube-api-access-sjpp8\") pod \"iptables-alerter-xkjrf\" (UID: \"842b0bc4-80e3-4936-a363-328c75f66265\") " pod="openshift-network-operator/iptables-alerter-xkjrf"
Sep 14 04:46:43.338809 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.338595    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/67d2dcbd-5882-4609-a9cd-527e8067af4b-etc-sysctl-d\") pod \"tuned-ptbx9\" (UID: \"67d2dcbd-5882-4609-a9cd-527e8067af4b\") " pod="openshift-cluster-node-tuning-operator/tuned-ptbx9"
Sep 14 04:46:43.338809 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.338596    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/67d2dcbd-5882-4609-a9cd-527e8067af4b-etc-sysctl-conf\") pod \"tuned-ptbx9\" (UID: \"67d2dcbd-5882-4609-a9cd-527e8067af4b\") " pod="openshift-cluster-node-tuning-operator/tuned-ptbx9"
Sep 14 04:46:43.338809 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.338629    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ca784040-efa0-4d69-8c25-2b284f3bbe65-dbus\") pod \"global-pull-secret-syncer-whcnp\" (UID: \"ca784040-efa0-4d69-8c25-2b284f3bbe65\") " pod="kube-system/global-pull-secret-syncer-whcnp"
Sep 14 04:46:43.338809 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.338658    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/67d2dcbd-5882-4609-a9cd-527e8067af4b-etc-sysconfig\") pod \"tuned-ptbx9\" (UID: \"67d2dcbd-5882-4609-a9cd-527e8067af4b\") " pod="openshift-cluster-node-tuning-operator/tuned-ptbx9"
Sep 14 04:46:43.338809 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.338685    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/67d2dcbd-5882-4609-a9cd-527e8067af4b-run\") pod \"tuned-ptbx9\" (UID: \"67d2dcbd-5882-4609-a9cd-527e8067af4b\") " pod="openshift-cluster-node-tuning-operator/tuned-ptbx9"
Sep 14 04:46:43.338809 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.338710    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/67d2dcbd-5882-4609-a9cd-527e8067af4b-etc-tuned\") pod \"tuned-ptbx9\" (UID: \"67d2dcbd-5882-4609-a9cd-527e8067af4b\") " pod="openshift-cluster-node-tuning-operator/tuned-ptbx9"
Sep 14 04:46:43.338809 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.338735    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ada14485-29e4-41d5-abe1-fea5303f42bf-registration-dir\") pod \"aws-ebs-csi-driver-node-mvvr2\" (UID: \"ada14485-29e4-41d5-abe1-fea5303f42bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvr2"
Sep 14 04:46:43.338809 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.338748    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-hostroot\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.338809 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.338768    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/67d2dcbd-5882-4609-a9cd-527e8067af4b-etc-sysconfig\") pod \"tuned-ptbx9\" (UID: \"67d2dcbd-5882-4609-a9cd-527e8067af4b\") " pod="openshift-cluster-node-tuning-operator/tuned-ptbx9"
Sep 14 04:46:43.338809 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.338761    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f7e9f24-ddd1-4c3b-acdf-371ea4c648e8-host\") pod \"node-ca-pfl42\" (UID: \"7f7e9f24-ddd1-4c3b-acdf-371ea4c648e8\") " pod="openshift-image-registry/node-ca-pfl42"
Sep 14 04:46:43.338809 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.338785    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ca784040-efa0-4d69-8c25-2b284f3bbe65-dbus\") pod \"global-pull-secret-syncer-whcnp\" (UID: \"ca784040-efa0-4d69-8c25-2b284f3bbe65\") " pod="kube-system/global-pull-secret-syncer-whcnp"
Sep 14 04:46:43.338809 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.338706    2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 14 04:46:43.339506 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.338866    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/67d2dcbd-5882-4609-a9cd-527e8067af4b-run\") pod \"tuned-ptbx9\" (UID: \"67d2dcbd-5882-4609-a9cd-527e8067af4b\") " pod="openshift-cluster-node-tuning-operator/tuned-ptbx9"
Sep 14 04:46:43.339506 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.338876    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ca784040-efa0-4d69-8c25-2b284f3bbe65-kubelet-config\") pod \"global-pull-secret-syncer-whcnp\" (UID: \"ca784040-efa0-4d69-8c25-2b284f3bbe65\") " pod="kube-system/global-pull-secret-syncer-whcnp"
Sep 14 04:46:43.339506 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.338915    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ada14485-29e4-41d5-abe1-fea5303f42bf-socket-dir\") pod \"aws-ebs-csi-driver-node-mvvr2\" (UID: \"ada14485-29e4-41d5-abe1-fea5303f42bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvr2"
Sep 14 04:46:43.339506 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.338952    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee8a9555-8b90-4ad8-a399-22d414487421-metrics-certs\") pod \"network-metrics-daemon-2gwsl\" (UID: \"ee8a9555-8b90-4ad8-a399-22d414487421\") " pod="openshift-multus/network-metrics-daemon-2gwsl"
Sep 14 04:46:43.339506 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.338973    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ca784040-efa0-4d69-8c25-2b284f3bbe65-kubelet-config\") pod \"global-pull-secret-syncer-whcnp\" (UID: \"ca784040-efa0-4d69-8c25-2b284f3bbe65\") " pod="kube-system/global-pull-secret-syncer-whcnp"
Sep 14 04:46:43.339506 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.338978    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-ovnkube-config\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.339506 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.338992    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-multus-socket-dir-parent\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.339506 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.339023    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-multus-daemon-config\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.339506 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.339046    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-ovn-node-metrics-cert\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.339506 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.339070    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67d2dcbd-5882-4609-a9cd-527e8067af4b-var-lib-kubelet\") pod \"tuned-ptbx9\" (UID: \"67d2dcbd-5882-4609-a9cd-527e8067af4b\") " pod="openshift-cluster-node-tuning-operator/tuned-ptbx9"
Sep 14 04:46:43.339506 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.339083    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-multus-socket-dir-parent\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.339506 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.339096    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqhzv\" (UniqueName: \"kubernetes.io/projected/ada14485-29e4-41d5-abe1-fea5303f42bf-kube-api-access-mqhzv\") pod \"aws-ebs-csi-driver-node-mvvr2\" (UID: \"ada14485-29e4-41d5-abe1-fea5303f42bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvr2"
Sep 14 04:46:43.339506 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.339121    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-ovnkube-script-lib\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.339506 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.339145    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-cni-binary-copy\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.339506 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.339144    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67d2dcbd-5882-4609-a9cd-527e8067af4b-var-lib-kubelet\") pod \"tuned-ptbx9\" (UID: \"67d2dcbd-5882-4609-a9cd-527e8067af4b\") " pod="openshift-cluster-node-tuning-operator/tuned-ptbx9"
Sep 14 04:46:43.339506 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.339197    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-host-var-lib-cni-multus\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.339506 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.339261    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-host-var-lib-cni-multus\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.340263 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.339294    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ada14485-29e4-41d5-abe1-fea5303f42bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-mvvr2\" (UID: \"ada14485-29e4-41d5-abe1-fea5303f42bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvr2"
Sep 14 04:46:43.340263 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.339335    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-host-var-lib-cni-bin\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.340263 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.339376    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-host-var-lib-cni-bin\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.340263 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.339425    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-host-var-lib-kubelet\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.340263 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.339465    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-host-var-lib-kubelet\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.340263 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.339494    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-multus-conf-dir\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.340263 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.339535    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-multus-conf-dir\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.340263 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.339568    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ada14485-29e4-41d5-abe1-fea5303f42bf-device-dir\") pod \"aws-ebs-csi-driver-node-mvvr2\" (UID: \"ada14485-29e4-41d5-abe1-fea5303f42bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvr2"
Sep 14 04:46:43.340263 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.339599    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-multus-daemon-config\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.340263 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.339602    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9f1e07bf-2d62-4d64-a473-150a0830ba42-konnectivity-ca\") pod \"konnectivity-agent-zv7p8\" (UID: \"9f1e07bf-2d62-4d64-a473-150a0830ba42\") " pod="kube-system/konnectivity-agent-zv7p8"
Sep 14 04:46:43.340263 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.339615    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-cni-binary-copy\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.340263 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.339643    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh4l8\" (UniqueName: \"kubernetes.io/projected/6b6a2ac6-52be-4747-b544-a61782e29691-kube-api-access-wh4l8\") pod \"network-check-target-6sw5h\" (UID: \"6b6a2ac6-52be-4747-b544-a61782e29691\") " pod="openshift-network-diagnostics/network-check-target-6sw5h"
Sep 14 04:46:43.340263 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.339665    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-host-kubelet\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.340263 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.339680    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-host-slash\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.340263 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.339695    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-host-cni-netd\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.340263 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.339710    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ada14485-29e4-41d5-abe1-fea5303f42bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mvvr2\" (UID: \"ada14485-29e4-41d5-abe1-fea5303f42bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvr2"
Sep 14 04:46:43.340263 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.339736    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7f7e9f24-ddd1-4c3b-acdf-371ea4c648e8-serviceca\") pod \"node-ca-pfl42\" (UID: \"7f7e9f24-ddd1-4c3b-acdf-371ea4c648e8\") " pod="openshift-image-registry/node-ca-pfl42"
Sep 14 04:46:43.341205 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.339762    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-host-run-k8s-cni-cncf-io\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.341205 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.339783    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-host-slash\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.341205 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.339790    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-host-run-netns\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.341205 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.339830    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-run-systemd\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.341205 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.339855    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-var-lib-openvswitch\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.341205 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.339878    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-run-ovn\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.341205 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.339909    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-env-overrides\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.341205 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.339933    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/842b0bc4-80e3-4936-a363-328c75f66265-iptables-alerter-script\") pod \"iptables-alerter-xkjrf\" (UID: \"842b0bc4-80e3-4936-a363-328c75f66265\") " pod="openshift-network-operator/iptables-alerter-xkjrf"
Sep 14 04:46:43.341205 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.339959    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl4pd\" (UniqueName: \"kubernetes.io/projected/7f7e9f24-ddd1-4c3b-acdf-371ea4c648e8-kube-api-access-fl4pd\") pod \"node-ca-pfl42\" (UID: \"7f7e9f24-ddd1-4c3b-acdf-371ea4c648e8\") " pod="openshift-image-registry/node-ca-pfl42"
Sep 14 04:46:43.341205 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.339986    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-system-cni-dir\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.341205 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.339996    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-host-run-netns\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.341205 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.340030    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-host-run-ovn-kubernetes\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.341205 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.340051    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-host-cni-netd\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.341205 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.340059    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.341205 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.340097    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-host-run-k8s-cni-cncf-io\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.341205 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.340107    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0cfe37ed-e18d-4e25-84d6-cba59e2b2d81-hosts-file\") pod \"node-resolver-rqw8q\" (UID: \"0cfe37ed-e18d-4e25-84d6-cba59e2b2d81\") " pod="openshift-dns/node-resolver-rqw8q"
Sep 14 04:46:43.341205 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.340136    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f6ac6666-f468-4757-8748-38b7e4fa90a9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z9dck\" (UID: \"f6ac6666-f468-4757-8748-38b7e4fa90a9\") " pod="openshift-multus/multus-additional-cni-plugins-z9dck"
Sep 14 04:46:43.342105 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.340143    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-var-lib-openvswitch\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.342105 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.340161    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9f1e07bf-2d62-4d64-a473-150a0830ba42-agent-certs\") pod \"konnectivity-agent-zv7p8\" (UID: \"9f1e07bf-2d62-4d64-a473-150a0830ba42\") " pod="kube-system/konnectivity-agent-zv7p8"
Sep 14 04:46:43.342105 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.340187    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-run-systemd\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.342105 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.340208    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-host-run-multus-certs\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.342105 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.340233    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-host-run-ovn-kubernetes\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.342105 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.340238    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-etc-kubernetes\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.342105 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.340276    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/67d2dcbd-5882-4609-a9cd-527e8067af4b-etc-modprobe-d\") pod \"tuned-ptbx9\" (UID: \"67d2dcbd-5882-4609-a9cd-527e8067af4b\") " pod="openshift-cluster-node-tuning-operator/tuned-ptbx9"
Sep 14 04:46:43.342105 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.340293    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-etc-kubernetes\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.342105 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.340303    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67d2dcbd-5882-4609-a9cd-527e8067af4b-host\") pod \"tuned-ptbx9\" (UID: \"67d2dcbd-5882-4609-a9cd-527e8067af4b\") " pod="openshift-cluster-node-tuning-operator/tuned-ptbx9"
Sep 14 04:46:43.342105 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.340329    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lc5b\" (UniqueName: \"kubernetes.io/projected/0cfe37ed-e18d-4e25-84d6-cba59e2b2d81-kube-api-access-4lc5b\") pod \"node-resolver-rqw8q\" (UID: \"0cfe37ed-e18d-4e25-84d6-cba59e2b2d81\") " pod="openshift-dns/node-resolver-rqw8q"
Sep 14 04:46:43.342105 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.340348    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.342105 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.340358    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f6ac6666-f468-4757-8748-38b7e4fa90a9-cnibin\") pod \"multus-additional-cni-plugins-z9dck\" (UID: \"f6ac6666-f468-4757-8748-38b7e4fa90a9\") " pod="openshift-multus/multus-additional-cni-plugins-z9dck"
Sep 14 04:46:43.342105 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.340376    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-env-overrides\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.342105 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.340386    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f6ac6666-f468-4757-8748-38b7e4fa90a9-cni-binary-copy\") pod \"multus-additional-cni-plugins-z9dck\" (UID: \"f6ac6666-f468-4757-8748-38b7e4fa90a9\") " pod="openshift-multus/multus-additional-cni-plugins-z9dck"
Sep 14 04:46:43.342105 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.340400    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0cfe37ed-e18d-4e25-84d6-cba59e2b2d81-hosts-file\") pod \"node-resolver-rqw8q\" (UID: \"0cfe37ed-e18d-4e25-84d6-cba59e2b2d81\") " pod="openshift-dns/node-resolver-rqw8q"
Sep 14 04:46:43.342105 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.340426    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8zlx\" (UniqueName: \"kubernetes.io/projected/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-kube-api-access-q8zlx\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.342105 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.340446    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-run-ovn\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.342926 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.340465    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67d2dcbd-5882-4609-a9cd-527e8067af4b-etc-kubernetes\") pod \"tuned-ptbx9\" (UID: \"67d2dcbd-5882-4609-a9cd-527e8067af4b\") " pod="openshift-cluster-node-tuning-operator/tuned-ptbx9"
Sep 14 04:46:43.342926 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.340485    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-host-kubelet\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.342926 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.340535    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67d2dcbd-5882-4609-a9cd-527e8067af4b-host\") pod \"tuned-ptbx9\" (UID: \"67d2dcbd-5882-4609-a9cd-527e8067af4b\") " pod="openshift-cluster-node-tuning-operator/tuned-ptbx9"
Sep 14 04:46:43.342926 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.340568    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f6ac6666-f468-4757-8748-38b7e4fa90a9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z9dck\" (UID: \"f6ac6666-f468-4757-8748-38b7e4fa90a9\") " pod="openshift-multus/multus-additional-cni-plugins-z9dck"
Sep 14 04:46:43.342926 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.340611    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67d2dcbd-5882-4609-a9cd-527e8067af4b-etc-kubernetes\") pod \"tuned-ptbx9\" (UID: \"67d2dcbd-5882-4609-a9cd-527e8067af4b\") " pod="openshift-cluster-node-tuning-operator/tuned-ptbx9"
Sep 14 04:46:43.342926 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.340599    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f6ac6666-f468-4757-8748-38b7e4fa90a9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z9dck\" (UID: \"f6ac6666-f468-4757-8748-38b7e4fa90a9\") " pod="openshift-multus/multus-additional-cni-plugins-z9dck"
Sep 14 04:46:43.342926 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.340648    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-host-run-multus-certs\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.342926 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.340651    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4k7j\" (UniqueName: \"kubernetes.io/projected/f6ac6666-f468-4757-8748-38b7e4fa90a9-kube-api-access-b4k7j\") pod \"multus-additional-cni-plugins-z9dck\" (UID: \"f6ac6666-f468-4757-8748-38b7e4fa90a9\") " pod="openshift-multus/multus-additional-cni-plugins-z9dck"
Sep 14 04:46:43.342926 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.340569    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-system-cni-dir\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.342926 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.340688    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqrt5\" (UniqueName: \"kubernetes.io/projected/ee8a9555-8b90-4ad8-a399-22d414487421-kube-api-access-pqrt5\") pod \"network-metrics-daemon-2gwsl\" (UID: \"ee8a9555-8b90-4ad8-a399-22d414487421\") " pod="openshift-multus/network-metrics-daemon-2gwsl"
Sep 14 04:46:43.342926 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.340696    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/67d2dcbd-5882-4609-a9cd-527e8067af4b-etc-modprobe-d\") pod \"tuned-ptbx9\" (UID: \"67d2dcbd-5882-4609-a9cd-527e8067af4b\") " pod="openshift-cluster-node-tuning-operator/tuned-ptbx9"
Sep 14 04:46:43.342926 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.340716    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/842b0bc4-80e3-4936-a363-328c75f66265-host-slash\") pod \"iptables-alerter-xkjrf\" (UID: \"842b0bc4-80e3-4936-a363-328c75f66265\") " pod="openshift-network-operator/iptables-alerter-xkjrf"
Sep 14 04:46:43.342926 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.340744    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/67d2dcbd-5882-4609-a9cd-527e8067af4b-sys\") pod \"tuned-ptbx9\" (UID: \"67d2dcbd-5882-4609-a9cd-527e8067af4b\") " pod="openshift-cluster-node-tuning-operator/tuned-ptbx9"
Sep 14 04:46:43.342926 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.340782    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/67d2dcbd-5882-4609-a9cd-527e8067af4b-lib-modules\") pod \"tuned-ptbx9\" (UID: \"67d2dcbd-5882-4609-a9cd-527e8067af4b\") " pod="openshift-cluster-node-tuning-operator/tuned-ptbx9"
Sep 14 04:46:43.342926 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.340881    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/67d2dcbd-5882-4609-a9cd-527e8067af4b-sys\") pod \"tuned-ptbx9\" (UID: \"67d2dcbd-5882-4609-a9cd-527e8067af4b\") " pod="openshift-cluster-node-tuning-operator/tuned-ptbx9"
Sep 14 04:46:43.342926 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.340883    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-host-run-netns\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.342926 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.340933    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-etc-openvswitch\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.343470 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.340958    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/67d2dcbd-5882-4609-a9cd-527e8067af4b-lib-modules\") pod \"tuned-ptbx9\" (UID: \"67d2dcbd-5882-4609-a9cd-527e8067af4b\") " pod="openshift-cluster-node-tuning-operator/tuned-ptbx9"
Sep 14 04:46:43.343470 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.340958    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-host-run-netns\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.343470 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.341005    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbd7g\" (UniqueName: \"kubernetes.io/projected/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-kube-api-access-xbd7g\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.343470 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.341010    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-etc-openvswitch\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.343470 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.341055    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ca784040-efa0-4d69-8c25-2b284f3bbe65-original-pull-secret\") pod \"global-pull-secret-syncer-whcnp\" (UID: \"ca784040-efa0-4d69-8c25-2b284f3bbe65\") " pod="kube-system/global-pull-secret-syncer-whcnp"
Sep 14 04:46:43.343470 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.341179    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ada14485-29e4-41d5-abe1-fea5303f42bf-sys-fs\") pod \"aws-ebs-csi-driver-node-mvvr2\" (UID: \"ada14485-29e4-41d5-abe1-fea5303f42bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvr2"
Sep 14 04:46:43.343470 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.341227    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-cnibin\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.343470 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.341315    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-cnibin\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.343470 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:43.341321    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 04:46:43.343470 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:43.341409    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca784040-efa0-4d69-8c25-2b284f3bbe65-original-pull-secret podName:ca784040-efa0-4d69-8c25-2b284f3bbe65 nodeName:}" failed. No retries permitted until 2026-09-14 04:46:43.841385967 +0000 UTC m=+2.017157528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ca784040-efa0-4d69-8c25-2b284f3bbe65-original-pull-secret") pod "global-pull-secret-syncer-whcnp" (UID: "ca784040-efa0-4d69-8c25-2b284f3bbe65") : object "kube-system"/"original-pull-secret" not registered
Sep 14 04:46:43.343470 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.341441    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-run-openvswitch\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.343470 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.341471    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-node-log\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.343470 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.341494    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-log-socket\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.343470 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.341517    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/67d2dcbd-5882-4609-a9cd-527e8067af4b-etc-systemd\") pod \"tuned-ptbx9\" (UID: \"67d2dcbd-5882-4609-a9cd-527e8067af4b\") " pod="openshift-cluster-node-tuning-operator/tuned-ptbx9"
Sep 14 04:46:43.343470 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.341521    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-run-openvswitch\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.343470 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.341542    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cfe37ed-e18d-4e25-84d6-cba59e2b2d81-tmp-dir\") pod \"node-resolver-rqw8q\" (UID: \"0cfe37ed-e18d-4e25-84d6-cba59e2b2d81\") " pod="openshift-dns/node-resolver-rqw8q"
Sep 14 04:46:43.343470 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.341550    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-node-log\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.344020 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.341590    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/67d2dcbd-5882-4609-a9cd-527e8067af4b-etc-systemd\") pod \"tuned-ptbx9\" (UID: \"67d2dcbd-5882-4609-a9cd-527e8067af4b\") " pod="openshift-cluster-node-tuning-operator/tuned-ptbx9"
Sep 14 04:46:43.344020 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.341591    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-log-socket\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.344020 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.341947    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cfe37ed-e18d-4e25-84d6-cba59e2b2d81-tmp-dir\") pod \"node-resolver-rqw8q\" (UID: \"0cfe37ed-e18d-4e25-84d6-cba59e2b2d81\") " pod="openshift-dns/node-resolver-rqw8q"
Sep 14 04:46:43.344020 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.342704    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/67d2dcbd-5882-4609-a9cd-527e8067af4b-etc-tuned\") pod \"tuned-ptbx9\" (UID: \"67d2dcbd-5882-4609-a9cd-527e8067af4b\") " pod="openshift-cluster-node-tuning-operator/tuned-ptbx9"
Sep 14 04:46:43.344020 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.342915    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67d2dcbd-5882-4609-a9cd-527e8067af4b-tmp\") pod \"tuned-ptbx9\" (UID: \"67d2dcbd-5882-4609-a9cd-527e8067af4b\") " pod="openshift-cluster-node-tuning-operator/tuned-ptbx9"
Sep 14 04:46:43.344020 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.343034    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-ovn-node-metrics-cert\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.347642 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:43.347620    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 04:46:43.347642 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:43.347642    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 04:46:43.347764 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:43.347654    2565 projected.go:196] Error preparing data for projected volume kube-api-access-wh4l8 for pod openshift-network-diagnostics/network-check-target-6sw5h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 04:46:43.347764 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:43.347732    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b6a2ac6-52be-4747-b544-a61782e29691-kube-api-access-wh4l8 podName:6b6a2ac6-52be-4747-b544-a61782e29691 nodeName:}" failed. No retries permitted until 2026-09-14 04:46:43.84771877 +0000 UTC m=+2.023490314 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wh4l8" (UniqueName: "kubernetes.io/projected/6b6a2ac6-52be-4747-b544-a61782e29691-kube-api-access-wh4l8") pod "network-check-target-6sw5h" (UID: "6b6a2ac6-52be-4747-b544-a61782e29691") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 04:46:43.347905 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.347887    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg9mv\" (UniqueName: \"kubernetes.io/projected/67d2dcbd-5882-4609-a9cd-527e8067af4b-kube-api-access-jg9mv\") pod \"tuned-ptbx9\" (UID: \"67d2dcbd-5882-4609-a9cd-527e8067af4b\") " pod="openshift-cluster-node-tuning-operator/tuned-ptbx9"
Sep 14 04:46:43.350928 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.350900    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8zlx\" (UniqueName: \"kubernetes.io/projected/2e7a5d9f-3918-4bba-995c-1dc18f9611ac-kube-api-access-q8zlx\") pod \"multus-6g897\" (UID: \"2e7a5d9f-3918-4bba-995c-1dc18f9611ac\") " pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.351640 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.351619    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbd7g\" (UniqueName: \"kubernetes.io/projected/e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae-kube-api-access-xbd7g\") pod \"ovnkube-node-kndxq\" (UID: \"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.352278 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.352261    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lc5b\" (UniqueName: \"kubernetes.io/projected/0cfe37ed-e18d-4e25-84d6-cba59e2b2d81-kube-api-access-4lc5b\") pod \"node-resolver-rqw8q\" (UID: \"0cfe37ed-e18d-4e25-84d6-cba59e2b2d81\") " pod="openshift-dns/node-resolver-rqw8q"
Sep 14 04:46:43.438580 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:43.438549    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53c18eb6ba51406b052ec7b9ec1cf464.slice/crio-3f5f0dac0e8abe02dc8a9ef7677a0f6cbd13827e833fdecc9829a3e46d0115b1 WatchSource:0}: Error finding container 3f5f0dac0e8abe02dc8a9ef7677a0f6cbd13827e833fdecc9829a3e46d0115b1: Status 404 returned error can't find the container with id 3f5f0dac0e8abe02dc8a9ef7677a0f6cbd13827e833fdecc9829a3e46d0115b1
Sep 14 04:46:43.442060 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.442030    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4k7j\" (UniqueName: \"kubernetes.io/projected/f6ac6666-f468-4757-8748-38b7e4fa90a9-kube-api-access-b4k7j\") pod \"multus-additional-cni-plugins-z9dck\" (UID: \"f6ac6666-f468-4757-8748-38b7e4fa90a9\") " pod="openshift-multus/multus-additional-cni-plugins-z9dck"
Sep 14 04:46:43.442177 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.442070    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqrt5\" (UniqueName: \"kubernetes.io/projected/ee8a9555-8b90-4ad8-a399-22d414487421-kube-api-access-pqrt5\") pod \"network-metrics-daemon-2gwsl\" (UID: \"ee8a9555-8b90-4ad8-a399-22d414487421\") " pod="openshift-multus/network-metrics-daemon-2gwsl"
Sep 14 04:46:43.442177 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.442103    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/842b0bc4-80e3-4936-a363-328c75f66265-host-slash\") pod \"iptables-alerter-xkjrf\" (UID: \"842b0bc4-80e3-4936-a363-328c75f66265\") " pod="openshift-network-operator/iptables-alerter-xkjrf"
Sep 14 04:46:43.442303 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.442274    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ada14485-29e4-41d5-abe1-fea5303f42bf-sys-fs\") pod \"aws-ebs-csi-driver-node-mvvr2\" (UID: \"ada14485-29e4-41d5-abe1-fea5303f42bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvr2"
Sep 14 04:46:43.442362 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.442330    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6ac6666-f468-4757-8748-38b7e4fa90a9-system-cni-dir\") pod \"multus-additional-cni-plugins-z9dck\" (UID: \"f6ac6666-f468-4757-8748-38b7e4fa90a9\") " pod="openshift-multus/multus-additional-cni-plugins-z9dck"
Sep 14 04:46:43.442362 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.442349    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/842b0bc4-80e3-4936-a363-328c75f66265-host-slash\") pod \"iptables-alerter-xkjrf\" (UID: \"842b0bc4-80e3-4936-a363-328c75f66265\") " pod="openshift-network-operator/iptables-alerter-xkjrf"
Sep 14 04:46:43.442465 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.442360    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjpp8\" (UniqueName: \"kubernetes.io/projected/842b0bc4-80e3-4936-a363-328c75f66265-kube-api-access-sjpp8\") pod \"iptables-alerter-xkjrf\" (UID: \"842b0bc4-80e3-4936-a363-328c75f66265\") " pod="openshift-network-operator/iptables-alerter-xkjrf"
Sep 14 04:46:43.442465 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.442399    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ada14485-29e4-41d5-abe1-fea5303f42bf-registration-dir\") pod \"aws-ebs-csi-driver-node-mvvr2\" (UID: \"ada14485-29e4-41d5-abe1-fea5303f42bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvr2"
Sep 14 04:46:43.442465 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.442423    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ada14485-29e4-41d5-abe1-fea5303f42bf-sys-fs\") pod \"aws-ebs-csi-driver-node-mvvr2\" (UID: \"ada14485-29e4-41d5-abe1-fea5303f42bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvr2"
Sep 14 04:46:43.442913 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.442760    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6ac6666-f468-4757-8748-38b7e4fa90a9-system-cni-dir\") pod \"multus-additional-cni-plugins-z9dck\" (UID: \"f6ac6666-f468-4757-8748-38b7e4fa90a9\") " pod="openshift-multus/multus-additional-cni-plugins-z9dck"
Sep 14 04:46:43.443014 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.442920    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f7e9f24-ddd1-4c3b-acdf-371ea4c648e8-host\") pod \"node-ca-pfl42\" (UID: \"7f7e9f24-ddd1-4c3b-acdf-371ea4c648e8\") " pod="openshift-image-registry/node-ca-pfl42"
Sep 14 04:46:43.443014 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.442967    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ada14485-29e4-41d5-abe1-fea5303f42bf-registration-dir\") pod \"aws-ebs-csi-driver-node-mvvr2\" (UID: \"ada14485-29e4-41d5-abe1-fea5303f42bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvr2"
Sep 14 04:46:43.443014 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.442988    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ada14485-29e4-41d5-abe1-fea5303f42bf-socket-dir\") pod \"aws-ebs-csi-driver-node-mvvr2\" (UID: \"ada14485-29e4-41d5-abe1-fea5303f42bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvr2"
Sep 14 04:46:43.443186 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.443044    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f7e9f24-ddd1-4c3b-acdf-371ea4c648e8-host\") pod \"node-ca-pfl42\" (UID: \"7f7e9f24-ddd1-4c3b-acdf-371ea4c648e8\") " pod="openshift-image-registry/node-ca-pfl42"
Sep 14 04:46:43.443186 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.443088    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee8a9555-8b90-4ad8-a399-22d414487421-metrics-certs\") pod \"network-metrics-daemon-2gwsl\" (UID: \"ee8a9555-8b90-4ad8-a399-22d414487421\") " pod="openshift-multus/network-metrics-daemon-2gwsl"
Sep 14 04:46:43.443186 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.443134    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqhzv\" (UniqueName: \"kubernetes.io/projected/ada14485-29e4-41d5-abe1-fea5303f42bf-kube-api-access-mqhzv\") pod \"aws-ebs-csi-driver-node-mvvr2\" (UID: \"ada14485-29e4-41d5-abe1-fea5303f42bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvr2"
Sep 14 04:46:43.443186 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.443170    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ada14485-29e4-41d5-abe1-fea5303f42bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-mvvr2\" (UID: \"ada14485-29e4-41d5-abe1-fea5303f42bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvr2"
Sep 14 04:46:43.443392 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.443202    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ada14485-29e4-41d5-abe1-fea5303f42bf-device-dir\") pod \"aws-ebs-csi-driver-node-mvvr2\" (UID: \"ada14485-29e4-41d5-abe1-fea5303f42bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvr2"
Sep 14 04:46:43.443392 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:43.443229    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 04:46:43.443392 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.443256    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9f1e07bf-2d62-4d64-a473-150a0830ba42-konnectivity-ca\") pod \"konnectivity-agent-zv7p8\" (UID: \"9f1e07bf-2d62-4d64-a473-150a0830ba42\") " pod="kube-system/konnectivity-agent-zv7p8"
Sep 14 04:46:43.443392 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.443257    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ada14485-29e4-41d5-abe1-fea5303f42bf-socket-dir\") pod \"aws-ebs-csi-driver-node-mvvr2\" (UID: \"ada14485-29e4-41d5-abe1-fea5303f42bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvr2"
Sep 14 04:46:43.443392 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:43.443301    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee8a9555-8b90-4ad8-a399-22d414487421-metrics-certs podName:ee8a9555-8b90-4ad8-a399-22d414487421 nodeName:}" failed. No retries permitted until 2026-09-14 04:46:43.943275018 +0000 UTC m=+2.119046573 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee8a9555-8b90-4ad8-a399-22d414487421-metrics-certs") pod "network-metrics-daemon-2gwsl" (UID: "ee8a9555-8b90-4ad8-a399-22d414487421") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 04:46:43.443392 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.443348    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ada14485-29e4-41d5-abe1-fea5303f42bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-mvvr2\" (UID: \"ada14485-29e4-41d5-abe1-fea5303f42bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvr2"
Sep 14 04:46:43.443392 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.443348    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ada14485-29e4-41d5-abe1-fea5303f42bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mvvr2\" (UID: \"ada14485-29e4-41d5-abe1-fea5303f42bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvr2"
Sep 14 04:46:43.443751 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.443398    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ada14485-29e4-41d5-abe1-fea5303f42bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mvvr2\" (UID: \"ada14485-29e4-41d5-abe1-fea5303f42bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvr2"
Sep 14 04:46:43.443806 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.443730    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ada14485-29e4-41d5-abe1-fea5303f42bf-device-dir\") pod \"aws-ebs-csi-driver-node-mvvr2\" (UID: \"ada14485-29e4-41d5-abe1-fea5303f42bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvr2"
Sep 14 04:46:43.443893 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.443408    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7f7e9f24-ddd1-4c3b-acdf-371ea4c648e8-serviceca\") pod \"node-ca-pfl42\" (UID: \"7f7e9f24-ddd1-4c3b-acdf-371ea4c648e8\") " pod="openshift-image-registry/node-ca-pfl42"
Sep 14 04:46:43.443893 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.443874    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7f7e9f24-ddd1-4c3b-acdf-371ea4c648e8-serviceca\") pod \"node-ca-pfl42\" (UID: \"7f7e9f24-ddd1-4c3b-acdf-371ea4c648e8\") " pod="openshift-image-registry/node-ca-pfl42"
Sep 14 04:46:43.444003 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.443906    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/842b0bc4-80e3-4936-a363-328c75f66265-iptables-alerter-script\") pod \"iptables-alerter-xkjrf\" (UID: \"842b0bc4-80e3-4936-a363-328c75f66265\") " pod="openshift-network-operator/iptables-alerter-xkjrf"
Sep 14 04:46:43.444003 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.443947    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl4pd\" (UniqueName: \"kubernetes.io/projected/7f7e9f24-ddd1-4c3b-acdf-371ea4c648e8-kube-api-access-fl4pd\") pod \"node-ca-pfl42\" (UID: \"7f7e9f24-ddd1-4c3b-acdf-371ea4c648e8\") " pod="openshift-image-registry/node-ca-pfl42"
Sep 14 04:46:43.444003 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.443984    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f6ac6666-f468-4757-8748-38b7e4fa90a9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z9dck\" (UID: \"f6ac6666-f468-4757-8748-38b7e4fa90a9\") " pod="openshift-multus/multus-additional-cni-plugins-z9dck"
Sep 14 04:46:43.444189 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.444023    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9f1e07bf-2d62-4d64-a473-150a0830ba42-agent-certs\") pod \"konnectivity-agent-zv7p8\" (UID: \"9f1e07bf-2d62-4d64-a473-150a0830ba42\") " pod="kube-system/konnectivity-agent-zv7p8"
Sep 14 04:46:43.444189 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.444094    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f6ac6666-f468-4757-8748-38b7e4fa90a9-cnibin\") pod \"multus-additional-cni-plugins-z9dck\" (UID: \"f6ac6666-f468-4757-8748-38b7e4fa90a9\") " pod="openshift-multus/multus-additional-cni-plugins-z9dck"
Sep 14 04:46:43.444189 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.444170    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f6ac6666-f468-4757-8748-38b7e4fa90a9-cni-binary-copy\") pod \"multus-additional-cni-plugins-z9dck\" (UID: \"f6ac6666-f468-4757-8748-38b7e4fa90a9\") " pod="openshift-multus/multus-additional-cni-plugins-z9dck"
Sep 14 04:46:43.444358 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.444237    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f6ac6666-f468-4757-8748-38b7e4fa90a9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z9dck\" (UID: \"f6ac6666-f468-4757-8748-38b7e4fa90a9\") " pod="openshift-multus/multus-additional-cni-plugins-z9dck"
Sep 14 04:46:43.444358 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.444268    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f6ac6666-f468-4757-8748-38b7e4fa90a9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z9dck\" (UID: \"f6ac6666-f468-4757-8748-38b7e4fa90a9\") " pod="openshift-multus/multus-additional-cni-plugins-z9dck"
Sep 14 04:46:43.444782 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.444753    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f6ac6666-f468-4757-8748-38b7e4fa90a9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z9dck\" (UID: \"f6ac6666-f468-4757-8748-38b7e4fa90a9\") " pod="openshift-multus/multus-additional-cni-plugins-z9dck"
Sep 14 04:46:43.444905 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.444850    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f6ac6666-f468-4757-8748-38b7e4fa90a9-cnibin\") pod \"multus-additional-cni-plugins-z9dck\" (UID: \"f6ac6666-f468-4757-8748-38b7e4fa90a9\") " pod="openshift-multus/multus-additional-cni-plugins-z9dck"
Sep 14 04:46:43.444971 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.444948    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/842b0bc4-80e3-4936-a363-328c75f66265-iptables-alerter-script\") pod \"iptables-alerter-xkjrf\" (UID: \"842b0bc4-80e3-4936-a363-328c75f66265\") " pod="openshift-network-operator/iptables-alerter-xkjrf"
Sep 14 04:46:43.445075 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.445049    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f6ac6666-f468-4757-8748-38b7e4fa90a9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z9dck\" (UID: \"f6ac6666-f468-4757-8748-38b7e4fa90a9\") " pod="openshift-multus/multus-additional-cni-plugins-z9dck"
Sep 14 04:46:43.445172 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.445148    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f6ac6666-f468-4757-8748-38b7e4fa90a9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z9dck\" (UID: \"f6ac6666-f468-4757-8748-38b7e4fa90a9\") " pod="openshift-multus/multus-additional-cni-plugins-z9dck"
Sep 14 04:46:43.447556 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.447534    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9f1e07bf-2d62-4d64-a473-150a0830ba42-konnectivity-ca\") pod \"konnectivity-agent-zv7p8\" (UID: \"9f1e07bf-2d62-4d64-a473-150a0830ba42\") " pod="kube-system/konnectivity-agent-zv7p8"
Sep 14 04:46:43.447745 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.447726    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9f1e07bf-2d62-4d64-a473-150a0830ba42-agent-certs\") pod \"konnectivity-agent-zv7p8\" (UID: \"9f1e07bf-2d62-4d64-a473-150a0830ba42\") " pod="kube-system/konnectivity-agent-zv7p8"
Sep 14 04:46:43.447868 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.447847    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f6ac6666-f468-4757-8748-38b7e4fa90a9-cni-binary-copy\") pod \"multus-additional-cni-plugins-z9dck\" (UID: \"f6ac6666-f468-4757-8748-38b7e4fa90a9\") " pod="openshift-multus/multus-additional-cni-plugins-z9dck"
Sep 14 04:46:43.449146 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.449129    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 04:46:43.452185 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:43.452162    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5acb9fe33c38810a5100cf8c524f161.slice/crio-88e67f2c2145b345fde9310425c7264881318969cc14c0818b2069764545a7d5 WatchSource:0}: Error finding container 88e67f2c2145b345fde9310425c7264881318969cc14c0818b2069764545a7d5: Status 404 returned error can't find the container with id 88e67f2c2145b345fde9310425c7264881318969cc14c0818b2069764545a7d5
Sep 14 04:46:43.452856 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.452826    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4k7j\" (UniqueName: \"kubernetes.io/projected/f6ac6666-f468-4757-8748-38b7e4fa90a9-kube-api-access-b4k7j\") pod \"multus-additional-cni-plugins-z9dck\" (UID: \"f6ac6666-f468-4757-8748-38b7e4fa90a9\") " pod="openshift-multus/multus-additional-cni-plugins-z9dck"
Sep 14 04:46:43.457049 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.457027    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl4pd\" (UniqueName: \"kubernetes.io/projected/7f7e9f24-ddd1-4c3b-acdf-371ea4c648e8-kube-api-access-fl4pd\") pod \"node-ca-pfl42\" (UID: \"7f7e9f24-ddd1-4c3b-acdf-371ea4c648e8\") " pod="openshift-image-registry/node-ca-pfl42"
Sep 14 04:46:43.457128 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.457093    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqhzv\" (UniqueName: \"kubernetes.io/projected/ada14485-29e4-41d5-abe1-fea5303f42bf-kube-api-access-mqhzv\") pod \"aws-ebs-csi-driver-node-mvvr2\" (UID: \"ada14485-29e4-41d5-abe1-fea5303f42bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvr2"
Sep 14 04:46:43.457128 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.457115    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqrt5\" (UniqueName: \"kubernetes.io/projected/ee8a9555-8b90-4ad8-a399-22d414487421-kube-api-access-pqrt5\") pod \"network-metrics-daemon-2gwsl\" (UID: \"ee8a9555-8b90-4ad8-a399-22d414487421\") " pod="openshift-multus/network-metrics-daemon-2gwsl"
Sep 14 04:46:43.457199 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.457139    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjpp8\" (UniqueName: \"kubernetes.io/projected/842b0bc4-80e3-4936-a363-328c75f66265-kube-api-access-sjpp8\") pod \"iptables-alerter-xkjrf\" (UID: \"842b0bc4-80e3-4936-a363-328c75f66265\") " pod="openshift-network-operator/iptables-alerter-xkjrf"
Sep 14 04:46:43.490895 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.490872    2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 14 04:46:43.549143 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.549086    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ptbx9"
Sep 14 04:46:43.559101 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:43.559079    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d2dcbd_5882_4609_a9cd_527e8067af4b.slice/crio-35906b9dbfb6909238b305f6c2d40b9a79d0843210e13e547a3e4f03a474b253 WatchSource:0}: Error finding container 35906b9dbfb6909238b305f6c2d40b9a79d0843210e13e547a3e4f03a474b253: Status 404 returned error can't find the container with id 35906b9dbfb6909238b305f6c2d40b9a79d0843210e13e547a3e4f03a474b253
Sep 14 04:46:43.567535 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.567516    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rqw8q"
Sep 14 04:46:43.575657 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:43.575635    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cfe37ed_e18d_4e25_84d6_cba59e2b2d81.slice/crio-78ac7ed52f4c1bc43598e46f683cc1ad93aa74cced2085c9523be3db80362245 WatchSource:0}: Error finding container 78ac7ed52f4c1bc43598e46f683cc1ad93aa74cced2085c9523be3db80362245: Status 404 returned error can't find the container with id 78ac7ed52f4c1bc43598e46f683cc1ad93aa74cced2085c9523be3db80362245
Sep 14 04:46:43.583867 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.583848    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6g897"
Sep 14 04:46:43.591866 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:43.591845    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e7a5d9f_3918_4bba_995c_1dc18f9611ac.slice/crio-9665e4563451d801e1d0b45f300e75c34f298af64b0b5c4765cc4915828ecb41 WatchSource:0}: Error finding container 9665e4563451d801e1d0b45f300e75c34f298af64b0b5c4765cc4915828ecb41: Status 404 returned error can't find the container with id 9665e4563451d801e1d0b45f300e75c34f298af64b0b5c4765cc4915828ecb41
Sep 14 04:46:43.596517 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.596499    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:46:43.604475 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:43.604459    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode46993d2_9dee_49c0_aa4f_57ecc2f3f9ae.slice/crio-4b6cb0be4f24c58469a6f9925afda45c5c850320ba6f476b4fce8041c05e22b9 WatchSource:0}: Error finding container 4b6cb0be4f24c58469a6f9925afda45c5c850320ba6f476b4fce8041c05e22b9: Status 404 returned error can't find the container with id 4b6cb0be4f24c58469a6f9925afda45c5c850320ba6f476b4fce8041c05e22b9
Sep 14 04:46:43.618998 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.618976    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zv7p8"
Sep 14 04:46:43.628375 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:43.628355    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f1e07bf_2d62_4d64_a473_150a0830ba42.slice/crio-482efd3d9b7884575d5c32a5443fc077c6bb82db46b2b28e23eff2be25672146 WatchSource:0}: Error finding container 482efd3d9b7884575d5c32a5443fc077c6bb82db46b2b28e23eff2be25672146: Status 404 returned error can't find the container with id 482efd3d9b7884575d5c32a5443fc077c6bb82db46b2b28e23eff2be25672146
Sep 14 04:46:43.631255 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.631236    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvr2"
Sep 14 04:46:43.639943 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:43.639918    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podada14485_29e4_41d5_abe1_fea5303f42bf.slice/crio-aeae4b8b0e97a4d7f47232fa3e84ba8e417344cc2e4fc5985d28a12b4e1d8a99 WatchSource:0}: Error finding container aeae4b8b0e97a4d7f47232fa3e84ba8e417344cc2e4fc5985d28a12b4e1d8a99: Status 404 returned error can't find the container with id aeae4b8b0e97a4d7f47232fa3e84ba8e417344cc2e4fc5985d28a12b4e1d8a99
Sep 14 04:46:43.644978 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.644960    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pfl42"
Sep 14 04:46:43.653025 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:43.653005    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f7e9f24_ddd1_4c3b_acdf_371ea4c648e8.slice/crio-e9259fb9b12e7fd3b97590d9c57a4d2be78aad149e7963769efa7959a7cf82f6 WatchSource:0}: Error finding container e9259fb9b12e7fd3b97590d9c57a4d2be78aad149e7963769efa7959a7cf82f6: Status 404 returned error can't find the container with id e9259fb9b12e7fd3b97590d9c57a4d2be78aad149e7963769efa7959a7cf82f6
Sep 14 04:46:43.672021 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.672004    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-z9dck"
Sep 14 04:46:43.678012 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.677996    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xkjrf"
Sep 14 04:46:43.680957 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:43.680867    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6ac6666_f468_4757_8748_38b7e4fa90a9.slice/crio-ad0fb8b252078255690015ec6dea60178ff560b586a427cc4f1c99ad412c4e3d WatchSource:0}: Error finding container ad0fb8b252078255690015ec6dea60178ff560b586a427cc4f1c99ad412c4e3d: Status 404 returned error can't find the container with id ad0fb8b252078255690015ec6dea60178ff560b586a427cc4f1c99ad412c4e3d
Sep 14 04:46:43.688533 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:46:43.688515    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod842b0bc4_80e3_4936_a363_328c75f66265.slice/crio-f8d22a277e07a29ed8ccb5b6cae37077d433e1c03fcdec398591e97723ec1ebf WatchSource:0}: Error finding container f8d22a277e07a29ed8ccb5b6cae37077d433e1c03fcdec398591e97723ec1ebf: Status 404 returned error can't find the container with id f8d22a277e07a29ed8ccb5b6cae37077d433e1c03fcdec398591e97723ec1ebf
Sep 14 04:46:43.712590 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.712569    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 14 04:46:43.846249 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.846171    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ca784040-efa0-4d69-8c25-2b284f3bbe65-original-pull-secret\") pod \"global-pull-secret-syncer-whcnp\" (UID: \"ca784040-efa0-4d69-8c25-2b284f3bbe65\") " pod="kube-system/global-pull-secret-syncer-whcnp"
Sep 14 04:46:43.846365 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:43.846299    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 04:46:43.846365 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:43.846346    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca784040-efa0-4d69-8c25-2b284f3bbe65-original-pull-secret podName:ca784040-efa0-4d69-8c25-2b284f3bbe65 nodeName:}" failed. No retries permitted until 2026-09-14 04:46:44.846333325 +0000 UTC m=+3.022104864 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ca784040-efa0-4d69-8c25-2b284f3bbe65-original-pull-secret") pod "global-pull-secret-syncer-whcnp" (UID: "ca784040-efa0-4d69-8c25-2b284f3bbe65") : object "kube-system"/"original-pull-secret" not registered
Sep 14 04:46:43.946832 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.946805    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee8a9555-8b90-4ad8-a399-22d414487421-metrics-certs\") pod \"network-metrics-daemon-2gwsl\" (UID: \"ee8a9555-8b90-4ad8-a399-22d414487421\") " pod="openshift-multus/network-metrics-daemon-2gwsl"
Sep 14 04:46:43.946977 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:43.946856    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh4l8\" (UniqueName: \"kubernetes.io/projected/6b6a2ac6-52be-4747-b544-a61782e29691-kube-api-access-wh4l8\") pod \"network-check-target-6sw5h\" (UID: \"6b6a2ac6-52be-4747-b544-a61782e29691\") " pod="openshift-network-diagnostics/network-check-target-6sw5h"
Sep 14 04:46:43.947055 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:43.946978    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 04:46:43.947055 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:43.947044    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 04:46:43.947168 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:43.947066    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 04:46:43.947168 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:43.947081    2565 projected.go:196] Error preparing data for projected volume kube-api-access-wh4l8 for pod openshift-network-diagnostics/network-check-target-6sw5h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 04:46:43.947168 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:43.947052    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee8a9555-8b90-4ad8-a399-22d414487421-metrics-certs podName:ee8a9555-8b90-4ad8-a399-22d414487421 nodeName:}" failed. No retries permitted until 2026-09-14 04:46:44.947028319 +0000 UTC m=+3.122799876 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee8a9555-8b90-4ad8-a399-22d414487421-metrics-certs") pod "network-metrics-daemon-2gwsl" (UID: "ee8a9555-8b90-4ad8-a399-22d414487421") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 04:46:43.947168 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:43.947139    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b6a2ac6-52be-4747-b544-a61782e29691-kube-api-access-wh4l8 podName:6b6a2ac6-52be-4747-b544-a61782e29691 nodeName:}" failed. No retries permitted until 2026-09-14 04:46:44.947122741 +0000 UTC m=+3.122894280 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wh4l8" (UniqueName: "kubernetes.io/projected/6b6a2ac6-52be-4747-b544-a61782e29691-kube-api-access-wh4l8") pod "network-check-target-6sw5h" (UID: "6b6a2ac6-52be-4747-b544-a61782e29691") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 04:46:44.294915 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:44.294670    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-13 04:41:43 +0000 UTC" deadline="2028-04-19 10:39:56.67731743 +0000 UTC"
Sep 14 04:46:44.294915 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:44.294713    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13997h53m12.382608914s"
Sep 14 04:46:44.331699 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:44.331667    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6sw5h"
Sep 14 04:46:44.331882 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:44.331798    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6sw5h" podUID="6b6a2ac6-52be-4747-b544-a61782e29691"
Sep 14 04:46:44.374280 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:44.374218    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rqw8q" event={"ID":"0cfe37ed-e18d-4e25-84d6-cba59e2b2d81","Type":"ContainerStarted","Data":"78ac7ed52f4c1bc43598e46f683cc1ad93aa74cced2085c9523be3db80362245"}
Sep 14 04:46:44.378218 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:44.378185    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-99.ec2.internal" event={"ID":"d5acb9fe33c38810a5100cf8c524f161","Type":"ContainerStarted","Data":"88e67f2c2145b345fde9310425c7264881318969cc14c0818b2069764545a7d5"}
Sep 14 04:46:44.388361 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:44.388327    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xkjrf" event={"ID":"842b0bc4-80e3-4936-a363-328c75f66265","Type":"ContainerStarted","Data":"f8d22a277e07a29ed8ccb5b6cae37077d433e1c03fcdec398591e97723ec1ebf"}
Sep 14 04:46:44.409051 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:44.406251    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9dck" event={"ID":"f6ac6666-f468-4757-8748-38b7e4fa90a9","Type":"ContainerStarted","Data":"ad0fb8b252078255690015ec6dea60178ff560b586a427cc4f1c99ad412c4e3d"}
Sep 14 04:46:44.411595 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:44.411540    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pfl42" event={"ID":"7f7e9f24-ddd1-4c3b-acdf-371ea4c648e8","Type":"ContainerStarted","Data":"e9259fb9b12e7fd3b97590d9c57a4d2be78aad149e7963769efa7959a7cf82f6"}
Sep 14 04:46:44.427187 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:44.427143    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kndxq" event={"ID":"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae","Type":"ContainerStarted","Data":"4b6cb0be4f24c58469a6f9925afda45c5c850320ba6f476b4fce8041c05e22b9"}
Sep 14 04:46:44.451007 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:44.450972    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ptbx9" event={"ID":"67d2dcbd-5882-4609-a9cd-527e8067af4b","Type":"ContainerStarted","Data":"35906b9dbfb6909238b305f6c2d40b9a79d0843210e13e547a3e4f03a474b253"}
Sep 14 04:46:44.451868 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:44.451849    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 14 04:46:44.467872 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:44.467826    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-99.ec2.internal" event={"ID":"53c18eb6ba51406b052ec7b9ec1cf464","Type":"ContainerStarted","Data":"3f5f0dac0e8abe02dc8a9ef7677a0f6cbd13827e833fdecc9829a3e46d0115b1"}
Sep 14 04:46:44.486776 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:44.486733    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvr2" event={"ID":"ada14485-29e4-41d5-abe1-fea5303f42bf","Type":"ContainerStarted","Data":"aeae4b8b0e97a4d7f47232fa3e84ba8e417344cc2e4fc5985d28a12b4e1d8a99"}
Sep 14 04:46:44.496072 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:44.495972    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zv7p8" event={"ID":"9f1e07bf-2d62-4d64-a473-150a0830ba42","Type":"ContainerStarted","Data":"482efd3d9b7884575d5c32a5443fc077c6bb82db46b2b28e23eff2be25672146"}
Sep 14 04:46:44.530184 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:44.530114    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6g897" event={"ID":"2e7a5d9f-3918-4bba-995c-1dc18f9611ac","Type":"ContainerStarted","Data":"9665e4563451d801e1d0b45f300e75c34f298af64b0b5c4765cc4915828ecb41"}
Sep 14 04:46:44.856390 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:44.856349    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ca784040-efa0-4d69-8c25-2b284f3bbe65-original-pull-secret\") pod \"global-pull-secret-syncer-whcnp\" (UID: \"ca784040-efa0-4d69-8c25-2b284f3bbe65\") " pod="kube-system/global-pull-secret-syncer-whcnp"
Sep 14 04:46:44.856605 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:44.856503    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 04:46:44.856605 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:44.856583    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca784040-efa0-4d69-8c25-2b284f3bbe65-original-pull-secret podName:ca784040-efa0-4d69-8c25-2b284f3bbe65 nodeName:}" failed. No retries permitted until 2026-09-14 04:46:46.856564996 +0000 UTC m=+5.032336538 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ca784040-efa0-4d69-8c25-2b284f3bbe65-original-pull-secret") pod "global-pull-secret-syncer-whcnp" (UID: "ca784040-efa0-4d69-8c25-2b284f3bbe65") : object "kube-system"/"original-pull-secret" not registered
Sep 14 04:46:44.958160 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:44.958001    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh4l8\" (UniqueName: \"kubernetes.io/projected/6b6a2ac6-52be-4747-b544-a61782e29691-kube-api-access-wh4l8\") pod \"network-check-target-6sw5h\" (UID: \"6b6a2ac6-52be-4747-b544-a61782e29691\") " pod="openshift-network-diagnostics/network-check-target-6sw5h"
Sep 14 04:46:44.958853 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:44.958525    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 04:46:44.958853 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:44.958556    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 04:46:44.958853 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:44.958578    2565 projected.go:196] Error preparing data for projected volume kube-api-access-wh4l8 for pod openshift-network-diagnostics/network-check-target-6sw5h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 04:46:44.958853 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:44.958650    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b6a2ac6-52be-4747-b544-a61782e29691-kube-api-access-wh4l8 podName:6b6a2ac6-52be-4747-b544-a61782e29691 nodeName:}" failed. No retries permitted until 2026-09-14 04:46:46.958621983 +0000 UTC m=+5.134393543 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wh4l8" (UniqueName: "kubernetes.io/projected/6b6a2ac6-52be-4747-b544-a61782e29691-kube-api-access-wh4l8") pod "network-check-target-6sw5h" (UID: "6b6a2ac6-52be-4747-b544-a61782e29691") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 04:46:44.964867 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:44.963122    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee8a9555-8b90-4ad8-a399-22d414487421-metrics-certs\") pod \"network-metrics-daemon-2gwsl\" (UID: \"ee8a9555-8b90-4ad8-a399-22d414487421\") " pod="openshift-multus/network-metrics-daemon-2gwsl"
Sep 14 04:46:44.964867 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:44.963268    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 04:46:44.964867 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:44.963333    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee8a9555-8b90-4ad8-a399-22d414487421-metrics-certs podName:ee8a9555-8b90-4ad8-a399-22d414487421 nodeName:}" failed. No retries permitted until 2026-09-14 04:46:46.963316206 +0000 UTC m=+5.139087750 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee8a9555-8b90-4ad8-a399-22d414487421-metrics-certs") pod "network-metrics-daemon-2gwsl" (UID: "ee8a9555-8b90-4ad8-a399-22d414487421") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 04:46:45.295551 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:45.295433    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-13 04:41:43 +0000 UTC" deadline="2028-03-03 07:58:57.522038197 +0000 UTC"
Sep 14 04:46:45.295551 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:45.295493    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12867h12m12.22654904s"
Sep 14 04:46:45.329009 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:45.328921    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-whcnp"
Sep 14 04:46:45.329182 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:45.329127    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-whcnp" podUID="ca784040-efa0-4d69-8c25-2b284f3bbe65"
Sep 14 04:46:45.329563 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:45.329538    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2gwsl"
Sep 14 04:46:45.329668 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:45.329646    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2gwsl" podUID="ee8a9555-8b90-4ad8-a399-22d414487421"
Sep 14 04:46:46.332115 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:46.330830    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6sw5h"
Sep 14 04:46:46.332115 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:46.330971    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6sw5h" podUID="6b6a2ac6-52be-4747-b544-a61782e29691"
Sep 14 04:46:46.879303 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:46.879265    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ca784040-efa0-4d69-8c25-2b284f3bbe65-original-pull-secret\") pod \"global-pull-secret-syncer-whcnp\" (UID: \"ca784040-efa0-4d69-8c25-2b284f3bbe65\") " pod="kube-system/global-pull-secret-syncer-whcnp"
Sep 14 04:46:46.879481 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:46.879399    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 04:46:46.879481 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:46.879459    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca784040-efa0-4d69-8c25-2b284f3bbe65-original-pull-secret podName:ca784040-efa0-4d69-8c25-2b284f3bbe65 nodeName:}" failed. No retries permitted until 2026-09-14 04:46:50.879440723 +0000 UTC m=+9.055212266 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ca784040-efa0-4d69-8c25-2b284f3bbe65-original-pull-secret") pod "global-pull-secret-syncer-whcnp" (UID: "ca784040-efa0-4d69-8c25-2b284f3bbe65") : object "kube-system"/"original-pull-secret" not registered
Sep 14 04:46:46.984739 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:46.979596    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee8a9555-8b90-4ad8-a399-22d414487421-metrics-certs\") pod \"network-metrics-daemon-2gwsl\" (UID: \"ee8a9555-8b90-4ad8-a399-22d414487421\") " pod="openshift-multus/network-metrics-daemon-2gwsl"
Sep 14 04:46:46.984739 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:46.979650    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh4l8\" (UniqueName: \"kubernetes.io/projected/6b6a2ac6-52be-4747-b544-a61782e29691-kube-api-access-wh4l8\") pod \"network-check-target-6sw5h\" (UID: \"6b6a2ac6-52be-4747-b544-a61782e29691\") " pod="openshift-network-diagnostics/network-check-target-6sw5h"
Sep 14 04:46:46.984739 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:46.979825    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 04:46:46.984739 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:46.979846    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 04:46:46.984739 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:46.979858    2565 projected.go:196] Error preparing data for projected volume kube-api-access-wh4l8 for pod openshift-network-diagnostics/network-check-target-6sw5h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 04:46:46.984739 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:46.979911    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b6a2ac6-52be-4747-b544-a61782e29691-kube-api-access-wh4l8 podName:6b6a2ac6-52be-4747-b544-a61782e29691 nodeName:}" failed. No retries permitted until 2026-09-14 04:46:50.979893509 +0000 UTC m=+9.155665048 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wh4l8" (UniqueName: "kubernetes.io/projected/6b6a2ac6-52be-4747-b544-a61782e29691-kube-api-access-wh4l8") pod "network-check-target-6sw5h" (UID: "6b6a2ac6-52be-4747-b544-a61782e29691") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 04:46:46.984739 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:46.984630    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 04:46:46.984739 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:46.984691    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee8a9555-8b90-4ad8-a399-22d414487421-metrics-certs podName:ee8a9555-8b90-4ad8-a399-22d414487421 nodeName:}" failed. No retries permitted until 2026-09-14 04:46:50.984673247 +0000 UTC m=+9.160444789 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee8a9555-8b90-4ad8-a399-22d414487421-metrics-certs") pod "network-metrics-daemon-2gwsl" (UID: "ee8a9555-8b90-4ad8-a399-22d414487421") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 04:46:47.328853 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:47.328522    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2gwsl"
Sep 14 04:46:47.328853 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:47.328594    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-whcnp"
Sep 14 04:46:47.329069 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:47.328975    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-whcnp" podUID="ca784040-efa0-4d69-8c25-2b284f3bbe65"
Sep 14 04:46:47.329446 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:47.329420    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2gwsl" podUID="ee8a9555-8b90-4ad8-a399-22d414487421"
Sep 14 04:46:48.329805 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:48.329044    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6sw5h"
Sep 14 04:46:48.330246 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:48.329911    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6sw5h" podUID="6b6a2ac6-52be-4747-b544-a61782e29691"
Sep 14 04:46:49.329017 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:49.328872    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2gwsl"
Sep 14 04:46:49.329017 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:49.328888    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-whcnp"
Sep 14 04:46:49.329241 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:49.329025    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2gwsl" podUID="ee8a9555-8b90-4ad8-a399-22d414487421"
Sep 14 04:46:49.329241 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:49.329176    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-whcnp" podUID="ca784040-efa0-4d69-8c25-2b284f3bbe65"
Sep 14 04:46:50.328426 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:50.328394    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6sw5h"
Sep 14 04:46:50.328874 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:50.328509    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6sw5h" podUID="6b6a2ac6-52be-4747-b544-a61782e29691"
Sep 14 04:46:50.916497 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:50.916450    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ca784040-efa0-4d69-8c25-2b284f3bbe65-original-pull-secret\") pod \"global-pull-secret-syncer-whcnp\" (UID: \"ca784040-efa0-4d69-8c25-2b284f3bbe65\") " pod="kube-system/global-pull-secret-syncer-whcnp"
Sep 14 04:46:50.916655 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:50.916635    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 04:46:50.916730 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:50.916719    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca784040-efa0-4d69-8c25-2b284f3bbe65-original-pull-secret podName:ca784040-efa0-4d69-8c25-2b284f3bbe65 nodeName:}" failed. No retries permitted until 2026-09-14 04:46:58.916697047 +0000 UTC m=+17.092468588 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ca784040-efa0-4d69-8c25-2b284f3bbe65-original-pull-secret") pod "global-pull-secret-syncer-whcnp" (UID: "ca784040-efa0-4d69-8c25-2b284f3bbe65") : object "kube-system"/"original-pull-secret" not registered
Sep 14 04:46:51.017523 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:51.017445    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee8a9555-8b90-4ad8-a399-22d414487421-metrics-certs\") pod \"network-metrics-daemon-2gwsl\" (UID: \"ee8a9555-8b90-4ad8-a399-22d414487421\") " pod="openshift-multus/network-metrics-daemon-2gwsl"
Sep 14 04:46:51.017523 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:51.017495    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh4l8\" (UniqueName: \"kubernetes.io/projected/6b6a2ac6-52be-4747-b544-a61782e29691-kube-api-access-wh4l8\") pod \"network-check-target-6sw5h\" (UID: \"6b6a2ac6-52be-4747-b544-a61782e29691\") " pod="openshift-network-diagnostics/network-check-target-6sw5h"
Sep 14 04:46:51.017797 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:51.017601    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 04:46:51.017797 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:51.017617    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 04:46:51.017797 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:51.017635    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 04:46:51.017797 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:51.017648    2565 projected.go:196] Error preparing data for projected volume kube-api-access-wh4l8 for pod openshift-network-diagnostics/network-check-target-6sw5h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 04:46:51.017797 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:51.017682    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee8a9555-8b90-4ad8-a399-22d414487421-metrics-certs podName:ee8a9555-8b90-4ad8-a399-22d414487421 nodeName:}" failed. No retries permitted until 2026-09-14 04:46:59.017661598 +0000 UTC m=+17.193433149 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee8a9555-8b90-4ad8-a399-22d414487421-metrics-certs") pod "network-metrics-daemon-2gwsl" (UID: "ee8a9555-8b90-4ad8-a399-22d414487421") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 04:46:51.017797 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:51.017702    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b6a2ac6-52be-4747-b544-a61782e29691-kube-api-access-wh4l8 podName:6b6a2ac6-52be-4747-b544-a61782e29691 nodeName:}" failed. No retries permitted until 2026-09-14 04:46:59.017692636 +0000 UTC m=+17.193464173 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wh4l8" (UniqueName: "kubernetes.io/projected/6b6a2ac6-52be-4747-b544-a61782e29691-kube-api-access-wh4l8") pod "network-check-target-6sw5h" (UID: "6b6a2ac6-52be-4747-b544-a61782e29691") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 04:46:51.328147 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:51.328054    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2gwsl"
Sep 14 04:46:51.328147 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:51.328079    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-whcnp"
Sep 14 04:46:51.328383 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:51.328204    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2gwsl" podUID="ee8a9555-8b90-4ad8-a399-22d414487421"
Sep 14 04:46:51.328383 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:51.328313    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-whcnp" podUID="ca784040-efa0-4d69-8c25-2b284f3bbe65"
Sep 14 04:46:52.329118 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:52.329082    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6sw5h"
Sep 14 04:46:52.329561 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:52.329210    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6sw5h" podUID="6b6a2ac6-52be-4747-b544-a61782e29691"
Sep 14 04:46:53.328096 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:53.328001    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2gwsl"
Sep 14 04:46:53.328275 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:53.328132    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2gwsl" podUID="ee8a9555-8b90-4ad8-a399-22d414487421"
Sep 14 04:46:53.328832 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:53.328803    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-whcnp"
Sep 14 04:46:53.328990 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:53.328962    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-whcnp" podUID="ca784040-efa0-4d69-8c25-2b284f3bbe65"
Sep 14 04:46:54.329060 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:54.329027    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6sw5h"
Sep 14 04:46:54.329528 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:54.329146    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6sw5h" podUID="6b6a2ac6-52be-4747-b544-a61782e29691"
Sep 14 04:46:55.328769 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:55.328678    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-whcnp"
Sep 14 04:46:55.328942 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:55.328679    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2gwsl"
Sep 14 04:46:55.328942 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:55.328823    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-whcnp" podUID="ca784040-efa0-4d69-8c25-2b284f3bbe65"
Sep 14 04:46:55.328942 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:55.328916    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2gwsl" podUID="ee8a9555-8b90-4ad8-a399-22d414487421"
Sep 14 04:46:56.328586 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:56.328553    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6sw5h"
Sep 14 04:46:56.329045 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:56.328657    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6sw5h" podUID="6b6a2ac6-52be-4747-b544-a61782e29691"
Sep 14 04:46:57.328087 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:57.328053    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2gwsl"
Sep 14 04:46:57.328250 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:57.328090    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-whcnp"
Sep 14 04:46:57.328250 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:57.328195    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2gwsl" podUID="ee8a9555-8b90-4ad8-a399-22d414487421"
Sep 14 04:46:57.328351 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:57.328327    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-whcnp" podUID="ca784040-efa0-4d69-8c25-2b284f3bbe65"
Sep 14 04:46:58.328637 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:58.328603    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6sw5h"
Sep 14 04:46:58.329071 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:58.328725    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6sw5h" podUID="6b6a2ac6-52be-4747-b544-a61782e29691"
Sep 14 04:46:58.977706 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:58.977670    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ca784040-efa0-4d69-8c25-2b284f3bbe65-original-pull-secret\") pod \"global-pull-secret-syncer-whcnp\" (UID: \"ca784040-efa0-4d69-8c25-2b284f3bbe65\") " pod="kube-system/global-pull-secret-syncer-whcnp"
Sep 14 04:46:58.977932 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:58.977833    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 04:46:58.977932 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:58.977900    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca784040-efa0-4d69-8c25-2b284f3bbe65-original-pull-secret podName:ca784040-efa0-4d69-8c25-2b284f3bbe65 nodeName:}" failed. No retries permitted until 2026-09-14 04:47:14.977878817 +0000 UTC m=+33.153650364 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ca784040-efa0-4d69-8c25-2b284f3bbe65-original-pull-secret") pod "global-pull-secret-syncer-whcnp" (UID: "ca784040-efa0-4d69-8c25-2b284f3bbe65") : object "kube-system"/"original-pull-secret" not registered
Sep 14 04:46:59.078230 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:59.078188    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh4l8\" (UniqueName: \"kubernetes.io/projected/6b6a2ac6-52be-4747-b544-a61782e29691-kube-api-access-wh4l8\") pod \"network-check-target-6sw5h\" (UID: \"6b6a2ac6-52be-4747-b544-a61782e29691\") " pod="openshift-network-diagnostics/network-check-target-6sw5h"
Sep 14 04:46:59.078381 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:59.078278    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee8a9555-8b90-4ad8-a399-22d414487421-metrics-certs\") pod \"network-metrics-daemon-2gwsl\" (UID: \"ee8a9555-8b90-4ad8-a399-22d414487421\") " pod="openshift-multus/network-metrics-daemon-2gwsl"
Sep 14 04:46:59.078381 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:59.078353    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 04:46:59.078381 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:59.078380    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 04:46:59.078511 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:59.078392    2565 projected.go:196] Error preparing data for projected volume kube-api-access-wh4l8 for pod openshift-network-diagnostics/network-check-target-6sw5h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 04:46:59.078511 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:59.078357    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 04:46:59.078511 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:59.078453    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b6a2ac6-52be-4747-b544-a61782e29691-kube-api-access-wh4l8 podName:6b6a2ac6-52be-4747-b544-a61782e29691 nodeName:}" failed. No retries permitted until 2026-09-14 04:47:15.078434502 +0000 UTC m=+33.254206078 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wh4l8" (UniqueName: "kubernetes.io/projected/6b6a2ac6-52be-4747-b544-a61782e29691-kube-api-access-wh4l8") pod "network-check-target-6sw5h" (UID: "6b6a2ac6-52be-4747-b544-a61782e29691") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 04:46:59.078511 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:59.078502    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee8a9555-8b90-4ad8-a399-22d414487421-metrics-certs podName:ee8a9555-8b90-4ad8-a399-22d414487421 nodeName:}" failed. No retries permitted until 2026-09-14 04:47:15.078487247 +0000 UTC m=+33.254258786 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee8a9555-8b90-4ad8-a399-22d414487421-metrics-certs") pod "network-metrics-daemon-2gwsl" (UID: "ee8a9555-8b90-4ad8-a399-22d414487421") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 04:46:59.328663 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:59.328573    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2gwsl"
Sep 14 04:46:59.329196 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:46:59.328587    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-whcnp"
Sep 14 04:46:59.329196 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:59.328698    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2gwsl" podUID="ee8a9555-8b90-4ad8-a399-22d414487421"
Sep 14 04:46:59.329196 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:46:59.328805    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-whcnp" podUID="ca784040-efa0-4d69-8c25-2b284f3bbe65"
Sep 14 04:47:00.328408 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:00.328372    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6sw5h"
Sep 14 04:47:00.328597 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:00.328513    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6sw5h" podUID="6b6a2ac6-52be-4747-b544-a61782e29691"
Sep 14 04:47:01.328424 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:01.328380    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2gwsl"
Sep 14 04:47:01.328875 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:01.328433    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-whcnp"
Sep 14 04:47:01.328875 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:01.328516    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2gwsl" podUID="ee8a9555-8b90-4ad8-a399-22d414487421"
Sep 14 04:47:01.328875 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:01.328634    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-whcnp" podUID="ca784040-efa0-4d69-8c25-2b284f3bbe65"
Sep 14 04:47:02.330010 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:02.328835    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6sw5h"
Sep 14 04:47:02.331463 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:02.330543    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6sw5h" podUID="6b6a2ac6-52be-4747-b544-a61782e29691"
Sep 14 04:47:03.329140 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:03.328846    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2gwsl"
Sep 14 04:47:03.329254 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:03.328899    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-whcnp"
Sep 14 04:47:03.329254 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:03.329223    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2gwsl" podUID="ee8a9555-8b90-4ad8-a399-22d414487421"
Sep 14 04:47:03.329355 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:03.329296    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-whcnp" podUID="ca784040-efa0-4d69-8c25-2b284f3bbe65"
Sep 14 04:47:03.576003 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:03.575970    2565 generic.go:356] "Generic (PLEG): container finished" podID="f6ac6666-f468-4757-8748-38b7e4fa90a9" containerID="9bceeb09558e184d4ef78064ffbc5a5b56be553d3f3cb222dd3533be521a3604" exitCode=0
Sep 14 04:47:03.576808 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:03.576040    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9dck" event={"ID":"f6ac6666-f468-4757-8748-38b7e4fa90a9","Type":"ContainerDied","Data":"9bceeb09558e184d4ef78064ffbc5a5b56be553d3f3cb222dd3533be521a3604"}
Sep 14 04:47:03.577881 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:03.577858    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pfl42" event={"ID":"7f7e9f24-ddd1-4c3b-acdf-371ea4c648e8","Type":"ContainerStarted","Data":"8a08b2725d9c87f75905b5b20354099290ef380a8d57416ae93e83353cd8cc03"}
Sep 14 04:47:03.580835 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:03.580815    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kndxq_e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae/ovn-acl-logging/0.log"
Sep 14 04:47:03.581177 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:03.581159    2565 generic.go:356] "Generic (PLEG): container finished" podID="e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae" containerID="e4e39ad0c96e0481bc17467dffac43d4e1c70ced35ef6fac5ce33888b3812d87" exitCode=1
Sep 14 04:47:03.581263 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:03.581226    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kndxq" event={"ID":"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae","Type":"ContainerStarted","Data":"33f98f4b9b330b4634a490c110aa359e6bb9d06911b7b0532babcffc8da19679"}
Sep 14 04:47:03.581313 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:03.581287    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kndxq" event={"ID":"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae","Type":"ContainerStarted","Data":"259cd749956d69615ca8e2f2f06d99abcc0d8ba22cc2269246c0efaa347e218e"}
Sep 14 04:47:03.581369 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:03.581334    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kndxq" event={"ID":"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae","Type":"ContainerStarted","Data":"e97f8a8369951e06bf66a95b4da5ff86b9952d1975ee9a36247e1dc7927c7fff"}
Sep 14 04:47:03.581426 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:03.581381    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kndxq" event={"ID":"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae","Type":"ContainerStarted","Data":"eaef8d767ad8a3515fdfd44ede5d52d267ba40f51521802d90d5c96c6f0e0a42"}
Sep 14 04:47:03.581481 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:03.581426    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kndxq" event={"ID":"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae","Type":"ContainerDied","Data":"e4e39ad0c96e0481bc17467dffac43d4e1c70ced35ef6fac5ce33888b3812d87"}
Sep 14 04:47:03.581481 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:03.581473    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kndxq" event={"ID":"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae","Type":"ContainerStarted","Data":"45457f369497741f4b50719c6afdce03b35a75a6f47eaed7a58f393b68fbb5ac"}
Sep 14 04:47:03.582639 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:03.582617    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ptbx9" event={"ID":"67d2dcbd-5882-4609-a9cd-527e8067af4b","Type":"ContainerStarted","Data":"3f3f49a41edea4858643bdfae0372d5abcf9cfc671341e6e7732e4377abe1188"}
Sep 14 04:47:03.584117 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:03.584096    2565 generic.go:356] "Generic (PLEG): container finished" podID="53c18eb6ba51406b052ec7b9ec1cf464" containerID="814dc71927a5b37454b5bd50f02a1c5d2690f499cc184a4db6579f5a46ac0daa" exitCode=0
Sep 14 04:47:03.584212 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:03.584157    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-99.ec2.internal" event={"ID":"53c18eb6ba51406b052ec7b9ec1cf464","Type":"ContainerDied","Data":"814dc71927a5b37454b5bd50f02a1c5d2690f499cc184a4db6579f5a46ac0daa"}
Sep 14 04:47:03.585694 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:03.585674    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvr2" event={"ID":"ada14485-29e4-41d5-abe1-fea5303f42bf","Type":"ContainerStarted","Data":"70efec3bcf1315c3f66fb59b2d2349373da04849ea26452f20793ba4a49c416c"}
Sep 14 04:47:03.587112 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:03.587093    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zv7p8" event={"ID":"9f1e07bf-2d62-4d64-a473-150a0830ba42","Type":"ContainerStarted","Data":"5c492b5e2426713d876a7b3ad24179c71e2ae47fd8711b322ad815e1e77acd64"}
Sep 14 04:47:03.588613 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:03.588595    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6g897" event={"ID":"2e7a5d9f-3918-4bba-995c-1dc18f9611ac","Type":"ContainerStarted","Data":"e4062a8077d62c37245c7c4eb7621f419fb93f8c657d0d4764cd9dad8f25f18f"}
Sep 14 04:47:03.590181 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:03.590162    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rqw8q" event={"ID":"0cfe37ed-e18d-4e25-84d6-cba59e2b2d81","Type":"ContainerStarted","Data":"e95d4f744a34a2859e59a9e9324050a2f97fa28ec5af67128731d42916fd81d0"}
Sep 14 04:47:03.591623 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:03.591606    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-99.ec2.internal" event={"ID":"d5acb9fe33c38810a5100cf8c524f161","Type":"ContainerStarted","Data":"32c5c74eb2a7183a95da3327b7c27997c60a42ad31bcad4a4688a7a3dd360a6d"}
Sep 14 04:47:03.656188 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:03.656138    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-99.ec2.internal" podStartSLOduration=20.656123572 podStartE2EDuration="20.656123572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 04:46:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:47:03.655464587 +0000 UTC m=+21.831236145" watchObservedRunningTime="2026-09-14 04:47:03.656123572 +0000 UTC m=+21.831895130"
Sep 14 04:47:03.678991 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:03.678947    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zv7p8" podStartSLOduration=2.587163827 podStartE2EDuration="21.67893254s" totalImagesPullingTime="19.091768713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:46:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:47:03.678639578 +0000 UTC m=+21.854411137" watchObservedRunningTime="2026-09-14 04:47:03.67893254 +0000 UTC m=+21.854704098"
Sep 14 04:47:03.704175 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:03.704133    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-pfl42" podStartSLOduration=2.638310316 podStartE2EDuration="21.704118379s" totalImagesPullingTime="19.065808063s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 04:46:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:47:03.70401328 +0000 UTC m=+21.879784838" watchObservedRunningTime="2026-09-14 04:47:03.704118379 +0000 UTC m=+21.879889941"
Sep 14 04:47:03.729292 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:03.729249    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ptbx9" podStartSLOduration=2.566603472 podStartE2EDuration="21.72923393s" totalImagesPullingTime="19.162630458s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 04:46:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:47:03.728804604 +0000 UTC m=+21.904576162" watchObservedRunningTime="2026-09-14 04:47:03.72923393 +0000 UTC m=+21.905005486"
Sep 14 04:47:03.745897 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:03.745852    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-rqw8q" podStartSLOduration=2.60067456 podStartE2EDuration="21.745836121s" totalImagesPullingTime="19.145161561s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 04:46:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:47:03.745573618 +0000 UTC m=+21.921345168" watchObservedRunningTime="2026-09-14 04:47:03.745836121 +0000 UTC m=+21.921607679"
Sep 14 04:47:03.766365 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:03.766321    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6g897" podStartSLOduration=2.57778162 podStartE2EDuration="21.766301956s" totalImagesPullingTime="19.188520336s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 04:46:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:47:03.766233705 +0000 UTC m=+21.942005267" watchObservedRunningTime="2026-09-14 04:47:03.766301956 +0000 UTC m=+21.942073518"
Sep 14 04:47:04.328211 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:04.328181    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6sw5h"
Sep 14 04:47:04.328393 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:04.328289    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6sw5h" podUID="6b6a2ac6-52be-4747-b544-a61782e29691"
Sep 14 04:47:04.562346 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:04.562316    2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 14 04:47:04.595645 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:04.595617    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xkjrf" event={"ID":"842b0bc4-80e3-4936-a363-328c75f66265","Type":"ContainerStarted","Data":"54b41bce768aedfae966e4e5fa54df2659b396781a57ad82630d35132ca5dcd5"}
Sep 14 04:47:04.597676 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:04.597649    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-99.ec2.internal" event={"ID":"53c18eb6ba51406b052ec7b9ec1cf464","Type":"ContainerStarted","Data":"004e8b3e47458aae00f6de72a48e5a8128a389eb760e8ad16e6e9fd64c7a57e2"}
Sep 14 04:47:04.599812 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:04.599742    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvr2" event={"ID":"ada14485-29e4-41d5-abe1-fea5303f42bf","Type":"ContainerStarted","Data":"4aad8e1f21d421569c20f6aedf85ea53f1f74d22b0a54cb50f5e90f0da436b75"}
Sep 14 04:47:04.619929 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:04.619893    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xkjrf" podStartSLOduration=3.588090424 podStartE2EDuration="22.619880231s" totalImagesPullingTime="19.031789807s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 04:46:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:47:04.619368605 +0000 UTC m=+22.795140165" watchObservedRunningTime="2026-09-14 04:47:04.619880231 +0000 UTC m=+22.795651791"
Sep 14 04:47:04.637913 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:04.637871    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-99.ec2.internal" podStartSLOduration=21.637855103 podStartE2EDuration="21.637855103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 04:46:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:47:04.637422164 +0000 UTC m=+22.813193728" watchObservedRunningTime="2026-09-14 04:47:04.637855103 +0000 UTC m=+22.813626662"
Sep 14 04:47:05.293067 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:05.292971    2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-14T04:47:04.562344635Z","UUID":"c5d8f61d-d5f9-4b92-b133-78badbdb92eb","Handler":null,"Name":"","Endpoint":""}
Sep 14 04:47:05.294814 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:05.294671    2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 14 04:47:05.294814 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:05.294711    2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 14 04:47:05.328424 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:05.328398    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-whcnp"
Sep 14 04:47:05.328596 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:05.328462    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2gwsl"
Sep 14 04:47:05.328673 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:05.328646    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2gwsl" podUID="ee8a9555-8b90-4ad8-a399-22d414487421"
Sep 14 04:47:05.328739 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:05.328685    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-whcnp" podUID="ca784040-efa0-4d69-8c25-2b284f3bbe65"
Sep 14 04:47:06.328261 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:06.328181    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6sw5h"
Sep 14 04:47:06.328637 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:06.328312    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6sw5h" podUID="6b6a2ac6-52be-4747-b544-a61782e29691"
Sep 14 04:47:06.556687 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:06.556654    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zv7p8"
Sep 14 04:47:06.557289 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:06.557262    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zv7p8"
Sep 14 04:47:06.606702 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:06.606619    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvr2" event={"ID":"ada14485-29e4-41d5-abe1-fea5303f42bf","Type":"ContainerStarted","Data":"0b8c40a3ee8f1461db0446c1d376b06a96782fd888b3e5eb81d04559286ffd3c"}
Sep 14 04:47:06.610141 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:06.610121    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kndxq_e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae/ovn-acl-logging/0.log"
Sep 14 04:47:06.610650 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:06.610622    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kndxq" event={"ID":"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae","Type":"ContainerStarted","Data":"a4688e4f4a48bb8e4a69d7aa351f84b6493224135f307b07e13f7dc6dde3cf97"}
Sep 14 04:47:06.610858 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:06.610839    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zv7p8"
Sep 14 04:47:06.611330 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:06.611313    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zv7p8"
Sep 14 04:47:06.630367 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:06.630324    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvr2" podStartSLOduration=3.077139275 podStartE2EDuration="24.630308872s" totalImagesPullingTime="21.553169597s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 04:46:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:47:06.63020892 +0000 UTC m=+24.805980480" watchObservedRunningTime="2026-09-14 04:47:06.630308872 +0000 UTC m=+24.806080432"
Sep 14 04:47:07.329075 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:07.328890    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2gwsl"
Sep 14 04:47:07.329524 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:07.328945    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-whcnp"
Sep 14 04:47:07.329524 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:07.329159    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2gwsl" podUID="ee8a9555-8b90-4ad8-a399-22d414487421"
Sep 14 04:47:07.329524 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:07.329265    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-whcnp" podUID="ca784040-efa0-4d69-8c25-2b284f3bbe65"
Sep 14 04:47:08.328983 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:08.328915    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6sw5h"
Sep 14 04:47:08.329110 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:08.329012    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6sw5h" podUID="6b6a2ac6-52be-4747-b544-a61782e29691"
Sep 14 04:47:08.617633 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:08.617566    2565 generic.go:356] "Generic (PLEG): container finished" podID="f6ac6666-f468-4757-8748-38b7e4fa90a9" containerID="b0b246945a5f9cd4e3402fa7675c774aa5e1e3d1e3bdb194bfd5aca323ef1e73" exitCode=0
Sep 14 04:47:08.617792 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:08.617652    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9dck" event={"ID":"f6ac6666-f468-4757-8748-38b7e4fa90a9","Type":"ContainerDied","Data":"b0b246945a5f9cd4e3402fa7675c774aa5e1e3d1e3bdb194bfd5aca323ef1e73"}
Sep 14 04:47:08.624039 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:08.624021    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kndxq_e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae/ovn-acl-logging/0.log"
Sep 14 04:47:08.624470 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:08.624441    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kndxq" event={"ID":"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae","Type":"ContainerStarted","Data":"6020c3aa2acb5083fa2775b99ce5d19b9cc56e8c1a7e86456ae136865d72468a"}
Sep 14 04:47:08.624903 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:08.624887    2565 scope.go:117] "RemoveContainer" containerID="e4e39ad0c96e0481bc17467dffac43d4e1c70ced35ef6fac5ce33888b3812d87"
Sep 14 04:47:09.329104 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:09.328924    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2gwsl"
Sep 14 04:47:09.329591 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:09.329184    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2gwsl" podUID="ee8a9555-8b90-4ad8-a399-22d414487421"
Sep 14 04:47:09.329591 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:09.329475    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-whcnp"
Sep 14 04:47:09.329591 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:09.329534    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-whcnp" podUID="ca784040-efa0-4d69-8c25-2b284f3bbe65"
Sep 14 04:47:09.631102 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:09.631070    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kndxq_e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae/ovn-acl-logging/0.log"
Sep 14 04:47:09.631461 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:09.631437    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kndxq" event={"ID":"e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae","Type":"ContainerStarted","Data":"c151d1061c78f2164a9a607b393090524281377b821d4295fbe5b2a54b3c69a8"}
Sep 14 04:47:09.631627 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:09.631613    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:47:09.631696 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:09.631655    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:47:09.631696 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:09.631692    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:47:09.651325 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:09.651303    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:47:09.651468 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:09.651452    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:47:09.667241 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:09.665102    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kndxq" podStartSLOduration=8.538129158 podStartE2EDuration="27.665084845s" totalImagesPullingTime="19.126955687s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 04:46:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:47:09.664850356 +0000 UTC m=+27.840621913" watchObservedRunningTime="2026-09-14 04:47:09.665084845 +0000 UTC m=+27.840856405"
Sep 14 04:47:09.816354 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:09.816326    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-whcnp"]
Sep 14 04:47:09.816489 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:09.816452    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-whcnp"
Sep 14 04:47:09.816554 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:09.816535    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-whcnp" podUID="ca784040-efa0-4d69-8c25-2b284f3bbe65"
Sep 14 04:47:09.820045 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:09.820017    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6sw5h"]
Sep 14 04:47:09.820164 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:09.820123    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6sw5h"
Sep 14 04:47:09.820247 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:09.820218    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6sw5h" podUID="6b6a2ac6-52be-4747-b544-a61782e29691"
Sep 14 04:47:09.820712 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:09.820676    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2gwsl"]
Sep 14 04:47:09.820843 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:09.820797    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2gwsl"
Sep 14 04:47:09.820919 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:09.820900    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2gwsl" podUID="ee8a9555-8b90-4ad8-a399-22d414487421"
Sep 14 04:47:10.636421 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:10.636389    2565 generic.go:356] "Generic (PLEG): container finished" podID="f6ac6666-f468-4757-8748-38b7e4fa90a9" containerID="c83660cb3eedfe0ad6fca3d3d3318fef2e2aca8e2efaf2bd6063ff17608a548f" exitCode=0
Sep 14 04:47:10.637466 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:10.636467    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9dck" event={"ID":"f6ac6666-f468-4757-8748-38b7e4fa90a9","Type":"ContainerDied","Data":"c83660cb3eedfe0ad6fca3d3d3318fef2e2aca8e2efaf2bd6063ff17608a548f"}
Sep 14 04:47:11.328534 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:11.328506    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6sw5h"
Sep 14 04:47:11.328699 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:11.328593    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-whcnp"
Sep 14 04:47:11.328699 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:11.328601    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6sw5h" podUID="6b6a2ac6-52be-4747-b544-a61782e29691"
Sep 14 04:47:11.328699 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:11.328673    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-whcnp" podUID="ca784040-efa0-4d69-8c25-2b284f3bbe65"
Sep 14 04:47:11.328699 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:11.328682    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2gwsl"
Sep 14 04:47:11.328877 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:11.328750    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2gwsl" podUID="ee8a9555-8b90-4ad8-a399-22d414487421"
Sep 14 04:47:12.643589 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:12.643405    2565 generic.go:356] "Generic (PLEG): container finished" podID="f6ac6666-f468-4757-8748-38b7e4fa90a9" containerID="540acc02c93c5f345e9c074f2b2e26b859374fd87f54d27daf9b9796af0f1d3e" exitCode=0
Sep 14 04:47:12.643951 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:12.643491    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9dck" event={"ID":"f6ac6666-f468-4757-8748-38b7e4fa90a9","Type":"ContainerDied","Data":"540acc02c93c5f345e9c074f2b2e26b859374fd87f54d27daf9b9796af0f1d3e"}
Sep 14 04:47:13.329012 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:13.328977    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-whcnp"
Sep 14 04:47:13.329171 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:13.329087    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2gwsl"
Sep 14 04:47:13.329171 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:13.329092    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-whcnp" podUID="ca784040-efa0-4d69-8c25-2b284f3bbe65"
Sep 14 04:47:13.329234 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:13.329189    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2gwsl" podUID="ee8a9555-8b90-4ad8-a399-22d414487421"
Sep 14 04:47:13.329234 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:13.329092    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6sw5h"
Sep 14 04:47:13.329294 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:13.329268    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6sw5h" podUID="6b6a2ac6-52be-4747-b544-a61782e29691"
Sep 14 04:47:14.992154 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:14.992124    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ca784040-efa0-4d69-8c25-2b284f3bbe65-original-pull-secret\") pod \"global-pull-secret-syncer-whcnp\" (UID: \"ca784040-efa0-4d69-8c25-2b284f3bbe65\") " pod="kube-system/global-pull-secret-syncer-whcnp"
Sep 14 04:47:14.992560 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:14.992230    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 04:47:14.992560 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:14.992285    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca784040-efa0-4d69-8c25-2b284f3bbe65-original-pull-secret podName:ca784040-efa0-4d69-8c25-2b284f3bbe65 nodeName:}" failed. No retries permitted until 2026-09-14 04:47:46.992266213 +0000 UTC m=+65.168037750 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ca784040-efa0-4d69-8c25-2b284f3bbe65-original-pull-secret") pod "global-pull-secret-syncer-whcnp" (UID: "ca784040-efa0-4d69-8c25-2b284f3bbe65") : object "kube-system"/"original-pull-secret" not registered
Sep 14 04:47:15.093167 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:15.093132    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee8a9555-8b90-4ad8-a399-22d414487421-metrics-certs\") pod \"network-metrics-daemon-2gwsl\" (UID: \"ee8a9555-8b90-4ad8-a399-22d414487421\") " pod="openshift-multus/network-metrics-daemon-2gwsl"
Sep 14 04:47:15.093331 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:15.093183    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh4l8\" (UniqueName: \"kubernetes.io/projected/6b6a2ac6-52be-4747-b544-a61782e29691-kube-api-access-wh4l8\") pod \"network-check-target-6sw5h\" (UID: \"6b6a2ac6-52be-4747-b544-a61782e29691\") " pod="openshift-network-diagnostics/network-check-target-6sw5h"
Sep 14 04:47:15.093331 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:15.093292    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 04:47:15.093450 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:15.093299    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 04:47:15.093450 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:15.093354    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee8a9555-8b90-4ad8-a399-22d414487421-metrics-certs podName:ee8a9555-8b90-4ad8-a399-22d414487421 nodeName:}" failed. No retries permitted until 2026-09-14 04:47:47.093334508 +0000 UTC m=+65.269106047 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee8a9555-8b90-4ad8-a399-22d414487421-metrics-certs") pod "network-metrics-daemon-2gwsl" (UID: "ee8a9555-8b90-4ad8-a399-22d414487421") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 04:47:15.093450 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:15.093355    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 04:47:15.093450 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:15.093372    2565 projected.go:196] Error preparing data for projected volume kube-api-access-wh4l8 for pod openshift-network-diagnostics/network-check-target-6sw5h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 04:47:15.093450 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:15.093418    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b6a2ac6-52be-4747-b544-a61782e29691-kube-api-access-wh4l8 podName:6b6a2ac6-52be-4747-b544-a61782e29691 nodeName:}" failed. No retries permitted until 2026-09-14 04:47:47.093403146 +0000 UTC m=+65.269174696 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wh4l8" (UniqueName: "kubernetes.io/projected/6b6a2ac6-52be-4747-b544-a61782e29691-kube-api-access-wh4l8") pod "network-check-target-6sw5h" (UID: "6b6a2ac6-52be-4747-b544-a61782e29691") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 04:47:15.328935 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:15.328864    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6sw5h"
Sep 14 04:47:15.329091 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:15.328865    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2gwsl"
Sep 14 04:47:15.329091 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:15.328965    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6sw5h" podUID="6b6a2ac6-52be-4747-b544-a61782e29691"
Sep 14 04:47:15.329091 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:15.328865    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-whcnp"
Sep 14 04:47:15.329091 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:15.329067    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2gwsl" podUID="ee8a9555-8b90-4ad8-a399-22d414487421"
Sep 14 04:47:15.329257 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:15.329119    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-whcnp" podUID="ca784040-efa0-4d69-8c25-2b284f3bbe65"
Sep 14 04:47:15.572692 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:15.572660    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-99.ec2.internal" event="NodeReady"
Sep 14 04:47:15.572914 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:15.572822    2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 14 04:47:15.625705 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:15.625672    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2ms2f"]
Sep 14 04:47:15.640294 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:15.640259    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zxrd4"]
Sep 14 04:47:15.640441 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:15.640383    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2ms2f"
Sep 14 04:47:15.643128 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:15.642963    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 14 04:47:15.643128 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:15.643029    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wwfwc\""
Sep 14 04:47:15.643128 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:15.643066    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 14 04:47:15.652351 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:15.652330    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2ms2f"]
Sep 14 04:47:15.652466 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:15.652364    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zxrd4"]
Sep 14 04:47:15.652466 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:15.652446    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zxrd4"
Sep 14 04:47:15.655032 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:15.655002    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 14 04:47:15.655032 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:15.655013    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 14 04:47:15.655201 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:15.655046    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-46brg\""
Sep 14 04:47:15.655201 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:15.655068    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 14 04:47:15.796725 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:15.796687    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb2c56d0-20d7-49ba-857c-febc67911fcd-tmp-dir\") pod \"dns-default-2ms2f\" (UID: \"bb2c56d0-20d7-49ba-857c-febc67911fcd\") " pod="openshift-dns/dns-default-2ms2f"
Sep 14 04:47:15.796896 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:15.796740    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f897b0d-d9b9-4ae8-b545-ce1f5dfa63c5-cert\") pod \"ingress-canary-zxrd4\" (UID: \"9f897b0d-d9b9-4ae8-b545-ce1f5dfa63c5\") " pod="openshift-ingress-canary/ingress-canary-zxrd4"
Sep 14 04:47:15.796896 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:15.796788    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d4zl\" (UniqueName: \"kubernetes.io/projected/9f897b0d-d9b9-4ae8-b545-ce1f5dfa63c5-kube-api-access-5d4zl\") pod \"ingress-canary-zxrd4\" (UID: \"9f897b0d-d9b9-4ae8-b545-ce1f5dfa63c5\") " pod="openshift-ingress-canary/ingress-canary-zxrd4"
Sep 14 04:47:15.796896 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:15.796814    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bb2c56d0-20d7-49ba-857c-febc67911fcd-config-volume\") pod \"dns-default-2ms2f\" (UID: \"bb2c56d0-20d7-49ba-857c-febc67911fcd\") " pod="openshift-dns/dns-default-2ms2f"
Sep 14 04:47:15.796896 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:15.796847    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt9lw\" (UniqueName: \"kubernetes.io/projected/bb2c56d0-20d7-49ba-857c-febc67911fcd-kube-api-access-dt9lw\") pod \"dns-default-2ms2f\" (UID: \"bb2c56d0-20d7-49ba-857c-febc67911fcd\") " pod="openshift-dns/dns-default-2ms2f"
Sep 14 04:47:15.797049 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:15.796902    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb2c56d0-20d7-49ba-857c-febc67911fcd-metrics-tls\") pod \"dns-default-2ms2f\" (UID: \"bb2c56d0-20d7-49ba-857c-febc67911fcd\") " pod="openshift-dns/dns-default-2ms2f"
Sep 14 04:47:15.898196 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:15.898111    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb2c56d0-20d7-49ba-857c-febc67911fcd-tmp-dir\") pod \"dns-default-2ms2f\" (UID: \"bb2c56d0-20d7-49ba-857c-febc67911fcd\") " pod="openshift-dns/dns-default-2ms2f"
Sep 14 04:47:15.898196 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:15.898176    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f897b0d-d9b9-4ae8-b545-ce1f5dfa63c5-cert\") pod \"ingress-canary-zxrd4\" (UID: \"9f897b0d-d9b9-4ae8-b545-ce1f5dfa63c5\") " pod="openshift-ingress-canary/ingress-canary-zxrd4"
Sep 14 04:47:15.898408 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:15.898198    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d4zl\" (UniqueName: \"kubernetes.io/projected/9f897b0d-d9b9-4ae8-b545-ce1f5dfa63c5-kube-api-access-5d4zl\") pod \"ingress-canary-zxrd4\" (UID: \"9f897b0d-d9b9-4ae8-b545-ce1f5dfa63c5\") " pod="openshift-ingress-canary/ingress-canary-zxrd4"
Sep 14 04:47:15.898408 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:15.898227    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bb2c56d0-20d7-49ba-857c-febc67911fcd-config-volume\") pod \"dns-default-2ms2f\" (UID: \"bb2c56d0-20d7-49ba-857c-febc67911fcd\") " pod="openshift-dns/dns-default-2ms2f"
Sep 14 04:47:15.898408 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:15.898268    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt9lw\" (UniqueName: \"kubernetes.io/projected/bb2c56d0-20d7-49ba-857c-febc67911fcd-kube-api-access-dt9lw\") pod \"dns-default-2ms2f\" (UID: \"bb2c56d0-20d7-49ba-857c-febc67911fcd\") " pod="openshift-dns/dns-default-2ms2f"
Sep 14 04:47:15.898408 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:15.898295    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb2c56d0-20d7-49ba-857c-febc67911fcd-metrics-tls\") pod \"dns-default-2ms2f\" (UID: \"bb2c56d0-20d7-49ba-857c-febc67911fcd\") " pod="openshift-dns/dns-default-2ms2f"
Sep 14 04:47:15.898597 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:15.898430    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 14 04:47:15.898597 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:15.898485    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb2c56d0-20d7-49ba-857c-febc67911fcd-metrics-tls podName:bb2c56d0-20d7-49ba-857c-febc67911fcd nodeName:}" failed. No retries permitted until 2026-09-14 04:47:16.398472332 +0000 UTC m=+34.574243869 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb2c56d0-20d7-49ba-857c-febc67911fcd-metrics-tls") pod "dns-default-2ms2f" (UID: "bb2c56d0-20d7-49ba-857c-febc67911fcd") : secret "dns-default-metrics-tls" not found
Sep 14 04:47:15.898597 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:15.898506    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb2c56d0-20d7-49ba-857c-febc67911fcd-tmp-dir\") pod \"dns-default-2ms2f\" (UID: \"bb2c56d0-20d7-49ba-857c-febc67911fcd\") " pod="openshift-dns/dns-default-2ms2f"
Sep 14 04:47:15.899134 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:15.899087    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bb2c56d0-20d7-49ba-857c-febc67911fcd-config-volume\") pod \"dns-default-2ms2f\" (UID: \"bb2c56d0-20d7-49ba-857c-febc67911fcd\") " pod="openshift-dns/dns-default-2ms2f"
Sep 14 04:47:15.903329 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:15.903287    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f897b0d-d9b9-4ae8-b545-ce1f5dfa63c5-cert\") pod \"ingress-canary-zxrd4\" (UID: \"9f897b0d-d9b9-4ae8-b545-ce1f5dfa63c5\") " pod="openshift-ingress-canary/ingress-canary-zxrd4"
Sep 14 04:47:15.907454 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:15.907431    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d4zl\" (UniqueName: \"kubernetes.io/projected/9f897b0d-d9b9-4ae8-b545-ce1f5dfa63c5-kube-api-access-5d4zl\") pod \"ingress-canary-zxrd4\" (UID: \"9f897b0d-d9b9-4ae8-b545-ce1f5dfa63c5\") " pod="openshift-ingress-canary/ingress-canary-zxrd4"
Sep 14 04:47:15.907837 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:15.907812    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt9lw\" (UniqueName: \"kubernetes.io/projected/bb2c56d0-20d7-49ba-857c-febc67911fcd-kube-api-access-dt9lw\") pod \"dns-default-2ms2f\" (UID: \"bb2c56d0-20d7-49ba-857c-febc67911fcd\") " pod="openshift-dns/dns-default-2ms2f"
Sep 14 04:47:15.962015 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:15.961975    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zxrd4"
Sep 14 04:47:16.402826 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:16.402796    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb2c56d0-20d7-49ba-857c-febc67911fcd-metrics-tls\") pod \"dns-default-2ms2f\" (UID: \"bb2c56d0-20d7-49ba-857c-febc67911fcd\") " pod="openshift-dns/dns-default-2ms2f"
Sep 14 04:47:16.403298 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:16.402913    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 14 04:47:16.403298 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:16.402973    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb2c56d0-20d7-49ba-857c-febc67911fcd-metrics-tls podName:bb2c56d0-20d7-49ba-857c-febc67911fcd nodeName:}" failed. No retries permitted until 2026-09-14 04:47:17.402958382 +0000 UTC m=+35.578729919 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb2c56d0-20d7-49ba-857c-febc67911fcd-metrics-tls") pod "dns-default-2ms2f" (UID: "bb2c56d0-20d7-49ba-857c-febc67911fcd") : secret "dns-default-metrics-tls" not found
Sep 14 04:47:17.230481 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:17.230272    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zxrd4"]
Sep 14 04:47:17.234256 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:47:17.234231    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f897b0d_d9b9_4ae8_b545_ce1f5dfa63c5.slice/crio-b2dc7692483e4f5e4fa96d351632a23b12f355ba0dbe3c32a808b9a21e725d9c WatchSource:0}: Error finding container b2dc7692483e4f5e4fa96d351632a23b12f355ba0dbe3c32a808b9a21e725d9c: Status 404 returned error can't find the container with id b2dc7692483e4f5e4fa96d351632a23b12f355ba0dbe3c32a808b9a21e725d9c
Sep 14 04:47:17.328266 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:17.328234    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6sw5h"
Sep 14 04:47:17.328266 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:17.328262    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2gwsl"
Sep 14 04:47:17.328439 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:17.328420    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-whcnp"
Sep 14 04:47:17.331804 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:17.331787    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 14 04:47:17.331890 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:17.331833    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 14 04:47:17.331890 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:17.331873    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kgkvn\""
Sep 14 04:47:17.331963 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:17.331912    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 14 04:47:17.331963 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:17.331835    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 14 04:47:17.332164 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:17.332150    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-54hj4\""
Sep 14 04:47:17.412457 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:17.412425    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb2c56d0-20d7-49ba-857c-febc67911fcd-metrics-tls\") pod \"dns-default-2ms2f\" (UID: \"bb2c56d0-20d7-49ba-857c-febc67911fcd\") " pod="openshift-dns/dns-default-2ms2f"
Sep 14 04:47:17.412825 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:17.412577    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 14 04:47:17.412825 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:17.412640    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb2c56d0-20d7-49ba-857c-febc67911fcd-metrics-tls podName:bb2c56d0-20d7-49ba-857c-febc67911fcd nodeName:}" failed. No retries permitted until 2026-09-14 04:47:19.412623739 +0000 UTC m=+37.588395276 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb2c56d0-20d7-49ba-857c-febc67911fcd-metrics-tls") pod "dns-default-2ms2f" (UID: "bb2c56d0-20d7-49ba-857c-febc67911fcd") : secret "dns-default-metrics-tls" not found
Sep 14 04:47:17.660240 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:17.660187    2565 generic.go:356] "Generic (PLEG): container finished" podID="f6ac6666-f468-4757-8748-38b7e4fa90a9" containerID="5a5fda9ef5d3941b252ddf416016e753720689372b0f9f434368ee0e90c84a14" exitCode=0
Sep 14 04:47:17.660406 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:17.660280    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9dck" event={"ID":"f6ac6666-f468-4757-8748-38b7e4fa90a9","Type":"ContainerDied","Data":"5a5fda9ef5d3941b252ddf416016e753720689372b0f9f434368ee0e90c84a14"}
Sep 14 04:47:17.661802 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:17.661776    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zxrd4" event={"ID":"9f897b0d-d9b9-4ae8-b545-ce1f5dfa63c5","Type":"ContainerStarted","Data":"b2dc7692483e4f5e4fa96d351632a23b12f355ba0dbe3c32a808b9a21e725d9c"}
Sep 14 04:47:18.668004 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:18.667973    2565 generic.go:356] "Generic (PLEG): container finished" podID="f6ac6666-f468-4757-8748-38b7e4fa90a9" containerID="f14dbd62a3970652cbca4dc940f7246b35ef830237c20e34277908a27a7df421" exitCode=0
Sep 14 04:47:18.668503 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:18.668017    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9dck" event={"ID":"f6ac6666-f468-4757-8748-38b7e4fa90a9","Type":"ContainerDied","Data":"f14dbd62a3970652cbca4dc940f7246b35ef830237c20e34277908a27a7df421"}
Sep 14 04:47:19.427144 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:19.427102    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb2c56d0-20d7-49ba-857c-febc67911fcd-metrics-tls\") pod \"dns-default-2ms2f\" (UID: \"bb2c56d0-20d7-49ba-857c-febc67911fcd\") " pod="openshift-dns/dns-default-2ms2f"
Sep 14 04:47:19.427312 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:19.427267    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 14 04:47:19.427364 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:19.427345    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb2c56d0-20d7-49ba-857c-febc67911fcd-metrics-tls podName:bb2c56d0-20d7-49ba-857c-febc67911fcd nodeName:}" failed. No retries permitted until 2026-09-14 04:47:23.427323413 +0000 UTC m=+41.603094964 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb2c56d0-20d7-49ba-857c-febc67911fcd-metrics-tls") pod "dns-default-2ms2f" (UID: "bb2c56d0-20d7-49ba-857c-febc67911fcd") : secret "dns-default-metrics-tls" not found
Sep 14 04:47:19.674598 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:19.674516    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9dck" event={"ID":"f6ac6666-f468-4757-8748-38b7e4fa90a9","Type":"ContainerStarted","Data":"ac39dec00d6ade3a9dc00206578756a06b50ef7ae971b25a66ccf0d7d327b30a"}
Sep 14 04:47:19.675964 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:19.675940    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zxrd4" event={"ID":"9f897b0d-d9b9-4ae8-b545-ce1f5dfa63c5","Type":"ContainerStarted","Data":"26e31f8dba785d9e65e830b2bfb4fd04e079632b269c2fd80636f5be552c23c6"}
Sep 14 04:47:19.698852 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:19.698806    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-z9dck" podStartSLOduration=7.921127972 podStartE2EDuration="37.698793327s" totalImagesPullingTime="29.777665355s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 04:46:42 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:47:19.697843616 +0000 UTC m=+37.873615177" watchObservedRunningTime="2026-09-14 04:47:19.698793327 +0000 UTC m=+37.874564885"
Sep 14 04:47:19.714392 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:19.714336    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zxrd4" podStartSLOduration=2.430903865 podStartE2EDuration="4.714320338s" totalImagesPullingTime="2.283416473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:47:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:47:19.71401526 +0000 UTC m=+37.889786819" watchObservedRunningTime="2026-09-14 04:47:19.714320338 +0000 UTC m=+37.890091897"
Sep 14 04:47:23.454936 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:23.454890    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb2c56d0-20d7-49ba-857c-febc67911fcd-metrics-tls\") pod \"dns-default-2ms2f\" (UID: \"bb2c56d0-20d7-49ba-857c-febc67911fcd\") " pod="openshift-dns/dns-default-2ms2f"
Sep 14 04:47:23.455315 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:23.455046    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 14 04:47:23.455315 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:23.455109    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb2c56d0-20d7-49ba-857c-febc67911fcd-metrics-tls podName:bb2c56d0-20d7-49ba-857c-febc67911fcd nodeName:}" failed. No retries permitted until 2026-09-14 04:47:31.45509378 +0000 UTC m=+49.630865317 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb2c56d0-20d7-49ba-857c-febc67911fcd-metrics-tls") pod "dns-default-2ms2f" (UID: "bb2c56d0-20d7-49ba-857c-febc67911fcd") : secret "dns-default-metrics-tls" not found
Sep 14 04:47:31.514346 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:31.514304    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb2c56d0-20d7-49ba-857c-febc67911fcd-metrics-tls\") pod \"dns-default-2ms2f\" (UID: \"bb2c56d0-20d7-49ba-857c-febc67911fcd\") " pod="openshift-dns/dns-default-2ms2f"
Sep 14 04:47:31.514806 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:31.514454    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 14 04:47:31.514806 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:31.514517    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb2c56d0-20d7-49ba-857c-febc67911fcd-metrics-tls podName:bb2c56d0-20d7-49ba-857c-febc67911fcd nodeName:}" failed. No retries permitted until 2026-09-14 04:47:47.514504004 +0000 UTC m=+65.690275541 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb2c56d0-20d7-49ba-857c-febc67911fcd-metrics-tls") pod "dns-default-2ms2f" (UID: "bb2c56d0-20d7-49ba-857c-febc67911fcd") : secret "dns-default-metrics-tls" not found
Sep 14 04:47:41.659768 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:41.659725    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kndxq"
Sep 14 04:47:47.019747 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:47.019704    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ca784040-efa0-4d69-8c25-2b284f3bbe65-original-pull-secret\") pod \"global-pull-secret-syncer-whcnp\" (UID: \"ca784040-efa0-4d69-8c25-2b284f3bbe65\") " pod="kube-system/global-pull-secret-syncer-whcnp"
Sep 14 04:47:47.026713 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:47.026689    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 14 04:47:47.033703 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:47.033680    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ca784040-efa0-4d69-8c25-2b284f3bbe65-original-pull-secret\") pod \"global-pull-secret-syncer-whcnp\" (UID: \"ca784040-efa0-4d69-8c25-2b284f3bbe65\") " pod="kube-system/global-pull-secret-syncer-whcnp"
Sep 14 04:47:47.051012 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:47.050991    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-whcnp"
Sep 14 04:47:47.120337 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:47.120307    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee8a9555-8b90-4ad8-a399-22d414487421-metrics-certs\") pod \"network-metrics-daemon-2gwsl\" (UID: \"ee8a9555-8b90-4ad8-a399-22d414487421\") " pod="openshift-multus/network-metrics-daemon-2gwsl"
Sep 14 04:47:47.120448 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:47.120348    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh4l8\" (UniqueName: \"kubernetes.io/projected/6b6a2ac6-52be-4747-b544-a61782e29691-kube-api-access-wh4l8\") pod \"network-check-target-6sw5h\" (UID: \"6b6a2ac6-52be-4747-b544-a61782e29691\") " pod="openshift-network-diagnostics/network-check-target-6sw5h"
Sep 14 04:47:47.123144 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:47.123123    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 14 04:47:47.123240 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:47.123220    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 14 04:47:47.130545 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:47.130529    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 14 04:47:47.130611 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:47.130584    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee8a9555-8b90-4ad8-a399-22d414487421-metrics-certs podName:ee8a9555-8b90-4ad8-a399-22d414487421 nodeName:}" failed. No retries permitted until 2026-09-14 04:48:51.130570117 +0000 UTC m=+129.306341668 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee8a9555-8b90-4ad8-a399-22d414487421-metrics-certs") pod "network-metrics-daemon-2gwsl" (UID: "ee8a9555-8b90-4ad8-a399-22d414487421") : secret "metrics-daemon-secret" not found
Sep 14 04:47:47.133150 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:47.133134    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 14 04:47:47.142797 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:47.142780    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh4l8\" (UniqueName: \"kubernetes.io/projected/6b6a2ac6-52be-4747-b544-a61782e29691-kube-api-access-wh4l8\") pod \"network-check-target-6sw5h\" (UID: \"6b6a2ac6-52be-4747-b544-a61782e29691\") " pod="openshift-network-diagnostics/network-check-target-6sw5h"
Sep 14 04:47:47.321918 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:47.321847    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-whcnp"]
Sep 14 04:47:47.325199 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:47:47.325177    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca784040_efa0_4d69_8c25_2b284f3bbe65.slice/crio-8ab4a544b1ab3f988f9b182ce8025c16cec335663925f65d40c416198a4fa3d2 WatchSource:0}: Error finding container 8ab4a544b1ab3f988f9b182ce8025c16cec335663925f65d40c416198a4fa3d2: Status 404 returned error can't find the container with id 8ab4a544b1ab3f988f9b182ce8025c16cec335663925f65d40c416198a4fa3d2
Sep 14 04:47:47.340272 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:47.340250    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-54hj4\""
Sep 14 04:47:47.348591 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:47.348572    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6sw5h"
Sep 14 04:47:47.524119 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:47.524090    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb2c56d0-20d7-49ba-857c-febc67911fcd-metrics-tls\") pod \"dns-default-2ms2f\" (UID: \"bb2c56d0-20d7-49ba-857c-febc67911fcd\") " pod="openshift-dns/dns-default-2ms2f"
Sep 14 04:47:47.524263 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:47.524200    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 14 04:47:47.524339 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:47:47.524265    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb2c56d0-20d7-49ba-857c-febc67911fcd-metrics-tls podName:bb2c56d0-20d7-49ba-857c-febc67911fcd nodeName:}" failed. No retries permitted until 2026-09-14 04:48:19.524250574 +0000 UTC m=+97.700022111 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb2c56d0-20d7-49ba-857c-febc67911fcd-metrics-tls") pod "dns-default-2ms2f" (UID: "bb2c56d0-20d7-49ba-857c-febc67911fcd") : secret "dns-default-metrics-tls" not found
Sep 14 04:47:47.613726 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:47.613702    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6sw5h"]
Sep 14 04:47:47.616585 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:47:47.616562    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b6a2ac6_52be_4747_b544_a61782e29691.slice/crio-5e9014a243dd84304d665fa81695843f14b69bf786add190c1928363385f3b00 WatchSource:0}: Error finding container 5e9014a243dd84304d665fa81695843f14b69bf786add190c1928363385f3b00: Status 404 returned error can't find the container with id 5e9014a243dd84304d665fa81695843f14b69bf786add190c1928363385f3b00
Sep 14 04:47:47.742094 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:47.742058    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6sw5h" event={"ID":"6b6a2ac6-52be-4747-b544-a61782e29691","Type":"ContainerStarted","Data":"5e9014a243dd84304d665fa81695843f14b69bf786add190c1928363385f3b00"}
Sep 14 04:47:47.743087 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:47.743065    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-whcnp" event={"ID":"ca784040-efa0-4d69-8c25-2b284f3bbe65","Type":"ContainerStarted","Data":"8ab4a544b1ab3f988f9b182ce8025c16cec335663925f65d40c416198a4fa3d2"}
Sep 14 04:47:52.756419 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:52.756384    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-whcnp" event={"ID":"ca784040-efa0-4d69-8c25-2b284f3bbe65","Type":"ContainerStarted","Data":"9d2a18f0eb71203366d32b149c16d7b7e74353e20e3ed05c64906d447c9357ce"}
Sep 14 04:47:52.757875 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:52.757850    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6sw5h" event={"ID":"6b6a2ac6-52be-4747-b544-a61782e29691","Type":"ContainerStarted","Data":"08596fd9fe6ea9a2cc6d80f74877931363c8ef91ddf23d2c9f44a46e56ebcc6c"}
Sep 14 04:47:52.757990 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:52.757972    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6sw5h"
Sep 14 04:47:52.774709 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:52.774665    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-whcnp" podStartSLOduration=66.008999001 podStartE2EDuration="1m10.774651885s" totalImagesPullingTime="4.765652884s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 04:46:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:47:52.773940567 +0000 UTC m=+70.949712126" watchObservedRunningTime="2026-09-14 04:47:52.774651885 +0000 UTC m=+70.950423442"
Sep 14 04:47:52.790839 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:47:52.790802    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6sw5h" podStartSLOduration=66.320595876 podStartE2EDuration="1m10.7907902s" totalImagesPullingTime="4.470194324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:46:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:47:52.790122426 +0000 UTC m=+70.965894007" watchObservedRunningTime="2026-09-14 04:47:52.7907902 +0000 UTC m=+70.966561759"
Sep 14 04:48:19.531107 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:19.531064    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb2c56d0-20d7-49ba-857c-febc67911fcd-metrics-tls\") pod \"dns-default-2ms2f\" (UID: \"bb2c56d0-20d7-49ba-857c-febc67911fcd\") " pod="openshift-dns/dns-default-2ms2f"
Sep 14 04:48:19.531451 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:19.531184    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 14 04:48:19.531451 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:19.531243    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb2c56d0-20d7-49ba-857c-febc67911fcd-metrics-tls podName:bb2c56d0-20d7-49ba-857c-febc67911fcd nodeName:}" failed. No retries permitted until 2026-09-14 04:49:23.531229369 +0000 UTC m=+161.707000905 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb2c56d0-20d7-49ba-857c-febc67911fcd-metrics-tls") pod "dns-default-2ms2f" (UID: "bb2c56d0-20d7-49ba-857c-febc67911fcd") : secret "dns-default-metrics-tls" not found
Sep 14 04:48:23.762642 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:23.762610    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6sw5h"
Sep 14 04:48:40.886067 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:40.886036    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-vhtjr"]
Sep 14 04:48:40.887738 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:40.887721    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vhtjr"
Sep 14 04:48:40.890243 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:40.890223    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 14 04:48:40.890351 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:40.890283    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 14 04:48:40.891289 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:40.891257    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 14 04:48:40.891386 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:40.891311    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-pjkxs\""
Sep 14 04:48:40.892433 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:40.892415    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 14 04:48:40.900983 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:40.900964    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-vhtjr"]
Sep 14 04:48:40.971019 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:40.970988    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-vhtjr\" (UID: \"ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vhtjr"
Sep 14 04:48:40.971198 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:40.971032    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-vhtjr\" (UID: \"ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vhtjr"
Sep 14 04:48:40.971198 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:40.971055    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5xgw\" (UniqueName: \"kubernetes.io/projected/ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc-kube-api-access-s5xgw\") pod \"cluster-monitoring-operator-6cd954b897-vhtjr\" (UID: \"ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vhtjr"
Sep 14 04:48:41.033657 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:41.033621    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-c9d6db5f8-lqphc"]
Sep 14 04:48:41.038128 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:41.038104    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-c9d6db5f8-lqphc"
Sep 14 04:48:41.040893 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:41.040871    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 14 04:48:41.041135 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:41.041118    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 14 04:48:41.041135 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:41.041127    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 14 04:48:41.041260 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:41.041159    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-8hwgw\""
Sep 14 04:48:41.041260 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:41.041205    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 14 04:48:41.041952 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:41.041932    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 14 04:48:41.042038 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:41.041970    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 14 04:48:41.054274 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:41.054253    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-c9d6db5f8-lqphc"]
Sep 14 04:48:41.072333 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:41.072307    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-vhtjr\" (UID: \"ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vhtjr"
Sep 14 04:48:41.072450 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:41.072334    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5xgw\" (UniqueName: \"kubernetes.io/projected/ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc-kube-api-access-s5xgw\") pod \"cluster-monitoring-operator-6cd954b897-vhtjr\" (UID: \"ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vhtjr"
Sep 14 04:48:41.072450 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:41.072395    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-vhtjr\" (UID: \"ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vhtjr"
Sep 14 04:48:41.072557 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:41.072485    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 14 04:48:41.072557 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:41.072542    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc-cluster-monitoring-operator-tls podName:ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc nodeName:}" failed. No retries permitted until 2026-09-14 04:48:41.572529633 +0000 UTC m=+119.748301170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-vhtjr" (UID: "ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc") : secret "cluster-monitoring-operator-tls" not found
Sep 14 04:48:41.073050 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:41.073026    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-vhtjr\" (UID: \"ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vhtjr"
Sep 14 04:48:41.080696 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:41.080675    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5xgw\" (UniqueName: \"kubernetes.io/projected/ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc-kube-api-access-s5xgw\") pod \"cluster-monitoring-operator-6cd954b897-vhtjr\" (UID: \"ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vhtjr"
Sep 14 04:48:41.173131 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:41.173060    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/93a9bb73-aa22-4294-b119-20aa06f4bfde-default-certificate\") pod \"router-default-c9d6db5f8-lqphc\" (UID: \"93a9bb73-aa22-4294-b119-20aa06f4bfde\") " pod="openshift-ingress/router-default-c9d6db5f8-lqphc"
Sep 14 04:48:41.173131 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:41.173093    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/93a9bb73-aa22-4294-b119-20aa06f4bfde-stats-auth\") pod \"router-default-c9d6db5f8-lqphc\" (UID: \"93a9bb73-aa22-4294-b119-20aa06f4bfde\") " pod="openshift-ingress/router-default-c9d6db5f8-lqphc"
Sep 14 04:48:41.173131 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:41.173110    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93a9bb73-aa22-4294-b119-20aa06f4bfde-metrics-certs\") pod \"router-default-c9d6db5f8-lqphc\" (UID: \"93a9bb73-aa22-4294-b119-20aa06f4bfde\") " pod="openshift-ingress/router-default-c9d6db5f8-lqphc"
Sep 14 04:48:41.173332 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:41.173135    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93a9bb73-aa22-4294-b119-20aa06f4bfde-service-ca-bundle\") pod \"router-default-c9d6db5f8-lqphc\" (UID: \"93a9bb73-aa22-4294-b119-20aa06f4bfde\") " pod="openshift-ingress/router-default-c9d6db5f8-lqphc"
Sep 14 04:48:41.173332 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:41.173192    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5925\" (UniqueName: \"kubernetes.io/projected/93a9bb73-aa22-4294-b119-20aa06f4bfde-kube-api-access-t5925\") pod \"router-default-c9d6db5f8-lqphc\" (UID: \"93a9bb73-aa22-4294-b119-20aa06f4bfde\") " pod="openshift-ingress/router-default-c9d6db5f8-lqphc"
Sep 14 04:48:41.273638 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:41.273594    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93a9bb73-aa22-4294-b119-20aa06f4bfde-service-ca-bundle\") pod \"router-default-c9d6db5f8-lqphc\" (UID: \"93a9bb73-aa22-4294-b119-20aa06f4bfde\") " pod="openshift-ingress/router-default-c9d6db5f8-lqphc"
Sep 14 04:48:41.273784 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:41.273652    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5925\" (UniqueName: \"kubernetes.io/projected/93a9bb73-aa22-4294-b119-20aa06f4bfde-kube-api-access-t5925\") pod \"router-default-c9d6db5f8-lqphc\" (UID: \"93a9bb73-aa22-4294-b119-20aa06f4bfde\") " pod="openshift-ingress/router-default-c9d6db5f8-lqphc"
Sep 14 04:48:41.273784 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:41.273724    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/93a9bb73-aa22-4294-b119-20aa06f4bfde-default-certificate\") pod \"router-default-c9d6db5f8-lqphc\" (UID: \"93a9bb73-aa22-4294-b119-20aa06f4bfde\") " pod="openshift-ingress/router-default-c9d6db5f8-lqphc"
Sep 14 04:48:41.273784 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:41.273772    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/93a9bb73-aa22-4294-b119-20aa06f4bfde-stats-auth\") pod \"router-default-c9d6db5f8-lqphc\" (UID: \"93a9bb73-aa22-4294-b119-20aa06f4bfde\") " pod="openshift-ingress/router-default-c9d6db5f8-lqphc"
Sep 14 04:48:41.273945 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:41.273800    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93a9bb73-aa22-4294-b119-20aa06f4bfde-metrics-certs\") pod \"router-default-c9d6db5f8-lqphc\" (UID: \"93a9bb73-aa22-4294-b119-20aa06f4bfde\") " pod="openshift-ingress/router-default-c9d6db5f8-lqphc"
Sep 14 04:48:41.273945 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:41.273875    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/93a9bb73-aa22-4294-b119-20aa06f4bfde-service-ca-bundle podName:93a9bb73-aa22-4294-b119-20aa06f4bfde nodeName:}" failed. No retries permitted until 2026-09-14 04:48:41.773845281 +0000 UTC m=+119.949616832 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/93a9bb73-aa22-4294-b119-20aa06f4bfde-service-ca-bundle") pod "router-default-c9d6db5f8-lqphc" (UID: "93a9bb73-aa22-4294-b119-20aa06f4bfde") : configmap references non-existent config key: service-ca.crt
Sep 14 04:48:41.273945 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:41.273908    2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 14 04:48:41.274068 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:41.273970    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93a9bb73-aa22-4294-b119-20aa06f4bfde-metrics-certs podName:93a9bb73-aa22-4294-b119-20aa06f4bfde nodeName:}" failed. No retries permitted until 2026-09-14 04:48:41.773953274 +0000 UTC m=+119.949724818 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93a9bb73-aa22-4294-b119-20aa06f4bfde-metrics-certs") pod "router-default-c9d6db5f8-lqphc" (UID: "93a9bb73-aa22-4294-b119-20aa06f4bfde") : secret "router-metrics-certs-default" not found
Sep 14 04:48:41.276235 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:41.276208    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/93a9bb73-aa22-4294-b119-20aa06f4bfde-default-certificate\") pod \"router-default-c9d6db5f8-lqphc\" (UID: \"93a9bb73-aa22-4294-b119-20aa06f4bfde\") " pod="openshift-ingress/router-default-c9d6db5f8-lqphc"
Sep 14 04:48:41.276351 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:41.276242    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/93a9bb73-aa22-4294-b119-20aa06f4bfde-stats-auth\") pod \"router-default-c9d6db5f8-lqphc\" (UID: \"93a9bb73-aa22-4294-b119-20aa06f4bfde\") " pod="openshift-ingress/router-default-c9d6db5f8-lqphc"
Sep 14 04:48:41.285304 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:41.285283    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5925\" (UniqueName: \"kubernetes.io/projected/93a9bb73-aa22-4294-b119-20aa06f4bfde-kube-api-access-t5925\") pod \"router-default-c9d6db5f8-lqphc\" (UID: \"93a9bb73-aa22-4294-b119-20aa06f4bfde\") " pod="openshift-ingress/router-default-c9d6db5f8-lqphc"
Sep 14 04:48:41.576429 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:41.576349    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-vhtjr\" (UID: \"ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vhtjr"
Sep 14 04:48:41.576568 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:41.576467    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 14 04:48:41.576568 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:41.576529    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc-cluster-monitoring-operator-tls podName:ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc nodeName:}" failed. No retries permitted until 2026-09-14 04:48:42.576515545 +0000 UTC m=+120.752287100 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-vhtjr" (UID: "ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc") : secret "cluster-monitoring-operator-tls" not found
Sep 14 04:48:41.777673 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:41.777637    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93a9bb73-aa22-4294-b119-20aa06f4bfde-metrics-certs\") pod \"router-default-c9d6db5f8-lqphc\" (UID: \"93a9bb73-aa22-4294-b119-20aa06f4bfde\") " pod="openshift-ingress/router-default-c9d6db5f8-lqphc"
Sep 14 04:48:41.777857 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:41.777681    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93a9bb73-aa22-4294-b119-20aa06f4bfde-service-ca-bundle\") pod \"router-default-c9d6db5f8-lqphc\" (UID: \"93a9bb73-aa22-4294-b119-20aa06f4bfde\") " pod="openshift-ingress/router-default-c9d6db5f8-lqphc"
Sep 14 04:48:41.777857 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:41.777842    2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 14 04:48:41.777931 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:41.777868    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/93a9bb73-aa22-4294-b119-20aa06f4bfde-service-ca-bundle podName:93a9bb73-aa22-4294-b119-20aa06f4bfde nodeName:}" failed. No retries permitted until 2026-09-14 04:48:42.777846785 +0000 UTC m=+120.953618322 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/93a9bb73-aa22-4294-b119-20aa06f4bfde-service-ca-bundle") pod "router-default-c9d6db5f8-lqphc" (UID: "93a9bb73-aa22-4294-b119-20aa06f4bfde") : configmap references non-existent config key: service-ca.crt
Sep 14 04:48:41.777931 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:41.777907    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93a9bb73-aa22-4294-b119-20aa06f4bfde-metrics-certs podName:93a9bb73-aa22-4294-b119-20aa06f4bfde nodeName:}" failed. No retries permitted until 2026-09-14 04:48:42.777893807 +0000 UTC m=+120.953665345 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93a9bb73-aa22-4294-b119-20aa06f4bfde-metrics-certs") pod "router-default-c9d6db5f8-lqphc" (UID: "93a9bb73-aa22-4294-b119-20aa06f4bfde") : secret "router-metrics-certs-default" not found
Sep 14 04:48:42.583288 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:42.583248    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-vhtjr\" (UID: \"ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vhtjr"
Sep 14 04:48:42.583658 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:42.583396    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 14 04:48:42.583658 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:42.583459    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc-cluster-monitoring-operator-tls podName:ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc nodeName:}" failed. No retries permitted until 2026-09-14 04:48:44.583441063 +0000 UTC m=+122.759212621 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-vhtjr" (UID: "ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc") : secret "cluster-monitoring-operator-tls" not found
Sep 14 04:48:42.785523 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:42.785478    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93a9bb73-aa22-4294-b119-20aa06f4bfde-metrics-certs\") pod \"router-default-c9d6db5f8-lqphc\" (UID: \"93a9bb73-aa22-4294-b119-20aa06f4bfde\") " pod="openshift-ingress/router-default-c9d6db5f8-lqphc"
Sep 14 04:48:42.785523 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:42.785529    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93a9bb73-aa22-4294-b119-20aa06f4bfde-service-ca-bundle\") pod \"router-default-c9d6db5f8-lqphc\" (UID: \"93a9bb73-aa22-4294-b119-20aa06f4bfde\") " pod="openshift-ingress/router-default-c9d6db5f8-lqphc"
Sep 14 04:48:42.785793 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:42.785627    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/93a9bb73-aa22-4294-b119-20aa06f4bfde-service-ca-bundle podName:93a9bb73-aa22-4294-b119-20aa06f4bfde nodeName:}" failed. No retries permitted until 2026-09-14 04:48:44.785614292 +0000 UTC m=+122.961385829 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/93a9bb73-aa22-4294-b119-20aa06f4bfde-service-ca-bundle") pod "router-default-c9d6db5f8-lqphc" (UID: "93a9bb73-aa22-4294-b119-20aa06f4bfde") : configmap references non-existent config key: service-ca.crt
Sep 14 04:48:42.785793 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:42.785635    2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 14 04:48:42.785793 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:42.785697    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93a9bb73-aa22-4294-b119-20aa06f4bfde-metrics-certs podName:93a9bb73-aa22-4294-b119-20aa06f4bfde nodeName:}" failed. No retries permitted until 2026-09-14 04:48:44.785680568 +0000 UTC m=+122.961452123 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93a9bb73-aa22-4294-b119-20aa06f4bfde-metrics-certs") pod "router-default-c9d6db5f8-lqphc" (UID: "93a9bb73-aa22-4294-b119-20aa06f4bfde") : secret "router-metrics-certs-default" not found
Sep 14 04:48:44.597682 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:44.597644    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-vhtjr\" (UID: \"ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vhtjr"
Sep 14 04:48:44.598149 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:44.597845    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 14 04:48:44.598149 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:44.597930    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc-cluster-monitoring-operator-tls podName:ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc nodeName:}" failed. No retries permitted until 2026-09-14 04:48:48.597908479 +0000 UTC m=+126.773680031 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-vhtjr" (UID: "ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc") : secret "cluster-monitoring-operator-tls" not found
Sep 14 04:48:44.799037 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:44.799003    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93a9bb73-aa22-4294-b119-20aa06f4bfde-metrics-certs\") pod \"router-default-c9d6db5f8-lqphc\" (UID: \"93a9bb73-aa22-4294-b119-20aa06f4bfde\") " pod="openshift-ingress/router-default-c9d6db5f8-lqphc"
Sep 14 04:48:44.799037 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:44.799042    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93a9bb73-aa22-4294-b119-20aa06f4bfde-service-ca-bundle\") pod \"router-default-c9d6db5f8-lqphc\" (UID: \"93a9bb73-aa22-4294-b119-20aa06f4bfde\") " pod="openshift-ingress/router-default-c9d6db5f8-lqphc"
Sep 14 04:48:44.799267 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:44.799158    2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 14 04:48:44.799267 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:44.799165    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/93a9bb73-aa22-4294-b119-20aa06f4bfde-service-ca-bundle podName:93a9bb73-aa22-4294-b119-20aa06f4bfde nodeName:}" failed. No retries permitted until 2026-09-14 04:48:48.799152451 +0000 UTC m=+126.974924002 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/93a9bb73-aa22-4294-b119-20aa06f4bfde-service-ca-bundle") pod "router-default-c9d6db5f8-lqphc" (UID: "93a9bb73-aa22-4294-b119-20aa06f4bfde") : configmap references non-existent config key: service-ca.crt
Sep 14 04:48:44.799267 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:44.799226    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93a9bb73-aa22-4294-b119-20aa06f4bfde-metrics-certs podName:93a9bb73-aa22-4294-b119-20aa06f4bfde nodeName:}" failed. No retries permitted until 2026-09-14 04:48:48.799210301 +0000 UTC m=+126.974981850 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93a9bb73-aa22-4294-b119-20aa06f4bfde-metrics-certs") pod "router-default-c9d6db5f8-lqphc" (UID: "93a9bb73-aa22-4294-b119-20aa06f4bfde") : secret "router-metrics-certs-default" not found
Sep 14 04:48:45.054381 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:45.054307    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2bkxf"]
Sep 14 04:48:45.056277 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:45.056262    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2bkxf"
Sep 14 04:48:45.058828 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:45.058795    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 14 04:48:45.058997 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:45.058979    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 14 04:48:45.059681 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:45.059664    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 14 04:48:45.059783 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:45.059685    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-d8h7x\""
Sep 14 04:48:45.059783 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:45.059751    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 14 04:48:45.079316 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:45.079292    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2bkxf"]
Sep 14 04:48:45.203456 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:45.203425    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2bkxf\" (UID: \"3bbc1798-b60c-4a19-bab3-c463a8bdd51a\") " pod="openshift-insights/insights-runtime-extractor-2bkxf"
Sep 14 04:48:45.203456 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:45.203455    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nh27\" (UniqueName: \"kubernetes.io/projected/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-kube-api-access-5nh27\") pod \"insights-runtime-extractor-2bkxf\" (UID: \"3bbc1798-b60c-4a19-bab3-c463a8bdd51a\") " pod="openshift-insights/insights-runtime-extractor-2bkxf"
Sep 14 04:48:45.203641 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:45.203480    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-data-volume\") pod \"insights-runtime-extractor-2bkxf\" (UID: \"3bbc1798-b60c-4a19-bab3-c463a8bdd51a\") " pod="openshift-insights/insights-runtime-extractor-2bkxf"
Sep 14 04:48:45.203641 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:45.203597    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-crio-socket\") pod \"insights-runtime-extractor-2bkxf\" (UID: \"3bbc1798-b60c-4a19-bab3-c463a8bdd51a\") " pod="openshift-insights/insights-runtime-extractor-2bkxf"
Sep 14 04:48:45.203641 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:45.203618    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2bkxf\" (UID: \"3bbc1798-b60c-4a19-bab3-c463a8bdd51a\") " pod="openshift-insights/insights-runtime-extractor-2bkxf"
Sep 14 04:48:45.304537 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:45.304487    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-crio-socket\") pod \"insights-runtime-extractor-2bkxf\" (UID: \"3bbc1798-b60c-4a19-bab3-c463a8bdd51a\") " pod="openshift-insights/insights-runtime-extractor-2bkxf"
Sep 14 04:48:45.304537 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:45.304515    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2bkxf\" (UID: \"3bbc1798-b60c-4a19-bab3-c463a8bdd51a\") " pod="openshift-insights/insights-runtime-extractor-2bkxf"
Sep 14 04:48:45.304537 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:45.304535    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2bkxf\" (UID: \"3bbc1798-b60c-4a19-bab3-c463a8bdd51a\") " pod="openshift-insights/insights-runtime-extractor-2bkxf"
Sep 14 04:48:45.304710 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:45.304550    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nh27\" (UniqueName: \"kubernetes.io/projected/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-kube-api-access-5nh27\") pod \"insights-runtime-extractor-2bkxf\" (UID: \"3bbc1798-b60c-4a19-bab3-c463a8bdd51a\") " pod="openshift-insights/insights-runtime-extractor-2bkxf"
Sep 14 04:48:45.304710 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:45.304566    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-data-volume\") pod \"insights-runtime-extractor-2bkxf\" (UID: \"3bbc1798-b60c-4a19-bab3-c463a8bdd51a\") " pod="openshift-insights/insights-runtime-extractor-2bkxf"
Sep 14 04:48:45.304710 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:45.304608    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-crio-socket\") pod \"insights-runtime-extractor-2bkxf\" (UID: \"3bbc1798-b60c-4a19-bab3-c463a8bdd51a\") " pod="openshift-insights/insights-runtime-extractor-2bkxf"
Sep 14 04:48:45.304710 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:45.304633    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 14 04:48:45.304710 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:45.304643    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 14 04:48:45.304710 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:45.304706    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-kube-rbac-proxy-cm podName:3bbc1798-b60c-4a19-bab3-c463a8bdd51a nodeName:}" failed. No retries permitted until 2026-09-14 04:48:45.804686944 +0000 UTC m=+123.980458485 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2bkxf" (UID: "3bbc1798-b60c-4a19-bab3-c463a8bdd51a") : configmap "kube-rbac-proxy" not found
Sep 14 04:48:45.304939 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:45.304727    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-insights-runtime-extractor-tls podName:3bbc1798-b60c-4a19-bab3-c463a8bdd51a nodeName:}" failed. No retries permitted until 2026-09-14 04:48:45.804717123 +0000 UTC m=+123.980488669 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2bkxf" (UID: "3bbc1798-b60c-4a19-bab3-c463a8bdd51a") : secret "insights-runtime-extractor-tls" not found
Sep 14 04:48:45.304939 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:45.304870    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-data-volume\") pod \"insights-runtime-extractor-2bkxf\" (UID: \"3bbc1798-b60c-4a19-bab3-c463a8bdd51a\") " pod="openshift-insights/insights-runtime-extractor-2bkxf"
Sep 14 04:48:45.315133 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:45.315113    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nh27\" (UniqueName: \"kubernetes.io/projected/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-kube-api-access-5nh27\") pod \"insights-runtime-extractor-2bkxf\" (UID: \"3bbc1798-b60c-4a19-bab3-c463a8bdd51a\") " pod="openshift-insights/insights-runtime-extractor-2bkxf"
Sep 14 04:48:45.808357 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:45.808321    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2bkxf\" (UID: \"3bbc1798-b60c-4a19-bab3-c463a8bdd51a\") " pod="openshift-insights/insights-runtime-extractor-2bkxf"
Sep 14 04:48:45.808357 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:45.808361    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2bkxf\" (UID: \"3bbc1798-b60c-4a19-bab3-c463a8bdd51a\") " pod="openshift-insights/insights-runtime-extractor-2bkxf"
Sep 14 04:48:45.808778 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:45.808474    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 14 04:48:45.808778 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:45.808482    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 14 04:48:45.808778 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:45.808535    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-insights-runtime-extractor-tls podName:3bbc1798-b60c-4a19-bab3-c463a8bdd51a nodeName:}" failed. No retries permitted until 2026-09-14 04:48:46.808521506 +0000 UTC m=+124.984293049 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2bkxf" (UID: "3bbc1798-b60c-4a19-bab3-c463a8bdd51a") : secret "insights-runtime-extractor-tls" not found
Sep 14 04:48:45.808778 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:45.808552    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-kube-rbac-proxy-cm podName:3bbc1798-b60c-4a19-bab3-c463a8bdd51a nodeName:}" failed. No retries permitted until 2026-09-14 04:48:46.808545157 +0000 UTC m=+124.984316694 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2bkxf" (UID: "3bbc1798-b60c-4a19-bab3-c463a8bdd51a") : configmap "kube-rbac-proxy" not found
Sep 14 04:48:46.107466 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:46.107435    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-7cxw6"]
Sep 14 04:48:46.109254 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:46.109239    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-7cxw6"
Sep 14 04:48:46.111611 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:46.111589    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 14 04:48:46.111811 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:46.111792    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 14 04:48:46.112409 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:46.112394    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-s62mx\""
Sep 14 04:48:46.124120 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:46.124097    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-7cxw6"]
Sep 14 04:48:46.211392 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:46.211358    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdlk7\" (UniqueName: \"kubernetes.io/projected/3684d0a2-72f8-48fa-92aa-45f04832a96d-kube-api-access-wdlk7\") pod \"migrator-8b4d697c5-7cxw6\" (UID: \"3684d0a2-72f8-48fa-92aa-45f04832a96d\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-7cxw6"
Sep 14 04:48:46.312399 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:46.312351    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdlk7\" (UniqueName: \"kubernetes.io/projected/3684d0a2-72f8-48fa-92aa-45f04832a96d-kube-api-access-wdlk7\") pod \"migrator-8b4d697c5-7cxw6\" (UID: \"3684d0a2-72f8-48fa-92aa-45f04832a96d\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-7cxw6"
Sep 14 04:48:46.320505 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:46.320476    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdlk7\" (UniqueName: \"kubernetes.io/projected/3684d0a2-72f8-48fa-92aa-45f04832a96d-kube-api-access-wdlk7\") pod \"migrator-8b4d697c5-7cxw6\" (UID: \"3684d0a2-72f8-48fa-92aa-45f04832a96d\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-7cxw6"
Sep 14 04:48:46.417787 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:46.417703    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-7cxw6"
Sep 14 04:48:46.684182 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:46.684106    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-7cxw6"]
Sep 14 04:48:46.686621 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:48:46.686582    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3684d0a2_72f8_48fa_92aa_45f04832a96d.slice/crio-cd8671d1cf8c25b957f6553276c69d8a00167560b07ff220813592df19da8911 WatchSource:0}: Error finding container cd8671d1cf8c25b957f6553276c69d8a00167560b07ff220813592df19da8911: Status 404 returned error can't find the container with id cd8671d1cf8c25b957f6553276c69d8a00167560b07ff220813592df19da8911
Sep 14 04:48:46.815626 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:46.815590    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2bkxf\" (UID: \"3bbc1798-b60c-4a19-bab3-c463a8bdd51a\") " pod="openshift-insights/insights-runtime-extractor-2bkxf"
Sep 14 04:48:46.815626 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:46.815627    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2bkxf\" (UID: \"3bbc1798-b60c-4a19-bab3-c463a8bdd51a\") " pod="openshift-insights/insights-runtime-extractor-2bkxf"
Sep 14 04:48:46.816047 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:46.815749    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 14 04:48:46.816047 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:46.815804    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 14 04:48:46.816047 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:46.815849    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-insights-runtime-extractor-tls podName:3bbc1798-b60c-4a19-bab3-c463a8bdd51a nodeName:}" failed. No retries permitted until 2026-09-14 04:48:48.815831678 +0000 UTC m=+126.991603215 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2bkxf" (UID: "3bbc1798-b60c-4a19-bab3-c463a8bdd51a") : secret "insights-runtime-extractor-tls" not found
Sep 14 04:48:46.816047 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:46.815865    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-kube-rbac-proxy-cm podName:3bbc1798-b60c-4a19-bab3-c463a8bdd51a nodeName:}" failed. No retries permitted until 2026-09-14 04:48:48.81585753 +0000 UTC m=+126.991629066 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2bkxf" (UID: "3bbc1798-b60c-4a19-bab3-c463a8bdd51a") : configmap "kube-rbac-proxy" not found
Sep 14 04:48:46.884796 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:46.884750    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-7cxw6" event={"ID":"3684d0a2-72f8-48fa-92aa-45f04832a96d","Type":"ContainerStarted","Data":"cd8671d1cf8c25b957f6553276c69d8a00167560b07ff220813592df19da8911"}
Sep 14 04:48:47.175493 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:47.175460    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rqw8q_0cfe37ed-e18d-4e25-84d6-cba59e2b2d81/dns-node-resolver/0.log"
Sep 14 04:48:47.813368 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:47.813343    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-sc7m8"]
Sep 14 04:48:47.815148 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:47.815134    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-sc7m8"
Sep 14 04:48:47.817981 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:47.817960    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 14 04:48:47.818353 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:47.818042    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 14 04:48:47.819054 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:47.819030    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-m2tlx\""
Sep 14 04:48:47.819130 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:47.819059    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 14 04:48:47.819130 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:47.819065    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 14 04:48:47.835748 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:47.835725    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-sc7m8"]
Sep 14 04:48:47.891460 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:47.891430    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-7cxw6" event={"ID":"3684d0a2-72f8-48fa-92aa-45f04832a96d","Type":"ContainerStarted","Data":"328289e32e7a2babadf9c7310b3a6283359ee2636a129b7bb02a67baa7d31cbf"}
Sep 14 04:48:47.891563 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:47.891474    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-7cxw6" event={"ID":"3684d0a2-72f8-48fa-92aa-45f04832a96d","Type":"ContainerStarted","Data":"6e2bb3b13ad103173a07805a336f5a9e40a650e990faf5111abf056e95d2ac7a"}
Sep 14 04:48:47.911171 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:47.911127    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-7cxw6" podStartSLOduration=0.934636351 podStartE2EDuration="1.911113169s" totalImagesPullingTime="976.476818ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:48:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:48:47.91067788 +0000 UTC m=+126.086449433" watchObservedRunningTime="2026-09-14 04:48:47.911113169 +0000 UTC m=+126.086884728"
Sep 14 04:48:47.926078 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:47.926028    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/00a0e388-48c4-46cd-8ab8-32fb46151779-signing-key\") pod \"service-ca-568869fdc5-sc7m8\" (UID: \"00a0e388-48c4-46cd-8ab8-32fb46151779\") " pod="openshift-service-ca/service-ca-568869fdc5-sc7m8"
Sep 14 04:48:47.926148 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:47.926085    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/00a0e388-48c4-46cd-8ab8-32fb46151779-signing-cabundle\") pod \"service-ca-568869fdc5-sc7m8\" (UID: \"00a0e388-48c4-46cd-8ab8-32fb46151779\") " pod="openshift-service-ca/service-ca-568869fdc5-sc7m8"
Sep 14 04:48:47.926148 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:47.926126    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwnn7\" (UniqueName: \"kubernetes.io/projected/00a0e388-48c4-46cd-8ab8-32fb46151779-kube-api-access-wwnn7\") pod \"service-ca-568869fdc5-sc7m8\" (UID: \"00a0e388-48c4-46cd-8ab8-32fb46151779\") " pod="openshift-service-ca/service-ca-568869fdc5-sc7m8"
Sep 14 04:48:48.027280 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:48.027251    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/00a0e388-48c4-46cd-8ab8-32fb46151779-signing-cabundle\") pod \"service-ca-568869fdc5-sc7m8\" (UID: \"00a0e388-48c4-46cd-8ab8-32fb46151779\") " pod="openshift-service-ca/service-ca-568869fdc5-sc7m8"
Sep 14 04:48:48.027367 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:48.027285    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwnn7\" (UniqueName: \"kubernetes.io/projected/00a0e388-48c4-46cd-8ab8-32fb46151779-kube-api-access-wwnn7\") pod \"service-ca-568869fdc5-sc7m8\" (UID: \"00a0e388-48c4-46cd-8ab8-32fb46151779\") " pod="openshift-service-ca/service-ca-568869fdc5-sc7m8"
Sep 14 04:48:48.027367 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:48.027348    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/00a0e388-48c4-46cd-8ab8-32fb46151779-signing-key\") pod \"service-ca-568869fdc5-sc7m8\" (UID: \"00a0e388-48c4-46cd-8ab8-32fb46151779\") " pod="openshift-service-ca/service-ca-568869fdc5-sc7m8"
Sep 14 04:48:48.028037 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:48.028018    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/00a0e388-48c4-46cd-8ab8-32fb46151779-signing-cabundle\") pod \"service-ca-568869fdc5-sc7m8\" (UID: \"00a0e388-48c4-46cd-8ab8-32fb46151779\") " pod="openshift-service-ca/service-ca-568869fdc5-sc7m8"
Sep 14 04:48:48.029750 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:48.029727    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/00a0e388-48c4-46cd-8ab8-32fb46151779-signing-key\") pod \"service-ca-568869fdc5-sc7m8\" (UID: \"00a0e388-48c4-46cd-8ab8-32fb46151779\") " pod="openshift-service-ca/service-ca-568869fdc5-sc7m8"
Sep 14 04:48:48.035445 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:48.035420    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwnn7\" (UniqueName: \"kubernetes.io/projected/00a0e388-48c4-46cd-8ab8-32fb46151779-kube-api-access-wwnn7\") pod \"service-ca-568869fdc5-sc7m8\" (UID: \"00a0e388-48c4-46cd-8ab8-32fb46151779\") " pod="openshift-service-ca/service-ca-568869fdc5-sc7m8"
Sep 14 04:48:48.126743 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:48.126716    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-sc7m8"
Sep 14 04:48:48.179303 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:48.179132    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pfl42_7f7e9f24-ddd1-4c3b-acdf-371ea4c648e8/node-ca/0.log"
Sep 14 04:48:48.394700 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:48.394674    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-sc7m8"]
Sep 14 04:48:48.398859 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:48:48.398833    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00a0e388_48c4_46cd_8ab8_32fb46151779.slice/crio-02ea5693dafa5b854e3a4a514e2e60f568ded9e8c52528ab67b0249f0e40a650 WatchSource:0}: Error finding container 02ea5693dafa5b854e3a4a514e2e60f568ded9e8c52528ab67b0249f0e40a650: Status 404 returned error can't find the container with id 02ea5693dafa5b854e3a4a514e2e60f568ded9e8c52528ab67b0249f0e40a650
Sep 14 04:48:48.631929 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:48.631903    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-vhtjr\" (UID: \"ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vhtjr"
Sep 14 04:48:48.632067 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:48.632044    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 14 04:48:48.632117 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:48.632107    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc-cluster-monitoring-operator-tls podName:ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc nodeName:}" failed. No retries permitted until 2026-09-14 04:48:56.632092454 +0000 UTC m=+134.807863995 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-vhtjr" (UID: "ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc") : secret "cluster-monitoring-operator-tls" not found
Sep 14 04:48:48.833450 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:48.833410    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93a9bb73-aa22-4294-b119-20aa06f4bfde-metrics-certs\") pod \"router-default-c9d6db5f8-lqphc\" (UID: \"93a9bb73-aa22-4294-b119-20aa06f4bfde\") " pod="openshift-ingress/router-default-c9d6db5f8-lqphc"
Sep 14 04:48:48.833922 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:48.833472    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93a9bb73-aa22-4294-b119-20aa06f4bfde-service-ca-bundle\") pod \"router-default-c9d6db5f8-lqphc\" (UID: \"93a9bb73-aa22-4294-b119-20aa06f4bfde\") " pod="openshift-ingress/router-default-c9d6db5f8-lqphc"
Sep 14 04:48:48.833922 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:48.833562    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2bkxf\" (UID: \"3bbc1798-b60c-4a19-bab3-c463a8bdd51a\") " pod="openshift-insights/insights-runtime-extractor-2bkxf"
Sep 14 04:48:48.833922 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:48.833594    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2bkxf\" (UID: \"3bbc1798-b60c-4a19-bab3-c463a8bdd51a\") " pod="openshift-insights/insights-runtime-extractor-2bkxf"
Sep 14 04:48:48.833922 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:48.833595    2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 14 04:48:48.833922 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:48.833651    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93a9bb73-aa22-4294-b119-20aa06f4bfde-metrics-certs podName:93a9bb73-aa22-4294-b119-20aa06f4bfde nodeName:}" failed. No retries permitted until 2026-09-14 04:48:56.833632244 +0000 UTC m=+135.009403800 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93a9bb73-aa22-4294-b119-20aa06f4bfde-metrics-certs") pod "router-default-c9d6db5f8-lqphc" (UID: "93a9bb73-aa22-4294-b119-20aa06f4bfde") : secret "router-metrics-certs-default" not found
Sep 14 04:48:48.833922 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:48.833672    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/93a9bb73-aa22-4294-b119-20aa06f4bfde-service-ca-bundle podName:93a9bb73-aa22-4294-b119-20aa06f4bfde nodeName:}" failed. No retries permitted until 2026-09-14 04:48:56.833663682 +0000 UTC m=+135.009435218 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/93a9bb73-aa22-4294-b119-20aa06f4bfde-service-ca-bundle") pod "router-default-c9d6db5f8-lqphc" (UID: "93a9bb73-aa22-4294-b119-20aa06f4bfde") : configmap references non-existent config key: service-ca.crt
Sep 14 04:48:48.833922 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:48.833677    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 14 04:48:48.833922 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:48.833703    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 14 04:48:48.833922 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:48.833747    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-kube-rbac-proxy-cm podName:3bbc1798-b60c-4a19-bab3-c463a8bdd51a nodeName:}" failed. No retries permitted until 2026-09-14 04:48:52.833731182 +0000 UTC m=+131.009502736 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2bkxf" (UID: "3bbc1798-b60c-4a19-bab3-c463a8bdd51a") : configmap "kube-rbac-proxy" not found
Sep 14 04:48:48.833922 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:48.833775    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-insights-runtime-extractor-tls podName:3bbc1798-b60c-4a19-bab3-c463a8bdd51a nodeName:}" failed. No retries permitted until 2026-09-14 04:48:52.833766256 +0000 UTC m=+131.009537793 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2bkxf" (UID: "3bbc1798-b60c-4a19-bab3-c463a8bdd51a") : secret "insights-runtime-extractor-tls" not found
Sep 14 04:48:48.896741 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:48.896657    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-sc7m8" event={"ID":"00a0e388-48c4-46cd-8ab8-32fb46151779","Type":"ContainerStarted","Data":"02ea5693dafa5b854e3a4a514e2e60f568ded9e8c52528ab67b0249f0e40a650"}
Sep 14 04:48:48.975460 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:48.975427    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zxrd4_9f897b0d-d9b9-4ae8-b545-ce1f5dfa63c5/serve-healthcheck-canary/0.log"
Sep 14 04:48:50.904788 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:50.904736    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-sc7m8" event={"ID":"00a0e388-48c4-46cd-8ab8-32fb46151779","Type":"ContainerStarted","Data":"d865a778e96c69516e19b331396d8633db766569dc599f083e0ba957b23811f5"}
Sep 14 04:48:50.922896 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:50.922848    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-sc7m8" podStartSLOduration=2.204848853 podStartE2EDuration="3.922833083s" totalImagesPullingTime="1.71798423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:48:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:48:50.921589546 +0000 UTC m=+129.097361098" watchObservedRunningTime="2026-09-14 04:48:50.922833083 +0000 UTC m=+129.098604641"
Sep 14 04:48:51.153381 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:51.153349    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee8a9555-8b90-4ad8-a399-22d414487421-metrics-certs\") pod \"network-metrics-daemon-2gwsl\" (UID: \"ee8a9555-8b90-4ad8-a399-22d414487421\") " pod="openshift-multus/network-metrics-daemon-2gwsl"
Sep 14 04:48:51.153533 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:51.153455    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 14 04:48:51.153533 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:51.153504    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee8a9555-8b90-4ad8-a399-22d414487421-metrics-certs podName:ee8a9555-8b90-4ad8-a399-22d414487421 nodeName:}" failed. No retries permitted until 2026-09-14 04:50:53.153491717 +0000 UTC m=+251.329263253 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee8a9555-8b90-4ad8-a399-22d414487421-metrics-certs") pod "network-metrics-daemon-2gwsl" (UID: "ee8a9555-8b90-4ad8-a399-22d414487421") : secret "metrics-daemon-secret" not found
Sep 14 04:48:52.865400 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:52.865362    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2bkxf\" (UID: \"3bbc1798-b60c-4a19-bab3-c463a8bdd51a\") " pod="openshift-insights/insights-runtime-extractor-2bkxf"
Sep 14 04:48:52.865400 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:52.865404    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2bkxf\" (UID: \"3bbc1798-b60c-4a19-bab3-c463a8bdd51a\") " pod="openshift-insights/insights-runtime-extractor-2bkxf"
Sep 14 04:48:52.865885 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:52.865479    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 14 04:48:52.865885 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:52.865525    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-kube-rbac-proxy-cm podName:3bbc1798-b60c-4a19-bab3-c463a8bdd51a nodeName:}" failed. No retries permitted until 2026-09-14 04:49:00.865512104 +0000 UTC m=+139.041283654 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2bkxf" (UID: "3bbc1798-b60c-4a19-bab3-c463a8bdd51a") : configmap "kube-rbac-proxy" not found
Sep 14 04:48:52.865885 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:52.865534    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 14 04:48:52.865885 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:52.865613    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-insights-runtime-extractor-tls podName:3bbc1798-b60c-4a19-bab3-c463a8bdd51a nodeName:}" failed. No retries permitted until 2026-09-14 04:49:00.865594927 +0000 UTC m=+139.041366479 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2bkxf" (UID: "3bbc1798-b60c-4a19-bab3-c463a8bdd51a") : secret "insights-runtime-extractor-tls" not found
Sep 14 04:48:56.694127 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:56.694088    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-vhtjr\" (UID: \"ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vhtjr"
Sep 14 04:48:56.694567 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:56.694228    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 14 04:48:56.694567 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:56.694290    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc-cluster-monitoring-operator-tls podName:ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc nodeName:}" failed. No retries permitted until 2026-09-14 04:49:12.694274699 +0000 UTC m=+150.870046236 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-vhtjr" (UID: "ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc") : secret "cluster-monitoring-operator-tls" not found
Sep 14 04:48:56.895588 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:56.895558    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93a9bb73-aa22-4294-b119-20aa06f4bfde-metrics-certs\") pod \"router-default-c9d6db5f8-lqphc\" (UID: \"93a9bb73-aa22-4294-b119-20aa06f4bfde\") " pod="openshift-ingress/router-default-c9d6db5f8-lqphc"
Sep 14 04:48:56.895749 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:56.895598    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93a9bb73-aa22-4294-b119-20aa06f4bfde-service-ca-bundle\") pod \"router-default-c9d6db5f8-lqphc\" (UID: \"93a9bb73-aa22-4294-b119-20aa06f4bfde\") " pod="openshift-ingress/router-default-c9d6db5f8-lqphc"
Sep 14 04:48:56.895749 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:48:56.895714    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/93a9bb73-aa22-4294-b119-20aa06f4bfde-service-ca-bundle podName:93a9bb73-aa22-4294-b119-20aa06f4bfde nodeName:}" failed. No retries permitted until 2026-09-14 04:49:12.895701024 +0000 UTC m=+151.071472560 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/93a9bb73-aa22-4294-b119-20aa06f4bfde-service-ca-bundle") pod "router-default-c9d6db5f8-lqphc" (UID: "93a9bb73-aa22-4294-b119-20aa06f4bfde") : configmap references non-existent config key: service-ca.crt
Sep 14 04:48:56.897908 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:48:56.897887    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93a9bb73-aa22-4294-b119-20aa06f4bfde-metrics-certs\") pod \"router-default-c9d6db5f8-lqphc\" (UID: \"93a9bb73-aa22-4294-b119-20aa06f4bfde\") " pod="openshift-ingress/router-default-c9d6db5f8-lqphc"
Sep 14 04:49:00.925337 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:00.925294    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2bkxf\" (UID: \"3bbc1798-b60c-4a19-bab3-c463a8bdd51a\") " pod="openshift-insights/insights-runtime-extractor-2bkxf"
Sep 14 04:49:00.925337 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:00.925340    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2bkxf\" (UID: \"3bbc1798-b60c-4a19-bab3-c463a8bdd51a\") " pod="openshift-insights/insights-runtime-extractor-2bkxf"
Sep 14 04:49:00.925775 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:49:00.925453    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 14 04:49:00.925775 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:49:00.925532    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-insights-runtime-extractor-tls podName:3bbc1798-b60c-4a19-bab3-c463a8bdd51a nodeName:}" failed. No retries permitted until 2026-09-14 04:49:16.925517299 +0000 UTC m=+155.101288839 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2bkxf" (UID: "3bbc1798-b60c-4a19-bab3-c463a8bdd51a") : secret "insights-runtime-extractor-tls" not found
Sep 14 04:49:00.925851 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:00.925774    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2bkxf\" (UID: \"3bbc1798-b60c-4a19-bab3-c463a8bdd51a\") " pod="openshift-insights/insights-runtime-extractor-2bkxf"
Sep 14 04:49:09.924404 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:09.924366    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-654b486d49-g55wq"]
Sep 14 04:49:09.927143 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:09.927126    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-654b486d49-g55wq"
Sep 14 04:49:09.930294 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:09.930272    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 14 04:49:09.931018 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:09.930999    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xgrqm\""
Sep 14 04:49:09.931066 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:09.931027    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 14 04:49:09.934275 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:09.934257    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 14 04:49:09.954244 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:09.954223    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 14 04:49:09.978209 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:09.978187    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-654b486d49-g55wq"]
Sep 14 04:49:10.093402 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:10.093363    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x57kc\" (UniqueName: \"kubernetes.io/projected/4d4725e3-157e-4b82-acc4-44492baf85a7-kube-api-access-x57kc\") pod \"image-registry-654b486d49-g55wq\" (UID: \"4d4725e3-157e-4b82-acc4-44492baf85a7\") " pod="openshift-image-registry/image-registry-654b486d49-g55wq"
Sep 14 04:49:10.093402 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:10.093406    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4d4725e3-157e-4b82-acc4-44492baf85a7-registry-tls\") pod \"image-registry-654b486d49-g55wq\" (UID: \"4d4725e3-157e-4b82-acc4-44492baf85a7\") " pod="openshift-image-registry/image-registry-654b486d49-g55wq"
Sep 14 04:49:10.093608 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:10.093450    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4d4725e3-157e-4b82-acc4-44492baf85a7-image-registry-private-configuration\") pod \"image-registry-654b486d49-g55wq\" (UID: \"4d4725e3-157e-4b82-acc4-44492baf85a7\") " pod="openshift-image-registry/image-registry-654b486d49-g55wq"
Sep 14 04:49:10.093608 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:10.093471    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4d4725e3-157e-4b82-acc4-44492baf85a7-registry-certificates\") pod \"image-registry-654b486d49-g55wq\" (UID: \"4d4725e3-157e-4b82-acc4-44492baf85a7\") " pod="openshift-image-registry/image-registry-654b486d49-g55wq"
Sep 14 04:49:10.093608 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:10.093505    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4d4725e3-157e-4b82-acc4-44492baf85a7-bound-sa-token\") pod \"image-registry-654b486d49-g55wq\" (UID: \"4d4725e3-157e-4b82-acc4-44492baf85a7\") " pod="openshift-image-registry/image-registry-654b486d49-g55wq"
Sep 14 04:49:10.093608 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:10.093531    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4d4725e3-157e-4b82-acc4-44492baf85a7-installation-pull-secrets\") pod \"image-registry-654b486d49-g55wq\" (UID: \"4d4725e3-157e-4b82-acc4-44492baf85a7\") " pod="openshift-image-registry/image-registry-654b486d49-g55wq"
Sep 14 04:49:10.093608 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:10.093551    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4d4725e3-157e-4b82-acc4-44492baf85a7-ca-trust-extracted\") pod \"image-registry-654b486d49-g55wq\" (UID: \"4d4725e3-157e-4b82-acc4-44492baf85a7\") " pod="openshift-image-registry/image-registry-654b486d49-g55wq"
Sep 14 04:49:10.093608 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:10.093565    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4d4725e3-157e-4b82-acc4-44492baf85a7-trusted-ca\") pod \"image-registry-654b486d49-g55wq\" (UID: \"4d4725e3-157e-4b82-acc4-44492baf85a7\") " pod="openshift-image-registry/image-registry-654b486d49-g55wq"
Sep 14 04:49:10.194082 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:10.194005    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x57kc\" (UniqueName: \"kubernetes.io/projected/4d4725e3-157e-4b82-acc4-44492baf85a7-kube-api-access-x57kc\") pod \"image-registry-654b486d49-g55wq\" (UID: \"4d4725e3-157e-4b82-acc4-44492baf85a7\") " pod="openshift-image-registry/image-registry-654b486d49-g55wq"
Sep 14 04:49:10.194082 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:10.194045    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4d4725e3-157e-4b82-acc4-44492baf85a7-registry-tls\") pod \"image-registry-654b486d49-g55wq\" (UID: \"4d4725e3-157e-4b82-acc4-44492baf85a7\") " pod="openshift-image-registry/image-registry-654b486d49-g55wq"
Sep 14 04:49:10.194270 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:10.194245    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4d4725e3-157e-4b82-acc4-44492baf85a7-image-registry-private-configuration\") pod \"image-registry-654b486d49-g55wq\" (UID: \"4d4725e3-157e-4b82-acc4-44492baf85a7\") " pod="openshift-image-registry/image-registry-654b486d49-g55wq"
Sep 14 04:49:10.194307 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:10.194295    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4d4725e3-157e-4b82-acc4-44492baf85a7-registry-certificates\") pod \"image-registry-654b486d49-g55wq\" (UID: \"4d4725e3-157e-4b82-acc4-44492baf85a7\") " pod="openshift-image-registry/image-registry-654b486d49-g55wq"
Sep 14 04:49:10.194363 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:10.194350    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4d4725e3-157e-4b82-acc4-44492baf85a7-bound-sa-token\") pod \"image-registry-654b486d49-g55wq\" (UID: \"4d4725e3-157e-4b82-acc4-44492baf85a7\") " pod="openshift-image-registry/image-registry-654b486d49-g55wq"
Sep 14 04:49:10.194417 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:10.194402    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4d4725e3-157e-4b82-acc4-44492baf85a7-installation-pull-secrets\") pod \"image-registry-654b486d49-g55wq\" (UID: \"4d4725e3-157e-4b82-acc4-44492baf85a7\") " pod="openshift-image-registry/image-registry-654b486d49-g55wq"
Sep 14 04:49:10.194455 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:10.194441    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4d4725e3-157e-4b82-acc4-44492baf85a7-ca-trust-extracted\") pod \"image-registry-654b486d49-g55wq\" (UID: \"4d4725e3-157e-4b82-acc4-44492baf85a7\") " pod="openshift-image-registry/image-registry-654b486d49-g55wq"
Sep 14 04:49:10.194503 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:10.194465    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4d4725e3-157e-4b82-acc4-44492baf85a7-trusted-ca\") pod \"image-registry-654b486d49-g55wq\" (UID: \"4d4725e3-157e-4b82-acc4-44492baf85a7\") " pod="openshift-image-registry/image-registry-654b486d49-g55wq"
Sep 14 04:49:10.195304 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:10.195272    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4d4725e3-157e-4b82-acc4-44492baf85a7-ca-trust-extracted\") pod \"image-registry-654b486d49-g55wq\" (UID: \"4d4725e3-157e-4b82-acc4-44492baf85a7\") " pod="openshift-image-registry/image-registry-654b486d49-g55wq"
Sep 14 04:49:10.195660 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:10.195639    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4d4725e3-157e-4b82-acc4-44492baf85a7-trusted-ca\") pod \"image-registry-654b486d49-g55wq\" (UID: \"4d4725e3-157e-4b82-acc4-44492baf85a7\") " pod="openshift-image-registry/image-registry-654b486d49-g55wq"
Sep 14 04:49:10.195660 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:10.195654    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4d4725e3-157e-4b82-acc4-44492baf85a7-registry-certificates\") pod \"image-registry-654b486d49-g55wq\" (UID: \"4d4725e3-157e-4b82-acc4-44492baf85a7\") " pod="openshift-image-registry/image-registry-654b486d49-g55wq"
Sep 14 04:49:10.196570 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:10.196554    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4d4725e3-157e-4b82-acc4-44492baf85a7-registry-tls\") pod \"image-registry-654b486d49-g55wq\" (UID: \"4d4725e3-157e-4b82-acc4-44492baf85a7\") " pod="openshift-image-registry/image-registry-654b486d49-g55wq"
Sep 14 04:49:10.197401 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:10.197387    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4d4725e3-157e-4b82-acc4-44492baf85a7-image-registry-private-configuration\") pod \"image-registry-654b486d49-g55wq\" (UID: \"4d4725e3-157e-4b82-acc4-44492baf85a7\") " pod="openshift-image-registry/image-registry-654b486d49-g55wq"
Sep 14 04:49:10.197596 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:10.197573    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4d4725e3-157e-4b82-acc4-44492baf85a7-installation-pull-secrets\") pod \"image-registry-654b486d49-g55wq\" (UID: \"4d4725e3-157e-4b82-acc4-44492baf85a7\") " pod="openshift-image-registry/image-registry-654b486d49-g55wq"
Sep 14 04:49:10.205833 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:10.205803    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4d4725e3-157e-4b82-acc4-44492baf85a7-bound-sa-token\") pod \"image-registry-654b486d49-g55wq\" (UID: \"4d4725e3-157e-4b82-acc4-44492baf85a7\") " pod="openshift-image-registry/image-registry-654b486d49-g55wq"
Sep 14 04:49:10.205988 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:10.205971    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x57kc\" (UniqueName: \"kubernetes.io/projected/4d4725e3-157e-4b82-acc4-44492baf85a7-kube-api-access-x57kc\") pod \"image-registry-654b486d49-g55wq\" (UID: \"4d4725e3-157e-4b82-acc4-44492baf85a7\") " pod="openshift-image-registry/image-registry-654b486d49-g55wq"
Sep 14 04:49:10.236875 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:10.236852    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-654b486d49-g55wq"
Sep 14 04:49:10.509503 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:10.509332    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-654b486d49-g55wq"]
Sep 14 04:49:10.511768 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:49:10.511734    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d4725e3_157e_4b82_acc4_44492baf85a7.slice/crio-a2b46a297737851ffe77a0c003b1bd81a3ee97fcaaa66e0c6705c079a57373fa WatchSource:0}: Error finding container a2b46a297737851ffe77a0c003b1bd81a3ee97fcaaa66e0c6705c079a57373fa: Status 404 returned error can't find the container with id a2b46a297737851ffe77a0c003b1bd81a3ee97fcaaa66e0c6705c079a57373fa
Sep 14 04:49:10.970072 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:10.970037    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-654b486d49-g55wq" event={"ID":"4d4725e3-157e-4b82-acc4-44492baf85a7","Type":"ContainerStarted","Data":"bc0ecdbf1283ca8d27be2db7bcbe5507e62f3e2d8504f549823d664ce95fa043"}
Sep 14 04:49:10.970440 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:10.970086    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-654b486d49-g55wq" event={"ID":"4d4725e3-157e-4b82-acc4-44492baf85a7","Type":"ContainerStarted","Data":"a2b46a297737851ffe77a0c003b1bd81a3ee97fcaaa66e0c6705c079a57373fa"}
Sep 14 04:49:10.970440 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:10.970147    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-654b486d49-g55wq"
Sep 14 04:49:10.997569 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:10.997522    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-654b486d49-g55wq" podStartSLOduration=1.997509874 podStartE2EDuration="1.997509874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:49:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:49:10.997090718 +0000 UTC m=+149.172862276" watchObservedRunningTime="2026-09-14 04:49:10.997509874 +0000 UTC m=+149.173281432"
Sep 14 04:49:12.711996 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:12.711957    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-vhtjr\" (UID: \"ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vhtjr"
Sep 14 04:49:12.714250 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:12.714231    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-vhtjr\" (UID: \"ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vhtjr"
Sep 14 04:49:12.913530 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:12.913492    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93a9bb73-aa22-4294-b119-20aa06f4bfde-service-ca-bundle\") pod \"router-default-c9d6db5f8-lqphc\" (UID: \"93a9bb73-aa22-4294-b119-20aa06f4bfde\") " pod="openshift-ingress/router-default-c9d6db5f8-lqphc"
Sep 14 04:49:12.914069 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:12.914050    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93a9bb73-aa22-4294-b119-20aa06f4bfde-service-ca-bundle\") pod \"router-default-c9d6db5f8-lqphc\" (UID: \"93a9bb73-aa22-4294-b119-20aa06f4bfde\") " pod="openshift-ingress/router-default-c9d6db5f8-lqphc"
Sep 14 04:49:12.999072 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:12.999002    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-pjkxs\""
Sep 14 04:49:13.007246 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:13.007230    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vhtjr"
Sep 14 04:49:13.149905 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:13.149877    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-8hwgw\""
Sep 14 04:49:13.157235 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:13.157218    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-c9d6db5f8-lqphc"
Sep 14 04:49:13.279965 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:13.279878    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-vhtjr"]
Sep 14 04:49:13.285841 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:49:13.285816    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce13cb9e_09d6_4a9a_b4d4_7db45242c8bc.slice/crio-fa2576cc57d9338d21b19bbd57a9230a4b69687de0769d5db46bfeba7034d038 WatchSource:0}: Error finding container fa2576cc57d9338d21b19bbd57a9230a4b69687de0769d5db46bfeba7034d038: Status 404 returned error can't find the container with id fa2576cc57d9338d21b19bbd57a9230a4b69687de0769d5db46bfeba7034d038
Sep 14 04:49:13.428487 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:13.428456    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-c9d6db5f8-lqphc"]
Sep 14 04:49:13.430993 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:49:13.430964    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93a9bb73_aa22_4294_b119_20aa06f4bfde.slice/crio-5b6a6ac0e2cb331a310b142b7b7205b5a87f88b60b2719a0ce489f01078885e6 WatchSource:0}: Error finding container 5b6a6ac0e2cb331a310b142b7b7205b5a87f88b60b2719a0ce489f01078885e6: Status 404 returned error can't find the container with id 5b6a6ac0e2cb331a310b142b7b7205b5a87f88b60b2719a0ce489f01078885e6
Sep 14 04:49:13.981052 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:13.981010    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-c9d6db5f8-lqphc" event={"ID":"93a9bb73-aa22-4294-b119-20aa06f4bfde","Type":"ContainerStarted","Data":"5cea01383307544d919a054bb82216dd51a2b2ba519bd201ff4cf11618ca5f4a"}
Sep 14 04:49:13.981513 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:13.981080    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-c9d6db5f8-lqphc" event={"ID":"93a9bb73-aa22-4294-b119-20aa06f4bfde","Type":"ContainerStarted","Data":"5b6a6ac0e2cb331a310b142b7b7205b5a87f88b60b2719a0ce489f01078885e6"}
Sep 14 04:49:13.982338 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:13.982312    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vhtjr" event={"ID":"ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc","Type":"ContainerStarted","Data":"fa2576cc57d9338d21b19bbd57a9230a4b69687de0769d5db46bfeba7034d038"}
Sep 14 04:49:14.002088 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:14.002039    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-c9d6db5f8-lqphc" podStartSLOduration=34.002021569 podStartE2EDuration="34.002021569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:48:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:49:14.001974809 +0000 UTC m=+152.177746367" watchObservedRunningTime="2026-09-14 04:49:14.002021569 +0000 UTC m=+152.177793127"
Sep 14 04:49:14.158221 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:14.158180    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-c9d6db5f8-lqphc"
Sep 14 04:49:14.160781 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:14.160740    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-c9d6db5f8-lqphc"
Sep 14 04:49:14.988003 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:14.987960    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vhtjr" event={"ID":"ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc","Type":"ContainerStarted","Data":"7edbe3c51eb35b48cb0b96a836ab054e6c8fe32415249827bf8dfc0a9e6717e2"}
Sep 14 04:49:14.988456 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:14.988188    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-c9d6db5f8-lqphc"
Sep 14 04:49:14.989314 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:14.989293    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-c9d6db5f8-lqphc"
Sep 14 04:49:15.007095 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:15.007052    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vhtjr" podStartSLOduration=33.472993054 podStartE2EDuration="35.007039221s" totalImagesPullingTime="1.534046167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:48:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:49:15.006391027 +0000 UTC m=+153.182162585" watchObservedRunningTime="2026-09-14 04:49:15.007039221 +0000 UTC m=+153.182810775"
Sep 14 04:49:16.944078 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:16.944037    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2bkxf\" (UID: \"3bbc1798-b60c-4a19-bab3-c463a8bdd51a\") " pod="openshift-insights/insights-runtime-extractor-2bkxf"
Sep 14 04:49:16.946339 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:16.946308    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3bbc1798-b60c-4a19-bab3-c463a8bdd51a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2bkxf\" (UID: \"3bbc1798-b60c-4a19-bab3-c463a8bdd51a\") " pod="openshift-insights/insights-runtime-extractor-2bkxf"
Sep 14 04:49:17.164344 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:17.164311    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2bkxf"
Sep 14 04:49:17.434619 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:17.434583    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2bkxf"]
Sep 14 04:49:17.437427 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:49:17.437400    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bbc1798_b60c_4a19_bab3_c463a8bdd51a.slice/crio-d65bbb57832f67792e34d78d731edc096ebb47bee0b42bcf69a167d2f4fe5e61 WatchSource:0}: Error finding container d65bbb57832f67792e34d78d731edc096ebb47bee0b42bcf69a167d2f4fe5e61: Status 404 returned error can't find the container with id d65bbb57832f67792e34d78d731edc096ebb47bee0b42bcf69a167d2f4fe5e61
Sep 14 04:49:17.998660 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:17.998625    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2bkxf" event={"ID":"3bbc1798-b60c-4a19-bab3-c463a8bdd51a","Type":"ContainerStarted","Data":"77f13b6fcd299d870d0e2ff5edb3ae7a7f60a7e9048aa974657c4f6ec209f578"}
Sep 14 04:49:17.999042 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:17.998686    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2bkxf" event={"ID":"3bbc1798-b60c-4a19-bab3-c463a8bdd51a","Type":"ContainerStarted","Data":"d65bbb57832f67792e34d78d731edc096ebb47bee0b42bcf69a167d2f4fe5e61"}
Sep 14 04:49:18.652790 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:49:18.652736    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-2ms2f" podUID="bb2c56d0-20d7-49ba-857c-febc67911fcd"
Sep 14 04:49:19.004587 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:19.004503    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2ms2f"
Sep 14 04:49:19.005029 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:19.004503    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2bkxf" event={"ID":"3bbc1798-b60c-4a19-bab3-c463a8bdd51a","Type":"ContainerStarted","Data":"280422adc5798d01fe65eae6c62ee817949a75ca6ddd699051f61779caf037a0"}
Sep 14 04:49:20.009637 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:20.009604    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2bkxf" event={"ID":"3bbc1798-b60c-4a19-bab3-c463a8bdd51a","Type":"ContainerStarted","Data":"8748657c9cdef09b6afea2e0b9e8b1f3e781aeaa887cdc6dd907798737dcac18"}
Sep 14 04:49:20.028641 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:20.028591    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2bkxf" podStartSLOduration=32.796807672 podStartE2EDuration="35.028576561s" totalImagesPullingTime="2.231768889s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 04:48:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:49:20.028265933 +0000 UTC m=+158.204037493" watchObservedRunningTime="2026-09-14 04:49:20.028576561 +0000 UTC m=+158.204348120"
Sep 14 04:49:20.346086 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:49:20.346053    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-2gwsl" podUID="ee8a9555-8b90-4ad8-a399-22d414487421"
Sep 14 04:49:23.591018 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:23.590984    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb2c56d0-20d7-49ba-857c-febc67911fcd-metrics-tls\") pod \"dns-default-2ms2f\" (UID: \"bb2c56d0-20d7-49ba-857c-febc67911fcd\") " pod="openshift-dns/dns-default-2ms2f"
Sep 14 04:49:23.593310 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:23.593286    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb2c56d0-20d7-49ba-857c-febc67911fcd-metrics-tls\") pod \"dns-default-2ms2f\" (UID: \"bb2c56d0-20d7-49ba-857c-febc67911fcd\") " pod="openshift-dns/dns-default-2ms2f"
Sep 14 04:49:23.808274 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:23.808243    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wwfwc\""
Sep 14 04:49:23.816393 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:23.816365    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2ms2f"
Sep 14 04:49:23.908083 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:23.908055    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xrqvz"]
Sep 14 04:49:23.913077 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:23.913056    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xrqvz"
Sep 14 04:49:23.915221 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:23.915201    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 14 04:49:23.915649 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:23.915635    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 14 04:49:23.916014 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:23.915993    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 14 04:49:23.916107 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:23.916055    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vr67q\""
Sep 14 04:49:23.916193 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:23.916132    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 14 04:49:23.994452 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:23.994423    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ad643666-9ad7-4c4f-9d4a-f18bdc21740d-node-exporter-wtmp\") pod \"node-exporter-xrqvz\" (UID: \"ad643666-9ad7-4c4f-9d4a-f18bdc21740d\") " pod="openshift-monitoring/node-exporter-xrqvz"
Sep 14 04:49:23.994452 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:23.994453    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ad643666-9ad7-4c4f-9d4a-f18bdc21740d-root\") pod \"node-exporter-xrqvz\" (UID: \"ad643666-9ad7-4c4f-9d4a-f18bdc21740d\") " pod="openshift-monitoring/node-exporter-xrqvz"
Sep 14 04:49:23.994630 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:23.994477    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad643666-9ad7-4c4f-9d4a-f18bdc21740d-sys\") pod \"node-exporter-xrqvz\" (UID: \"ad643666-9ad7-4c4f-9d4a-f18bdc21740d\") " pod="openshift-monitoring/node-exporter-xrqvz"
Sep 14 04:49:23.994630 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:23.994494    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ad643666-9ad7-4c4f-9d4a-f18bdc21740d-node-exporter-accelerators-collector-config\") pod \"node-exporter-xrqvz\" (UID: \"ad643666-9ad7-4c4f-9d4a-f18bdc21740d\") " pod="openshift-monitoring/node-exporter-xrqvz"
Sep 14 04:49:23.994630 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:23.994560    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad643666-9ad7-4c4f-9d4a-f18bdc21740d-metrics-client-ca\") pod \"node-exporter-xrqvz\" (UID: \"ad643666-9ad7-4c4f-9d4a-f18bdc21740d\") " pod="openshift-monitoring/node-exporter-xrqvz"
Sep 14 04:49:23.994630 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:23.994586    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ad643666-9ad7-4c4f-9d4a-f18bdc21740d-node-exporter-tls\") pod \"node-exporter-xrqvz\" (UID: \"ad643666-9ad7-4c4f-9d4a-f18bdc21740d\") " pod="openshift-monitoring/node-exporter-xrqvz"
Sep 14 04:49:23.994630 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:23.994615    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ad643666-9ad7-4c4f-9d4a-f18bdc21740d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xrqvz\" (UID: \"ad643666-9ad7-4c4f-9d4a-f18bdc21740d\") " pod="openshift-monitoring/node-exporter-xrqvz"
Sep 14 04:49:23.994812 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:23.994634    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ad643666-9ad7-4c4f-9d4a-f18bdc21740d-node-exporter-textfile\") pod \"node-exporter-xrqvz\" (UID: \"ad643666-9ad7-4c4f-9d4a-f18bdc21740d\") " pod="openshift-monitoring/node-exporter-xrqvz"
Sep 14 04:49:23.994812 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:23.994665    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gknk\" (UniqueName: \"kubernetes.io/projected/ad643666-9ad7-4c4f-9d4a-f18bdc21740d-kube-api-access-8gknk\") pod \"node-exporter-xrqvz\" (UID: \"ad643666-9ad7-4c4f-9d4a-f18bdc21740d\") " pod="openshift-monitoring/node-exporter-xrqvz"
Sep 14 04:49:24.095397 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:24.095362    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ad643666-9ad7-4c4f-9d4a-f18bdc21740d-node-exporter-wtmp\") pod \"node-exporter-xrqvz\" (UID: \"ad643666-9ad7-4c4f-9d4a-f18bdc21740d\") " pod="openshift-monitoring/node-exporter-xrqvz"
Sep 14 04:49:24.095397 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:24.095388    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2ms2f"]
Sep 14 04:49:24.095397 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:24.095408    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ad643666-9ad7-4c4f-9d4a-f18bdc21740d-root\") pod \"node-exporter-xrqvz\" (UID: \"ad643666-9ad7-4c4f-9d4a-f18bdc21740d\") " pod="openshift-monitoring/node-exporter-xrqvz"
Sep 14 04:49:24.095689 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:24.095454    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad643666-9ad7-4c4f-9d4a-f18bdc21740d-sys\") pod \"node-exporter-xrqvz\" (UID: \"ad643666-9ad7-4c4f-9d4a-f18bdc21740d\") " pod="openshift-monitoring/node-exporter-xrqvz"
Sep 14 04:49:24.095689 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:24.095480    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ad643666-9ad7-4c4f-9d4a-f18bdc21740d-node-exporter-accelerators-collector-config\") pod \"node-exporter-xrqvz\" (UID: \"ad643666-9ad7-4c4f-9d4a-f18bdc21740d\") " pod="openshift-monitoring/node-exporter-xrqvz"
Sep 14 04:49:24.095689 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:24.095518    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad643666-9ad7-4c4f-9d4a-f18bdc21740d-metrics-client-ca\") pod \"node-exporter-xrqvz\" (UID: \"ad643666-9ad7-4c4f-9d4a-f18bdc21740d\") " pod="openshift-monitoring/node-exporter-xrqvz"
Sep 14 04:49:24.095689 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:24.095522    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ad643666-9ad7-4c4f-9d4a-f18bdc21740d-root\") pod \"node-exporter-xrqvz\" (UID: \"ad643666-9ad7-4c4f-9d4a-f18bdc21740d\") " pod="openshift-monitoring/node-exporter-xrqvz"
Sep 14 04:49:24.095689 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:24.095542    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ad643666-9ad7-4c4f-9d4a-f18bdc21740d-node-exporter-tls\") pod \"node-exporter-xrqvz\" (UID: \"ad643666-9ad7-4c4f-9d4a-f18bdc21740d\") " pod="openshift-monitoring/node-exporter-xrqvz"
Sep 14 04:49:24.095689 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:24.095570    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ad643666-9ad7-4c4f-9d4a-f18bdc21740d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xrqvz\" (UID: \"ad643666-9ad7-4c4f-9d4a-f18bdc21740d\") " pod="openshift-monitoring/node-exporter-xrqvz"
Sep 14 04:49:24.095689 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:24.095600    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ad643666-9ad7-4c4f-9d4a-f18bdc21740d-node-exporter-textfile\") pod \"node-exporter-xrqvz\" (UID: \"ad643666-9ad7-4c4f-9d4a-f18bdc21740d\") " pod="openshift-monitoring/node-exporter-xrqvz"
Sep 14 04:49:24.095689 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:24.095630    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gknk\" (UniqueName: \"kubernetes.io/projected/ad643666-9ad7-4c4f-9d4a-f18bdc21740d-kube-api-access-8gknk\") pod \"node-exporter-xrqvz\" (UID: \"ad643666-9ad7-4c4f-9d4a-f18bdc21740d\") " pod="openshift-monitoring/node-exporter-xrqvz"
Sep 14 04:49:24.096203 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:24.096174    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad643666-9ad7-4c4f-9d4a-f18bdc21740d-sys\") pod \"node-exporter-xrqvz\" (UID: \"ad643666-9ad7-4c4f-9d4a-f18bdc21740d\") " pod="openshift-monitoring/node-exporter-xrqvz"
Sep 14 04:49:24.096203 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:24.095570    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ad643666-9ad7-4c4f-9d4a-f18bdc21740d-node-exporter-wtmp\") pod \"node-exporter-xrqvz\" (UID: \"ad643666-9ad7-4c4f-9d4a-f18bdc21740d\") " pod="openshift-monitoring/node-exporter-xrqvz"
Sep 14 04:49:24.096384 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:24.096223    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad643666-9ad7-4c4f-9d4a-f18bdc21740d-metrics-client-ca\") pod \"node-exporter-xrqvz\" (UID: \"ad643666-9ad7-4c4f-9d4a-f18bdc21740d\") " pod="openshift-monitoring/node-exporter-xrqvz"
Sep 14 04:49:24.096530 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:24.096503    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ad643666-9ad7-4c4f-9d4a-f18bdc21740d-node-exporter-textfile\") pod \"node-exporter-xrqvz\" (UID: \"ad643666-9ad7-4c4f-9d4a-f18bdc21740d\") " pod="openshift-monitoring/node-exporter-xrqvz"
Sep 14 04:49:24.097001 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:24.096978    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ad643666-9ad7-4c4f-9d4a-f18bdc21740d-node-exporter-accelerators-collector-config\") pod \"node-exporter-xrqvz\" (UID: \"ad643666-9ad7-4c4f-9d4a-f18bdc21740d\") " pod="openshift-monitoring/node-exporter-xrqvz"
Sep 14 04:49:24.098568 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:24.098522    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ad643666-9ad7-4c4f-9d4a-f18bdc21740d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xrqvz\" (UID: \"ad643666-9ad7-4c4f-9d4a-f18bdc21740d\") " pod="openshift-monitoring/node-exporter-xrqvz"
Sep 14 04:49:24.098800 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:24.098744    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ad643666-9ad7-4c4f-9d4a-f18bdc21740d-node-exporter-tls\") pod \"node-exporter-xrqvz\" (UID: \"ad643666-9ad7-4c4f-9d4a-f18bdc21740d\") " pod="openshift-monitoring/node-exporter-xrqvz"
Sep 14 04:49:24.098879 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:49:24.098856    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2c56d0_20d7_49ba_857c_febc67911fcd.slice/crio-a901a8c37f1d32d37fa06b073c4c801bf3c5b0a6c3bea68f8e2d265c044ef3d7 WatchSource:0}: Error finding container a901a8c37f1d32d37fa06b073c4c801bf3c5b0a6c3bea68f8e2d265c044ef3d7: Status 404 returned error can't find the container with id a901a8c37f1d32d37fa06b073c4c801bf3c5b0a6c3bea68f8e2d265c044ef3d7
Sep 14 04:49:24.106536 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:24.106516    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gknk\" (UniqueName: \"kubernetes.io/projected/ad643666-9ad7-4c4f-9d4a-f18bdc21740d-kube-api-access-8gknk\") pod \"node-exporter-xrqvz\" (UID: \"ad643666-9ad7-4c4f-9d4a-f18bdc21740d\") " pod="openshift-monitoring/node-exporter-xrqvz"
Sep 14 04:49:24.223193 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:24.223121    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xrqvz"
Sep 14 04:49:24.237078 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:49:24.237056    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad643666_9ad7_4c4f_9d4a_f18bdc21740d.slice/crio-eff86f0f23f410cd302b36c6734531ae37f3d0f1ea34c93f92d3992aa5266513 WatchSource:0}: Error finding container eff86f0f23f410cd302b36c6734531ae37f3d0f1ea34c93f92d3992aa5266513: Status 404 returned error can't find the container with id eff86f0f23f410cd302b36c6734531ae37f3d0f1ea34c93f92d3992aa5266513
Sep 14 04:49:25.029332 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:25.029015    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xrqvz" event={"ID":"ad643666-9ad7-4c4f-9d4a-f18bdc21740d","Type":"ContainerStarted","Data":"eff86f0f23f410cd302b36c6734531ae37f3d0f1ea34c93f92d3992aa5266513"}
Sep 14 04:49:25.030875 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:25.030787    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2ms2f" event={"ID":"bb2c56d0-20d7-49ba-857c-febc67911fcd","Type":"ContainerStarted","Data":"a901a8c37f1d32d37fa06b073c4c801bf3c5b0a6c3bea68f8e2d265c044ef3d7"}
Sep 14 04:49:26.036206 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:26.036142    2565 generic.go:356] "Generic (PLEG): container finished" podID="ad643666-9ad7-4c4f-9d4a-f18bdc21740d" containerID="7336a2accff45b55ca0bf2dafa0d8fb15e48fa38405c3073df36757052f8ccb5" exitCode=0
Sep 14 04:49:26.036599 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:26.036230    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xrqvz" event={"ID":"ad643666-9ad7-4c4f-9d4a-f18bdc21740d","Type":"ContainerDied","Data":"7336a2accff45b55ca0bf2dafa0d8fb15e48fa38405c3073df36757052f8ccb5"}
Sep 14 04:49:26.038844 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:26.038822    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2ms2f" event={"ID":"bb2c56d0-20d7-49ba-857c-febc67911fcd","Type":"ContainerStarted","Data":"ba393352caaaf462ee35fef3d9fdcd8c5cc690c6ff2d907fabd13e1af7d57adf"}
Sep 14 04:49:26.038962 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:26.038861    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2ms2f" event={"ID":"bb2c56d0-20d7-49ba-857c-febc67911fcd","Type":"ContainerStarted","Data":"820fe20c7c32168ba280ce81f532adfc5beb376a4163928dd8a7bb7fe012cfd0"}
Sep 14 04:49:26.038962 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:26.038945    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2ms2f"
Sep 14 04:49:26.079834 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:26.079792    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-2ms2f" podStartSLOduration=129.395927462 podStartE2EDuration="2m11.079775744s" totalImagesPullingTime="1.683848282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:47:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:49:26.07927889 +0000 UTC m=+164.255050448" watchObservedRunningTime="2026-09-14 04:49:26.079775744 +0000 UTC m=+164.255547294"
Sep 14 04:49:27.044162 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:27.044113    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xrqvz" event={"ID":"ad643666-9ad7-4c4f-9d4a-f18bdc21740d","Type":"ContainerStarted","Data":"8c71ecc76caa1925ec8511e8bf6ad5513ee471971d25ab9827f2ce3d0d56287f"}
Sep 14 04:49:27.044516 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:27.044184    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xrqvz" event={"ID":"ad643666-9ad7-4c4f-9d4a-f18bdc21740d","Type":"ContainerStarted","Data":"e7f7c3ee283d06fcc65628a70c7736b0dba3af052f075830b4b3ca66145a11ab"}
Sep 14 04:49:27.072198 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:27.072119    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xrqvz" podStartSLOduration=2.451172346 podStartE2EDuration="4.072101342s" totalImagesPullingTime="1.541960186s" totalInitContainerRuntime="78.96881ms" isStatefulPod=true podCreationTimestamp="2026-09-14 04:49:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:49:27.071929893 +0000 UTC m=+165.247701448" watchObservedRunningTime="2026-09-14 04:49:27.072101342 +0000 UTC m=+165.247872903"
Sep 14 04:49:30.241057 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:30.241014    2565 patch_prober.go:28] interesting pod/image-registry-654b486d49-g55wq container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 14 04:49:30.241521 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:30.241070    2565 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-654b486d49-g55wq" podUID="4d4725e3-157e-4b82-acc4-44492baf85a7" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 14 04:49:31.979479 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:31.979451    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-654b486d49-g55wq"
Sep 14 04:49:33.328814 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:33.328776    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2gwsl"
Sep 14 04:49:36.046110 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:49:36.046029    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2ms2f"
Sep 14 04:50:53.162462 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:50:53.162428    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee8a9555-8b90-4ad8-a399-22d414487421-metrics-certs\") pod \"network-metrics-daemon-2gwsl\" (UID: \"ee8a9555-8b90-4ad8-a399-22d414487421\") " pod="openshift-multus/network-metrics-daemon-2gwsl"
Sep 14 04:50:53.164652 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:50:53.164634    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee8a9555-8b90-4ad8-a399-22d414487421-metrics-certs\") pod \"network-metrics-daemon-2gwsl\" (UID: \"ee8a9555-8b90-4ad8-a399-22d414487421\") " pod="openshift-multus/network-metrics-daemon-2gwsl"
Sep 14 04:50:53.432436 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:50:53.432353    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kgkvn\""
Sep 14 04:50:53.440417 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:50:53.440390    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2gwsl"
Sep 14 04:50:53.712072 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:50:53.711977    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2gwsl"]
Sep 14 04:50:53.714481 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:50:53.714447    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee8a9555_8b90_4ad8_a399_22d414487421.slice/crio-cee703bb3f007db7ddf79ef698cafe4acab11a0628966da624482466235c0fa8 WatchSource:0}: Error finding container cee703bb3f007db7ddf79ef698cafe4acab11a0628966da624482466235c0fa8: Status 404 returned error can't find the container with id cee703bb3f007db7ddf79ef698cafe4acab11a0628966da624482466235c0fa8
Sep 14 04:50:54.320233 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:50:54.320199    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2gwsl" event={"ID":"ee8a9555-8b90-4ad8-a399-22d414487421","Type":"ContainerStarted","Data":"cee703bb3f007db7ddf79ef698cafe4acab11a0628966da624482466235c0fa8"}
Sep 14 04:50:55.325477 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:50:55.325445    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2gwsl" event={"ID":"ee8a9555-8b90-4ad8-a399-22d414487421","Type":"ContainerStarted","Data":"b751d24303f373c172db0b3be963689862038d7ecd39be6c8ce6c333f9a9227b"}
Sep 14 04:50:55.325857 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:50:55.325499    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2gwsl" event={"ID":"ee8a9555-8b90-4ad8-a399-22d414487421","Type":"ContainerStarted","Data":"74a6003ca3ecb57a9d8c525490a3344ab7c54f02d91bd792876c205ef829e382"}
Sep 14 04:50:55.349340 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:50:55.349292    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-2gwsl" podStartSLOduration=252.423596264 podStartE2EDuration="4m13.349275767s" totalImagesPullingTime="925.679503ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:46:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:50:55.347779717 +0000 UTC m=+253.523551276" watchObservedRunningTime="2026-09-14 04:50:55.349275767 +0000 UTC m=+253.525047369"
Sep 14 04:51:42.241873 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:51:42.241844    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kndxq_e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae/ovn-acl-logging/0.log"
Sep 14 04:51:42.247267 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:51:42.247246    2565 kubelet.go:1648] "Image garbage collection succeeded"
Sep 14 04:51:42.250101 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:51:42.250078    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kndxq_e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae/ovn-acl-logging/0.log"
Sep 14 04:52:44.621775 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:52:44.621688    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-79c9967cd5-zq8dx"]
Sep 14 04:52:44.624724 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:52:44.624702    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79c9967cd5-zq8dx"
Sep 14 04:52:44.628658 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:52:44.628637    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 14 04:52:44.628790 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:52:44.628682    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 14 04:52:44.628790 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:52:44.628693    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 14 04:52:44.628790 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:52:44.628689    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 14 04:52:44.635235 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:52:44.635212    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-79c9967cd5-zq8dx"]
Sep 14 04:52:44.765698 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:52:44.765665    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d15a3722-043a-48d5-849e-e854cd401ab1-tmp\") pod \"klusterlet-addon-workmgr-79c9967cd5-zq8dx\" (UID: \"d15a3722-043a-48d5-849e-e854cd401ab1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79c9967cd5-zq8dx"
Sep 14 04:52:44.765698 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:52:44.765703    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxspn\" (UniqueName: \"kubernetes.io/projected/d15a3722-043a-48d5-849e-e854cd401ab1-kube-api-access-jxspn\") pod \"klusterlet-addon-workmgr-79c9967cd5-zq8dx\" (UID: \"d15a3722-043a-48d5-849e-e854cd401ab1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79c9967cd5-zq8dx"
Sep 14 04:52:44.765929 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:52:44.765741    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d15a3722-043a-48d5-849e-e854cd401ab1-klusterlet-config\") pod \"klusterlet-addon-workmgr-79c9967cd5-zq8dx\" (UID: \"d15a3722-043a-48d5-849e-e854cd401ab1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79c9967cd5-zq8dx"
Sep 14 04:52:44.866960 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:52:44.866926    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d15a3722-043a-48d5-849e-e854cd401ab1-klusterlet-config\") pod \"klusterlet-addon-workmgr-79c9967cd5-zq8dx\" (UID: \"d15a3722-043a-48d5-849e-e854cd401ab1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79c9967cd5-zq8dx"
Sep 14 04:52:44.867128 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:52:44.866979    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d15a3722-043a-48d5-849e-e854cd401ab1-tmp\") pod \"klusterlet-addon-workmgr-79c9967cd5-zq8dx\" (UID: \"d15a3722-043a-48d5-849e-e854cd401ab1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79c9967cd5-zq8dx"
Sep 14 04:52:44.867128 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:52:44.867004    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxspn\" (UniqueName: \"kubernetes.io/projected/d15a3722-043a-48d5-849e-e854cd401ab1-kube-api-access-jxspn\") pod \"klusterlet-addon-workmgr-79c9967cd5-zq8dx\" (UID: \"d15a3722-043a-48d5-849e-e854cd401ab1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79c9967cd5-zq8dx"
Sep 14 04:52:44.867321 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:52:44.867305    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d15a3722-043a-48d5-849e-e854cd401ab1-tmp\") pod \"klusterlet-addon-workmgr-79c9967cd5-zq8dx\" (UID: \"d15a3722-043a-48d5-849e-e854cd401ab1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79c9967cd5-zq8dx"
Sep 14 04:52:44.869587 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:52:44.869562    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d15a3722-043a-48d5-849e-e854cd401ab1-klusterlet-config\") pod \"klusterlet-addon-workmgr-79c9967cd5-zq8dx\" (UID: \"d15a3722-043a-48d5-849e-e854cd401ab1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79c9967cd5-zq8dx"
Sep 14 04:52:44.877435 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:52:44.877381    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxspn\" (UniqueName: \"kubernetes.io/projected/d15a3722-043a-48d5-849e-e854cd401ab1-kube-api-access-jxspn\") pod \"klusterlet-addon-workmgr-79c9967cd5-zq8dx\" (UID: \"d15a3722-043a-48d5-849e-e854cd401ab1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79c9967cd5-zq8dx"
Sep 14 04:52:44.933953 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:52:44.933924    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79c9967cd5-zq8dx"
Sep 14 04:52:45.202313 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:52:45.202224    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-79c9967cd5-zq8dx"]
Sep 14 04:52:45.204993 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:52:45.204960    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd15a3722_043a_48d5_849e_e854cd401ab1.slice/crio-c2b70054d3aacc99c3c80fb9ca5847e5d3b7d2fc99cb9bb23fffd91eef4d97a5 WatchSource:0}: Error finding container c2b70054d3aacc99c3c80fb9ca5847e5d3b7d2fc99cb9bb23fffd91eef4d97a5: Status 404 returned error can't find the container with id c2b70054d3aacc99c3c80fb9ca5847e5d3b7d2fc99cb9bb23fffd91eef4d97a5
Sep 14 04:52:45.206862 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:52:45.206847    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 04:52:45.675579 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:52:45.675543    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79c9967cd5-zq8dx" event={"ID":"d15a3722-043a-48d5-849e-e854cd401ab1","Type":"ContainerStarted","Data":"c2b70054d3aacc99c3c80fb9ca5847e5d3b7d2fc99cb9bb23fffd91eef4d97a5"}
Sep 14 04:52:49.692526 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:52:49.692488    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79c9967cd5-zq8dx" event={"ID":"d15a3722-043a-48d5-849e-e854cd401ab1","Type":"ContainerStarted","Data":"1f10500dfeffe90ddb0284de402482ba01f0c7f9abf9746e153268d100ddbc8f"}
Sep 14 04:52:49.692982 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:52:49.692681    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79c9967cd5-zq8dx"
Sep 14 04:52:49.694585 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:52:49.694564    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79c9967cd5-zq8dx"
Sep 14 04:52:49.709478 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:52:49.709317    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79c9967cd5-zq8dx" podStartSLOduration=2.047588326 podStartE2EDuration="5.709299657s" totalImagesPullingTime="3.661711331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:52:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:52:49.708405669 +0000 UTC m=+367.884177241" watchObservedRunningTime="2026-09-14 04:52:49.709299657 +0000 UTC m=+367.885071217"
Sep 14 04:53:23.482329 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:23.482284    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ndbqd"]
Sep 14 04:53:23.486122 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:23.486103    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ndbqd"
Sep 14 04:53:23.488475 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:23.488443    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 14 04:53:23.488591 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:23.488522    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 14 04:53:23.489358 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:23.489339    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-szxlp\""
Sep 14 04:53:23.493970 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:23.493951    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ndbqd"]
Sep 14 04:53:23.650508 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:23.650464    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9cdd4ce5-8946-4af1-be44-93650764e39e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ndbqd\" (UID: \"9cdd4ce5-8946-4af1-be44-93650764e39e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ndbqd"
Sep 14 04:53:23.650683 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:23.650518    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrsgw\" (UniqueName: \"kubernetes.io/projected/9cdd4ce5-8946-4af1-be44-93650764e39e-kube-api-access-zrsgw\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ndbqd\" (UID: \"9cdd4ce5-8946-4af1-be44-93650764e39e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ndbqd"
Sep 14 04:53:23.650683 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:23.650634    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9cdd4ce5-8946-4af1-be44-93650764e39e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ndbqd\" (UID: \"9cdd4ce5-8946-4af1-be44-93650764e39e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ndbqd"
Sep 14 04:53:23.751972 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:23.751915    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9cdd4ce5-8946-4af1-be44-93650764e39e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ndbqd\" (UID: \"9cdd4ce5-8946-4af1-be44-93650764e39e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ndbqd"
Sep 14 04:53:23.751972 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:23.751954    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9cdd4ce5-8946-4af1-be44-93650764e39e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ndbqd\" (UID: \"9cdd4ce5-8946-4af1-be44-93650764e39e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ndbqd"
Sep 14 04:53:23.752133 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:23.751979    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrsgw\" (UniqueName: \"kubernetes.io/projected/9cdd4ce5-8946-4af1-be44-93650764e39e-kube-api-access-zrsgw\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ndbqd\" (UID: \"9cdd4ce5-8946-4af1-be44-93650764e39e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ndbqd"
Sep 14 04:53:23.752334 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:23.752311    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9cdd4ce5-8946-4af1-be44-93650764e39e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ndbqd\" (UID: \"9cdd4ce5-8946-4af1-be44-93650764e39e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ndbqd"
Sep 14 04:53:23.752413 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:23.752365    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9cdd4ce5-8946-4af1-be44-93650764e39e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ndbqd\" (UID: \"9cdd4ce5-8946-4af1-be44-93650764e39e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ndbqd"
Sep 14 04:53:23.760638 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:23.760609    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrsgw\" (UniqueName: \"kubernetes.io/projected/9cdd4ce5-8946-4af1-be44-93650764e39e-kube-api-access-zrsgw\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ndbqd\" (UID: \"9cdd4ce5-8946-4af1-be44-93650764e39e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ndbqd"
Sep 14 04:53:23.795182 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:23.795162    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ndbqd"
Sep 14 04:53:24.062900 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:24.062837    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ndbqd"]
Sep 14 04:53:24.064971 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:53:24.064944    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cdd4ce5_8946_4af1_be44_93650764e39e.slice/crio-d2f76858a1dcbb620c7928eb87750a5c976b299ff5231f06098ec831d2ea59a7 WatchSource:0}: Error finding container d2f76858a1dcbb620c7928eb87750a5c976b299ff5231f06098ec831d2ea59a7: Status 404 returned error can't find the container with id d2f76858a1dcbb620c7928eb87750a5c976b299ff5231f06098ec831d2ea59a7
Sep 14 04:53:24.810060 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:24.810028    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ndbqd" event={"ID":"9cdd4ce5-8946-4af1-be44-93650764e39e","Type":"ContainerStarted","Data":"d2f76858a1dcbb620c7928eb87750a5c976b299ff5231f06098ec831d2ea59a7"}
Sep 14 04:53:30.832914 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:30.832843    2565 generic.go:356] "Generic (PLEG): container finished" podID="9cdd4ce5-8946-4af1-be44-93650764e39e" containerID="04955b9c51d9c1c2b84ad641a719e2a9b0d179b6e4fd586601eb14d44722be8b" exitCode=0
Sep 14 04:53:30.833401 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:30.832924    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ndbqd" event={"ID":"9cdd4ce5-8946-4af1-be44-93650764e39e","Type":"ContainerDied","Data":"04955b9c51d9c1c2b84ad641a719e2a9b0d179b6e4fd586601eb14d44722be8b"}
Sep 14 04:53:40.867232 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:40.867194    2565 generic.go:356] "Generic (PLEG): container finished" podID="9cdd4ce5-8946-4af1-be44-93650764e39e" containerID="411946182abcc60d9893594c364a6b072a4cd0bf022758c3b2a83ce59e734552" exitCode=0
Sep 14 04:53:40.867685 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:40.867274    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ndbqd" event={"ID":"9cdd4ce5-8946-4af1-be44-93650764e39e","Type":"ContainerDied","Data":"411946182abcc60d9893594c364a6b072a4cd0bf022758c3b2a83ce59e734552"}
Sep 14 04:53:47.895575 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:47.895540    2565 generic.go:356] "Generic (PLEG): container finished" podID="9cdd4ce5-8946-4af1-be44-93650764e39e" containerID="6a422fe4e1993301b17147febf631b8a651ac17cc29ef3ab17faf53df163af36" exitCode=0
Sep 14 04:53:47.895985 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:47.895622    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ndbqd" event={"ID":"9cdd4ce5-8946-4af1-be44-93650764e39e","Type":"ContainerDied","Data":"6a422fe4e1993301b17147febf631b8a651ac17cc29ef3ab17faf53df163af36"}
Sep 14 04:53:49.024083 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:49.024059    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ndbqd"
Sep 14 04:53:49.141996 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:49.141966    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9cdd4ce5-8946-4af1-be44-93650764e39e-util\") pod \"9cdd4ce5-8946-4af1-be44-93650764e39e\" (UID: \"9cdd4ce5-8946-4af1-be44-93650764e39e\") "
Sep 14 04:53:49.142161 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:49.142007    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrsgw\" (UniqueName: \"kubernetes.io/projected/9cdd4ce5-8946-4af1-be44-93650764e39e-kube-api-access-zrsgw\") pod \"9cdd4ce5-8946-4af1-be44-93650764e39e\" (UID: \"9cdd4ce5-8946-4af1-be44-93650764e39e\") "
Sep 14 04:53:49.142161 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:49.142026    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9cdd4ce5-8946-4af1-be44-93650764e39e-bundle\") pod \"9cdd4ce5-8946-4af1-be44-93650764e39e\" (UID: \"9cdd4ce5-8946-4af1-be44-93650764e39e\") "
Sep 14 04:53:49.142855 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:49.142825    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cdd4ce5-8946-4af1-be44-93650764e39e-bundle" (OuterVolumeSpecName: "bundle") pod "9cdd4ce5-8946-4af1-be44-93650764e39e" (UID: "9cdd4ce5-8946-4af1-be44-93650764e39e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 04:53:49.144261 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:49.144238    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cdd4ce5-8946-4af1-be44-93650764e39e-kube-api-access-zrsgw" (OuterVolumeSpecName: "kube-api-access-zrsgw") pod "9cdd4ce5-8946-4af1-be44-93650764e39e" (UID: "9cdd4ce5-8946-4af1-be44-93650764e39e"). InnerVolumeSpecName "kube-api-access-zrsgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 04:53:49.146556 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:49.146537    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cdd4ce5-8946-4af1-be44-93650764e39e-util" (OuterVolumeSpecName: "util") pod "9cdd4ce5-8946-4af1-be44-93650764e39e" (UID: "9cdd4ce5-8946-4af1-be44-93650764e39e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 04:53:49.243187 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:49.243106    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9cdd4ce5-8946-4af1-be44-93650764e39e-util\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 04:53:49.243187 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:49.243136    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrsgw\" (UniqueName: \"kubernetes.io/projected/9cdd4ce5-8946-4af1-be44-93650764e39e-kube-api-access-zrsgw\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 04:53:49.243187 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:49.243148    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9cdd4ce5-8946-4af1-be44-93650764e39e-bundle\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 04:53:49.904700 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:49.904648    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ndbqd" event={"ID":"9cdd4ce5-8946-4af1-be44-93650764e39e","Type":"ContainerDied","Data":"d2f76858a1dcbb620c7928eb87750a5c976b299ff5231f06098ec831d2ea59a7"}
Sep 14 04:53:49.904869 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:49.904712    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2f76858a1dcbb620c7928eb87750a5c976b299ff5231f06098ec831d2ea59a7"
Sep 14 04:53:49.904869 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:49.904676    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ndbqd"
Sep 14 04:53:56.437859 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:56.437830    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hhbmf"]
Sep 14 04:53:56.438298 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:56.438055    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cdd4ce5-8946-4af1-be44-93650764e39e" containerName="pull"
Sep 14 04:53:56.438298 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:56.438065    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cdd4ce5-8946-4af1-be44-93650764e39e" containerName="pull"
Sep 14 04:53:56.438298 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:56.438081    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cdd4ce5-8946-4af1-be44-93650764e39e" containerName="util"
Sep 14 04:53:56.438298 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:56.438086    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cdd4ce5-8946-4af1-be44-93650764e39e" containerName="util"
Sep 14 04:53:56.438298 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:56.438091    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cdd4ce5-8946-4af1-be44-93650764e39e" containerName="extract"
Sep 14 04:53:56.438298 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:56.438097    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cdd4ce5-8946-4af1-be44-93650764e39e" containerName="extract"
Sep 14 04:53:56.438298 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:56.438135    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cdd4ce5-8946-4af1-be44-93650764e39e" containerName="extract"
Sep 14 04:53:56.441231 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:56.441215    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hhbmf"
Sep 14 04:53:56.443602 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:56.443576    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 14 04:53:56.443789 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:56.443772    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 14 04:53:56.443876 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:56.443841    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-fz4f7\""
Sep 14 04:53:56.453709 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:56.453688    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hhbmf"]
Sep 14 04:53:56.495180 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:56.495155    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2r6p\" (UniqueName: \"kubernetes.io/projected/37991dbe-1c9a-45b5-88ab-27e8abd77f8d-kube-api-access-m2r6p\") pod \"cert-manager-operator-controller-manager-55c444bf86-hhbmf\" (UID: \"37991dbe-1c9a-45b5-88ab-27e8abd77f8d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hhbmf"
Sep 14 04:53:56.495272 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:56.495203    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37991dbe-1c9a-45b5-88ab-27e8abd77f8d-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-hhbmf\" (UID: \"37991dbe-1c9a-45b5-88ab-27e8abd77f8d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hhbmf"
Sep 14 04:53:56.596542 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:56.596519    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37991dbe-1c9a-45b5-88ab-27e8abd77f8d-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-hhbmf\" (UID: \"37991dbe-1c9a-45b5-88ab-27e8abd77f8d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hhbmf"
Sep 14 04:53:56.596644 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:56.596576    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2r6p\" (UniqueName: \"kubernetes.io/projected/37991dbe-1c9a-45b5-88ab-27e8abd77f8d-kube-api-access-m2r6p\") pod \"cert-manager-operator-controller-manager-55c444bf86-hhbmf\" (UID: \"37991dbe-1c9a-45b5-88ab-27e8abd77f8d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hhbmf"
Sep 14 04:53:56.596910 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:56.596894    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37991dbe-1c9a-45b5-88ab-27e8abd77f8d-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-hhbmf\" (UID: \"37991dbe-1c9a-45b5-88ab-27e8abd77f8d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hhbmf"
Sep 14 04:53:56.605802 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:56.605783    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2r6p\" (UniqueName: \"kubernetes.io/projected/37991dbe-1c9a-45b5-88ab-27e8abd77f8d-kube-api-access-m2r6p\") pod \"cert-manager-operator-controller-manager-55c444bf86-hhbmf\" (UID: \"37991dbe-1c9a-45b5-88ab-27e8abd77f8d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hhbmf"
Sep 14 04:53:56.750330 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:56.750272    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hhbmf"
Sep 14 04:53:57.021955 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:57.021745    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hhbmf"]
Sep 14 04:53:57.024900 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:53:57.024872    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37991dbe_1c9a_45b5_88ab_27e8abd77f8d.slice/crio-e8d58593e9f5872355786b0271de3c22a7f9eb9a455d052d875ba64de72df3ed WatchSource:0}: Error finding container e8d58593e9f5872355786b0271de3c22a7f9eb9a455d052d875ba64de72df3ed: Status 404 returned error can't find the container with id e8d58593e9f5872355786b0271de3c22a7f9eb9a455d052d875ba64de72df3ed
Sep 14 04:53:57.932694 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:57.932659    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hhbmf" event={"ID":"37991dbe-1c9a-45b5-88ab-27e8abd77f8d","Type":"ContainerStarted","Data":"e8d58593e9f5872355786b0271de3c22a7f9eb9a455d052d875ba64de72df3ed"}
Sep 14 04:53:58.937722 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:58.937643    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hhbmf" event={"ID":"37991dbe-1c9a-45b5-88ab-27e8abd77f8d","Type":"ContainerStarted","Data":"591c467996419b286ccf62c94b65ae1cfbe795e3c4131392ccd49acf3685d46f"}
Sep 14 04:53:58.961104 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:53:58.961048    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hhbmf" podStartSLOduration=1.357143615 podStartE2EDuration="2.961027726s" totalImagesPullingTime="1.603884111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:53:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:53:58.958844228 +0000 UTC m=+437.134615788" watchObservedRunningTime="2026-09-14 04:53:58.961027726 +0000 UTC m=+437.136799285"
Sep 14 04:54:00.378232 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:00.378187    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvmsl5"]
Sep 14 04:54:00.380273 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:00.380258    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvmsl5"
Sep 14 04:54:00.382921 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:00.382899    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 14 04:54:00.383749 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:00.383733    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 14 04:54:00.383822 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:00.383767    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-szxlp\""
Sep 14 04:54:00.392238 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:00.392216    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvmsl5"]
Sep 14 04:54:00.424742 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:00.424708    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjlk5\" (UniqueName: \"kubernetes.io/projected/7073cdf8-bd7e-4b45-807c-498d24a14dbb-kube-api-access-wjlk5\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvmsl5\" (UID: \"7073cdf8-bd7e-4b45-807c-498d24a14dbb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvmsl5"
Sep 14 04:54:00.424896 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:00.424819    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7073cdf8-bd7e-4b45-807c-498d24a14dbb-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvmsl5\" (UID: \"7073cdf8-bd7e-4b45-807c-498d24a14dbb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvmsl5"
Sep 14 04:54:00.424896 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:00.424855    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7073cdf8-bd7e-4b45-807c-498d24a14dbb-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvmsl5\" (UID: \"7073cdf8-bd7e-4b45-807c-498d24a14dbb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvmsl5"
Sep 14 04:54:00.525907 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:00.525868    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjlk5\" (UniqueName: \"kubernetes.io/projected/7073cdf8-bd7e-4b45-807c-498d24a14dbb-kube-api-access-wjlk5\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvmsl5\" (UID: \"7073cdf8-bd7e-4b45-807c-498d24a14dbb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvmsl5"
Sep 14 04:54:00.526073 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:00.525930    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7073cdf8-bd7e-4b45-807c-498d24a14dbb-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvmsl5\" (UID: \"7073cdf8-bd7e-4b45-807c-498d24a14dbb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvmsl5"
Sep 14 04:54:00.526073 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:00.525953    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7073cdf8-bd7e-4b45-807c-498d24a14dbb-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvmsl5\" (UID: \"7073cdf8-bd7e-4b45-807c-498d24a14dbb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvmsl5"
Sep 14 04:54:00.526300 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:00.526283    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7073cdf8-bd7e-4b45-807c-498d24a14dbb-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvmsl5\" (UID: \"7073cdf8-bd7e-4b45-807c-498d24a14dbb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvmsl5"
Sep 14 04:54:00.526355 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:00.526332    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7073cdf8-bd7e-4b45-807c-498d24a14dbb-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvmsl5\" (UID: \"7073cdf8-bd7e-4b45-807c-498d24a14dbb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvmsl5"
Sep 14 04:54:00.540033 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:00.540011    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjlk5\" (UniqueName: \"kubernetes.io/projected/7073cdf8-bd7e-4b45-807c-498d24a14dbb-kube-api-access-wjlk5\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvmsl5\" (UID: \"7073cdf8-bd7e-4b45-807c-498d24a14dbb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvmsl5"
Sep 14 04:54:00.689263 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:00.689181    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvmsl5"
Sep 14 04:54:00.957443 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:00.957418    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvmsl5"]
Sep 14 04:54:00.959776 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:54:00.959729    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7073cdf8_bd7e_4b45_807c_498d24a14dbb.slice/crio-45b0d1e4729c098711fbd0c1a09d01834c4b50639d10df5eb01df663d45d5118 WatchSource:0}: Error finding container 45b0d1e4729c098711fbd0c1a09d01834c4b50639d10df5eb01df663d45d5118: Status 404 returned error can't find the container with id 45b0d1e4729c098711fbd0c1a09d01834c4b50639d10df5eb01df663d45d5118
Sep 14 04:54:01.950205 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:01.950168    2565 generic.go:356] "Generic (PLEG): container finished" podID="7073cdf8-bd7e-4b45-807c-498d24a14dbb" containerID="9bdf255a8dcc1c17e2a8cb8e01c8aefbd36ecbc7aa3708585c5651bbe20a8b26" exitCode=0
Sep 14 04:54:01.950596 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:01.950255    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvmsl5" event={"ID":"7073cdf8-bd7e-4b45-807c-498d24a14dbb","Type":"ContainerDied","Data":"9bdf255a8dcc1c17e2a8cb8e01c8aefbd36ecbc7aa3708585c5651bbe20a8b26"}
Sep 14 04:54:01.950596 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:01.950304    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvmsl5" event={"ID":"7073cdf8-bd7e-4b45-807c-498d24a14dbb","Type":"ContainerStarted","Data":"45b0d1e4729c098711fbd0c1a09d01834c4b50639d10df5eb01df663d45d5118"}
Sep 14 04:54:03.338955 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:03.338922    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-fn57m"]
Sep 14 04:54:03.342195 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:03.342166    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fn57m"
Sep 14 04:54:03.345227 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:03.345202    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 14 04:54:03.345339 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:03.345202    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-szh8p\""
Sep 14 04:54:03.345339 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:03.345202    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 14 04:54:03.355967 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:03.355910    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-fn57m"]
Sep 14 04:54:03.448547 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:03.448502    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bccf762c-2452-4df0-a3aa-5117ccb0c256-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-fn57m\" (UID: \"bccf762c-2452-4df0-a3aa-5117ccb0c256\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fn57m"
Sep 14 04:54:03.448691 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:03.448566    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlgj4\" (UniqueName: \"kubernetes.io/projected/bccf762c-2452-4df0-a3aa-5117ccb0c256-kube-api-access-zlgj4\") pod \"cert-manager-cainjector-559cdd8c87-fn57m\" (UID: \"bccf762c-2452-4df0-a3aa-5117ccb0c256\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fn57m"
Sep 14 04:54:03.550025 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:03.549993    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bccf762c-2452-4df0-a3aa-5117ccb0c256-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-fn57m\" (UID: \"bccf762c-2452-4df0-a3aa-5117ccb0c256\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fn57m"
Sep 14 04:54:03.550025 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:03.550033    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlgj4\" (UniqueName: \"kubernetes.io/projected/bccf762c-2452-4df0-a3aa-5117ccb0c256-kube-api-access-zlgj4\") pod \"cert-manager-cainjector-559cdd8c87-fn57m\" (UID: \"bccf762c-2452-4df0-a3aa-5117ccb0c256\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fn57m"
Sep 14 04:54:03.559091 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:03.559064    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bccf762c-2452-4df0-a3aa-5117ccb0c256-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-fn57m\" (UID: \"bccf762c-2452-4df0-a3aa-5117ccb0c256\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fn57m"
Sep 14 04:54:03.559313 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:03.559291    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlgj4\" (UniqueName: \"kubernetes.io/projected/bccf762c-2452-4df0-a3aa-5117ccb0c256-kube-api-access-zlgj4\") pod \"cert-manager-cainjector-559cdd8c87-fn57m\" (UID: \"bccf762c-2452-4df0-a3aa-5117ccb0c256\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fn57m"
Sep 14 04:54:03.662859 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:03.662777    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fn57m"
Sep 14 04:54:04.239702 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:04.239612    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-fn57m"]
Sep 14 04:54:04.242162 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:54:04.242134    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbccf762c_2452_4df0_a3aa_5117ccb0c256.slice/crio-d2d4ca49420b16529e0ede5f52236058788f93920b5bcda9119f0a531246b627 WatchSource:0}: Error finding container d2d4ca49420b16529e0ede5f52236058788f93920b5bcda9119f0a531246b627: Status 404 returned error can't find the container with id d2d4ca49420b16529e0ede5f52236058788f93920b5bcda9119f0a531246b627
Sep 14 04:54:04.965116 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:04.965076    2565 generic.go:356] "Generic (PLEG): container finished" podID="7073cdf8-bd7e-4b45-807c-498d24a14dbb" containerID="f4b2213e02726acd3f713402f6a7dde506440db41c0ef0c69c0e691dd6ae4804" exitCode=0
Sep 14 04:54:04.965552 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:04.965159    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvmsl5" event={"ID":"7073cdf8-bd7e-4b45-807c-498d24a14dbb","Type":"ContainerDied","Data":"f4b2213e02726acd3f713402f6a7dde506440db41c0ef0c69c0e691dd6ae4804"}
Sep 14 04:54:04.966552 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:04.966532    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fn57m" event={"ID":"bccf762c-2452-4df0-a3aa-5117ccb0c256","Type":"ContainerStarted","Data":"d2d4ca49420b16529e0ede5f52236058788f93920b5bcda9119f0a531246b627"}
Sep 14 04:54:05.972280 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:05.972242    2565 generic.go:356] "Generic (PLEG): container finished" podID="7073cdf8-bd7e-4b45-807c-498d24a14dbb" containerID="c90956458492ad1c53e35a9ba2effe7ce23bb6842052c8707184752a43c38bb2" exitCode=0
Sep 14 04:54:05.972280 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:05.972284    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvmsl5" event={"ID":"7073cdf8-bd7e-4b45-807c-498d24a14dbb","Type":"ContainerDied","Data":"c90956458492ad1c53e35a9ba2effe7ce23bb6842052c8707184752a43c38bb2"}
Sep 14 04:54:07.097718 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:07.097693    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvmsl5"
Sep 14 04:54:07.178769 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:07.178738    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7073cdf8-bd7e-4b45-807c-498d24a14dbb-util\") pod \"7073cdf8-bd7e-4b45-807c-498d24a14dbb\" (UID: \"7073cdf8-bd7e-4b45-807c-498d24a14dbb\") "
Sep 14 04:54:07.178923 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:07.178787    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7073cdf8-bd7e-4b45-807c-498d24a14dbb-bundle\") pod \"7073cdf8-bd7e-4b45-807c-498d24a14dbb\" (UID: \"7073cdf8-bd7e-4b45-807c-498d24a14dbb\") "
Sep 14 04:54:07.178923 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:07.178831    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjlk5\" (UniqueName: \"kubernetes.io/projected/7073cdf8-bd7e-4b45-807c-498d24a14dbb-kube-api-access-wjlk5\") pod \"7073cdf8-bd7e-4b45-807c-498d24a14dbb\" (UID: \"7073cdf8-bd7e-4b45-807c-498d24a14dbb\") "
Sep 14 04:54:07.179188 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:07.179165    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7073cdf8-bd7e-4b45-807c-498d24a14dbb-bundle" (OuterVolumeSpecName: "bundle") pod "7073cdf8-bd7e-4b45-807c-498d24a14dbb" (UID: "7073cdf8-bd7e-4b45-807c-498d24a14dbb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 04:54:07.180906 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:07.180870    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7073cdf8-bd7e-4b45-807c-498d24a14dbb-kube-api-access-wjlk5" (OuterVolumeSpecName: "kube-api-access-wjlk5") pod "7073cdf8-bd7e-4b45-807c-498d24a14dbb" (UID: "7073cdf8-bd7e-4b45-807c-498d24a14dbb"). InnerVolumeSpecName "kube-api-access-wjlk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 04:54:07.183304 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:07.183267    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7073cdf8-bd7e-4b45-807c-498d24a14dbb-util" (OuterVolumeSpecName: "util") pod "7073cdf8-bd7e-4b45-807c-498d24a14dbb" (UID: "7073cdf8-bd7e-4b45-807c-498d24a14dbb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 04:54:07.280125 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:07.280050    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7073cdf8-bd7e-4b45-807c-498d24a14dbb-util\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 04:54:07.280125 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:07.280076    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7073cdf8-bd7e-4b45-807c-498d24a14dbb-bundle\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 04:54:07.280125 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:07.280085    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjlk5\" (UniqueName: \"kubernetes.io/projected/7073cdf8-bd7e-4b45-807c-498d24a14dbb-kube-api-access-wjlk5\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 04:54:07.981702 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:07.981665    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvmsl5" event={"ID":"7073cdf8-bd7e-4b45-807c-498d24a14dbb","Type":"ContainerDied","Data":"45b0d1e4729c098711fbd0c1a09d01834c4b50639d10df5eb01df663d45d5118"}
Sep 14 04:54:07.981702 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:07.981699    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvmsl5"
Sep 14 04:54:07.981915 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:07.981716    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45b0d1e4729c098711fbd0c1a09d01834c4b50639d10df5eb01df663d45d5118"
Sep 14 04:54:10.763259 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:10.763227    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dnr7w"]
Sep 14 04:54:10.763620 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:10.763476    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7073cdf8-bd7e-4b45-807c-498d24a14dbb" containerName="pull"
Sep 14 04:54:10.763620 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:10.763487    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7073cdf8-bd7e-4b45-807c-498d24a14dbb" containerName="pull"
Sep 14 04:54:10.763620 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:10.763505    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7073cdf8-bd7e-4b45-807c-498d24a14dbb" containerName="extract"
Sep 14 04:54:10.763620 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:10.763511    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7073cdf8-bd7e-4b45-807c-498d24a14dbb" containerName="extract"
Sep 14 04:54:10.763620 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:10.763518    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7073cdf8-bd7e-4b45-807c-498d24a14dbb" containerName="util"
Sep 14 04:54:10.763620 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:10.763524    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7073cdf8-bd7e-4b45-807c-498d24a14dbb" containerName="util"
Sep 14 04:54:10.763620 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:10.763566    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7073cdf8-bd7e-4b45-807c-498d24a14dbb" containerName="extract"
Sep 14 04:54:10.767202 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:10.767182    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-dnr7w"
Sep 14 04:54:10.769612 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:10.769593    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-lr56n\""
Sep 14 04:54:10.777041 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:10.777021    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dnr7w"]
Sep 14 04:54:10.907146 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:10.907108    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c388e828-1907-4316-8090-760cf6166c22-bound-sa-token\") pod \"cert-manager-55b4d954d5-dnr7w\" (UID: \"c388e828-1907-4316-8090-760cf6166c22\") " pod="cert-manager/cert-manager-55b4d954d5-dnr7w"
Sep 14 04:54:10.907290 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:10.907156    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrg7g\" (UniqueName: \"kubernetes.io/projected/c388e828-1907-4316-8090-760cf6166c22-kube-api-access-vrg7g\") pod \"cert-manager-55b4d954d5-dnr7w\" (UID: \"c388e828-1907-4316-8090-760cf6166c22\") " pod="cert-manager/cert-manager-55b4d954d5-dnr7w"
Sep 14 04:54:11.008182 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:11.008147    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c388e828-1907-4316-8090-760cf6166c22-bound-sa-token\") pod \"cert-manager-55b4d954d5-dnr7w\" (UID: \"c388e828-1907-4316-8090-760cf6166c22\") " pod="cert-manager/cert-manager-55b4d954d5-dnr7w"
Sep 14 04:54:11.008373 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:11.008198    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrg7g\" (UniqueName: \"kubernetes.io/projected/c388e828-1907-4316-8090-760cf6166c22-kube-api-access-vrg7g\") pod \"cert-manager-55b4d954d5-dnr7w\" (UID: \"c388e828-1907-4316-8090-760cf6166c22\") " pod="cert-manager/cert-manager-55b4d954d5-dnr7w"
Sep 14 04:54:11.017526 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:11.017447    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c388e828-1907-4316-8090-760cf6166c22-bound-sa-token\") pod \"cert-manager-55b4d954d5-dnr7w\" (UID: \"c388e828-1907-4316-8090-760cf6166c22\") " pod="cert-manager/cert-manager-55b4d954d5-dnr7w"
Sep 14 04:54:11.017526 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:11.017496    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrg7g\" (UniqueName: \"kubernetes.io/projected/c388e828-1907-4316-8090-760cf6166c22-kube-api-access-vrg7g\") pod \"cert-manager-55b4d954d5-dnr7w\" (UID: \"c388e828-1907-4316-8090-760cf6166c22\") " pod="cert-manager/cert-manager-55b4d954d5-dnr7w"
Sep 14 04:54:11.075419 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:11.075390    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-dnr7w"
Sep 14 04:54:11.345172 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:11.345151    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dnr7w"]
Sep 14 04:54:11.347577 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:54:11.347547    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc388e828_1907_4316_8090_760cf6166c22.slice/crio-a49281b84203c8b55686cc21fa8f7c0036a03f7b9bcfa3609ee53b4f5118e330 WatchSource:0}: Error finding container a49281b84203c8b55686cc21fa8f7c0036a03f7b9bcfa3609ee53b4f5118e330: Status 404 returned error can't find the container with id a49281b84203c8b55686cc21fa8f7c0036a03f7b9bcfa3609ee53b4f5118e330
Sep 14 04:54:11.999545 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:11.999502    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-dnr7w" event={"ID":"c388e828-1907-4316-8090-760cf6166c22","Type":"ContainerStarted","Data":"a49281b84203c8b55686cc21fa8f7c0036a03f7b9bcfa3609ee53b4f5118e330"}
Sep 14 04:54:15.013214 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:15.013182    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fn57m" event={"ID":"bccf762c-2452-4df0-a3aa-5117ccb0c256","Type":"ContainerStarted","Data":"1657c35e697a019a391fd0589d7975588cc8769347778522a37eafe94d35b98e"}
Sep 14 04:54:15.014701 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:15.014667    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-dnr7w" event={"ID":"c388e828-1907-4316-8090-760cf6166c22","Type":"ContainerStarted","Data":"d0bcfe27b7769bd3ad8dcadfce2b0c7b64d0aa03aa8fec228db10187eb31ba0c"}
Sep 14 04:54:15.078272 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:15.078213    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fn57m" podStartSLOduration=2.275377537 podStartE2EDuration="12.078194807s" totalImagesPullingTime="9.80281727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:54:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:54:15.076544206 +0000 UTC m=+453.252315766" watchObservedRunningTime="2026-09-14 04:54:15.078194807 +0000 UTC m=+453.253966366"
Sep 14 04:54:21.195455 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:21.195399    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-dnr7w" podStartSLOduration=8.503991548 podStartE2EDuration="11.19537888s" totalImagesPullingTime="2.691387332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:54:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:54:15.143564515 +0000 UTC m=+453.319336073" watchObservedRunningTime="2026-09-14 04:54:21.19537888 +0000 UTC m=+459.371150439"
Sep 14 04:54:21.195908 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:21.195776    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqrgsp"]
Sep 14 04:54:21.199819 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:21.199800    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqrgsp"
Sep 14 04:54:21.205208 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:21.205184    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 14 04:54:21.206146 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:21.205738    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-szxlp\""
Sep 14 04:54:21.208643 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:21.207097    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 14 04:54:21.228646 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:21.228622    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqrgsp"]
Sep 14 04:54:21.294522 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:21.294497    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/793b3b94-c677-4426-8de2-5e902f0bcdaf-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqrgsp\" (UID: \"793b3b94-c677-4426-8de2-5e902f0bcdaf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqrgsp"
Sep 14 04:54:21.294634 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:21.294539    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx9v8\" (UniqueName: \"kubernetes.io/projected/793b3b94-c677-4426-8de2-5e902f0bcdaf-kube-api-access-tx9v8\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqrgsp\" (UID: \"793b3b94-c677-4426-8de2-5e902f0bcdaf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqrgsp"
Sep 14 04:54:21.294634 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:21.294602    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/793b3b94-c677-4426-8de2-5e902f0bcdaf-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqrgsp\" (UID: \"793b3b94-c677-4426-8de2-5e902f0bcdaf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqrgsp"
Sep 14 04:54:21.395500 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:21.395470    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/793b3b94-c677-4426-8de2-5e902f0bcdaf-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqrgsp\" (UID: \"793b3b94-c677-4426-8de2-5e902f0bcdaf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqrgsp"
Sep 14 04:54:21.395616 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:21.395514    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx9v8\" (UniqueName: \"kubernetes.io/projected/793b3b94-c677-4426-8de2-5e902f0bcdaf-kube-api-access-tx9v8\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqrgsp\" (UID: \"793b3b94-c677-4426-8de2-5e902f0bcdaf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqrgsp"
Sep 14 04:54:21.395616 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:21.395564    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/793b3b94-c677-4426-8de2-5e902f0bcdaf-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqrgsp\" (UID: \"793b3b94-c677-4426-8de2-5e902f0bcdaf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqrgsp"
Sep 14 04:54:21.395970 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:21.395948    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/793b3b94-c677-4426-8de2-5e902f0bcdaf-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqrgsp\" (UID: \"793b3b94-c677-4426-8de2-5e902f0bcdaf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqrgsp"
Sep 14 04:54:21.396009 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:21.395956    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/793b3b94-c677-4426-8de2-5e902f0bcdaf-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqrgsp\" (UID: \"793b3b94-c677-4426-8de2-5e902f0bcdaf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqrgsp"
Sep 14 04:54:21.405162 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:21.405137    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx9v8\" (UniqueName: \"kubernetes.io/projected/793b3b94-c677-4426-8de2-5e902f0bcdaf-kube-api-access-tx9v8\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqrgsp\" (UID: \"793b3b94-c677-4426-8de2-5e902f0bcdaf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqrgsp"
Sep 14 04:54:21.508882 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:21.508819    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqrgsp"
Sep 14 04:54:21.781931 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:21.781860    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqrgsp"]
Sep 14 04:54:21.785063 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:54:21.785038    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod793b3b94_c677_4426_8de2_5e902f0bcdaf.slice/crio-8873ae80d955a4d59cf1e55f40aac85d64adca0d7555c6edbfa3adeccbdd8e21 WatchSource:0}: Error finding container 8873ae80d955a4d59cf1e55f40aac85d64adca0d7555c6edbfa3adeccbdd8e21: Status 404 returned error can't find the container with id 8873ae80d955a4d59cf1e55f40aac85d64adca0d7555c6edbfa3adeccbdd8e21
Sep 14 04:54:22.042924 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:22.042850    2565 generic.go:356] "Generic (PLEG): container finished" podID="793b3b94-c677-4426-8de2-5e902f0bcdaf" containerID="6d19bacb2e24991f2f1e4832db17f1864c648ad161dec03c2672a447f548a642" exitCode=0
Sep 14 04:54:22.043057 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:22.042942    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqrgsp" event={"ID":"793b3b94-c677-4426-8de2-5e902f0bcdaf","Type":"ContainerDied","Data":"6d19bacb2e24991f2f1e4832db17f1864c648ad161dec03c2672a447f548a642"}
Sep 14 04:54:22.043057 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:22.042999    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqrgsp" event={"ID":"793b3b94-c677-4426-8de2-5e902f0bcdaf","Type":"ContainerStarted","Data":"8873ae80d955a4d59cf1e55f40aac85d64adca0d7555c6edbfa3adeccbdd8e21"}
Sep 14 04:54:23.048983 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:23.048898    2565 generic.go:356] "Generic (PLEG): container finished" podID="793b3b94-c677-4426-8de2-5e902f0bcdaf" containerID="1d11492b55d4a75ad262bfc8fd3c97002d053b781b1e9fdd537114b3d2e1fdd9" exitCode=0
Sep 14 04:54:23.048983 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:23.048968    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqrgsp" event={"ID":"793b3b94-c677-4426-8de2-5e902f0bcdaf","Type":"ContainerDied","Data":"1d11492b55d4a75ad262bfc8fd3c97002d053b781b1e9fdd537114b3d2e1fdd9"}
Sep 14 04:54:24.054775 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:24.054728    2565 generic.go:356] "Generic (PLEG): container finished" podID="793b3b94-c677-4426-8de2-5e902f0bcdaf" containerID="d1dec88329bf6f634c37939b8a9dcc99556fe30b949528e291f7fd87d7b58935" exitCode=0
Sep 14 04:54:24.055140 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:24.054799    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqrgsp" event={"ID":"793b3b94-c677-4426-8de2-5e902f0bcdaf","Type":"ContainerDied","Data":"d1dec88329bf6f634c37939b8a9dcc99556fe30b949528e291f7fd87d7b58935"}
Sep 14 04:54:25.181644 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:25.181612    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqrgsp"
Sep 14 04:54:25.327111 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:25.327024    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tx9v8\" (UniqueName: \"kubernetes.io/projected/793b3b94-c677-4426-8de2-5e902f0bcdaf-kube-api-access-tx9v8\") pod \"793b3b94-c677-4426-8de2-5e902f0bcdaf\" (UID: \"793b3b94-c677-4426-8de2-5e902f0bcdaf\") "
Sep 14 04:54:25.327111 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:25.327071    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/793b3b94-c677-4426-8de2-5e902f0bcdaf-bundle\") pod \"793b3b94-c677-4426-8de2-5e902f0bcdaf\" (UID: \"793b3b94-c677-4426-8de2-5e902f0bcdaf\") "
Sep 14 04:54:25.327321 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:25.327117    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/793b3b94-c677-4426-8de2-5e902f0bcdaf-util\") pod \"793b3b94-c677-4426-8de2-5e902f0bcdaf\" (UID: \"793b3b94-c677-4426-8de2-5e902f0bcdaf\") "
Sep 14 04:54:25.327820 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:25.327790    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/793b3b94-c677-4426-8de2-5e902f0bcdaf-bundle" (OuterVolumeSpecName: "bundle") pod "793b3b94-c677-4426-8de2-5e902f0bcdaf" (UID: "793b3b94-c677-4426-8de2-5e902f0bcdaf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 04:54:25.329172 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:25.329148    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/793b3b94-c677-4426-8de2-5e902f0bcdaf-kube-api-access-tx9v8" (OuterVolumeSpecName: "kube-api-access-tx9v8") pod "793b3b94-c677-4426-8de2-5e902f0bcdaf" (UID: "793b3b94-c677-4426-8de2-5e902f0bcdaf"). InnerVolumeSpecName "kube-api-access-tx9v8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 04:54:25.332694 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:25.332671    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/793b3b94-c677-4426-8de2-5e902f0bcdaf-util" (OuterVolumeSpecName: "util") pod "793b3b94-c677-4426-8de2-5e902f0bcdaf" (UID: "793b3b94-c677-4426-8de2-5e902f0bcdaf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 04:54:25.428032 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:25.428006    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/793b3b94-c677-4426-8de2-5e902f0bcdaf-util\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 04:54:25.428032 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:25.428029    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx9v8\" (UniqueName: \"kubernetes.io/projected/793b3b94-c677-4426-8de2-5e902f0bcdaf-kube-api-access-tx9v8\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 04:54:25.428152 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:25.428040    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/793b3b94-c677-4426-8de2-5e902f0bcdaf-bundle\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 04:54:26.064809 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:26.064781    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqrgsp"
Sep 14 04:54:26.064992 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:26.064773    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqrgsp" event={"ID":"793b3b94-c677-4426-8de2-5e902f0bcdaf","Type":"ContainerDied","Data":"8873ae80d955a4d59cf1e55f40aac85d64adca0d7555c6edbfa3adeccbdd8e21"}
Sep 14 04:54:26.064992 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:26.064898    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8873ae80d955a4d59cf1e55f40aac85d64adca0d7555c6edbfa3adeccbdd8e21"
Sep 14 04:54:32.335819 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:32.335779    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09b4pp"]
Sep 14 04:54:32.336219 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:32.336110    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="793b3b94-c677-4426-8de2-5e902f0bcdaf" containerName="util"
Sep 14 04:54:32.336219 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:32.336124    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="793b3b94-c677-4426-8de2-5e902f0bcdaf" containerName="util"
Sep 14 04:54:32.336219 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:32.336139    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="793b3b94-c677-4426-8de2-5e902f0bcdaf" containerName="pull"
Sep 14 04:54:32.336219 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:32.336147    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="793b3b94-c677-4426-8de2-5e902f0bcdaf" containerName="pull"
Sep 14 04:54:32.336219 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:32.336167    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="793b3b94-c677-4426-8de2-5e902f0bcdaf" containerName="extract"
Sep 14 04:54:32.336219 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:32.336183    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="793b3b94-c677-4426-8de2-5e902f0bcdaf" containerName="extract"
Sep 14 04:54:32.339451 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:32.336250    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="793b3b94-c677-4426-8de2-5e902f0bcdaf" containerName="extract"
Sep 14 04:54:32.340145 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:32.340117    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09b4pp"
Sep 14 04:54:32.343441 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:32.343418    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-szxlp\""
Sep 14 04:54:32.343548 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:32.343493    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 14 04:54:32.343548 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:32.343523    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 14 04:54:32.346615 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:32.346596    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09b4pp"]
Sep 14 04:54:32.481674 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:32.481646    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxkxb\" (UniqueName: \"kubernetes.io/projected/02e0973e-c6c3-456e-a5f7-34e153f2db15-kube-api-access-nxkxb\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09b4pp\" (UID: \"02e0973e-c6c3-456e-a5f7-34e153f2db15\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09b4pp"
Sep 14 04:54:32.481852 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:32.481705    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02e0973e-c6c3-456e-a5f7-34e153f2db15-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09b4pp\" (UID: \"02e0973e-c6c3-456e-a5f7-34e153f2db15\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09b4pp"
Sep 14 04:54:32.481852 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:32.481808    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02e0973e-c6c3-456e-a5f7-34e153f2db15-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09b4pp\" (UID: \"02e0973e-c6c3-456e-a5f7-34e153f2db15\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09b4pp"
Sep 14 04:54:32.583117 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:32.583088    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02e0973e-c6c3-456e-a5f7-34e153f2db15-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09b4pp\" (UID: \"02e0973e-c6c3-456e-a5f7-34e153f2db15\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09b4pp"
Sep 14 04:54:32.583117 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:32.583121    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02e0973e-c6c3-456e-a5f7-34e153f2db15-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09b4pp\" (UID: \"02e0973e-c6c3-456e-a5f7-34e153f2db15\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09b4pp"
Sep 14 04:54:32.583301 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:32.583159    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxkxb\" (UniqueName: \"kubernetes.io/projected/02e0973e-c6c3-456e-a5f7-34e153f2db15-kube-api-access-nxkxb\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09b4pp\" (UID: \"02e0973e-c6c3-456e-a5f7-34e153f2db15\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09b4pp"
Sep 14 04:54:32.583473 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:32.583452    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02e0973e-c6c3-456e-a5f7-34e153f2db15-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09b4pp\" (UID: \"02e0973e-c6c3-456e-a5f7-34e153f2db15\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09b4pp"
Sep 14 04:54:32.583506 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:32.583474    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02e0973e-c6c3-456e-a5f7-34e153f2db15-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09b4pp\" (UID: \"02e0973e-c6c3-456e-a5f7-34e153f2db15\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09b4pp"
Sep 14 04:54:32.602542 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:32.602483    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxkxb\" (UniqueName: \"kubernetes.io/projected/02e0973e-c6c3-456e-a5f7-34e153f2db15-kube-api-access-nxkxb\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09b4pp\" (UID: \"02e0973e-c6c3-456e-a5f7-34e153f2db15\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09b4pp"
Sep 14 04:54:32.650345 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:32.650315    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09b4pp"
Sep 14 04:54:32.922742 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:32.922718    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09b4pp"]
Sep 14 04:54:32.924624 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:54:32.924594    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02e0973e_c6c3_456e_a5f7_34e153f2db15.slice/crio-6ec786814e2b0808895f2694b9610118556577bd0b3f9864e368aef86d882eea WatchSource:0}: Error finding container 6ec786814e2b0808895f2694b9610118556577bd0b3f9864e368aef86d882eea: Status 404 returned error can't find the container with id 6ec786814e2b0808895f2694b9610118556577bd0b3f9864e368aef86d882eea
Sep 14 04:54:33.093343 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:33.093311    2565 generic.go:356] "Generic (PLEG): container finished" podID="02e0973e-c6c3-456e-a5f7-34e153f2db15" containerID="c4d0ffd65563f1616683cacf6c84ec07b7741aa800c3fa99db7af7a3758972c4" exitCode=0
Sep 14 04:54:33.093503 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:33.093349    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09b4pp" event={"ID":"02e0973e-c6c3-456e-a5f7-34e153f2db15","Type":"ContainerDied","Data":"c4d0ffd65563f1616683cacf6c84ec07b7741aa800c3fa99db7af7a3758972c4"}
Sep 14 04:54:33.093503 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:33.093389    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09b4pp" event={"ID":"02e0973e-c6c3-456e-a5f7-34e153f2db15","Type":"ContainerStarted","Data":"6ec786814e2b0808895f2694b9610118556577bd0b3f9864e368aef86d882eea"}
Sep 14 04:54:34.019850 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:34.019773    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-78b6456dc5-v2m8w"]
Sep 14 04:54:34.022508 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:34.022493    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-v2m8w"
Sep 14 04:54:34.025299 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:34.025275    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 14 04:54:34.025441 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:34.025422    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 14 04:54:34.025514 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:34.025455    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-q4vrb\""
Sep 14 04:54:34.025571 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:34.025543    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 14 04:54:34.025802 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:34.025786    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 14 04:54:34.039349 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:34.039327    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-78b6456dc5-v2m8w"]
Sep 14 04:54:34.100701 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:34.100675    2565 generic.go:356] "Generic (PLEG): container finished" podID="02e0973e-c6c3-456e-a5f7-34e153f2db15" containerID="6239fd275939875186ac6ba8a583f2d745c362639980cd92aa8144f5db21be0f" exitCode=0
Sep 14 04:54:34.100843 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:34.100741    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09b4pp" event={"ID":"02e0973e-c6c3-456e-a5f7-34e153f2db15","Type":"ContainerDied","Data":"6239fd275939875186ac6ba8a583f2d745c362639980cd92aa8144f5db21be0f"}
Sep 14 04:54:34.195053 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:34.195024    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/03e983db-f240-4e1f-af0a-726228526dab-apiservice-cert\") pod \"opendatahub-operator-controller-manager-78b6456dc5-v2m8w\" (UID: \"03e983db-f240-4e1f-af0a-726228526dab\") " pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-v2m8w"
Sep 14 04:54:34.195208 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:34.195066    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/03e983db-f240-4e1f-af0a-726228526dab-manifests\") pod \"opendatahub-operator-controller-manager-78b6456dc5-v2m8w\" (UID: \"03e983db-f240-4e1f-af0a-726228526dab\") " pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-v2m8w"
Sep 14 04:54:34.195208 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:34.195097    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdh67\" (UniqueName: \"kubernetes.io/projected/03e983db-f240-4e1f-af0a-726228526dab-kube-api-access-jdh67\") pod \"opendatahub-operator-controller-manager-78b6456dc5-v2m8w\" (UID: \"03e983db-f240-4e1f-af0a-726228526dab\") " pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-v2m8w"
Sep 14 04:54:34.195208 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:34.195124    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/03e983db-f240-4e1f-af0a-726228526dab-webhook-cert\") pod \"opendatahub-operator-controller-manager-78b6456dc5-v2m8w\" (UID: \"03e983db-f240-4e1f-af0a-726228526dab\") " pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-v2m8w"
Sep 14 04:54:34.195208 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:34.195161    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03e983db-f240-4e1f-af0a-726228526dab-tmp\") pod \"opendatahub-operator-controller-manager-78b6456dc5-v2m8w\" (UID: \"03e983db-f240-4e1f-af0a-726228526dab\") " pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-v2m8w"
Sep 14 04:54:34.295939 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:34.295908    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/03e983db-f240-4e1f-af0a-726228526dab-apiservice-cert\") pod \"opendatahub-operator-controller-manager-78b6456dc5-v2m8w\" (UID: \"03e983db-f240-4e1f-af0a-726228526dab\") " pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-v2m8w"
Sep 14 04:54:34.296040 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:34.295949    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/03e983db-f240-4e1f-af0a-726228526dab-manifests\") pod \"opendatahub-operator-controller-manager-78b6456dc5-v2m8w\" (UID: \"03e983db-f240-4e1f-af0a-726228526dab\") " pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-v2m8w"
Sep 14 04:54:34.296040 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:34.295978    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdh67\" (UniqueName: \"kubernetes.io/projected/03e983db-f240-4e1f-af0a-726228526dab-kube-api-access-jdh67\") pod \"opendatahub-operator-controller-manager-78b6456dc5-v2m8w\" (UID: \"03e983db-f240-4e1f-af0a-726228526dab\") " pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-v2m8w"
Sep 14 04:54:34.296040 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:34.295995    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/03e983db-f240-4e1f-af0a-726228526dab-webhook-cert\") pod \"opendatahub-operator-controller-manager-78b6456dc5-v2m8w\" (UID: \"03e983db-f240-4e1f-af0a-726228526dab\") " pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-v2m8w"
Sep 14 04:54:34.296040 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:34.296018    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03e983db-f240-4e1f-af0a-726228526dab-tmp\") pod \"opendatahub-operator-controller-manager-78b6456dc5-v2m8w\" (UID: \"03e983db-f240-4e1f-af0a-726228526dab\") " pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-v2m8w"
Sep 14 04:54:34.296434 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:34.296406    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/03e983db-f240-4e1f-af0a-726228526dab-manifests\") pod \"opendatahub-operator-controller-manager-78b6456dc5-v2m8w\" (UID: \"03e983db-f240-4e1f-af0a-726228526dab\") " pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-v2m8w"
Sep 14 04:54:34.298122 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:34.298092    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03e983db-f240-4e1f-af0a-726228526dab-tmp\") pod \"opendatahub-operator-controller-manager-78b6456dc5-v2m8w\" (UID: \"03e983db-f240-4e1f-af0a-726228526dab\") " pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-v2m8w"
Sep 14 04:54:34.298230 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:34.298149    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/03e983db-f240-4e1f-af0a-726228526dab-apiservice-cert\") pod \"opendatahub-operator-controller-manager-78b6456dc5-v2m8w\" (UID: \"03e983db-f240-4e1f-af0a-726228526dab\") " pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-v2m8w"
Sep 14 04:54:34.298376 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:34.298356    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/03e983db-f240-4e1f-af0a-726228526dab-webhook-cert\") pod \"opendatahub-operator-controller-manager-78b6456dc5-v2m8w\" (UID: \"03e983db-f240-4e1f-af0a-726228526dab\") " pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-v2m8w"
Sep 14 04:54:34.305117 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:34.305099    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdh67\" (UniqueName: \"kubernetes.io/projected/03e983db-f240-4e1f-af0a-726228526dab-kube-api-access-jdh67\") pod \"opendatahub-operator-controller-manager-78b6456dc5-v2m8w\" (UID: \"03e983db-f240-4e1f-af0a-726228526dab\") " pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-v2m8w"
Sep 14 04:54:34.331618 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:34.331598    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-v2m8w"
Sep 14 04:54:34.608955 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:34.608933    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-78b6456dc5-v2m8w"]
Sep 14 04:54:34.611498 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:54:34.611465    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03e983db_f240_4e1f_af0a_726228526dab.slice/crio-0cfe6aa83267e71bcd73664fa136deeeea5afeb45e1e4ece18286e7559f4640c WatchSource:0}: Error finding container 0cfe6aa83267e71bcd73664fa136deeeea5afeb45e1e4ece18286e7559f4640c: Status 404 returned error can't find the container with id 0cfe6aa83267e71bcd73664fa136deeeea5afeb45e1e4ece18286e7559f4640c
Sep 14 04:54:35.106579 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:35.106538    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-v2m8w" event={"ID":"03e983db-f240-4e1f-af0a-726228526dab","Type":"ContainerStarted","Data":"0cfe6aa83267e71bcd73664fa136deeeea5afeb45e1e4ece18286e7559f4640c"}
Sep 14 04:54:35.109014 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:35.108981    2565 generic.go:356] "Generic (PLEG): container finished" podID="02e0973e-c6c3-456e-a5f7-34e153f2db15" containerID="c55ef09bfa249d283b18fc1c67d5e3756f2b1e2f0053eddfb891973a59f1f744" exitCode=0
Sep 14 04:54:35.109147 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:35.109022    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09b4pp" event={"ID":"02e0973e-c6c3-456e-a5f7-34e153f2db15","Type":"ContainerDied","Data":"c55ef09bfa249d283b18fc1c67d5e3756f2b1e2f0053eddfb891973a59f1f744"}
Sep 14 04:54:37.458352 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:37.458329    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09b4pp"
Sep 14 04:54:37.622563 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:37.622537    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxkxb\" (UniqueName: \"kubernetes.io/projected/02e0973e-c6c3-456e-a5f7-34e153f2db15-kube-api-access-nxkxb\") pod \"02e0973e-c6c3-456e-a5f7-34e153f2db15\" (UID: \"02e0973e-c6c3-456e-a5f7-34e153f2db15\") "
Sep 14 04:54:37.622674 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:37.622578    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02e0973e-c6c3-456e-a5f7-34e153f2db15-util\") pod \"02e0973e-c6c3-456e-a5f7-34e153f2db15\" (UID: \"02e0973e-c6c3-456e-a5f7-34e153f2db15\") "
Sep 14 04:54:37.622674 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:37.622604    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02e0973e-c6c3-456e-a5f7-34e153f2db15-bundle\") pod \"02e0973e-c6c3-456e-a5f7-34e153f2db15\" (UID: \"02e0973e-c6c3-456e-a5f7-34e153f2db15\") "
Sep 14 04:54:37.623280 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:37.623258    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02e0973e-c6c3-456e-a5f7-34e153f2db15-bundle" (OuterVolumeSpecName: "bundle") pod "02e0973e-c6c3-456e-a5f7-34e153f2db15" (UID: "02e0973e-c6c3-456e-a5f7-34e153f2db15"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 04:54:37.624478 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:37.624449    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02e0973e-c6c3-456e-a5f7-34e153f2db15-kube-api-access-nxkxb" (OuterVolumeSpecName: "kube-api-access-nxkxb") pod "02e0973e-c6c3-456e-a5f7-34e153f2db15" (UID: "02e0973e-c6c3-456e-a5f7-34e153f2db15"). InnerVolumeSpecName "kube-api-access-nxkxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 04:54:37.627699 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:37.627681    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02e0973e-c6c3-456e-a5f7-34e153f2db15-util" (OuterVolumeSpecName: "util") pod "02e0973e-c6c3-456e-a5f7-34e153f2db15" (UID: "02e0973e-c6c3-456e-a5f7-34e153f2db15"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 04:54:37.723610 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:37.723525    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxkxb\" (UniqueName: \"kubernetes.io/projected/02e0973e-c6c3-456e-a5f7-34e153f2db15-kube-api-access-nxkxb\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 04:54:37.723610 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:37.723565    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02e0973e-c6c3-456e-a5f7-34e153f2db15-util\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 04:54:37.723610 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:37.723576    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02e0973e-c6c3-456e-a5f7-34e153f2db15-bundle\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 04:54:38.125006 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:38.124970    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-v2m8w" event={"ID":"03e983db-f240-4e1f-af0a-726228526dab","Type":"ContainerStarted","Data":"cf8ee0c9c691e4cb0c6f4f550cfebcb56a4c66dd2caecdaf712db48c03d58f78"}
Sep 14 04:54:38.126887 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:38.126857    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09b4pp" event={"ID":"02e0973e-c6c3-456e-a5f7-34e153f2db15","Type":"ContainerDied","Data":"6ec786814e2b0808895f2694b9610118556577bd0b3f9864e368aef86d882eea"}
Sep 14 04:54:38.127001 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:38.126883    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09b4pp"
Sep 14 04:54:38.127001 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:38.126894    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ec786814e2b0808895f2694b9610118556577bd0b3f9864e368aef86d882eea"
Sep 14 04:54:39.132562 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:39.132528    2565 generic.go:356] "Generic (PLEG): container finished" podID="03e983db-f240-4e1f-af0a-726228526dab" containerID="cf8ee0c9c691e4cb0c6f4f550cfebcb56a4c66dd2caecdaf712db48c03d58f78" exitCode=0
Sep 14 04:54:39.133051 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:39.132581    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-v2m8w" event={"ID":"03e983db-f240-4e1f-af0a-726228526dab","Type":"ContainerDied","Data":"cf8ee0c9c691e4cb0c6f4f550cfebcb56a4c66dd2caecdaf712db48c03d58f78"}
Sep 14 04:54:40.139059 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:40.139021    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-v2m8w" event={"ID":"03e983db-f240-4e1f-af0a-726228526dab","Type":"ContainerStarted","Data":"1d70d9483658c1f259ede4effd4b80b9feea7416b2424ad1b778bb24b8604cdd"}
Sep 14 04:54:40.139465 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:40.139167    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-v2m8w"
Sep 14 04:54:40.244046 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:40.243995    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-v2m8w" podStartSLOduration=3.040155057 podStartE2EDuration="7.243980483s" totalImagesPullingTime="3.213200813s" totalInitContainerRuntime="990.624613ms" isStatefulPod=false podCreationTimestamp="2026-09-14 04:54:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:54:40.238830971 +0000 UTC m=+478.414602530" watchObservedRunningTime="2026-09-14 04:54:40.243980483 +0000 UTC m=+478.419752042"
Sep 14 04:54:51.145472 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:51.145442    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-78b6456dc5-v2m8w"
Sep 14 04:54:54.766704 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:54.766667    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-96745c74f-xnltl"]
Sep 14 04:54:54.767171 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:54.766936    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02e0973e-c6c3-456e-a5f7-34e153f2db15" containerName="pull"
Sep 14 04:54:54.767171 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:54.766947    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="02e0973e-c6c3-456e-a5f7-34e153f2db15" containerName="pull"
Sep 14 04:54:54.767171 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:54.766963    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02e0973e-c6c3-456e-a5f7-34e153f2db15" containerName="extract"
Sep 14 04:54:54.767171 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:54.766969    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="02e0973e-c6c3-456e-a5f7-34e153f2db15" containerName="extract"
Sep 14 04:54:54.767171 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:54.766977    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02e0973e-c6c3-456e-a5f7-34e153f2db15" containerName="util"
Sep 14 04:54:54.767171 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:54.766983    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="02e0973e-c6c3-456e-a5f7-34e153f2db15" containerName="util"
Sep 14 04:54:54.767171 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:54.767024    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="02e0973e-c6c3-456e-a5f7-34e153f2db15" containerName="extract"
Sep 14 04:54:54.772074 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:54.772047    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-96745c74f-xnltl"
Sep 14 04:54:54.775927 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:54.775897    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 14 04:54:54.776176 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:54.776140    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 14 04:54:54.804622 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:54.804589    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-96745c74f-xnltl"]
Sep 14 04:54:54.846108 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:54.846082    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq7n7\" (UniqueName: \"kubernetes.io/projected/2a5dca15-e3df-4cab-898d-6716ee6f4615-kube-api-access-zq7n7\") pod \"kube-auth-proxy-96745c74f-xnltl\" (UID: \"2a5dca15-e3df-4cab-898d-6716ee6f4615\") " pod="openshift-ingress/kube-auth-proxy-96745c74f-xnltl"
Sep 14 04:54:54.846250 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:54.846138    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a5dca15-e3df-4cab-898d-6716ee6f4615-tls-certs\") pod \"kube-auth-proxy-96745c74f-xnltl\" (UID: \"2a5dca15-e3df-4cab-898d-6716ee6f4615\") " pod="openshift-ingress/kube-auth-proxy-96745c74f-xnltl"
Sep 14 04:54:54.846250 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:54.846161    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a5dca15-e3df-4cab-898d-6716ee6f4615-tmp\") pod \"kube-auth-proxy-96745c74f-xnltl\" (UID: \"2a5dca15-e3df-4cab-898d-6716ee6f4615\") " pod="openshift-ingress/kube-auth-proxy-96745c74f-xnltl"
Sep 14 04:54:54.947235 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:54.947200    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a5dca15-e3df-4cab-898d-6716ee6f4615-tls-certs\") pod \"kube-auth-proxy-96745c74f-xnltl\" (UID: \"2a5dca15-e3df-4cab-898d-6716ee6f4615\") " pod="openshift-ingress/kube-auth-proxy-96745c74f-xnltl"
Sep 14 04:54:54.947403 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:54.947246    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a5dca15-e3df-4cab-898d-6716ee6f4615-tmp\") pod \"kube-auth-proxy-96745c74f-xnltl\" (UID: \"2a5dca15-e3df-4cab-898d-6716ee6f4615\") " pod="openshift-ingress/kube-auth-proxy-96745c74f-xnltl"
Sep 14 04:54:54.947403 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:54.947288    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq7n7\" (UniqueName: \"kubernetes.io/projected/2a5dca15-e3df-4cab-898d-6716ee6f4615-kube-api-access-zq7n7\") pod \"kube-auth-proxy-96745c74f-xnltl\" (UID: \"2a5dca15-e3df-4cab-898d-6716ee6f4615\") " pod="openshift-ingress/kube-auth-proxy-96745c74f-xnltl"
Sep 14 04:54:54.947517 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:54:54.947404    2565 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 14 04:54:54.947517 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:54:54.947464    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a5dca15-e3df-4cab-898d-6716ee6f4615-tls-certs podName:2a5dca15-e3df-4cab-898d-6716ee6f4615 nodeName:}" failed. No retries permitted until 2026-09-14 04:54:55.447449336 +0000 UTC m=+493.623220872 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2a5dca15-e3df-4cab-898d-6716ee6f4615-tls-certs") pod "kube-auth-proxy-96745c74f-xnltl" (UID: "2a5dca15-e3df-4cab-898d-6716ee6f4615") : secret "kube-auth-proxy-tls" not found
Sep 14 04:54:54.949496 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:54.949463    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a5dca15-e3df-4cab-898d-6716ee6f4615-tmp\") pod \"kube-auth-proxy-96745c74f-xnltl\" (UID: \"2a5dca15-e3df-4cab-898d-6716ee6f4615\") " pod="openshift-ingress/kube-auth-proxy-96745c74f-xnltl"
Sep 14 04:54:54.984375 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:54.984347    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq7n7\" (UniqueName: \"kubernetes.io/projected/2a5dca15-e3df-4cab-898d-6716ee6f4615-kube-api-access-zq7n7\") pod \"kube-auth-proxy-96745c74f-xnltl\" (UID: \"2a5dca15-e3df-4cab-898d-6716ee6f4615\") " pod="openshift-ingress/kube-auth-proxy-96745c74f-xnltl"
Sep 14 04:54:55.450274 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:55.450242    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a5dca15-e3df-4cab-898d-6716ee6f4615-tls-certs\") pod \"kube-auth-proxy-96745c74f-xnltl\" (UID: \"2a5dca15-e3df-4cab-898d-6716ee6f4615\") " pod="openshift-ingress/kube-auth-proxy-96745c74f-xnltl"
Sep 14 04:54:55.452580 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:55.452553    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a5dca15-e3df-4cab-898d-6716ee6f4615-tls-certs\") pod \"kube-auth-proxy-96745c74f-xnltl\" (UID: \"2a5dca15-e3df-4cab-898d-6716ee6f4615\") " pod="openshift-ingress/kube-auth-proxy-96745c74f-xnltl"
Sep 14 04:54:55.681606 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:55.681569    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-96745c74f-xnltl"
Sep 14 04:54:55.738428 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:55.738401    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5854c46596-shm4c"]
Sep 14 04:54:55.742008 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:55.741989    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5854c46596-shm4c"
Sep 14 04:54:55.756450 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:55.756428    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 14 04:54:55.759090 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:55.759071    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 14 04:54:55.759202 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:55.759089    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-tmngr\""
Sep 14 04:54:55.759653 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:55.759638    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 14 04:54:55.759732 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:55.759663    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 14 04:54:55.759814 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:55.759743    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 14 04:54:55.768804 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:55.768782    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5854c46596-shm4c"]
Sep 14 04:54:55.853448 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:55.853420    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/989234dc-b2f0-4b28-95f6-df049e7fbe4b-metrics-cert\") pod \"lws-controller-manager-5854c46596-shm4c\" (UID: \"989234dc-b2f0-4b28-95f6-df049e7fbe4b\") " pod="openshift-lws-operator/lws-controller-manager-5854c46596-shm4c"
Sep 14 04:54:55.853583 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:55.853456    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/989234dc-b2f0-4b28-95f6-df049e7fbe4b-cert\") pod \"lws-controller-manager-5854c46596-shm4c\" (UID: \"989234dc-b2f0-4b28-95f6-df049e7fbe4b\") " pod="openshift-lws-operator/lws-controller-manager-5854c46596-shm4c"
Sep 14 04:54:55.853583 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:55.853537    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p58ch\" (UniqueName: \"kubernetes.io/projected/989234dc-b2f0-4b28-95f6-df049e7fbe4b-kube-api-access-p58ch\") pod \"lws-controller-manager-5854c46596-shm4c\" (UID: \"989234dc-b2f0-4b28-95f6-df049e7fbe4b\") " pod="openshift-lws-operator/lws-controller-manager-5854c46596-shm4c"
Sep 14 04:54:55.853653 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:55.853584    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/989234dc-b2f0-4b28-95f6-df049e7fbe4b-manager-config\") pod \"lws-controller-manager-5854c46596-shm4c\" (UID: \"989234dc-b2f0-4b28-95f6-df049e7fbe4b\") " pod="openshift-lws-operator/lws-controller-manager-5854c46596-shm4c"
Sep 14 04:54:55.954793 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:55.954738    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p58ch\" (UniqueName: \"kubernetes.io/projected/989234dc-b2f0-4b28-95f6-df049e7fbe4b-kube-api-access-p58ch\") pod \"lws-controller-manager-5854c46596-shm4c\" (UID: \"989234dc-b2f0-4b28-95f6-df049e7fbe4b\") " pod="openshift-lws-operator/lws-controller-manager-5854c46596-shm4c"
Sep 14 04:54:55.954963 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:55.954808    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/989234dc-b2f0-4b28-95f6-df049e7fbe4b-manager-config\") pod \"lws-controller-manager-5854c46596-shm4c\" (UID: \"989234dc-b2f0-4b28-95f6-df049e7fbe4b\") " pod="openshift-lws-operator/lws-controller-manager-5854c46596-shm4c"
Sep 14 04:54:55.954963 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:55.954841    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/989234dc-b2f0-4b28-95f6-df049e7fbe4b-metrics-cert\") pod \"lws-controller-manager-5854c46596-shm4c\" (UID: \"989234dc-b2f0-4b28-95f6-df049e7fbe4b\") " pod="openshift-lws-operator/lws-controller-manager-5854c46596-shm4c"
Sep 14 04:54:55.954963 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:55.954869    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/989234dc-b2f0-4b28-95f6-df049e7fbe4b-cert\") pod \"lws-controller-manager-5854c46596-shm4c\" (UID: \"989234dc-b2f0-4b28-95f6-df049e7fbe4b\") " pod="openshift-lws-operator/lws-controller-manager-5854c46596-shm4c"
Sep 14 04:54:55.955438 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:55.955416    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/989234dc-b2f0-4b28-95f6-df049e7fbe4b-manager-config\") pod \"lws-controller-manager-5854c46596-shm4c\" (UID: \"989234dc-b2f0-4b28-95f6-df049e7fbe4b\") " pod="openshift-lws-operator/lws-controller-manager-5854c46596-shm4c"
Sep 14 04:54:55.957380 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:55.957354    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/989234dc-b2f0-4b28-95f6-df049e7fbe4b-metrics-cert\") pod \"lws-controller-manager-5854c46596-shm4c\" (UID: \"989234dc-b2f0-4b28-95f6-df049e7fbe4b\") " pod="openshift-lws-operator/lws-controller-manager-5854c46596-shm4c"
Sep 14 04:54:55.957464 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:55.957437    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/989234dc-b2f0-4b28-95f6-df049e7fbe4b-cert\") pod \"lws-controller-manager-5854c46596-shm4c\" (UID: \"989234dc-b2f0-4b28-95f6-df049e7fbe4b\") " pod="openshift-lws-operator/lws-controller-manager-5854c46596-shm4c"
Sep 14 04:54:55.972481 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:55.972418    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-96745c74f-xnltl"]
Sep 14 04:54:55.974621 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:55.974598    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p58ch\" (UniqueName: \"kubernetes.io/projected/989234dc-b2f0-4b28-95f6-df049e7fbe4b-kube-api-access-p58ch\") pod \"lws-controller-manager-5854c46596-shm4c\" (UID: \"989234dc-b2f0-4b28-95f6-df049e7fbe4b\") " pod="openshift-lws-operator/lws-controller-manager-5854c46596-shm4c"
Sep 14 04:54:55.974717 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:54:55.974699    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a5dca15_e3df_4cab_898d_6716ee6f4615.slice/crio-91ae1beb4ff4b7aaf81fe1a06a4161bc9ba67ef1c11cf1471ba5165846a4fbcf WatchSource:0}: Error finding container 91ae1beb4ff4b7aaf81fe1a06a4161bc9ba67ef1c11cf1471ba5165846a4fbcf: Status 404 returned error can't find the container with id 91ae1beb4ff4b7aaf81fe1a06a4161bc9ba67ef1c11cf1471ba5165846a4fbcf
Sep 14 04:54:56.052956 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:56.052925    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5854c46596-shm4c"
Sep 14 04:54:56.206013 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:56.205979    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-96745c74f-xnltl" event={"ID":"2a5dca15-e3df-4cab-898d-6716ee6f4615","Type":"ContainerStarted","Data":"91ae1beb4ff4b7aaf81fe1a06a4161bc9ba67ef1c11cf1471ba5165846a4fbcf"}
Sep 14 04:54:56.368591 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:56.368566    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5854c46596-shm4c"]
Sep 14 04:54:56.369643 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:54:56.369619    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod989234dc_b2f0_4b28_95f6_df049e7fbe4b.slice/crio-824510c3fcb637fadaa735abc425b78639b1a7cdb5a86e7cb3a40bb9c57c4e5b WatchSource:0}: Error finding container 824510c3fcb637fadaa735abc425b78639b1a7cdb5a86e7cb3a40bb9c57c4e5b: Status 404 returned error can't find the container with id 824510c3fcb637fadaa735abc425b78639b1a7cdb5a86e7cb3a40bb9c57c4e5b
Sep 14 04:54:57.213928 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:57.213882    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5854c46596-shm4c" event={"ID":"989234dc-b2f0-4b28-95f6-df049e7fbe4b","Type":"ContainerStarted","Data":"824510c3fcb637fadaa735abc425b78639b1a7cdb5a86e7cb3a40bb9c57c4e5b"}
Sep 14 04:54:59.225333 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:59.225233    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-96745c74f-xnltl" event={"ID":"2a5dca15-e3df-4cab-898d-6716ee6f4615","Type":"ContainerStarted","Data":"bdcb82233cf06bc1d3f225e04d219aa4028f28921a5cb40b56443e035c88bee1"}
Sep 14 04:54:59.226884 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:59.226860    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5854c46596-shm4c" event={"ID":"989234dc-b2f0-4b28-95f6-df049e7fbe4b","Type":"ContainerStarted","Data":"9ce1d5561a7e57600e70ced3195640dd198a40bbaecc4b938c57ddd84a4c2ba4"}
Sep 14 04:54:59.227001 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:59.226921    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5854c46596-shm4c"
Sep 14 04:54:59.251995 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:59.251945    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-96745c74f-xnltl" podStartSLOduration=2.327543071 podStartE2EDuration="5.251930139s" totalImagesPullingTime="2.924387068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:54:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:54:59.250515249 +0000 UTC m=+497.426286831" watchObservedRunningTime="2026-09-14 04:54:59.251930139 +0000 UTC m=+497.427701697"
Sep 14 04:54:59.272793 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:54:59.272734    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5854c46596-shm4c" podStartSLOduration=1.749473683 podStartE2EDuration="4.272719893s" totalImagesPullingTime="2.52324621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:54:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:54:59.271075956 +0000 UTC m=+497.446847517" watchObservedRunningTime="2026-09-14 04:54:59.272719893 +0000 UTC m=+497.448491452"
Sep 14 04:55:10.232471 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:55:10.232438    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5854c46596-shm4c"
Sep 14 04:56:09.366516 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.366438    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avszg4"]
Sep 14 04:56:09.368509 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.368492    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avszg4"
Sep 14 04:56:09.371075 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.371052    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 14 04:56:09.371211 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.371109    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 14 04:56:09.371211 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.371147    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-szxlp\""
Sep 14 04:56:09.379323 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.379302    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avszg4"]
Sep 14 04:56:09.400347 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.400320    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bce0b2ab-632f-481c-83f8-a6e2c3fd741e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avszg4\" (UID: \"bce0b2ab-632f-481c-83f8-a6e2c3fd741e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avszg4"
Sep 14 04:56:09.400456 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.400370    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bce0b2ab-632f-481c-83f8-a6e2c3fd741e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avszg4\" (UID: \"bce0b2ab-632f-481c-83f8-a6e2c3fd741e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avszg4"
Sep 14 04:56:09.400456 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.400396    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxtfz\" (UniqueName: \"kubernetes.io/projected/bce0b2ab-632f-481c-83f8-a6e2c3fd741e-kube-api-access-lxtfz\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avszg4\" (UID: \"bce0b2ab-632f-481c-83f8-a6e2c3fd741e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avszg4"
Sep 14 04:56:09.471642 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.471607    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvnwl4"]
Sep 14 04:56:09.473846 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.473830    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvnwl4"
Sep 14 04:56:09.488568 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.488546    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvnwl4"]
Sep 14 04:56:09.500860 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.500833    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvnwl4\" (UID: \"f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvnwl4"
Sep 14 04:56:09.500953 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.500875    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bce0b2ab-632f-481c-83f8-a6e2c3fd741e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avszg4\" (UID: \"bce0b2ab-632f-481c-83f8-a6e2c3fd741e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avszg4"
Sep 14 04:56:09.500953 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.500903    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5n74\" (UniqueName: \"kubernetes.io/projected/f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3-kube-api-access-b5n74\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvnwl4\" (UID: \"f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvnwl4"
Sep 14 04:56:09.500953 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.500939    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvnwl4\" (UID: \"f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvnwl4"
Sep 14 04:56:09.501055 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.500973    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bce0b2ab-632f-481c-83f8-a6e2c3fd741e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avszg4\" (UID: \"bce0b2ab-632f-481c-83f8-a6e2c3fd741e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avszg4"
Sep 14 04:56:09.501055 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.501006    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxtfz\" (UniqueName: \"kubernetes.io/projected/bce0b2ab-632f-481c-83f8-a6e2c3fd741e-kube-api-access-lxtfz\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avszg4\" (UID: \"bce0b2ab-632f-481c-83f8-a6e2c3fd741e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avszg4"
Sep 14 04:56:09.501192 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.501172    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bce0b2ab-632f-481c-83f8-a6e2c3fd741e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avszg4\" (UID: \"bce0b2ab-632f-481c-83f8-a6e2c3fd741e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avszg4"
Sep 14 04:56:09.501258 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.501240    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bce0b2ab-632f-481c-83f8-a6e2c3fd741e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avszg4\" (UID: \"bce0b2ab-632f-481c-83f8-a6e2c3fd741e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avszg4"
Sep 14 04:56:09.510432 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.510404    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxtfz\" (UniqueName: \"kubernetes.io/projected/bce0b2ab-632f-481c-83f8-a6e2c3fd741e-kube-api-access-lxtfz\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avszg4\" (UID: \"bce0b2ab-632f-481c-83f8-a6e2c3fd741e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avszg4"
Sep 14 04:56:09.568845 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.568815    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78sxk4"]
Sep 14 04:56:09.570989 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.570970    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78sxk4"
Sep 14 04:56:09.582155 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.582130    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78sxk4"]
Sep 14 04:56:09.601991 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.601967    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5n74\" (UniqueName: \"kubernetes.io/projected/f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3-kube-api-access-b5n74\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvnwl4\" (UID: \"f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvnwl4"
Sep 14 04:56:09.602117 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.602010    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03430fbf-e65a-4535-82ce-766e347dc6d1-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78sxk4\" (UID: \"03430fbf-e65a-4535-82ce-766e347dc6d1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78sxk4"
Sep 14 04:56:09.602117 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.602039    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03430fbf-e65a-4535-82ce-766e347dc6d1-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78sxk4\" (UID: \"03430fbf-e65a-4535-82ce-766e347dc6d1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78sxk4"
Sep 14 04:56:09.602117 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.602089    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtk86\" (UniqueName: \"kubernetes.io/projected/03430fbf-e65a-4535-82ce-766e347dc6d1-kube-api-access-xtk86\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78sxk4\" (UID: \"03430fbf-e65a-4535-82ce-766e347dc6d1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78sxk4"
Sep 14 04:56:09.602283 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.602121    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvnwl4\" (UID: \"f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvnwl4"
Sep 14 04:56:09.602283 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.602219    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvnwl4\" (UID: \"f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvnwl4"
Sep 14 04:56:09.602450 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.602430    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvnwl4\" (UID: \"f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvnwl4"
Sep 14 04:56:09.602565 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.602513    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvnwl4\" (UID: \"f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvnwl4"
Sep 14 04:56:09.609801 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.609781    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5n74\" (UniqueName: \"kubernetes.io/projected/f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3-kube-api-access-b5n74\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvnwl4\" (UID: \"f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvnwl4"
Sep 14 04:56:09.665896 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.665833    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69xxht"]
Sep 14 04:56:09.667970 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.667954    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69xxht"
Sep 14 04:56:09.677155 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.677120    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avszg4"
Sep 14 04:56:09.678248 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.678226    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69xxht"]
Sep 14 04:56:09.703212 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.703183    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/081a98d9-f05c-4cbf-9fe2-a737b786c61d-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69xxht\" (UID: \"081a98d9-f05c-4cbf-9fe2-a737b786c61d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69xxht"
Sep 14 04:56:09.703336 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.703228    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvr6s\" (UniqueName: \"kubernetes.io/projected/081a98d9-f05c-4cbf-9fe2-a737b786c61d-kube-api-access-rvr6s\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69xxht\" (UID: \"081a98d9-f05c-4cbf-9fe2-a737b786c61d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69xxht"
Sep 14 04:56:09.703336 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.703285    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03430fbf-e65a-4535-82ce-766e347dc6d1-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78sxk4\" (UID: \"03430fbf-e65a-4535-82ce-766e347dc6d1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78sxk4"
Sep 14 04:56:09.703336 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.703307    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03430fbf-e65a-4535-82ce-766e347dc6d1-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78sxk4\" (UID: \"03430fbf-e65a-4535-82ce-766e347dc6d1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78sxk4"
Sep 14 04:56:09.703500 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.703340    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtk86\" (UniqueName: \"kubernetes.io/projected/03430fbf-e65a-4535-82ce-766e347dc6d1-kube-api-access-xtk86\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78sxk4\" (UID: \"03430fbf-e65a-4535-82ce-766e347dc6d1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78sxk4"
Sep 14 04:56:09.703500 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.703378    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/081a98d9-f05c-4cbf-9fe2-a737b786c61d-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69xxht\" (UID: \"081a98d9-f05c-4cbf-9fe2-a737b786c61d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69xxht"
Sep 14 04:56:09.703686 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.703667    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03430fbf-e65a-4535-82ce-766e347dc6d1-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78sxk4\" (UID: \"03430fbf-e65a-4535-82ce-766e347dc6d1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78sxk4"
Sep 14 04:56:09.703738 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.703669    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03430fbf-e65a-4535-82ce-766e347dc6d1-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78sxk4\" (UID: \"03430fbf-e65a-4535-82ce-766e347dc6d1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78sxk4"
Sep 14 04:56:09.712201 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.712179    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtk86\" (UniqueName: \"kubernetes.io/projected/03430fbf-e65a-4535-82ce-766e347dc6d1-kube-api-access-xtk86\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78sxk4\" (UID: \"03430fbf-e65a-4535-82ce-766e347dc6d1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78sxk4"
Sep 14 04:56:09.782129 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.782093    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvnwl4"
Sep 14 04:56:09.803837 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.803807    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/081a98d9-f05c-4cbf-9fe2-a737b786c61d-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69xxht\" (UID: \"081a98d9-f05c-4cbf-9fe2-a737b786c61d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69xxht"
Sep 14 04:56:09.803955 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.803868    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/081a98d9-f05c-4cbf-9fe2-a737b786c61d-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69xxht\" (UID: \"081a98d9-f05c-4cbf-9fe2-a737b786c61d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69xxht"
Sep 14 04:56:09.803955 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.803916    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvr6s\" (UniqueName: \"kubernetes.io/projected/081a98d9-f05c-4cbf-9fe2-a737b786c61d-kube-api-access-rvr6s\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69xxht\" (UID: \"081a98d9-f05c-4cbf-9fe2-a737b786c61d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69xxht"
Sep 14 04:56:09.804230 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.804206    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/081a98d9-f05c-4cbf-9fe2-a737b786c61d-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69xxht\" (UID: \"081a98d9-f05c-4cbf-9fe2-a737b786c61d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69xxht"
Sep 14 04:56:09.804323 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.804215    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/081a98d9-f05c-4cbf-9fe2-a737b786c61d-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69xxht\" (UID: \"081a98d9-f05c-4cbf-9fe2-a737b786c61d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69xxht"
Sep 14 04:56:09.813108 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.813083    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvr6s\" (UniqueName: \"kubernetes.io/projected/081a98d9-f05c-4cbf-9fe2-a737b786c61d-kube-api-access-rvr6s\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69xxht\" (UID: \"081a98d9-f05c-4cbf-9fe2-a737b786c61d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69xxht"
Sep 14 04:56:09.879736 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.879708    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78sxk4"
Sep 14 04:56:09.949586 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.949559    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avszg4"]
Sep 14 04:56:09.952107 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:56:09.952076    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbce0b2ab_632f_481c_83f8_a6e2c3fd741e.slice/crio-79d477a52201d09403c9ef691ef7e32923f3f25d74a4a63d780590a98983225d WatchSource:0}: Error finding container 79d477a52201d09403c9ef691ef7e32923f3f25d74a4a63d780590a98983225d: Status 404 returned error can't find the container with id 79d477a52201d09403c9ef691ef7e32923f3f25d74a4a63d780590a98983225d
Sep 14 04:56:09.977214 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:09.977191    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69xxht"
Sep 14 04:56:10.053691 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:10.053657    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvnwl4"]
Sep 14 04:56:10.055966 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:56:10.055931    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2e4e46b_b9d5_4d6c_bbb9_d981b7b14fd3.slice/crio-a067a8f3a9704f8bfa20ba89bcb955a4d0c9ac110d99edda571d24012336943f WatchSource:0}: Error finding container a067a8f3a9704f8bfa20ba89bcb955a4d0c9ac110d99edda571d24012336943f: Status 404 returned error can't find the container with id a067a8f3a9704f8bfa20ba89bcb955a4d0c9ac110d99edda571d24012336943f
Sep 14 04:56:10.147979 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:10.147953    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78sxk4"]
Sep 14 04:56:10.150228 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:56:10.150201    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03430fbf_e65a_4535_82ce_766e347dc6d1.slice/crio-c2d8789e571a30a92f4255fc48ca8e5ed1cbc21ab82d4fcb465ce6d225c4afff WatchSource:0}: Error finding container c2d8789e571a30a92f4255fc48ca8e5ed1cbc21ab82d4fcb465ce6d225c4afff: Status 404 returned error can't find the container with id c2d8789e571a30a92f4255fc48ca8e5ed1cbc21ab82d4fcb465ce6d225c4afff
Sep 14 04:56:10.247191 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:10.247166    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69xxht"]
Sep 14 04:56:10.249349 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:56:10.249322    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod081a98d9_f05c_4cbf_9fe2_a737b786c61d.slice/crio-9f776b7097526a241c6773d8e0822c021b874c83c65fec0f6c1540f810d720d5 WatchSource:0}: Error finding container 9f776b7097526a241c6773d8e0822c021b874c83c65fec0f6c1540f810d720d5: Status 404 returned error can't find the container with id 9f776b7097526a241c6773d8e0822c021b874c83c65fec0f6c1540f810d720d5
Sep 14 04:56:10.524907 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:10.524824    2565 generic.go:356] "Generic (PLEG): container finished" podID="081a98d9-f05c-4cbf-9fe2-a737b786c61d" containerID="3dc20356bc19f50d8bd6ab9c937cc797ad6cb5cb8545fc65fe719d74c814936b" exitCode=0
Sep 14 04:56:10.525290 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:10.524915    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69xxht" event={"ID":"081a98d9-f05c-4cbf-9fe2-a737b786c61d","Type":"ContainerDied","Data":"3dc20356bc19f50d8bd6ab9c937cc797ad6cb5cb8545fc65fe719d74c814936b"}
Sep 14 04:56:10.525290 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:10.524961    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69xxht" event={"ID":"081a98d9-f05c-4cbf-9fe2-a737b786c61d","Type":"ContainerStarted","Data":"9f776b7097526a241c6773d8e0822c021b874c83c65fec0f6c1540f810d720d5"}
Sep 14 04:56:10.526527 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:10.526509    2565 generic.go:356] "Generic (PLEG): container finished" podID="03430fbf-e65a-4535-82ce-766e347dc6d1" containerID="d82bde27d8f13b1c62fe25c43ed93f8a36df6ed3d1f437dc7f653a3260ac3fc2" exitCode=0
Sep 14 04:56:10.526616 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:10.526591    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78sxk4" event={"ID":"03430fbf-e65a-4535-82ce-766e347dc6d1","Type":"ContainerDied","Data":"d82bde27d8f13b1c62fe25c43ed93f8a36df6ed3d1f437dc7f653a3260ac3fc2"}
Sep 14 04:56:10.526672 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:10.526641    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78sxk4" event={"ID":"03430fbf-e65a-4535-82ce-766e347dc6d1","Type":"ContainerStarted","Data":"c2d8789e571a30a92f4255fc48ca8e5ed1cbc21ab82d4fcb465ce6d225c4afff"}
Sep 14 04:56:10.528097 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:10.528065    2565 generic.go:356] "Generic (PLEG): container finished" podID="f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3" containerID="214a05236ba9c6baa5426f0525a94c46c9098d1cc6392075982a3a7970b10327" exitCode=0
Sep 14 04:56:10.528164 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:10.528117    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvnwl4" event={"ID":"f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3","Type":"ContainerDied","Data":"214a05236ba9c6baa5426f0525a94c46c9098d1cc6392075982a3a7970b10327"}
Sep 14 04:56:10.528164 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:10.528158    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvnwl4" event={"ID":"f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3","Type":"ContainerStarted","Data":"a067a8f3a9704f8bfa20ba89bcb955a4d0c9ac110d99edda571d24012336943f"}
Sep 14 04:56:10.529827 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:10.529805    2565 generic.go:356] "Generic (PLEG): container finished" podID="bce0b2ab-632f-481c-83f8-a6e2c3fd741e" containerID="a7f8526767d35cbfc81de00a12c083757120ef5774ff7bbb397f4a30b93c952d" exitCode=0
Sep 14 04:56:10.529922 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:10.529866    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avszg4" event={"ID":"bce0b2ab-632f-481c-83f8-a6e2c3fd741e","Type":"ContainerDied","Data":"a7f8526767d35cbfc81de00a12c083757120ef5774ff7bbb397f4a30b93c952d"}
Sep 14 04:56:10.529922 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:10.529898    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avszg4" event={"ID":"bce0b2ab-632f-481c-83f8-a6e2c3fd741e","Type":"ContainerStarted","Data":"79d477a52201d09403c9ef691ef7e32923f3f25d74a4a63d780590a98983225d"}
Sep 14 04:56:11.536233 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:11.536207    2565 generic.go:356] "Generic (PLEG): container finished" podID="081a98d9-f05c-4cbf-9fe2-a737b786c61d" containerID="9a46fc1561e0c0619d1eac0aceca1c8274b7ea5aa8e6cae67aede0deb92ee65c" exitCode=0
Sep 14 04:56:11.536550 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:11.536257    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69xxht" event={"ID":"081a98d9-f05c-4cbf-9fe2-a737b786c61d","Type":"ContainerDied","Data":"9a46fc1561e0c0619d1eac0aceca1c8274b7ea5aa8e6cae67aede0deb92ee65c"}
Sep 14 04:56:11.543691 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:11.543652    2565 generic.go:356] "Generic (PLEG): container finished" podID="f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3" containerID="2d1d1a3f7175349532f17873d3867f794c288fb5855fc8160bca383b887bb407" exitCode=0
Sep 14 04:56:11.543807 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:11.543711    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvnwl4" event={"ID":"f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3","Type":"ContainerDied","Data":"2d1d1a3f7175349532f17873d3867f794c288fb5855fc8160bca383b887bb407"}
Sep 14 04:56:12.549330 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:12.549299    2565 generic.go:356] "Generic (PLEG): container finished" podID="081a98d9-f05c-4cbf-9fe2-a737b786c61d" containerID="cb586c29a4c4c417dd0191d5aa0b2088bec2609db5b6778d469cc2d3148c9a58" exitCode=0
Sep 14 04:56:12.549735 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:12.549367    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69xxht" event={"ID":"081a98d9-f05c-4cbf-9fe2-a737b786c61d","Type":"ContainerDied","Data":"cb586c29a4c4c417dd0191d5aa0b2088bec2609db5b6778d469cc2d3148c9a58"}
Sep 14 04:56:12.551184 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:12.551162    2565 generic.go:356] "Generic (PLEG): container finished" podID="03430fbf-e65a-4535-82ce-766e347dc6d1" containerID="e2bb54a58fa13418dddef739d05d37af38707a91d724af058f6e1ebbbf91feb8" exitCode=0
Sep 14 04:56:12.551259 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:12.551228    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78sxk4" event={"ID":"03430fbf-e65a-4535-82ce-766e347dc6d1","Type":"ContainerDied","Data":"e2bb54a58fa13418dddef739d05d37af38707a91d724af058f6e1ebbbf91feb8"}
Sep 14 04:56:12.553418 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:12.553396    2565 generic.go:356] "Generic (PLEG): container finished" podID="f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3" containerID="3ed400f906a1bdd1dcc9f761fbc491975145c7b1235cd82cfbad4b5b9f08811f" exitCode=0
Sep 14 04:56:12.553509 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:12.553483    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvnwl4" event={"ID":"f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3","Type":"ContainerDied","Data":"3ed400f906a1bdd1dcc9f761fbc491975145c7b1235cd82cfbad4b5b9f08811f"}
Sep 14 04:56:13.560679 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:13.560646    2565 generic.go:356] "Generic (PLEG): container finished" podID="03430fbf-e65a-4535-82ce-766e347dc6d1" containerID="cb404cdaa884b5eb0a117ccc40c94e6e85a83fa5b3df43bf91f926fed14aa165" exitCode=0
Sep 14 04:56:13.561075 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:13.560728    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78sxk4" event={"ID":"03430fbf-e65a-4535-82ce-766e347dc6d1","Type":"ContainerDied","Data":"cb404cdaa884b5eb0a117ccc40c94e6e85a83fa5b3df43bf91f926fed14aa165"}
Sep 14 04:56:13.562479 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:13.562453    2565 generic.go:356] "Generic (PLEG): container finished" podID="bce0b2ab-632f-481c-83f8-a6e2c3fd741e" containerID="67e885800e12c9442fa7307a8cdb7f2c395fafd8c5ba913662c7b8b07ed7c22b" exitCode=0
Sep 14 04:56:13.562609 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:13.562540    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avszg4" event={"ID":"bce0b2ab-632f-481c-83f8-a6e2c3fd741e","Type":"ContainerDied","Data":"67e885800e12c9442fa7307a8cdb7f2c395fafd8c5ba913662c7b8b07ed7c22b"}
Sep 14 04:56:13.702647 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:13.702624    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvnwl4"
Sep 14 04:56:13.719512 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:13.719491    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69xxht"
Sep 14 04:56:13.731363 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:13.731342    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/081a98d9-f05c-4cbf-9fe2-a737b786c61d-bundle\") pod \"081a98d9-f05c-4cbf-9fe2-a737b786c61d\" (UID: \"081a98d9-f05c-4cbf-9fe2-a737b786c61d\") "
Sep 14 04:56:13.731475 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:13.731379    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvr6s\" (UniqueName: \"kubernetes.io/projected/081a98d9-f05c-4cbf-9fe2-a737b786c61d-kube-api-access-rvr6s\") pod \"081a98d9-f05c-4cbf-9fe2-a737b786c61d\" (UID: \"081a98d9-f05c-4cbf-9fe2-a737b786c61d\") "
Sep 14 04:56:13.731475 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:13.731405    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/081a98d9-f05c-4cbf-9fe2-a737b786c61d-util\") pod \"081a98d9-f05c-4cbf-9fe2-a737b786c61d\" (UID: \"081a98d9-f05c-4cbf-9fe2-a737b786c61d\") "
Sep 14 04:56:13.731475 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:13.731440    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3-util\") pod \"f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3\" (UID: \"f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3\") "
Sep 14 04:56:13.731631 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:13.731491    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3-bundle\") pod \"f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3\" (UID: \"f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3\") "
Sep 14 04:56:13.731631 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:13.731525    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5n74\" (UniqueName: \"kubernetes.io/projected/f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3-kube-api-access-b5n74\") pod \"f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3\" (UID: \"f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3\") "
Sep 14 04:56:13.732214 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:13.732188    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/081a98d9-f05c-4cbf-9fe2-a737b786c61d-bundle" (OuterVolumeSpecName: "bundle") pod "081a98d9-f05c-4cbf-9fe2-a737b786c61d" (UID: "081a98d9-f05c-4cbf-9fe2-a737b786c61d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 04:56:13.732389 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:13.732346    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3-bundle" (OuterVolumeSpecName: "bundle") pod "f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3" (UID: "f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 04:56:13.734527 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:13.734500    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3-kube-api-access-b5n74" (OuterVolumeSpecName: "kube-api-access-b5n74") pod "f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3" (UID: "f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3"). InnerVolumeSpecName "kube-api-access-b5n74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 04:56:13.734715 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:13.734690    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/081a98d9-f05c-4cbf-9fe2-a737b786c61d-kube-api-access-rvr6s" (OuterVolumeSpecName: "kube-api-access-rvr6s") pod "081a98d9-f05c-4cbf-9fe2-a737b786c61d" (UID: "081a98d9-f05c-4cbf-9fe2-a737b786c61d"). InnerVolumeSpecName "kube-api-access-rvr6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 04:56:13.737797 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:13.737769    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/081a98d9-f05c-4cbf-9fe2-a737b786c61d-util" (OuterVolumeSpecName: "util") pod "081a98d9-f05c-4cbf-9fe2-a737b786c61d" (UID: "081a98d9-f05c-4cbf-9fe2-a737b786c61d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 04:56:13.738394 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:13.738366    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3-util" (OuterVolumeSpecName: "util") pod "f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3" (UID: "f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 04:56:13.832622 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:13.832560    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/081a98d9-f05c-4cbf-9fe2-a737b786c61d-util\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 04:56:13.832622 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:13.832583    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3-util\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 04:56:13.832622 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:13.832591    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3-bundle\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 04:56:13.832622 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:13.832600    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5n74\" (UniqueName: \"kubernetes.io/projected/f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3-kube-api-access-b5n74\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 04:56:13.832622 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:13.832610    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/081a98d9-f05c-4cbf-9fe2-a737b786c61d-bundle\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 04:56:13.832622 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:13.832619    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvr6s\" (UniqueName: \"kubernetes.io/projected/081a98d9-f05c-4cbf-9fe2-a737b786c61d-kube-api-access-rvr6s\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 04:56:14.568744 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:14.568710    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69xxht"
Sep 14 04:56:14.569134 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:14.568708    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69xxht" event={"ID":"081a98d9-f05c-4cbf-9fe2-a737b786c61d","Type":"ContainerDied","Data":"9f776b7097526a241c6773d8e0822c021b874c83c65fec0f6c1540f810d720d5"}
Sep 14 04:56:14.569134 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:14.568813    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f776b7097526a241c6773d8e0822c021b874c83c65fec0f6c1540f810d720d5"
Sep 14 04:56:14.570809 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:14.570783    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvnwl4" event={"ID":"f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3","Type":"ContainerDied","Data":"a067a8f3a9704f8bfa20ba89bcb955a4d0c9ac110d99edda571d24012336943f"}
Sep 14 04:56:14.570809 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:14.570798    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvnwl4"
Sep 14 04:56:14.570998 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:14.570835    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a067a8f3a9704f8bfa20ba89bcb955a4d0c9ac110d99edda571d24012336943f"
Sep 14 04:56:14.573155 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:14.573131    2565 generic.go:356] "Generic (PLEG): container finished" podID="bce0b2ab-632f-481c-83f8-a6e2c3fd741e" containerID="72805c4e1d4136eb4eb5ce2fc7bc64522e0b86829916228407d25ac51811b38e" exitCode=0
Sep 14 04:56:14.573155 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:14.573150    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avszg4" event={"ID":"bce0b2ab-632f-481c-83f8-a6e2c3fd741e","Type":"ContainerDied","Data":"72805c4e1d4136eb4eb5ce2fc7bc64522e0b86829916228407d25ac51811b38e"}
Sep 14 04:56:14.699144 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:14.699124    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78sxk4"
Sep 14 04:56:14.740190 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:14.740157    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03430fbf-e65a-4535-82ce-766e347dc6d1-util\") pod \"03430fbf-e65a-4535-82ce-766e347dc6d1\" (UID: \"03430fbf-e65a-4535-82ce-766e347dc6d1\") "
Sep 14 04:56:14.740322 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:14.740210    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03430fbf-e65a-4535-82ce-766e347dc6d1-bundle\") pod \"03430fbf-e65a-4535-82ce-766e347dc6d1\" (UID: \"03430fbf-e65a-4535-82ce-766e347dc6d1\") "
Sep 14 04:56:14.740322 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:14.740240    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtk86\" (UniqueName: \"kubernetes.io/projected/03430fbf-e65a-4535-82ce-766e347dc6d1-kube-api-access-xtk86\") pod \"03430fbf-e65a-4535-82ce-766e347dc6d1\" (UID: \"03430fbf-e65a-4535-82ce-766e347dc6d1\") "
Sep 14 04:56:14.740664 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:14.740638    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03430fbf-e65a-4535-82ce-766e347dc6d1-bundle" (OuterVolumeSpecName: "bundle") pod "03430fbf-e65a-4535-82ce-766e347dc6d1" (UID: "03430fbf-e65a-4535-82ce-766e347dc6d1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 04:56:14.742357 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:14.742337    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03430fbf-e65a-4535-82ce-766e347dc6d1-kube-api-access-xtk86" (OuterVolumeSpecName: "kube-api-access-xtk86") pod "03430fbf-e65a-4535-82ce-766e347dc6d1" (UID: "03430fbf-e65a-4535-82ce-766e347dc6d1"). InnerVolumeSpecName "kube-api-access-xtk86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 04:56:14.745365 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:14.745333    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03430fbf-e65a-4535-82ce-766e347dc6d1-util" (OuterVolumeSpecName: "util") pod "03430fbf-e65a-4535-82ce-766e347dc6d1" (UID: "03430fbf-e65a-4535-82ce-766e347dc6d1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 04:56:14.841651 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:14.841625    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03430fbf-e65a-4535-82ce-766e347dc6d1-util\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 04:56:14.841651 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:14.841645    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03430fbf-e65a-4535-82ce-766e347dc6d1-bundle\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 04:56:14.841651 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:14.841654    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtk86\" (UniqueName: \"kubernetes.io/projected/03430fbf-e65a-4535-82ce-766e347dc6d1-kube-api-access-xtk86\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 04:56:15.581096 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:15.581066    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78sxk4"
Sep 14 04:56:15.581482 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:15.581100    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78sxk4" event={"ID":"03430fbf-e65a-4535-82ce-766e347dc6d1","Type":"ContainerDied","Data":"c2d8789e571a30a92f4255fc48ca8e5ed1cbc21ab82d4fcb465ce6d225c4afff"}
Sep 14 04:56:15.581482 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:15.581144    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2d8789e571a30a92f4255fc48ca8e5ed1cbc21ab82d4fcb465ce6d225c4afff"
Sep 14 04:56:15.701729 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:15.701707    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avszg4"
Sep 14 04:56:15.749483 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:15.749455    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bce0b2ab-632f-481c-83f8-a6e2c3fd741e-util\") pod \"bce0b2ab-632f-481c-83f8-a6e2c3fd741e\" (UID: \"bce0b2ab-632f-481c-83f8-a6e2c3fd741e\") "
Sep 14 04:56:15.749483 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:15.749487    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxtfz\" (UniqueName: \"kubernetes.io/projected/bce0b2ab-632f-481c-83f8-a6e2c3fd741e-kube-api-access-lxtfz\") pod \"bce0b2ab-632f-481c-83f8-a6e2c3fd741e\" (UID: \"bce0b2ab-632f-481c-83f8-a6e2c3fd741e\") "
Sep 14 04:56:15.749637 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:15.749540    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bce0b2ab-632f-481c-83f8-a6e2c3fd741e-bundle\") pod \"bce0b2ab-632f-481c-83f8-a6e2c3fd741e\" (UID: \"bce0b2ab-632f-481c-83f8-a6e2c3fd741e\") "
Sep 14 04:56:15.750164 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:15.750135    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce0b2ab-632f-481c-83f8-a6e2c3fd741e-bundle" (OuterVolumeSpecName: "bundle") pod "bce0b2ab-632f-481c-83f8-a6e2c3fd741e" (UID: "bce0b2ab-632f-481c-83f8-a6e2c3fd741e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 04:56:15.751434 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:15.751415    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bce0b2ab-632f-481c-83f8-a6e2c3fd741e-kube-api-access-lxtfz" (OuterVolumeSpecName: "kube-api-access-lxtfz") pod "bce0b2ab-632f-481c-83f8-a6e2c3fd741e" (UID: "bce0b2ab-632f-481c-83f8-a6e2c3fd741e"). InnerVolumeSpecName "kube-api-access-lxtfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 04:56:15.756651 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:15.756627    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce0b2ab-632f-481c-83f8-a6e2c3fd741e-util" (OuterVolumeSpecName: "util") pod "bce0b2ab-632f-481c-83f8-a6e2c3fd741e" (UID: "bce0b2ab-632f-481c-83f8-a6e2c3fd741e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 04:56:15.850909 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:15.850883    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bce0b2ab-632f-481c-83f8-a6e2c3fd741e-bundle\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 04:56:15.850909 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:15.850907    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bce0b2ab-632f-481c-83f8-a6e2c3fd741e-util\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 04:56:15.851108 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:15.850915    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxtfz\" (UniqueName: \"kubernetes.io/projected/bce0b2ab-632f-481c-83f8-a6e2c3fd741e-kube-api-access-lxtfz\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 04:56:16.587454 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:16.587424    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avszg4"
Sep 14 04:56:16.587454 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:16.587446    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avszg4" event={"ID":"bce0b2ab-632f-481c-83f8-a6e2c3fd741e","Type":"ContainerDied","Data":"79d477a52201d09403c9ef691ef7e32923f3f25d74a4a63d780590a98983225d"}
Sep 14 04:56:16.587888 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:16.587495    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79d477a52201d09403c9ef691ef7e32923f3f25d74a4a63d780590a98983225d"
Sep 14 04:56:42.281890 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:42.281855    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kndxq_e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae/ovn-acl-logging/0.log"
Sep 14 04:56:42.285514 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:56:42.285495    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kndxq_e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae/ovn-acl-logging/0.log"
Sep 14 04:57:29.134567 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.134534    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 04:57:29.135028 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.134830    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03430fbf-e65a-4535-82ce-766e347dc6d1" containerName="pull"
Sep 14 04:57:29.135028 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.134842    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="03430fbf-e65a-4535-82ce-766e347dc6d1" containerName="pull"
Sep 14 04:57:29.135028 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.134851    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bce0b2ab-632f-481c-83f8-a6e2c3fd741e" containerName="pull"
Sep 14 04:57:29.135028 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.134857    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bce0b2ab-632f-481c-83f8-a6e2c3fd741e" containerName="pull"
Sep 14 04:57:29.135028 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.134864    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3" containerName="util"
Sep 14 04:57:29.135028 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.134870    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3" containerName="util"
Sep 14 04:57:29.135028 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.134875    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="081a98d9-f05c-4cbf-9fe2-a737b786c61d" containerName="extract"
Sep 14 04:57:29.135028 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.134880    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="081a98d9-f05c-4cbf-9fe2-a737b786c61d" containerName="extract"
Sep 14 04:57:29.135028 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.134887    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bce0b2ab-632f-481c-83f8-a6e2c3fd741e" containerName="util"
Sep 14 04:57:29.135028 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.134892    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bce0b2ab-632f-481c-83f8-a6e2c3fd741e" containerName="util"
Sep 14 04:57:29.135028 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.134900    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03430fbf-e65a-4535-82ce-766e347dc6d1" containerName="util"
Sep 14 04:57:29.135028 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.134906    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="03430fbf-e65a-4535-82ce-766e347dc6d1" containerName="util"
Sep 14 04:57:29.135028 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.134911    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="081a98d9-f05c-4cbf-9fe2-a737b786c61d" containerName="pull"
Sep 14 04:57:29.135028 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.134916    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="081a98d9-f05c-4cbf-9fe2-a737b786c61d" containerName="pull"
Sep 14 04:57:29.135028 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.134923    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03430fbf-e65a-4535-82ce-766e347dc6d1" containerName="extract"
Sep 14 04:57:29.135028 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.134928    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="03430fbf-e65a-4535-82ce-766e347dc6d1" containerName="extract"
Sep 14 04:57:29.135028 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.134938    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3" containerName="pull"
Sep 14 04:57:29.135028 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.134943    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3" containerName="pull"
Sep 14 04:57:29.135028 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.134953    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="081a98d9-f05c-4cbf-9fe2-a737b786c61d" containerName="util"
Sep 14 04:57:29.135028 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.134958    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="081a98d9-f05c-4cbf-9fe2-a737b786c61d" containerName="util"
Sep 14 04:57:29.135028 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.134964    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3" containerName="extract"
Sep 14 04:57:29.135028 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.134969    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3" containerName="extract"
Sep 14 04:57:29.135028 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.134975    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bce0b2ab-632f-481c-83f8-a6e2c3fd741e" containerName="extract"
Sep 14 04:57:29.135028 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.134979    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bce0b2ab-632f-481c-83f8-a6e2c3fd741e" containerName="extract"
Sep 14 04:57:29.135028 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.135031    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2e4e46b-b9d5-4d6c-bbb9-d981b7b14fd3" containerName="extract"
Sep 14 04:57:29.135028 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.135038    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="03430fbf-e65a-4535-82ce-766e347dc6d1" containerName="extract"
Sep 14 04:57:29.135028 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.135044    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bce0b2ab-632f-481c-83f8-a6e2c3fd741e" containerName="extract"
Sep 14 04:57:29.135803 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.135049    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="081a98d9-f05c-4cbf-9fe2-a737b786c61d" containerName="extract"
Sep 14 04:57:29.137987 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.137968    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"
Sep 14 04:57:29.140711 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.140686    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 14 04:57:29.140839 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.140819    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 14 04:57:29.140959 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.140944    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-74g7q\""
Sep 14 04:57:29.141554 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.141535    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 14 04:57:29.148203 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.148179    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 04:57:29.186260 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.186224    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 04:57:29.193469 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.193446    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sb5s\" (UniqueName: \"kubernetes.io/projected/98d95388-675b-41a4-a488-c76222746864-kube-api-access-9sb5s\") pod \"limitador-limitador-84994f7ffc-dbf8d\" (UID: \"98d95388-675b-41a4-a488-c76222746864\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"
Sep 14 04:57:29.193555 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.193480    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/98d95388-675b-41a4-a488-c76222746864-config-file\") pod \"limitador-limitador-84994f7ffc-dbf8d\" (UID: \"98d95388-675b-41a4-a488-c76222746864\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"
Sep 14 04:57:29.294600 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.294565    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sb5s\" (UniqueName: \"kubernetes.io/projected/98d95388-675b-41a4-a488-c76222746864-kube-api-access-9sb5s\") pod \"limitador-limitador-84994f7ffc-dbf8d\" (UID: \"98d95388-675b-41a4-a488-c76222746864\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"
Sep 14 04:57:29.294600 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.294606    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/98d95388-675b-41a4-a488-c76222746864-config-file\") pod \"limitador-limitador-84994f7ffc-dbf8d\" (UID: \"98d95388-675b-41a4-a488-c76222746864\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"
Sep 14 04:57:29.295230 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.295212    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/98d95388-675b-41a4-a488-c76222746864-config-file\") pod \"limitador-limitador-84994f7ffc-dbf8d\" (UID: \"98d95388-675b-41a4-a488-c76222746864\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"
Sep 14 04:57:29.303225 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.303201    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sb5s\" (UniqueName: \"kubernetes.io/projected/98d95388-675b-41a4-a488-c76222746864-kube-api-access-9sb5s\") pod \"limitador-limitador-84994f7ffc-dbf8d\" (UID: \"98d95388-675b-41a4-a488-c76222746864\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"
Sep 14 04:57:29.449410 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.449317    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"
Sep 14 04:57:29.605103 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.605070    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-bkqbq"]
Sep 14 04:57:29.608415 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.608398    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-bkqbq"
Sep 14 04:57:29.610941 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.610920    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-pkggr\""
Sep 14 04:57:29.616786 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.616747    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-bkqbq"]
Sep 14 04:57:29.698343 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.698311    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7jh6\" (UniqueName: \"kubernetes.io/projected/03afd611-91a2-4c94-998d-9a1640e9b5a9-kube-api-access-z7jh6\") pod \"authorino-59c974bd48-bkqbq\" (UID: \"03afd611-91a2-4c94-998d-9a1640e9b5a9\") " pod="kuadrant-system/authorino-59c974bd48-bkqbq"
Sep 14 04:57:29.720204 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.720182    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 04:57:29.721802 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:57:29.721737    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98d95388_675b_41a4_a488_c76222746864.slice/crio-77d893fd7472e51d4dee89a09341875f7f1e29a037033ce0a1d9478e188b7621 WatchSource:0}: Error finding container 77d893fd7472e51d4dee89a09341875f7f1e29a037033ce0a1d9478e188b7621: Status 404 returned error can't find the container with id 77d893fd7472e51d4dee89a09341875f7f1e29a037033ce0a1d9478e188b7621
Sep 14 04:57:29.799189 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.799160    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7jh6\" (UniqueName: \"kubernetes.io/projected/03afd611-91a2-4c94-998d-9a1640e9b5a9-kube-api-access-z7jh6\") pod \"authorino-59c974bd48-bkqbq\" (UID: \"03afd611-91a2-4c94-998d-9a1640e9b5a9\") " pod="kuadrant-system/authorino-59c974bd48-bkqbq"
Sep 14 04:57:29.819336 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.819311    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7jh6\" (UniqueName: \"kubernetes.io/projected/03afd611-91a2-4c94-998d-9a1640e9b5a9-kube-api-access-z7jh6\") pod \"authorino-59c974bd48-bkqbq\" (UID: \"03afd611-91a2-4c94-998d-9a1640e9b5a9\") " pod="kuadrant-system/authorino-59c974bd48-bkqbq"
Sep 14 04:57:29.909483 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.909450    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-dbf8d" event={"ID":"98d95388-675b-41a4-a488-c76222746864","Type":"ContainerStarted","Data":"77d893fd7472e51d4dee89a09341875f7f1e29a037033ce0a1d9478e188b7621"}
Sep 14 04:57:29.918178 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:29.918145    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-bkqbq"
Sep 14 04:57:30.187633 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:30.187605    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-bkqbq"]
Sep 14 04:57:30.189076 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:57:30.189044    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03afd611_91a2_4c94_998d_9a1640e9b5a9.slice/crio-d3fc6a9bea6316f4dc35509ee7f8d2b75be367ae3b807fd098f66a3de31650d1 WatchSource:0}: Error finding container d3fc6a9bea6316f4dc35509ee7f8d2b75be367ae3b807fd098f66a3de31650d1: Status 404 returned error can't find the container with id d3fc6a9bea6316f4dc35509ee7f8d2b75be367ae3b807fd098f66a3de31650d1
Sep 14 04:57:30.917368 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:30.917309    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-bkqbq" event={"ID":"03afd611-91a2-4c94-998d-9a1640e9b5a9","Type":"ContainerStarted","Data":"d3fc6a9bea6316f4dc35509ee7f8d2b75be367ae3b807fd098f66a3de31650d1"}
Sep 14 04:57:31.922949 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:31.922906    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-dbf8d" event={"ID":"98d95388-675b-41a4-a488-c76222746864","Type":"ContainerStarted","Data":"d1e0b6b1843d49a313efe9e6a63035cdda66bdb4542dcc37a71842310e7d3553"}
Sep 14 04:57:31.923391 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:31.922985    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"
Sep 14 04:57:31.942553 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:31.942402    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-dbf8d" podStartSLOduration=0.955160936 podStartE2EDuration="2.942301817s" totalImagesPullingTime="1.987140881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:57:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:57:31.941685954 +0000 UTC m=+650.117457517" watchObservedRunningTime="2026-09-14 04:57:31.942301817 +0000 UTC m=+650.118073371"
Sep 14 04:57:33.935404 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:33.935366    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-bkqbq" event={"ID":"03afd611-91a2-4c94-998d-9a1640e9b5a9","Type":"ContainerStarted","Data":"f7e557694c4ecab5bcbafc642db95b7f0c7d6ec58dc5336aa6c4d2329aa601fa"}
Sep 14 04:57:33.951296 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:33.951246    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-bkqbq" podStartSLOduration=1.723665332 podStartE2EDuration="4.951232109s" totalImagesPullingTime="3.227566777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:57:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:57:33.949826049 +0000 UTC m=+652.125597609" watchObservedRunningTime="2026-09-14 04:57:33.951232109 +0000 UTC m=+652.127003667"
Sep 14 04:57:42.930412 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:57:42.930380    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"
Sep 14 04:58:01.197855 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:58:01.197819    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-bkqbq"]
Sep 14 04:58:01.198334 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:58:01.198020    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-bkqbq" podUID="03afd611-91a2-4c94-998d-9a1640e9b5a9" containerName="authorino" containerID="cri-o://f7e557694c4ecab5bcbafc642db95b7f0c7d6ec58dc5336aa6c4d2329aa601fa" gracePeriod=30
Sep 14 04:58:01.285066 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:58:01.285034    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7db5c68f5b-vpzpz"]
Sep 14 04:58:01.288117 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:58:01.288101    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7db5c68f5b-vpzpz"
Sep 14 04:58:01.290949 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:58:01.290929    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 14 04:58:01.295947 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:58:01.295925    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7db5c68f5b-vpzpz"]
Sep 14 04:58:01.344690 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:58:01.344660    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpvhg\" (UniqueName: \"kubernetes.io/projected/77a36ad6-cbb3-4fdf-9793-8494102b2f0c-kube-api-access-jpvhg\") pod \"authorino-7db5c68f5b-vpzpz\" (UID: \"77a36ad6-cbb3-4fdf-9793-8494102b2f0c\") " pod="kuadrant-system/authorino-7db5c68f5b-vpzpz"
Sep 14 04:58:01.344690 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:58:01.344694    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/77a36ad6-cbb3-4fdf-9793-8494102b2f0c-tls-cert\") pod \"authorino-7db5c68f5b-vpzpz\" (UID: \"77a36ad6-cbb3-4fdf-9793-8494102b2f0c\") " pod="kuadrant-system/authorino-7db5c68f5b-vpzpz"
Sep 14 04:58:01.438885 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:58:01.438822    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-bkqbq"
Sep 14 04:58:01.445169 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:58:01.445147    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpvhg\" (UniqueName: \"kubernetes.io/projected/77a36ad6-cbb3-4fdf-9793-8494102b2f0c-kube-api-access-jpvhg\") pod \"authorino-7db5c68f5b-vpzpz\" (UID: \"77a36ad6-cbb3-4fdf-9793-8494102b2f0c\") " pod="kuadrant-system/authorino-7db5c68f5b-vpzpz"
Sep 14 04:58:01.445271 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:58:01.445174    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/77a36ad6-cbb3-4fdf-9793-8494102b2f0c-tls-cert\") pod \"authorino-7db5c68f5b-vpzpz\" (UID: \"77a36ad6-cbb3-4fdf-9793-8494102b2f0c\") " pod="kuadrant-system/authorino-7db5c68f5b-vpzpz"
Sep 14 04:58:01.447399 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:58:01.447373    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/77a36ad6-cbb3-4fdf-9793-8494102b2f0c-tls-cert\") pod \"authorino-7db5c68f5b-vpzpz\" (UID: \"77a36ad6-cbb3-4fdf-9793-8494102b2f0c\") " pod="kuadrant-system/authorino-7db5c68f5b-vpzpz"
Sep 14 04:58:01.454078 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:58:01.453973    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpvhg\" (UniqueName: \"kubernetes.io/projected/77a36ad6-cbb3-4fdf-9793-8494102b2f0c-kube-api-access-jpvhg\") pod \"authorino-7db5c68f5b-vpzpz\" (UID: \"77a36ad6-cbb3-4fdf-9793-8494102b2f0c\") " pod="kuadrant-system/authorino-7db5c68f5b-vpzpz"
Sep 14 04:58:01.545542 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:58:01.545509    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7jh6\" (UniqueName: \"kubernetes.io/projected/03afd611-91a2-4c94-998d-9a1640e9b5a9-kube-api-access-z7jh6\") pod \"03afd611-91a2-4c94-998d-9a1640e9b5a9\" (UID: \"03afd611-91a2-4c94-998d-9a1640e9b5a9\") "
Sep 14 04:58:01.547677 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:58:01.547654    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03afd611-91a2-4c94-998d-9a1640e9b5a9-kube-api-access-z7jh6" (OuterVolumeSpecName: "kube-api-access-z7jh6") pod "03afd611-91a2-4c94-998d-9a1640e9b5a9" (UID: "03afd611-91a2-4c94-998d-9a1640e9b5a9"). InnerVolumeSpecName "kube-api-access-z7jh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 04:58:01.597259 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:58:01.597227    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7db5c68f5b-vpzpz"
Sep 14 04:58:01.646357 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:58:01.646324    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7jh6\" (UniqueName: \"kubernetes.io/projected/03afd611-91a2-4c94-998d-9a1640e9b5a9-kube-api-access-z7jh6\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 04:58:01.868078 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:58:01.868046    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7db5c68f5b-vpzpz"]
Sep 14 04:58:01.871157 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:58:01.871135    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77a36ad6_cbb3_4fdf_9793_8494102b2f0c.slice/crio-fdf1452ed91d5ad453a89415247761a5c91d80cb656dac379becf8299337b273 WatchSource:0}: Error finding container fdf1452ed91d5ad453a89415247761a5c91d80cb656dac379becf8299337b273: Status 404 returned error can't find the container with id fdf1452ed91d5ad453a89415247761a5c91d80cb656dac379becf8299337b273
Sep 14 04:58:01.872450 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:58:01.872432    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 04:58:02.075687 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:58:02.075648    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7db5c68f5b-vpzpz" event={"ID":"77a36ad6-cbb3-4fdf-9793-8494102b2f0c","Type":"ContainerStarted","Data":"fdf1452ed91d5ad453a89415247761a5c91d80cb656dac379becf8299337b273"}
Sep 14 04:58:02.076896 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:58:02.076874    2565 generic.go:356] "Generic (PLEG): container finished" podID="03afd611-91a2-4c94-998d-9a1640e9b5a9" containerID="f7e557694c4ecab5bcbafc642db95b7f0c7d6ec58dc5336aa6c4d2329aa601fa" exitCode=0
Sep 14 04:58:02.076974 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:58:02.076916    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-bkqbq"
Sep 14 04:58:02.076974 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:58:02.076951    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-bkqbq" event={"ID":"03afd611-91a2-4c94-998d-9a1640e9b5a9","Type":"ContainerDied","Data":"f7e557694c4ecab5bcbafc642db95b7f0c7d6ec58dc5336aa6c4d2329aa601fa"}
Sep 14 04:58:02.077037 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:58:02.076985    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-bkqbq" event={"ID":"03afd611-91a2-4c94-998d-9a1640e9b5a9","Type":"ContainerDied","Data":"d3fc6a9bea6316f4dc35509ee7f8d2b75be367ae3b807fd098f66a3de31650d1"}
Sep 14 04:58:02.077037 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:58:02.077002    2565 scope.go:117] "RemoveContainer" containerID="f7e557694c4ecab5bcbafc642db95b7f0c7d6ec58dc5336aa6c4d2329aa601fa"
Sep 14 04:58:02.087833 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:58:02.087304    2565 scope.go:117] "RemoveContainer" containerID="f7e557694c4ecab5bcbafc642db95b7f0c7d6ec58dc5336aa6c4d2329aa601fa"
Sep 14 04:58:02.087833 ip-10-0-133-99 kubenswrapper[2565]: E0914 04:58:02.087797    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7e557694c4ecab5bcbafc642db95b7f0c7d6ec58dc5336aa6c4d2329aa601fa\": container with ID starting with f7e557694c4ecab5bcbafc642db95b7f0c7d6ec58dc5336aa6c4d2329aa601fa not found: ID does not exist" containerID="f7e557694c4ecab5bcbafc642db95b7f0c7d6ec58dc5336aa6c4d2329aa601fa"
Sep 14 04:58:02.088055 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:58:02.087829    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7e557694c4ecab5bcbafc642db95b7f0c7d6ec58dc5336aa6c4d2329aa601fa"} err="failed to get container status \"f7e557694c4ecab5bcbafc642db95b7f0c7d6ec58dc5336aa6c4d2329aa601fa\": rpc error: code = NotFound desc = could not find container \"f7e557694c4ecab5bcbafc642db95b7f0c7d6ec58dc5336aa6c4d2329aa601fa\": container with ID starting with f7e557694c4ecab5bcbafc642db95b7f0c7d6ec58dc5336aa6c4d2329aa601fa not found: ID does not exist"
Sep 14 04:58:02.108330 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:58:02.108305    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-bkqbq"]
Sep 14 04:58:02.113433 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:58:02.113409    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-bkqbq"]
Sep 14 04:58:02.335597 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:58:02.335570    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03afd611-91a2-4c94-998d-9a1640e9b5a9" path="/var/lib/kubelet/pods/03afd611-91a2-4c94-998d-9a1640e9b5a9/volumes"
Sep 14 04:58:03.083024 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:58:03.082989    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7db5c68f5b-vpzpz" event={"ID":"77a36ad6-cbb3-4fdf-9793-8494102b2f0c","Type":"ContainerStarted","Data":"ca4c30304ba970da0eef965b8c2de97957fedc6b9800152286172af1d29d1e86"}
Sep 14 04:58:03.101969 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:58:03.101914    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7db5c68f5b-vpzpz" podStartSLOduration=1.6623657029999999 podStartE2EDuration="2.101895422s" totalImagesPullingTime="439.529719ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:58:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:58:03.099960528 +0000 UTC m=+681.275732090" watchObservedRunningTime="2026-09-14 04:58:03.101895422 +0000 UTC m=+681.277666981"
Sep 14 04:58:44.469236 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:58:44.469198    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 04:59:00.544503 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:59:00.544466    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 04:59:10.132062 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:59:10.132032    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 04:59:19.729953 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:59:19.729868    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 04:59:41.538684 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:59:41.538650    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 04:59:47.148279 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:59:47.148245    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976"]
Sep 14 04:59:47.148633 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:59:47.148520    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03afd611-91a2-4c94-998d-9a1640e9b5a9" containerName="authorino"
Sep 14 04:59:47.148633 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:59:47.148531    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="03afd611-91a2-4c94-998d-9a1640e9b5a9" containerName="authorino"
Sep 14 04:59:47.148633 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:59:47.148599    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="03afd611-91a2-4c94-998d-9a1640e9b5a9" containerName="authorino"
Sep 14 04:59:47.150374 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:59:47.150357    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976"
Sep 14 04:59:47.153874 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:59:47.153848    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 14 04:59:47.154007 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:59:47.153913    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-c7z9x\""
Sep 14 04:59:47.154007 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:59:47.153961    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 14 04:59:47.154007 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:59:47.153848    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 14 04:59:47.162693 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:59:47.162671    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976"]
Sep 14 04:59:47.234603 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:59:47.234566    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/114d8063-89e7-4b4c-8ae2-20e344640805-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976\" (UID: \"114d8063-89e7-4b4c-8ae2-20e344640805\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976"
Sep 14 04:59:47.234789 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:59:47.234634    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/114d8063-89e7-4b4c-8ae2-20e344640805-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976\" (UID: \"114d8063-89e7-4b4c-8ae2-20e344640805\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976"
Sep 14 04:59:47.234789 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:59:47.234662    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/114d8063-89e7-4b4c-8ae2-20e344640805-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976\" (UID: \"114d8063-89e7-4b4c-8ae2-20e344640805\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976"
Sep 14 04:59:47.234789 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:59:47.234695    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp9qm\" (UniqueName: \"kubernetes.io/projected/114d8063-89e7-4b4c-8ae2-20e344640805-kube-api-access-kp9qm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976\" (UID: \"114d8063-89e7-4b4c-8ae2-20e344640805\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976"
Sep 14 04:59:47.234939 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:59:47.234783    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/114d8063-89e7-4b4c-8ae2-20e344640805-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976\" (UID: \"114d8063-89e7-4b4c-8ae2-20e344640805\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976"
Sep 14 04:59:47.234939 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:59:47.234819    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/114d8063-89e7-4b4c-8ae2-20e344640805-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976\" (UID: \"114d8063-89e7-4b4c-8ae2-20e344640805\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976"
Sep 14 04:59:47.336049 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:59:47.336003    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/114d8063-89e7-4b4c-8ae2-20e344640805-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976\" (UID: \"114d8063-89e7-4b4c-8ae2-20e344640805\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976"
Sep 14 04:59:47.336049 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:59:47.336052    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/114d8063-89e7-4b4c-8ae2-20e344640805-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976\" (UID: \"114d8063-89e7-4b4c-8ae2-20e344640805\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976"
Sep 14 04:59:47.336226 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:59:47.336080    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp9qm\" (UniqueName: \"kubernetes.io/projected/114d8063-89e7-4b4c-8ae2-20e344640805-kube-api-access-kp9qm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976\" (UID: \"114d8063-89e7-4b4c-8ae2-20e344640805\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976"
Sep 14 04:59:47.336226 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:59:47.336100    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/114d8063-89e7-4b4c-8ae2-20e344640805-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976\" (UID: \"114d8063-89e7-4b4c-8ae2-20e344640805\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976"
Sep 14 04:59:47.336226 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:59:47.336119    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/114d8063-89e7-4b4c-8ae2-20e344640805-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976\" (UID: \"114d8063-89e7-4b4c-8ae2-20e344640805\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976"
Sep 14 04:59:47.336226 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:59:47.336153    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/114d8063-89e7-4b4c-8ae2-20e344640805-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976\" (UID: \"114d8063-89e7-4b4c-8ae2-20e344640805\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976"
Sep 14 04:59:47.336434 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:59:47.336415    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/114d8063-89e7-4b4c-8ae2-20e344640805-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976\" (UID: \"114d8063-89e7-4b4c-8ae2-20e344640805\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976"
Sep 14 04:59:47.336518 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:59:47.336482    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/114d8063-89e7-4b4c-8ae2-20e344640805-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976\" (UID: \"114d8063-89e7-4b4c-8ae2-20e344640805\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976"
Sep 14 04:59:47.336606 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:59:47.336585    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/114d8063-89e7-4b4c-8ae2-20e344640805-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976\" (UID: \"114d8063-89e7-4b4c-8ae2-20e344640805\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976"
Sep 14 04:59:47.338390 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:59:47.338366    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/114d8063-89e7-4b4c-8ae2-20e344640805-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976\" (UID: \"114d8063-89e7-4b4c-8ae2-20e344640805\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976"
Sep 14 04:59:47.338638 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:59:47.338622    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/114d8063-89e7-4b4c-8ae2-20e344640805-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976\" (UID: \"114d8063-89e7-4b4c-8ae2-20e344640805\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976"
Sep 14 04:59:47.344584 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:59:47.344557    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp9qm\" (UniqueName: \"kubernetes.io/projected/114d8063-89e7-4b4c-8ae2-20e344640805-kube-api-access-kp9qm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976\" (UID: \"114d8063-89e7-4b4c-8ae2-20e344640805\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976"
Sep 14 04:59:47.460435 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:59:47.460376    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976"
Sep 14 04:59:47.727361 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:59:47.727331    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976"]
Sep 14 04:59:47.729777 ip-10-0-133-99 kubenswrapper[2565]: W0914 04:59:47.729731    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod114d8063_89e7_4b4c_8ae2_20e344640805.slice/crio-1a1c11123a6c801b4f03be12b7d3dd5ef49bf30d99b91394834f4483d39bec37 WatchSource:0}: Error finding container 1a1c11123a6c801b4f03be12b7d3dd5ef49bf30d99b91394834f4483d39bec37: Status 404 returned error can't find the container with id 1a1c11123a6c801b4f03be12b7d3dd5ef49bf30d99b91394834f4483d39bec37
Sep 14 04:59:48.595940 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:59:48.595881    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976" event={"ID":"114d8063-89e7-4b4c-8ae2-20e344640805","Type":"ContainerStarted","Data":"1a1c11123a6c801b4f03be12b7d3dd5ef49bf30d99b91394834f4483d39bec37"}
Sep 14 04:59:48.634216 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:59:48.634177    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 04:59:49.603819 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:59:49.603773    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976" event={"ID":"114d8063-89e7-4b4c-8ae2-20e344640805","Type":"ContainerStarted","Data":"2987a64745aea81742ad8fb7243e47cb02ae9e1b933ea7ff5bb539d081902a42"}
Sep 14 04:59:49.626376 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:59:49.626309    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976" podStartSLOduration=1.642479248 podStartE2EDuration="2.626283495s" totalImagesPullingTime="983.804247ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 04:59:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 04:59:49.625198902 +0000 UTC m=+787.800970462" watchObservedRunningTime="2026-09-14 04:59:49.626283495 +0000 UTC m=+787.802055119"
Sep 14 04:59:56.029512 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:59:56.029478    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 04:59:57.461208 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:59:57.461177    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976"
Sep 14 04:59:57.461692 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:59:57.461271    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976"
Sep 14 04:59:57.473903 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:59:57.473874    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976"
Sep 14 04:59:57.651234 ip-10-0-133-99 kubenswrapper[2565]: I0914 04:59:57.651207    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976"
Sep 14 05:00:00.136515 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:00.136480    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29822700-zxqrq"]
Sep 14 05:00:00.139869 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:00.139849    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29822700-zxqrq"
Sep 14 05:00:00.142076 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:00.142054    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k6hpm\""
Sep 14 05:00:00.147115 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:00.147079    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29822700-zxqrq"]
Sep 14 05:00:00.151061 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:00.151037    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29822700-zxqrq"
Sep 14 05:00:00.421893 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:00.421711    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29822700-zxqrq"]
Sep 14 05:00:00.424436 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:00:00.424410    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c78c401_caf2_41eb_8729_1113340b8f2e.slice/crio-603d3f27713eb6ca61fb4c4b50e3ca91cc9a28dcb22536908e9e97e02c902d4c WatchSource:0}: Error finding container 603d3f27713eb6ca61fb4c4b50e3ca91cc9a28dcb22536908e9e97e02c902d4c: Status 404 returned error can't find the container with id 603d3f27713eb6ca61fb4c4b50e3ca91cc9a28dcb22536908e9e97e02c902d4c
Sep 14 05:00:00.655944 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:00.655909    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29822700-zxqrq" event={"ID":"5c78c401-caf2-41eb-8729-1113340b8f2e","Type":"ContainerStarted","Data":"603d3f27713eb6ca61fb4c4b50e3ca91cc9a28dcb22536908e9e97e02c902d4c"}
Sep 14 05:00:03.321056 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:03.321023    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd"]
Sep 14 05:00:03.324669 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:03.324647    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd"
Sep 14 05:00:03.327059 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:03.327035    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 14 05:00:03.335874 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:03.335845    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd"]
Sep 14 05:00:03.372399 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:03.372355    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e7b27aa-4d5a-4784-b6ee-ed3471d5fecf-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd\" (UID: \"8e7b27aa-4d5a-4784-b6ee-ed3471d5fecf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd"
Sep 14 05:00:03.372574 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:03.372418    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e7b27aa-4d5a-4784-b6ee-ed3471d5fecf-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd\" (UID: \"8e7b27aa-4d5a-4784-b6ee-ed3471d5fecf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd"
Sep 14 05:00:03.372574 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:03.372453    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e7b27aa-4d5a-4784-b6ee-ed3471d5fecf-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd\" (UID: \"8e7b27aa-4d5a-4784-b6ee-ed3471d5fecf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd"
Sep 14 05:00:03.372574 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:03.372494    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e7b27aa-4d5a-4784-b6ee-ed3471d5fecf-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd\" (UID: \"8e7b27aa-4d5a-4784-b6ee-ed3471d5fecf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd"
Sep 14 05:00:03.372574 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:03.372534    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp2q2\" (UniqueName: \"kubernetes.io/projected/8e7b27aa-4d5a-4784-b6ee-ed3471d5fecf-kube-api-access-tp2q2\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd\" (UID: \"8e7b27aa-4d5a-4784-b6ee-ed3471d5fecf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd"
Sep 14 05:00:03.372729 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:03.372583    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e7b27aa-4d5a-4784-b6ee-ed3471d5fecf-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd\" (UID: \"8e7b27aa-4d5a-4784-b6ee-ed3471d5fecf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd"
Sep 14 05:00:03.473318 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:03.473281    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e7b27aa-4d5a-4784-b6ee-ed3471d5fecf-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd\" (UID: \"8e7b27aa-4d5a-4784-b6ee-ed3471d5fecf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd"
Sep 14 05:00:03.473318 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:03.473320    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp2q2\" (UniqueName: \"kubernetes.io/projected/8e7b27aa-4d5a-4784-b6ee-ed3471d5fecf-kube-api-access-tp2q2\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd\" (UID: \"8e7b27aa-4d5a-4784-b6ee-ed3471d5fecf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd"
Sep 14 05:00:03.473553 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:03.473344    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e7b27aa-4d5a-4784-b6ee-ed3471d5fecf-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd\" (UID: \"8e7b27aa-4d5a-4784-b6ee-ed3471d5fecf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd"
Sep 14 05:00:03.473553 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:03.473375    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e7b27aa-4d5a-4784-b6ee-ed3471d5fecf-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd\" (UID: \"8e7b27aa-4d5a-4784-b6ee-ed3471d5fecf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd"
Sep 14 05:00:03.473553 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:03.473407    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e7b27aa-4d5a-4784-b6ee-ed3471d5fecf-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd\" (UID: \"8e7b27aa-4d5a-4784-b6ee-ed3471d5fecf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd"
Sep 14 05:00:03.473553 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:03.473433    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e7b27aa-4d5a-4784-b6ee-ed3471d5fecf-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd\" (UID: \"8e7b27aa-4d5a-4784-b6ee-ed3471d5fecf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd"
Sep 14 05:00:03.473787 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:03.473745    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e7b27aa-4d5a-4784-b6ee-ed3471d5fecf-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd\" (UID: \"8e7b27aa-4d5a-4784-b6ee-ed3471d5fecf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd"
Sep 14 05:00:03.473854 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:03.473804    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e7b27aa-4d5a-4784-b6ee-ed3471d5fecf-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd\" (UID: \"8e7b27aa-4d5a-4784-b6ee-ed3471d5fecf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd"
Sep 14 05:00:03.473904 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:03.473867    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e7b27aa-4d5a-4784-b6ee-ed3471d5fecf-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd\" (UID: \"8e7b27aa-4d5a-4784-b6ee-ed3471d5fecf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd"
Sep 14 05:00:03.475708 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:03.475682    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e7b27aa-4d5a-4784-b6ee-ed3471d5fecf-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd\" (UID: \"8e7b27aa-4d5a-4784-b6ee-ed3471d5fecf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd"
Sep 14 05:00:03.475907 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:03.475892    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e7b27aa-4d5a-4784-b6ee-ed3471d5fecf-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd\" (UID: \"8e7b27aa-4d5a-4784-b6ee-ed3471d5fecf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd"
Sep 14 05:00:03.481715 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:03.481681    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp2q2\" (UniqueName: \"kubernetes.io/projected/8e7b27aa-4d5a-4784-b6ee-ed3471d5fecf-kube-api-access-tp2q2\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd\" (UID: \"8e7b27aa-4d5a-4784-b6ee-ed3471d5fecf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd"
Sep 14 05:00:03.634646 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:03.634606    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd"
Sep 14 05:00:03.676351 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:03.676311    2565 generic.go:356] "Generic (PLEG): container finished" podID="5c78c401-caf2-41eb-8729-1113340b8f2e" containerID="d290aad89ace53a0efd9c1a7b01c89c3a0afddc5597d82ea97893e14de48505e" exitCode=0
Sep 14 05:00:03.676533 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:03.676400    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29822700-zxqrq" event={"ID":"5c78c401-caf2-41eb-8729-1113340b8f2e","Type":"ContainerDied","Data":"d290aad89ace53a0efd9c1a7b01c89c3a0afddc5597d82ea97893e14de48505e"}
Sep 14 05:00:03.908345 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:03.908315    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd"]
Sep 14 05:00:03.910122 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:00:03.910094    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e7b27aa_4d5a_4784_b6ee_ed3471d5fecf.slice/crio-298f3f33b10f2618da213ef1bbd92722d8a0d512bf19543b0c8206cd410ffbe0 WatchSource:0}: Error finding container 298f3f33b10f2618da213ef1bbd92722d8a0d512bf19543b0c8206cd410ffbe0: Status 404 returned error can't find the container with id 298f3f33b10f2618da213ef1bbd92722d8a0d512bf19543b0c8206cd410ffbe0
Sep 14 05:00:04.682036 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:04.681997    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd" event={"ID":"8e7b27aa-4d5a-4784-b6ee-ed3471d5fecf","Type":"ContainerStarted","Data":"298f3f33b10f2618da213ef1bbd92722d8a0d512bf19543b0c8206cd410ffbe0"}
Sep 14 05:00:04.804927 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:04.804899    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29822700-zxqrq"
Sep 14 05:00:05.635907 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:05.635870    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:00:05.689489 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:05.689455    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd" event={"ID":"8e7b27aa-4d5a-4784-b6ee-ed3471d5fecf","Type":"ContainerStarted","Data":"aba325ff3434cb5d39ea7cc03b2d49cb7089dde7f6c30c3c32a38f45c6e51b69"}
Sep 14 05:00:05.690993 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:05.690964    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29822700-zxqrq" event={"ID":"5c78c401-caf2-41eb-8729-1113340b8f2e","Type":"ContainerDied","Data":"603d3f27713eb6ca61fb4c4b50e3ca91cc9a28dcb22536908e9e97e02c902d4c"}
Sep 14 05:00:05.691062 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:05.691020    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="603d3f27713eb6ca61fb4c4b50e3ca91cc9a28dcb22536908e9e97e02c902d4c"
Sep 14 05:00:05.691062 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:05.690975    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29822700-zxqrq"
Sep 14 05:00:05.709632 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:05.709580    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd" podStartSLOduration=1.761762276 podStartE2EDuration="2.709562991s" totalImagesPullingTime="947.800715ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:00:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:00:05.708984851 +0000 UTC m=+803.884756412" watchObservedRunningTime="2026-09-14 05:00:05.709562991 +0000 UTC m=+803.885334584"
Sep 14 05:00:13.635211 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:13.635172    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd"
Sep 14 05:00:13.635575 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:13.635234    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd"
Sep 14 05:00:13.647690 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:13.647663    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd"
Sep 14 05:00:13.737748 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:00:13.737717    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd"
Sep 14 05:01:01.543373 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:01.543336    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-78cfd7c89-xbsm5"]
Sep 14 05:01:01.543850 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:01.543622    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c78c401-caf2-41eb-8729-1113340b8f2e" containerName="cleanup"
Sep 14 05:01:01.543850 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:01.543633    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c78c401-caf2-41eb-8729-1113340b8f2e" containerName="cleanup"
Sep 14 05:01:01.543850 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:01.543688    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c78c401-caf2-41eb-8729-1113340b8f2e" containerName="cleanup"
Sep 14 05:01:01.545495 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:01.545479    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78cfd7c89-xbsm5"
Sep 14 05:01:01.553931 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:01.553902    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-78cfd7c89-xbsm5"]
Sep 14 05:01:01.654309 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:01.654270    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b16d10bd-7c66-48bb-b9a4-011e1f9f9e19-tls-cert\") pod \"authorino-78cfd7c89-xbsm5\" (UID: \"b16d10bd-7c66-48bb-b9a4-011e1f9f9e19\") " pod="kuadrant-system/authorino-78cfd7c89-xbsm5"
Sep 14 05:01:01.654309 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:01.654315    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thvcq\" (UniqueName: \"kubernetes.io/projected/b16d10bd-7c66-48bb-b9a4-011e1f9f9e19-kube-api-access-thvcq\") pod \"authorino-78cfd7c89-xbsm5\" (UID: \"b16d10bd-7c66-48bb-b9a4-011e1f9f9e19\") " pod="kuadrant-system/authorino-78cfd7c89-xbsm5"
Sep 14 05:01:01.754661 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:01.754624    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thvcq\" (UniqueName: \"kubernetes.io/projected/b16d10bd-7c66-48bb-b9a4-011e1f9f9e19-kube-api-access-thvcq\") pod \"authorino-78cfd7c89-xbsm5\" (UID: \"b16d10bd-7c66-48bb-b9a4-011e1f9f9e19\") " pod="kuadrant-system/authorino-78cfd7c89-xbsm5"
Sep 14 05:01:01.754847 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:01.754727    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b16d10bd-7c66-48bb-b9a4-011e1f9f9e19-tls-cert\") pod \"authorino-78cfd7c89-xbsm5\" (UID: \"b16d10bd-7c66-48bb-b9a4-011e1f9f9e19\") " pod="kuadrant-system/authorino-78cfd7c89-xbsm5"
Sep 14 05:01:01.757146 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:01.757130    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b16d10bd-7c66-48bb-b9a4-011e1f9f9e19-tls-cert\") pod \"authorino-78cfd7c89-xbsm5\" (UID: \"b16d10bd-7c66-48bb-b9a4-011e1f9f9e19\") " pod="kuadrant-system/authorino-78cfd7c89-xbsm5"
Sep 14 05:01:01.763246 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:01.763226    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thvcq\" (UniqueName: \"kubernetes.io/projected/b16d10bd-7c66-48bb-b9a4-011e1f9f9e19-kube-api-access-thvcq\") pod \"authorino-78cfd7c89-xbsm5\" (UID: \"b16d10bd-7c66-48bb-b9a4-011e1f9f9e19\") " pod="kuadrant-system/authorino-78cfd7c89-xbsm5"
Sep 14 05:01:01.854269 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:01.854231    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78cfd7c89-xbsm5"
Sep 14 05:01:02.120548 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:02.120523    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-78cfd7c89-xbsm5"]
Sep 14 05:01:02.122981 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:01:02.122953    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb16d10bd_7c66_48bb_b9a4_011e1f9f9e19.slice/crio-1c1449a6d6419ae886f435cea24b5308b9e2c668f897efe5eeceeaa91def294e WatchSource:0}: Error finding container 1c1449a6d6419ae886f435cea24b5308b9e2c668f897efe5eeceeaa91def294e: Status 404 returned error can't find the container with id 1c1449a6d6419ae886f435cea24b5308b9e2c668f897efe5eeceeaa91def294e
Sep 14 05:01:02.964982 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:02.964941    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78cfd7c89-xbsm5" event={"ID":"b16d10bd-7c66-48bb-b9a4-011e1f9f9e19","Type":"ContainerStarted","Data":"1c1449a6d6419ae886f435cea24b5308b9e2c668f897efe5eeceeaa91def294e"}
Sep 14 05:01:03.974245 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:03.974196    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78cfd7c89-xbsm5" event={"ID":"b16d10bd-7c66-48bb-b9a4-011e1f9f9e19","Type":"ContainerStarted","Data":"74e087b5636543354967d3e3e1e06b46e917ce0264eb9329b717090b0a5c1266"}
Sep 14 05:01:03.995124 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:03.995042    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-78cfd7c89-xbsm5" podStartSLOduration=2.091131602 podStartE2EDuration="2.995016058s" totalImagesPullingTime="903.884456ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:01:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:01:03.991378173 +0000 UTC m=+862.167149733" watchObservedRunningTime="2026-09-14 05:01:03.995016058 +0000 UTC m=+862.170787688"
Sep 14 05:01:04.021568 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:04.021531    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7db5c68f5b-vpzpz"]
Sep 14 05:01:04.021738 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:04.021716    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7db5c68f5b-vpzpz" podUID="77a36ad6-cbb3-4fdf-9793-8494102b2f0c" containerName="authorino" containerID="cri-o://ca4c30304ba970da0eef965b8c2de97957fedc6b9800152286172af1d29d1e86" gracePeriod=30
Sep 14 05:01:04.267981 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:04.267952    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7db5c68f5b-vpzpz"
Sep 14 05:01:04.375333 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:04.375295    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpvhg\" (UniqueName: \"kubernetes.io/projected/77a36ad6-cbb3-4fdf-9793-8494102b2f0c-kube-api-access-jpvhg\") pod \"77a36ad6-cbb3-4fdf-9793-8494102b2f0c\" (UID: \"77a36ad6-cbb3-4fdf-9793-8494102b2f0c\") "
Sep 14 05:01:04.375505 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:04.375357    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/77a36ad6-cbb3-4fdf-9793-8494102b2f0c-tls-cert\") pod \"77a36ad6-cbb3-4fdf-9793-8494102b2f0c\" (UID: \"77a36ad6-cbb3-4fdf-9793-8494102b2f0c\") "
Sep 14 05:01:04.377551 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:04.377515    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77a36ad6-cbb3-4fdf-9793-8494102b2f0c-kube-api-access-jpvhg" (OuterVolumeSpecName: "kube-api-access-jpvhg") pod "77a36ad6-cbb3-4fdf-9793-8494102b2f0c" (UID: "77a36ad6-cbb3-4fdf-9793-8494102b2f0c"). InnerVolumeSpecName "kube-api-access-jpvhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:01:04.385255 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:04.385226    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77a36ad6-cbb3-4fdf-9793-8494102b2f0c-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "77a36ad6-cbb3-4fdf-9793-8494102b2f0c" (UID: "77a36ad6-cbb3-4fdf-9793-8494102b2f0c"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 05:01:04.476579 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:04.476542    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpvhg\" (UniqueName: \"kubernetes.io/projected/77a36ad6-cbb3-4fdf-9793-8494102b2f0c-kube-api-access-jpvhg\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:01:04.476579 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:04.476573    2565 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/77a36ad6-cbb3-4fdf-9793-8494102b2f0c-tls-cert\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:01:04.979929 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:04.979886    2565 generic.go:356] "Generic (PLEG): container finished" podID="77a36ad6-cbb3-4fdf-9793-8494102b2f0c" containerID="ca4c30304ba970da0eef965b8c2de97957fedc6b9800152286172af1d29d1e86" exitCode=0
Sep 14 05:01:04.979929 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:04.979930    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7db5c68f5b-vpzpz"
Sep 14 05:01:04.980427 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:04.979929    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7db5c68f5b-vpzpz" event={"ID":"77a36ad6-cbb3-4fdf-9793-8494102b2f0c","Type":"ContainerDied","Data":"ca4c30304ba970da0eef965b8c2de97957fedc6b9800152286172af1d29d1e86"}
Sep 14 05:01:04.980427 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:04.979983    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7db5c68f5b-vpzpz" event={"ID":"77a36ad6-cbb3-4fdf-9793-8494102b2f0c","Type":"ContainerDied","Data":"fdf1452ed91d5ad453a89415247761a5c91d80cb656dac379becf8299337b273"}
Sep 14 05:01:04.980427 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:04.980006    2565 scope.go:117] "RemoveContainer" containerID="ca4c30304ba970da0eef965b8c2de97957fedc6b9800152286172af1d29d1e86"
Sep 14 05:01:04.988905 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:04.988886    2565 scope.go:117] "RemoveContainer" containerID="ca4c30304ba970da0eef965b8c2de97957fedc6b9800152286172af1d29d1e86"
Sep 14 05:01:04.989217 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:01:04.989189    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca4c30304ba970da0eef965b8c2de97957fedc6b9800152286172af1d29d1e86\": container with ID starting with ca4c30304ba970da0eef965b8c2de97957fedc6b9800152286172af1d29d1e86 not found: ID does not exist" containerID="ca4c30304ba970da0eef965b8c2de97957fedc6b9800152286172af1d29d1e86"
Sep 14 05:01:04.989305 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:04.989227    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca4c30304ba970da0eef965b8c2de97957fedc6b9800152286172af1d29d1e86"} err="failed to get container status \"ca4c30304ba970da0eef965b8c2de97957fedc6b9800152286172af1d29d1e86\": rpc error: code = NotFound desc = could not find container \"ca4c30304ba970da0eef965b8c2de97957fedc6b9800152286172af1d29d1e86\": container with ID starting with ca4c30304ba970da0eef965b8c2de97957fedc6b9800152286172af1d29d1e86 not found: ID does not exist"
Sep 14 05:01:05.004944 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:05.004917    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7db5c68f5b-vpzpz"]
Sep 14 05:01:05.007221 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:05.007198    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7db5c68f5b-vpzpz"]
Sep 14 05:01:06.334088 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:06.334054    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77a36ad6-cbb3-4fdf-9793-8494102b2f0c" path="/var/lib/kubelet/pods/77a36ad6-cbb3-4fdf-9793-8494102b2f0c/volumes"
Sep 14 05:01:30.289961 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:30.289920    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7545-a8d0bb"]
Sep 14 05:01:30.290355 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:30.290205    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77a36ad6-cbb3-4fdf-9793-8494102b2f0c" containerName="authorino"
Sep 14 05:01:30.290355 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:30.290216    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="77a36ad6-cbb3-4fdf-9793-8494102b2f0c" containerName="authorino"
Sep 14 05:01:30.290355 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:30.290287    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="77a36ad6-cbb3-4fdf-9793-8494102b2f0c" containerName="authorino"
Sep 14 05:01:30.293351 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:30.293328    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7545-a8d0bb"
Sep 14 05:01:30.297186 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:30.296316    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dcz7p\""
Sep 14 05:01:30.298662 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:30.298638    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7545-a8d0bb"]
Sep 14 05:01:30.398280 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:30.398244    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmm7j\" (UniqueName: \"kubernetes.io/projected/ad86ca06-7981-4a47-9a2a-90717d8c565b-kube-api-access-kmm7j\") pod \"test-curl-7545-a8d0bb\" (UID: \"ad86ca06-7981-4a47-9a2a-90717d8c565b\") " pod="opendatahub/test-curl-7545-a8d0bb"
Sep 14 05:01:30.499399 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:30.499366    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmm7j\" (UniqueName: \"kubernetes.io/projected/ad86ca06-7981-4a47-9a2a-90717d8c565b-kube-api-access-kmm7j\") pod \"test-curl-7545-a8d0bb\" (UID: \"ad86ca06-7981-4a47-9a2a-90717d8c565b\") " pod="opendatahub/test-curl-7545-a8d0bb"
Sep 14 05:01:30.507378 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:30.507351    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmm7j\" (UniqueName: \"kubernetes.io/projected/ad86ca06-7981-4a47-9a2a-90717d8c565b-kube-api-access-kmm7j\") pod \"test-curl-7545-a8d0bb\" (UID: \"ad86ca06-7981-4a47-9a2a-90717d8c565b\") " pod="opendatahub/test-curl-7545-a8d0bb"
Sep 14 05:01:30.622151 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:30.622126    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7545-a8d0bb"
Sep 14 05:01:30.892614 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:30.892587    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7545-a8d0bb"]
Sep 14 05:01:30.894499 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:01:30.894467    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad86ca06_7981_4a47_9a2a_90717d8c565b.slice/crio-e0c4b8552bfb0c40fea70306cef47d0bfddc37950aa9c68c8c7b1d002bb59721 WatchSource:0}: Error finding container e0c4b8552bfb0c40fea70306cef47d0bfddc37950aa9c68c8c7b1d002bb59721: Status 404 returned error can't find the container with id e0c4b8552bfb0c40fea70306cef47d0bfddc37950aa9c68c8c7b1d002bb59721
Sep 14 05:01:31.104750 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:31.104715    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7545-a8d0bb" event={"ID":"ad86ca06-7981-4a47-9a2a-90717d8c565b","Type":"ContainerStarted","Data":"e0c4b8552bfb0c40fea70306cef47d0bfddc37950aa9c68c8c7b1d002bb59721"}
Sep 14 05:01:32.705069 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:32.705029    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:01:33.741510 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:33.741476    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-667664-5bd57bd994-52n8z"]
Sep 14 05:01:33.746282 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:33.746265    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-667664-5bd57bd994-52n8z"
Sep 14 05:01:33.748734 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:33.748711    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-667664-dockercfg-mvs58\""
Sep 14 05:01:33.748868 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:33.748741    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-667664\""
Sep 14 05:01:33.754885 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:33.754862    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-667664-5bd57bd994-52n8z"]
Sep 14 05:01:33.835234 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:33.835202    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eec5515b-3b3f-4fdd-9cf0-a8f2cf77a226-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-667664-5bd57bd994-52n8z\" (UID: \"eec5515b-3b3f-4fdd-9cf0-a8f2cf77a226\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-667664-5bd57bd994-52n8z"
Sep 14 05:01:33.835408 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:33.835270    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jphlh\" (UniqueName: \"kubernetes.io/projected/eec5515b-3b3f-4fdd-9cf0-a8f2cf77a226-kube-api-access-jphlh\") pod \"payload-pre-processing-e2e-inf-a-667664-5bd57bd994-52n8z\" (UID: \"eec5515b-3b3f-4fdd-9cf0-a8f2cf77a226\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-667664-5bd57bd994-52n8z"
Sep 14 05:01:33.936589 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:33.936550    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jphlh\" (UniqueName: \"kubernetes.io/projected/eec5515b-3b3f-4fdd-9cf0-a8f2cf77a226-kube-api-access-jphlh\") pod \"payload-pre-processing-e2e-inf-a-667664-5bd57bd994-52n8z\" (UID: \"eec5515b-3b3f-4fdd-9cf0-a8f2cf77a226\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-667664-5bd57bd994-52n8z"
Sep 14 05:01:33.936805 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:33.936618    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eec5515b-3b3f-4fdd-9cf0-a8f2cf77a226-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-667664-5bd57bd994-52n8z\" (UID: \"eec5515b-3b3f-4fdd-9cf0-a8f2cf77a226\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-667664-5bd57bd994-52n8z"
Sep 14 05:01:33.937363 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:33.937342    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eec5515b-3b3f-4fdd-9cf0-a8f2cf77a226-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-667664-5bd57bd994-52n8z\" (UID: \"eec5515b-3b3f-4fdd-9cf0-a8f2cf77a226\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-667664-5bd57bd994-52n8z"
Sep 14 05:01:33.944644 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:33.944617    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jphlh\" (UniqueName: \"kubernetes.io/projected/eec5515b-3b3f-4fdd-9cf0-a8f2cf77a226-kube-api-access-jphlh\") pod \"payload-pre-processing-e2e-inf-a-667664-5bd57bd994-52n8z\" (UID: \"eec5515b-3b3f-4fdd-9cf0-a8f2cf77a226\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-667664-5bd57bd994-52n8z"
Sep 14 05:01:34.058072 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:34.057979    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-667664-5bd57bd994-52n8z"
Sep 14 05:01:34.345320 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:34.345296    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-667664-5bd57bd994-52n8z"]
Sep 14 05:01:34.347650 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:01:34.347624    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeec5515b_3b3f_4fdd_9cf0_a8f2cf77a226.slice/crio-6d95c4a1106b32bcf469a2e5accdd7462e7516d2f95d64256e1d8ef5c1389fab WatchSource:0}: Error finding container 6d95c4a1106b32bcf469a2e5accdd7462e7516d2f95d64256e1d8ef5c1389fab: Status 404 returned error can't find the container with id 6d95c4a1106b32bcf469a2e5accdd7462e7516d2f95d64256e1d8ef5c1389fab
Sep 14 05:01:35.128584 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:35.128544    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-667664-5bd57bd994-52n8z" event={"ID":"eec5515b-3b3f-4fdd-9cf0-a8f2cf77a226","Type":"ContainerStarted","Data":"6d95c4a1106b32bcf469a2e5accdd7462e7516d2f95d64256e1d8ef5c1389fab"}
Sep 14 05:01:36.135086 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:36.135049    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-667664-5bd57bd994-52n8z" event={"ID":"eec5515b-3b3f-4fdd-9cf0-a8f2cf77a226","Type":"ContainerStarted","Data":"bd753d2797174af7a089b061f9a5221f9b5601272b4d5ce0e92bd39708639016"}
Sep 14 05:01:36.135479 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:36.135154    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-667664-5bd57bd994-52n8z"
Sep 14 05:01:36.152224 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:36.152178    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-667664-5bd57bd994-52n8z" podStartSLOduration=1.896009955 podStartE2EDuration="3.152156228s" totalImagesPullingTime="1.256146273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:01:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:01:36.151039291 +0000 UTC m=+894.326810841" watchObservedRunningTime="2026-09-14 05:01:36.152156228 +0000 UTC m=+894.327927786"
Sep 14 05:01:36.987888 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:36.987853    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c366d3-668d8688df-dtphc"]
Sep 14 05:01:36.991283 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:36.991259    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c366d3-668d8688df-dtphc"
Sep 14 05:01:36.994914 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:36.994885    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-c366d3-dockercfg-vv4f4\""
Sep 14 05:01:36.995000 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:36.994923    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-c366d3\""
Sep 14 05:01:37.002654 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:37.002630    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c366d3-668d8688df-dtphc"]
Sep 14 05:01:37.167284 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:37.167255    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57w26\" (UniqueName: \"kubernetes.io/projected/5feb03be-ccbd-4401-8d8e-95b8942e9270-kube-api-access-57w26\") pod \"payload-processing-e2e-worker-w0-c366d3-668d8688df-dtphc\" (UID: \"5feb03be-ccbd-4401-8d8e-95b8942e9270\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c366d3-668d8688df-dtphc"
Sep 14 05:01:37.167664 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:37.167321    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5feb03be-ccbd-4401-8d8e-95b8942e9270-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-c366d3-668d8688df-dtphc\" (UID: \"5feb03be-ccbd-4401-8d8e-95b8942e9270\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c366d3-668d8688df-dtphc"
Sep 14 05:01:37.268191 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:37.268096    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5feb03be-ccbd-4401-8d8e-95b8942e9270-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-c366d3-668d8688df-dtphc\" (UID: \"5feb03be-ccbd-4401-8d8e-95b8942e9270\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c366d3-668d8688df-dtphc"
Sep 14 05:01:37.268350 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:37.268221    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57w26\" (UniqueName: \"kubernetes.io/projected/5feb03be-ccbd-4401-8d8e-95b8942e9270-kube-api-access-57w26\") pod \"payload-processing-e2e-worker-w0-c366d3-668d8688df-dtphc\" (UID: \"5feb03be-ccbd-4401-8d8e-95b8942e9270\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c366d3-668d8688df-dtphc"
Sep 14 05:01:37.269113 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:37.269089    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5feb03be-ccbd-4401-8d8e-95b8942e9270-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-c366d3-668d8688df-dtphc\" (UID: \"5feb03be-ccbd-4401-8d8e-95b8942e9270\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c366d3-668d8688df-dtphc"
Sep 14 05:01:37.277547 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:37.277525    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57w26\" (UniqueName: \"kubernetes.io/projected/5feb03be-ccbd-4401-8d8e-95b8942e9270-kube-api-access-57w26\") pod \"payload-processing-e2e-worker-w0-c366d3-668d8688df-dtphc\" (UID: \"5feb03be-ccbd-4401-8d8e-95b8942e9270\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c366d3-668d8688df-dtphc"
Sep 14 05:01:37.301031 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:37.301010    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c366d3-668d8688df-dtphc"
Sep 14 05:01:37.575963 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:37.575937    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c366d3-668d8688df-dtphc"]
Sep 14 05:01:37.577644 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:01:37.577617    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5feb03be_ccbd_4401_8d8e_95b8942e9270.slice/crio-4e0779c11500013ca09552f8b5aa6d27a4f906e562a4010e3759363002e735fb WatchSource:0}: Error finding container 4e0779c11500013ca09552f8b5aa6d27a4f906e562a4010e3759363002e735fb: Status 404 returned error can't find the container with id 4e0779c11500013ca09552f8b5aa6d27a4f906e562a4010e3759363002e735fb
Sep 14 05:01:38.147000 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:38.146966    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c366d3-668d8688df-dtphc" event={"ID":"5feb03be-ccbd-4401-8d8e-95b8942e9270","Type":"ContainerStarted","Data":"fe3673f24c33cf6a2e98521d830455b9b56b6ee4c3388db97f9cd5d8e5aa9219"}
Sep 14 05:01:38.147193 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:38.147023    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c366d3-668d8688df-dtphc" event={"ID":"5feb03be-ccbd-4401-8d8e-95b8942e9270","Type":"ContainerStarted","Data":"4e0779c11500013ca09552f8b5aa6d27a4f906e562a4010e3759363002e735fb"}
Sep 14 05:01:38.147193 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:38.147056    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-c366d3-668d8688df-dtphc"
Sep 14 05:01:38.164821 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:38.164749    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-c366d3-668d8688df-dtphc" podStartSLOduration=2.1647336 podStartE2EDuration="2.1647336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:01:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:01:38.163506904 +0000 UTC m=+896.339278463" watchObservedRunningTime="2026-09-14 05:01:38.1647336 +0000 UTC m=+896.340505156"
Sep 14 05:01:42.319688 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:42.319658    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kndxq_e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae/ovn-acl-logging/0.log"
Sep 14 05:01:42.320180 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:42.319717    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kndxq_e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae/ovn-acl-logging/0.log"
Sep 14 05:01:45.002126 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:45.002092    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-654e8bde-5c9d48dbf4-mqjqq"]
Sep 14 05:01:45.006843 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:45.006821    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-654e8bde-5c9d48dbf4-mqjqq"
Sep 14 05:01:45.009489 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:45.009471    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-654e8bde-dockercfg-zlzbb\""
Sep 14 05:01:45.009697 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:45.009684    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-654e8bde\""
Sep 14 05:01:45.015358 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:45.015336    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-654e8bde-5c9d48dbf4-mqjqq"]
Sep 14 05:01:45.028501 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:45.028462    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-654e8bde-f5c84745-5s7k2"]
Sep 14 05:01:45.033225 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:45.033200    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-654e8bde-f5c84745-5s7k2"
Sep 14 05:01:45.040715 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:45.040691    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-654e8bde-f5c84745-5s7k2"]
Sep 14 05:01:45.132027 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:45.131993    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc2nv\" (UniqueName: \"kubernetes.io/projected/9d2bd35b-18c9-4944-b616-65a8ef81f4d8-kube-api-access-jc2nv\") pod \"payload-processing-e2e-ait-654e8bde-f5c84745-5s7k2\" (UID: \"9d2bd35b-18c9-4944-b616-65a8ef81f4d8\") " pod="openshift-ingress/payload-processing-e2e-ait-654e8bde-f5c84745-5s7k2"
Sep 14 05:01:45.132183 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:45.132034    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ftdn\" (UniqueName: \"kubernetes.io/projected/e78c59a2-18da-4216-8d59-f267b844cf72-kube-api-access-8ftdn\") pod \"payload-pre-processing-e2e-ait-654e8bde-5c9d48dbf4-mqjqq\" (UID: \"e78c59a2-18da-4216-8d59-f267b844cf72\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-654e8bde-5c9d48dbf4-mqjqq"
Sep 14 05:01:45.132183 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:45.132053    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d2bd35b-18c9-4944-b616-65a8ef81f4d8-plugins-config-volume\") pod \"payload-processing-e2e-ait-654e8bde-f5c84745-5s7k2\" (UID: \"9d2bd35b-18c9-4944-b616-65a8ef81f4d8\") " pod="openshift-ingress/payload-processing-e2e-ait-654e8bde-f5c84745-5s7k2"
Sep 14 05:01:45.132183 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:45.132106    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e78c59a2-18da-4216-8d59-f267b844cf72-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-654e8bde-5c9d48dbf4-mqjqq\" (UID: \"e78c59a2-18da-4216-8d59-f267b844cf72\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-654e8bde-5c9d48dbf4-mqjqq"
Sep 14 05:01:45.233092 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:45.233056    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ftdn\" (UniqueName: \"kubernetes.io/projected/e78c59a2-18da-4216-8d59-f267b844cf72-kube-api-access-8ftdn\") pod \"payload-pre-processing-e2e-ait-654e8bde-5c9d48dbf4-mqjqq\" (UID: \"e78c59a2-18da-4216-8d59-f267b844cf72\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-654e8bde-5c9d48dbf4-mqjqq"
Sep 14 05:01:45.233092 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:45.233088    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d2bd35b-18c9-4944-b616-65a8ef81f4d8-plugins-config-volume\") pod \"payload-processing-e2e-ait-654e8bde-f5c84745-5s7k2\" (UID: \"9d2bd35b-18c9-4944-b616-65a8ef81f4d8\") " pod="openshift-ingress/payload-processing-e2e-ait-654e8bde-f5c84745-5s7k2"
Sep 14 05:01:45.233324 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:45.233110    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e78c59a2-18da-4216-8d59-f267b844cf72-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-654e8bde-5c9d48dbf4-mqjqq\" (UID: \"e78c59a2-18da-4216-8d59-f267b844cf72\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-654e8bde-5c9d48dbf4-mqjqq"
Sep 14 05:01:45.233324 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:45.233172    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc2nv\" (UniqueName: \"kubernetes.io/projected/9d2bd35b-18c9-4944-b616-65a8ef81f4d8-kube-api-access-jc2nv\") pod \"payload-processing-e2e-ait-654e8bde-f5c84745-5s7k2\" (UID: \"9d2bd35b-18c9-4944-b616-65a8ef81f4d8\") " pod="openshift-ingress/payload-processing-e2e-ait-654e8bde-f5c84745-5s7k2"
Sep 14 05:01:45.233859 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:45.233838    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d2bd35b-18c9-4944-b616-65a8ef81f4d8-plugins-config-volume\") pod \"payload-processing-e2e-ait-654e8bde-f5c84745-5s7k2\" (UID: \"9d2bd35b-18c9-4944-b616-65a8ef81f4d8\") " pod="openshift-ingress/payload-processing-e2e-ait-654e8bde-f5c84745-5s7k2"
Sep 14 05:01:45.233930 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:45.233876    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e78c59a2-18da-4216-8d59-f267b844cf72-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-654e8bde-5c9d48dbf4-mqjqq\" (UID: \"e78c59a2-18da-4216-8d59-f267b844cf72\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-654e8bde-5c9d48dbf4-mqjqq"
Sep 14 05:01:45.242591 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:45.242559    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc2nv\" (UniqueName: \"kubernetes.io/projected/9d2bd35b-18c9-4944-b616-65a8ef81f4d8-kube-api-access-jc2nv\") pod \"payload-processing-e2e-ait-654e8bde-f5c84745-5s7k2\" (UID: \"9d2bd35b-18c9-4944-b616-65a8ef81f4d8\") " pod="openshift-ingress/payload-processing-e2e-ait-654e8bde-f5c84745-5s7k2"
Sep 14 05:01:45.242689 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:45.242557    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ftdn\" (UniqueName: \"kubernetes.io/projected/e78c59a2-18da-4216-8d59-f267b844cf72-kube-api-access-8ftdn\") pod \"payload-pre-processing-e2e-ait-654e8bde-5c9d48dbf4-mqjqq\" (UID: \"e78c59a2-18da-4216-8d59-f267b844cf72\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-654e8bde-5c9d48dbf4-mqjqq"
Sep 14 05:01:45.318484 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:45.318413    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-654e8bde-5c9d48dbf4-mqjqq"
Sep 14 05:01:45.344532 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:45.344496    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-654e8bde-f5c84745-5s7k2"
Sep 14 05:01:45.589683 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:45.589634    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-654e8bde-5c9d48dbf4-mqjqq"]
Sep 14 05:01:45.590994 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:01:45.590967    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode78c59a2_18da_4216_8d59_f267b844cf72.slice/crio-5399dd69db3afcc5cb897fefcdd6516becfd064ac8cb6cbf08918b51f4a1942a WatchSource:0}: Error finding container 5399dd69db3afcc5cb897fefcdd6516becfd064ac8cb6cbf08918b51f4a1942a: Status 404 returned error can't find the container with id 5399dd69db3afcc5cb897fefcdd6516becfd064ac8cb6cbf08918b51f4a1942a
Sep 14 05:01:45.622835 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:45.622774    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-654e8bde-f5c84745-5s7k2"]
Sep 14 05:01:45.624904 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:01:45.624873    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d2bd35b_18c9_4944_b616_65a8ef81f4d8.slice/crio-6ec45984430ad0e2a06df2f96749735023f5389fa3059d7261f0c83921f7062a WatchSource:0}: Error finding container 6ec45984430ad0e2a06df2f96749735023f5389fa3059d7261f0c83921f7062a: Status 404 returned error can't find the container with id 6ec45984430ad0e2a06df2f96749735023f5389fa3059d7261f0c83921f7062a
Sep 14 05:01:46.187275 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:46.187235    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-654e8bde-f5c84745-5s7k2" event={"ID":"9d2bd35b-18c9-4944-b616-65a8ef81f4d8","Type":"ContainerStarted","Data":"e39812e2ad01daf8ccb944b2a011279e37220851ce2b7141904c2ffb54b8fb8f"}
Sep 14 05:01:46.187739 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:46.187293    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-654e8bde-f5c84745-5s7k2" event={"ID":"9d2bd35b-18c9-4944-b616-65a8ef81f4d8","Type":"ContainerStarted","Data":"6ec45984430ad0e2a06df2f96749735023f5389fa3059d7261f0c83921f7062a"}
Sep 14 05:01:46.187739 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:46.187335    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-654e8bde-f5c84745-5s7k2"
Sep 14 05:01:46.188809 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:46.188784    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-654e8bde-5c9d48dbf4-mqjqq" event={"ID":"e78c59a2-18da-4216-8d59-f267b844cf72","Type":"ContainerStarted","Data":"1d32565df3964e832a89cc4d3ad17e89097db63bc4843ccdae1f67a9775e69c2"}
Sep 14 05:01:46.188914 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:46.188826    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-654e8bde-5c9d48dbf4-mqjqq" event={"ID":"e78c59a2-18da-4216-8d59-f267b844cf72","Type":"ContainerStarted","Data":"5399dd69db3afcc5cb897fefcdd6516becfd064ac8cb6cbf08918b51f4a1942a"}
Sep 14 05:01:46.188914 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:46.188882    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-654e8bde-5c9d48dbf4-mqjqq"
Sep 14 05:01:46.205766 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:46.205708    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-654e8bde-f5c84745-5s7k2" podStartSLOduration=1.20569151 podStartE2EDuration="1.20569151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:01:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:01:46.202522268 +0000 UTC m=+904.378293826" watchObservedRunningTime="2026-09-14 05:01:46.20569151 +0000 UTC m=+904.381463074"
Sep 14 05:01:46.219084 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:46.219039    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-654e8bde-5c9d48dbf4-mqjqq" podStartSLOduration=2.219023613 podStartE2EDuration="2.219023613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:01:46.216543106 +0000 UTC m=+904.392314667" watchObservedRunningTime="2026-09-14 05:01:46.219023613 +0000 UTC m=+904.394795176"
Sep 14 05:01:47.142106 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:47.142071    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-667664-5bd57bd994-52n8z"
Sep 14 05:01:49.153679 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:49.153649    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-c366d3-668d8688df-dtphc"
Sep 14 05:01:50.236586 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:50.236550    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:01:54.500826 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:54.500789    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"]
Sep 14 05:01:54.505342 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:54.505319    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"
Sep 14 05:01:54.507932 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:54.507910    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 14 05:01:54.508220 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:54.508203    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-f14475-openshift-default-dockercfg-28xc2\""
Sep 14 05:01:54.513171 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:54.513143    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38ea6dfe-3dce-404e-988c-8b3d8f912061-istio-data\") pod \"e2e-autores-f14475-openshift-default-96796fcc6-bfzx9\" (UID: \"38ea6dfe-3dce-404e-988c-8b3d8f912061\") " pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"
Sep 14 05:01:54.513362 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:54.513334    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ea6dfe-3dce-404e-988c-8b3d8f912061-workload-socket\") pod \"e2e-autores-f14475-openshift-default-96796fcc6-bfzx9\" (UID: \"38ea6dfe-3dce-404e-988c-8b3d8f912061\") " pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"
Sep 14 05:01:54.513474 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:54.513396    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38ea6dfe-3dce-404e-988c-8b3d8f912061-istio-podinfo\") pod \"e2e-autores-f14475-openshift-default-96796fcc6-bfzx9\" (UID: \"38ea6dfe-3dce-404e-988c-8b3d8f912061\") " pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"
Sep 14 05:01:54.513474 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:54.513426    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38ea6dfe-3dce-404e-988c-8b3d8f912061-istiod-ca-cert\") pod \"e2e-autores-f14475-openshift-default-96796fcc6-bfzx9\" (UID: \"38ea6dfe-3dce-404e-988c-8b3d8f912061\") " pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"
Sep 14 05:01:54.513474 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:54.513458    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38ea6dfe-3dce-404e-988c-8b3d8f912061-workload-certs\") pod \"e2e-autores-f14475-openshift-default-96796fcc6-bfzx9\" (UID: \"38ea6dfe-3dce-404e-988c-8b3d8f912061\") " pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"
Sep 14 05:01:54.513645 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:54.513486    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlbwk\" (UniqueName: \"kubernetes.io/projected/38ea6dfe-3dce-404e-988c-8b3d8f912061-kube-api-access-xlbwk\") pod \"e2e-autores-f14475-openshift-default-96796fcc6-bfzx9\" (UID: \"38ea6dfe-3dce-404e-988c-8b3d8f912061\") " pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"
Sep 14 05:01:54.513645 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:54.513542    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ea6dfe-3dce-404e-988c-8b3d8f912061-credential-socket\") pod \"e2e-autores-f14475-openshift-default-96796fcc6-bfzx9\" (UID: \"38ea6dfe-3dce-404e-988c-8b3d8f912061\") " pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"
Sep 14 05:01:54.513825 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:54.513628    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38ea6dfe-3dce-404e-988c-8b3d8f912061-istio-token\") pod \"e2e-autores-f14475-openshift-default-96796fcc6-bfzx9\" (UID: \"38ea6dfe-3dce-404e-988c-8b3d8f912061\") " pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"
Sep 14 05:01:54.513825 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:54.513705    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38ea6dfe-3dce-404e-988c-8b3d8f912061-istio-envoy\") pod \"e2e-autores-f14475-openshift-default-96796fcc6-bfzx9\" (UID: \"38ea6dfe-3dce-404e-988c-8b3d8f912061\") " pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"
Sep 14 05:01:54.517279 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:54.517258    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"]
Sep 14 05:01:54.614302 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:54.614268    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ea6dfe-3dce-404e-988c-8b3d8f912061-workload-socket\") pod \"e2e-autores-f14475-openshift-default-96796fcc6-bfzx9\" (UID: \"38ea6dfe-3dce-404e-988c-8b3d8f912061\") " pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"
Sep 14 05:01:54.614488 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:54.614317    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38ea6dfe-3dce-404e-988c-8b3d8f912061-istio-podinfo\") pod \"e2e-autores-f14475-openshift-default-96796fcc6-bfzx9\" (UID: \"38ea6dfe-3dce-404e-988c-8b3d8f912061\") " pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"
Sep 14 05:01:54.614570 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:54.614539    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38ea6dfe-3dce-404e-988c-8b3d8f912061-istiod-ca-cert\") pod \"e2e-autores-f14475-openshift-default-96796fcc6-bfzx9\" (UID: \"38ea6dfe-3dce-404e-988c-8b3d8f912061\") " pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"
Sep 14 05:01:54.614632 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:54.614591    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38ea6dfe-3dce-404e-988c-8b3d8f912061-workload-certs\") pod \"e2e-autores-f14475-openshift-default-96796fcc6-bfzx9\" (UID: \"38ea6dfe-3dce-404e-988c-8b3d8f912061\") " pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"
Sep 14 05:01:54.614632 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:54.614623    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlbwk\" (UniqueName: \"kubernetes.io/projected/38ea6dfe-3dce-404e-988c-8b3d8f912061-kube-api-access-xlbwk\") pod \"e2e-autores-f14475-openshift-default-96796fcc6-bfzx9\" (UID: \"38ea6dfe-3dce-404e-988c-8b3d8f912061\") " pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"
Sep 14 05:01:54.614743 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:54.614651    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ea6dfe-3dce-404e-988c-8b3d8f912061-credential-socket\") pod \"e2e-autores-f14475-openshift-default-96796fcc6-bfzx9\" (UID: \"38ea6dfe-3dce-404e-988c-8b3d8f912061\") " pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"
Sep 14 05:01:54.614743 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:54.614693    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38ea6dfe-3dce-404e-988c-8b3d8f912061-istio-token\") pod \"e2e-autores-f14475-openshift-default-96796fcc6-bfzx9\" (UID: \"38ea6dfe-3dce-404e-988c-8b3d8f912061\") " pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"
Sep 14 05:01:54.614876 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:54.614740    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38ea6dfe-3dce-404e-988c-8b3d8f912061-istio-envoy\") pod \"e2e-autores-f14475-openshift-default-96796fcc6-bfzx9\" (UID: \"38ea6dfe-3dce-404e-988c-8b3d8f912061\") " pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"
Sep 14 05:01:54.614876 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:54.614783    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38ea6dfe-3dce-404e-988c-8b3d8f912061-istio-data\") pod \"e2e-autores-f14475-openshift-default-96796fcc6-bfzx9\" (UID: \"38ea6dfe-3dce-404e-988c-8b3d8f912061\") " pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"
Sep 14 05:01:54.614876 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:54.614867    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ea6dfe-3dce-404e-988c-8b3d8f912061-workload-socket\") pod \"e2e-autores-f14475-openshift-default-96796fcc6-bfzx9\" (UID: \"38ea6dfe-3dce-404e-988c-8b3d8f912061\") " pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"
Sep 14 05:01:54.615037 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:54.614962    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38ea6dfe-3dce-404e-988c-8b3d8f912061-workload-certs\") pod \"e2e-autores-f14475-openshift-default-96796fcc6-bfzx9\" (UID: \"38ea6dfe-3dce-404e-988c-8b3d8f912061\") " pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"
Sep 14 05:01:54.615379 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:54.615348    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ea6dfe-3dce-404e-988c-8b3d8f912061-credential-socket\") pod \"e2e-autores-f14475-openshift-default-96796fcc6-bfzx9\" (UID: \"38ea6dfe-3dce-404e-988c-8b3d8f912061\") " pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"
Sep 14 05:01:54.615673 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:54.615648    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38ea6dfe-3dce-404e-988c-8b3d8f912061-istio-data\") pod \"e2e-autores-f14475-openshift-default-96796fcc6-bfzx9\" (UID: \"38ea6dfe-3dce-404e-988c-8b3d8f912061\") " pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"
Sep 14 05:01:54.615965 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:54.615944    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38ea6dfe-3dce-404e-988c-8b3d8f912061-istiod-ca-cert\") pod \"e2e-autores-f14475-openshift-default-96796fcc6-bfzx9\" (UID: \"38ea6dfe-3dce-404e-988c-8b3d8f912061\") " pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"
Sep 14 05:01:54.617493 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:54.617468    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38ea6dfe-3dce-404e-988c-8b3d8f912061-istio-podinfo\") pod \"e2e-autores-f14475-openshift-default-96796fcc6-bfzx9\" (UID: \"38ea6dfe-3dce-404e-988c-8b3d8f912061\") " pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"
Sep 14 05:01:54.618066 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:54.618048    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38ea6dfe-3dce-404e-988c-8b3d8f912061-istio-envoy\") pod \"e2e-autores-f14475-openshift-default-96796fcc6-bfzx9\" (UID: \"38ea6dfe-3dce-404e-988c-8b3d8f912061\") " pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"
Sep 14 05:01:54.623439 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:54.623417    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlbwk\" (UniqueName: \"kubernetes.io/projected/38ea6dfe-3dce-404e-988c-8b3d8f912061-kube-api-access-xlbwk\") pod \"e2e-autores-f14475-openshift-default-96796fcc6-bfzx9\" (UID: \"38ea6dfe-3dce-404e-988c-8b3d8f912061\") " pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"
Sep 14 05:01:54.624028 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:54.624004    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38ea6dfe-3dce-404e-988c-8b3d8f912061-istio-token\") pod \"e2e-autores-f14475-openshift-default-96796fcc6-bfzx9\" (UID: \"38ea6dfe-3dce-404e-988c-8b3d8f912061\") " pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"
Sep 14 05:01:54.817919 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:54.817807    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"
Sep 14 05:01:55.111339 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:55.111302    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"]
Sep 14 05:01:55.113563 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:01:55.113537    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38ea6dfe_3dce_404e_988c_8b3d8f912061.slice/crio-e8c3869a8af64b72e37685016c37439bbcc84912339d004fb1aded14a7236cfa WatchSource:0}: Error finding container e8c3869a8af64b72e37685016c37439bbcc84912339d004fb1aded14a7236cfa: Status 404 returned error can't find the container with id e8c3869a8af64b72e37685016c37439bbcc84912339d004fb1aded14a7236cfa
Sep 14 05:01:55.242188 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:55.242143    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9" event={"ID":"38ea6dfe-3dce-404e-988c-8b3d8f912061","Type":"ContainerStarted","Data":"e8c3869a8af64b72e37685016c37439bbcc84912339d004fb1aded14a7236cfa"}
Sep 14 05:01:55.618219 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:55.618187    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e7fdd6-56c7655575-4ltrp"]
Sep 14 05:01:55.627448 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:55.627414    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e7fdd6-56c7655575-4ltrp"
Sep 14 05:01:55.630265 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:55.629957    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-e7fdd6-dockercfg-chprd\""
Sep 14 05:01:55.630265 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:55.630162    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e7fdd6-56c7655575-4ltrp"]
Sep 14 05:01:55.630265 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:55.630200    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-e7fdd6\""
Sep 14 05:01:55.726191 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:55.726156    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/170ea597-3f51-42e8-8590-497cd6f7761b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e7fdd6-56c7655575-4ltrp\" (UID: \"170ea597-3f51-42e8-8590-497cd6f7761b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e7fdd6-56c7655575-4ltrp"
Sep 14 05:01:55.726345 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:55.726253    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nv2t\" (UniqueName: \"kubernetes.io/projected/170ea597-3f51-42e8-8590-497cd6f7761b-kube-api-access-7nv2t\") pod \"payload-processing-e2e-inf-b-e7fdd6-56c7655575-4ltrp\" (UID: \"170ea597-3f51-42e8-8590-497cd6f7761b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e7fdd6-56c7655575-4ltrp"
Sep 14 05:01:55.826891 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:55.826857    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nv2t\" (UniqueName: \"kubernetes.io/projected/170ea597-3f51-42e8-8590-497cd6f7761b-kube-api-access-7nv2t\") pod \"payload-processing-e2e-inf-b-e7fdd6-56c7655575-4ltrp\" (UID: \"170ea597-3f51-42e8-8590-497cd6f7761b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e7fdd6-56c7655575-4ltrp"
Sep 14 05:01:55.827081 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:55.826975    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/170ea597-3f51-42e8-8590-497cd6f7761b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e7fdd6-56c7655575-4ltrp\" (UID: \"170ea597-3f51-42e8-8590-497cd6f7761b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e7fdd6-56c7655575-4ltrp"
Sep 14 05:01:55.827682 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:55.827658    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/170ea597-3f51-42e8-8590-497cd6f7761b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e7fdd6-56c7655575-4ltrp\" (UID: \"170ea597-3f51-42e8-8590-497cd6f7761b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e7fdd6-56c7655575-4ltrp"
Sep 14 05:01:55.837254 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:55.837224    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nv2t\" (UniqueName: \"kubernetes.io/projected/170ea597-3f51-42e8-8590-497cd6f7761b-kube-api-access-7nv2t\") pod \"payload-processing-e2e-inf-b-e7fdd6-56c7655575-4ltrp\" (UID: \"170ea597-3f51-42e8-8590-497cd6f7761b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e7fdd6-56c7655575-4ltrp"
Sep 14 05:01:55.942351 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:55.942284    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e7fdd6-56c7655575-4ltrp"
Sep 14 05:01:56.216236 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:56.216206    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e7fdd6-56c7655575-4ltrp"]
Sep 14 05:01:56.218399 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:01:56.218371    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod170ea597_3f51_42e8_8590_497cd6f7761b.slice/crio-e59a93c4de827d5f7b71134b9720509d6018764542f12925f3284b1257b54e5a WatchSource:0}: Error finding container e59a93c4de827d5f7b71134b9720509d6018764542f12925f3284b1257b54e5a: Status 404 returned error can't find the container with id e59a93c4de827d5f7b71134b9720509d6018764542f12925f3284b1257b54e5a
Sep 14 05:01:56.244538 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:56.244510    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-654e8bde-bvll5"]
Sep 14 05:01:56.249359 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:56.249332    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e7fdd6-56c7655575-4ltrp" event={"ID":"170ea597-3f51-42e8-8590-497cd6f7761b","Type":"ContainerStarted","Data":"e59a93c4de827d5f7b71134b9720509d6018764542f12925f3284b1257b54e5a"}
Sep 14 05:01:56.249474 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:56.249434    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-654e8bde-bvll5"
Sep 14 05:01:56.250366 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:56.250340    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7545-a8d0bb" event={"ID":"ad86ca06-7981-4a47-9a2a-90717d8c565b","Type":"ContainerStarted","Data":"095fe7cd7bcd135d2d69386b4545064b5658c261f653b3264faa27ace4fe0e2f"}
Sep 14 05:01:56.251937 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:56.251912    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k6hpm\""
Sep 14 05:01:56.252149 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:56.251912    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 05:01:56.256250 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:56.256229    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-654e8bde-bvll5"]
Sep 14 05:01:56.276952 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:56.276906    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7545-a8d0bb" podStartSLOduration=1.159126203 podStartE2EDuration="26.2768865s" totalImagesPullingTime="25.117760297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:01:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:01:56.275699853 +0000 UTC m=+914.451471412" watchObservedRunningTime="2026-09-14 05:01:56.2768865 +0000 UTC m=+914.452658060"
Sep 14 05:01:56.331891 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:56.331860    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edd0f06e-d2a0-4435-a79d-f245fc2c0960-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-654e8bde-bvll5\" (UID: \"edd0f06e-d2a0-4435-a79d-f245fc2c0960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-654e8bde-bvll5"
Sep 14 05:01:56.432639 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:56.432606    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edd0f06e-d2a0-4435-a79d-f245fc2c0960-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-654e8bde-bvll5\" (UID: \"edd0f06e-d2a0-4435-a79d-f245fc2c0960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-654e8bde-bvll5"
Sep 14 05:01:56.433294 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:56.433269    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edd0f06e-d2a0-4435-a79d-f245fc2c0960-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-654e8bde-bvll5\" (UID: \"edd0f06e-d2a0-4435-a79d-f245fc2c0960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-654e8bde-bvll5"
Sep 14 05:01:56.581607 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:56.581509    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-654e8bde-bvll5"
Sep 14 05:01:56.863249 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:56.863208    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-654e8bde-bvll5"]
Sep 14 05:01:56.864886 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:01:56.864857    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedd0f06e_d2a0_4435_a79d_f245fc2c0960.slice/crio-b5eb7c24a98871354f9c234ef80de30e1d67e9ebfb4bde46b83d15f1901301d6 WatchSource:0}: Error finding container b5eb7c24a98871354f9c234ef80de30e1d67e9ebfb4bde46b83d15f1901301d6: Status 404 returned error can't find the container with id b5eb7c24a98871354f9c234ef80de30e1d67e9ebfb4bde46b83d15f1901301d6
Sep 14 05:01:56.984984 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:56.984946    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7545-a8d0bb"]
Sep 14 05:01:56.988364 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:56.988299    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7545-a8d0bb"]
Sep 14 05:01:57.110105 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:57.110072    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7545-ba9b29"]
Sep 14 05:01:57.110519 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:57.110497    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad86ca06-7981-4a47-9a2a-90717d8c565b" containerName="test-curl-7545-a8d0bb"
Sep 14 05:01:57.110519 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:57.110521    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad86ca06-7981-4a47-9a2a-90717d8c565b" containerName="test-curl-7545-a8d0bb"
Sep 14 05:01:57.110678 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:57.110630    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad86ca06-7981-4a47-9a2a-90717d8c565b" containerName="test-curl-7545-a8d0bb"
Sep 14 05:01:57.117040 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:57.116982    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7545-ba9b29"
Sep 14 05:01:57.118401 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:57.118378    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7545-ba9b29"]
Sep 14 05:01:57.140867 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:57.140845    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvhtd\" (UniqueName: \"kubernetes.io/projected/30b99dbe-ea2e-475a-9607-7598a8d3ca01-kube-api-access-nvhtd\") pod \"test-curl-7545-ba9b29\" (UID: \"30b99dbe-ea2e-475a-9607-7598a8d3ca01\") " pod="opendatahub/test-curl-7545-ba9b29"
Sep 14 05:01:57.196890 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:57.196861    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-654e8bde-f5c84745-5s7k2"
Sep 14 05:01:57.197027 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:57.196939    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-654e8bde-5c9d48dbf4-mqjqq"
Sep 14 05:01:57.241836 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:57.241734    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvhtd\" (UniqueName: \"kubernetes.io/projected/30b99dbe-ea2e-475a-9607-7598a8d3ca01-kube-api-access-nvhtd\") pod \"test-curl-7545-ba9b29\" (UID: \"30b99dbe-ea2e-475a-9607-7598a8d3ca01\") " pod="opendatahub/test-curl-7545-ba9b29"
Sep 14 05:01:57.253712 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:57.253649    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvhtd\" (UniqueName: \"kubernetes.io/projected/30b99dbe-ea2e-475a-9607-7598a8d3ca01-kube-api-access-nvhtd\") pod \"test-curl-7545-ba9b29\" (UID: \"30b99dbe-ea2e-475a-9607-7598a8d3ca01\") " pod="opendatahub/test-curl-7545-ba9b29"
Sep 14 05:01:57.260881 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:57.260798    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e7fdd6-56c7655575-4ltrp" event={"ID":"170ea597-3f51-42e8-8590-497cd6f7761b","Type":"ContainerStarted","Data":"749d63e628ef63857628fc56081150138140d3f03dca308b3b7b64cf95f72c70"}
Sep 14 05:01:57.261074 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:57.261053    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e7fdd6-56c7655575-4ltrp"
Sep 14 05:01:57.263512 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:57.263487    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-654e8bde-bvll5" event={"ID":"edd0f06e-d2a0-4435-a79d-f245fc2c0960","Type":"ContainerStarted","Data":"888de0ac61ea0e0b08467f3ba2357497e666265d769abed3cd1b53c143a7e0b4"}
Sep 14 05:01:57.263687 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:57.263673    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-654e8bde-bvll5" event={"ID":"edd0f06e-d2a0-4435-a79d-f245fc2c0960","Type":"ContainerStarted","Data":"b5eb7c24a98871354f9c234ef80de30e1d67e9ebfb4bde46b83d15f1901301d6"}
Sep 14 05:01:57.263965 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:01:57.263947    2565 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7545-a8d0bb" container="test-curl-7545-a8d0bb"
Sep 14 05:01:57.278642 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:57.278598    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-e7fdd6-56c7655575-4ltrp" podStartSLOduration=2.2785826719999998 podStartE2EDuration="2.278582672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:01:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:01:57.277692569 +0000 UTC m=+915.453464129" watchObservedRunningTime="2026-09-14 05:01:57.278582672 +0000 UTC m=+915.454354246"
Sep 14 05:01:57.293666 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:57.293608    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-654e8bde-bvll5" podStartSLOduration=1.293586761 podStartE2EDuration="1.293586761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:01:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:01:57.292492536 +0000 UTC m=+915.468264097" watchObservedRunningTime="2026-09-14 05:01:57.293586761 +0000 UTC m=+915.469358320"
Sep 14 05:01:57.429723 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:57.429638    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7545-ba9b29"
Sep 14 05:01:58.115908 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:58.115866    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:01:58.116226 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:58.115960    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:01:58.116226 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:58.115999    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:01:58.271440 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:58.271409    2565 generic.go:356] "Generic (PLEG): container finished" podID="edd0f06e-d2a0-4435-a79d-f245fc2c0960" containerID="888de0ac61ea0e0b08467f3ba2357497e666265d769abed3cd1b53c143a7e0b4" exitCode=0
Sep 14 05:01:58.271626 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:58.271510    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-654e8bde-bvll5" event={"ID":"edd0f06e-d2a0-4435-a79d-f245fc2c0960","Type":"ContainerDied","Data":"888de0ac61ea0e0b08467f3ba2357497e666265d769abed3cd1b53c143a7e0b4"}
Sep 14 05:01:58.273275 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:58.273238    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9" event={"ID":"38ea6dfe-3dce-404e-988c-8b3d8f912061","Type":"ContainerStarted","Data":"ab07c93f774034c727d587b22c01c02ec9cc6d1f897e8c3fbb65960d1fc1fd4f"}
Sep 14 05:01:58.273387 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:58.273290    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7545-a8d0bb" podUID="ad86ca06-7981-4a47-9a2a-90717d8c565b" containerName="test-curl-7545-a8d0bb" containerID="cri-o://095fe7cd7bcd135d2d69386b4545064b5658c261f653b3264faa27ace4fe0e2f" gracePeriod=2
Sep 14 05:01:58.307445 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:58.307360    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9" podStartSLOduration=1.307804124 podStartE2EDuration="4.307345102s" totalImagesPullingTime="2.999540978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:01:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:01:58.306503671 +0000 UTC m=+916.482275232" watchObservedRunningTime="2026-09-14 05:01:58.307345102 +0000 UTC m=+916.483116660"
Sep 14 05:01:58.309054 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:01:58.309021    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7545-a8d0bb\" is forbidden: User \"system:node:ip-10-0-133-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-99.ec2.internal' and this object" podUID="ad86ca06-7981-4a47-9a2a-90717d8c565b" pod="opendatahub/test-curl-7545-a8d0bb"
Sep 14 05:01:58.313348 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:58.313266    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7545-ba9b29"]
Sep 14 05:01:58.314322 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:01:58.314299    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30b99dbe_ea2e_475a_9607_7598a8d3ca01.slice/crio-615373e98a920f8f4d6509502afa71b5684dd8e53d55734e84d19f1433334bd6 WatchSource:0}: Error finding container 615373e98a920f8f4d6509502afa71b5684dd8e53d55734e84d19f1433334bd6: Status 404 returned error can't find the container with id 615373e98a920f8f4d6509502afa71b5684dd8e53d55734e84d19f1433334bd6
Sep 14 05:01:58.817962 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:58.817931    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"
Sep 14 05:01:59.280134 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:59.280092    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7545-ba9b29" event={"ID":"30b99dbe-ea2e-475a-9607-7598a8d3ca01","Type":"ContainerStarted","Data":"f494897c3f84ab3e9197ec2eddcc860973f4630f929109a7fdf70aaaa11bd33f"}
Sep 14 05:01:59.280510 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:59.280146    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7545-ba9b29" event={"ID":"30b99dbe-ea2e-475a-9607-7598a8d3ca01","Type":"ContainerStarted","Data":"615373e98a920f8f4d6509502afa71b5684dd8e53d55734e84d19f1433334bd6"}
Sep 14 05:01:59.293930 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:59.293875    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7545-ba9b29" podStartSLOduration=2.293859366 podStartE2EDuration="2.293859366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:01:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:01:59.293033493 +0000 UTC m=+917.468805053" watchObservedRunningTime="2026-09-14 05:01:59.293859366 +0000 UTC m=+917.469630924"
Sep 14 05:01:59.408322 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:59.408298    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-654e8bde-bvll5"
Sep 14 05:01:59.463466 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:59.463432    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edd0f06e-d2a0-4435-a79d-f245fc2c0960-maas-api-service-ca\") pod \"edd0f06e-d2a0-4435-a79d-f245fc2c0960\" (UID: \"edd0f06e-d2a0-4435-a79d-f245fc2c0960\") "
Sep 14 05:01:59.463824 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:59.463800    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edd0f06e-d2a0-4435-a79d-f245fc2c0960-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "edd0f06e-d2a0-4435-a79d-f245fc2c0960" (UID: "edd0f06e-d2a0-4435-a79d-f245fc2c0960"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 05:01:59.565081 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:59.565007    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edd0f06e-d2a0-4435-a79d-f245fc2c0960-maas-api-service-ca\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:01:59.825415 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:59.825301    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"
Sep 14 05:01:59.865823 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:59.865789    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7545-ba9b29"]
Sep 14 05:01:59.868389 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:01:59.868362    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7545-ba9b29"]
Sep 14 05:02:00.287833 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:00.287774    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-654e8bde-bvll5"
Sep 14 05:02:00.287833 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:00.287800    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-654e8bde-bvll5" event={"ID":"edd0f06e-d2a0-4435-a79d-f245fc2c0960","Type":"ContainerDied","Data":"b5eb7c24a98871354f9c234ef80de30e1d67e9ebfb4bde46b83d15f1901301d6"}
Sep 14 05:02:00.288361 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:00.287869    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5eb7c24a98871354f9c234ef80de30e1d67e9ebfb4bde46b83d15f1901301d6"
Sep 14 05:02:00.288700 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:00.288592    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"
Sep 14 05:02:00.288836 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:02:00.288789    2565 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7545-ba9b29" container="test-curl-7545-ba9b29"
Sep 14 05:02:00.289556 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:00.289536    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"
Sep 14 05:02:00.458290 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:00.458264    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7545-a8d0bb"
Sep 14 05:02:00.573744 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:00.573658    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmm7j\" (UniqueName: \"kubernetes.io/projected/ad86ca06-7981-4a47-9a2a-90717d8c565b-kube-api-access-kmm7j\") pod \"ad86ca06-7981-4a47-9a2a-90717d8c565b\" (UID: \"ad86ca06-7981-4a47-9a2a-90717d8c565b\") "
Sep 14 05:02:00.575888 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:00.575865    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad86ca06-7981-4a47-9a2a-90717d8c565b-kube-api-access-kmm7j" (OuterVolumeSpecName: "kube-api-access-kmm7j") pod "ad86ca06-7981-4a47-9a2a-90717d8c565b" (UID: "ad86ca06-7981-4a47-9a2a-90717d8c565b"). InnerVolumeSpecName "kube-api-access-kmm7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:02:00.674608 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:00.674571    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmm7j\" (UniqueName: \"kubernetes.io/projected/ad86ca06-7981-4a47-9a2a-90717d8c565b-kube-api-access-kmm7j\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:02:01.294541 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:01.294506    2565 generic.go:356] "Generic (PLEG): container finished" podID="ad86ca06-7981-4a47-9a2a-90717d8c565b" containerID="095fe7cd7bcd135d2d69386b4545064b5658c261f653b3264faa27ace4fe0e2f" exitCode=137
Sep 14 05:02:01.294993 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:01.294549    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7545-a8d0bb"
Sep 14 05:02:01.294993 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:01.294597    2565 scope.go:117] "RemoveContainer" containerID="095fe7cd7bcd135d2d69386b4545064b5658c261f653b3264faa27ace4fe0e2f"
Sep 14 05:02:01.294993 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:01.294787    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7545-ba9b29" podUID="30b99dbe-ea2e-475a-9607-7598a8d3ca01" containerName="test-curl-7545-ba9b29" containerID="cri-o://f494897c3f84ab3e9197ec2eddcc860973f4630f929109a7fdf70aaaa11bd33f" gracePeriod=2
Sep 14 05:02:01.303305 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:01.303289    2565 scope.go:117] "RemoveContainer" containerID="095fe7cd7bcd135d2d69386b4545064b5658c261f653b3264faa27ace4fe0e2f"
Sep 14 05:02:01.303541 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:02:01.303520    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"095fe7cd7bcd135d2d69386b4545064b5658c261f653b3264faa27ace4fe0e2f\": container with ID starting with 095fe7cd7bcd135d2d69386b4545064b5658c261f653b3264faa27ace4fe0e2f not found: ID does not exist" containerID="095fe7cd7bcd135d2d69386b4545064b5658c261f653b3264faa27ace4fe0e2f"
Sep 14 05:02:01.303594 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:01.303549    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"095fe7cd7bcd135d2d69386b4545064b5658c261f653b3264faa27ace4fe0e2f"} err="failed to get container status \"095fe7cd7bcd135d2d69386b4545064b5658c261f653b3264faa27ace4fe0e2f\": rpc error: code = NotFound desc = could not find container \"095fe7cd7bcd135d2d69386b4545064b5658c261f653b3264faa27ace4fe0e2f\": container with ID starting with 095fe7cd7bcd135d2d69386b4545064b5658c261f653b3264faa27ace4fe0e2f not found: ID does not exist"
Sep 14 05:02:01.304209 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:02:01.304191    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7545-a8d0bb\" is forbidden: User \"system:node:ip-10-0-133-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-99.ec2.internal' and this object" podUID="ad86ca06-7981-4a47-9a2a-90717d8c565b" pod="opendatahub/test-curl-7545-a8d0bb"
Sep 14 05:02:02.334484 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:02:02.334439    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7545-a8d0bb\" is forbidden: User \"system:node:ip-10-0-133-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-99.ec2.internal' and this object" podUID="ad86ca06-7981-4a47-9a2a-90717d8c565b" pod="opendatahub/test-curl-7545-a8d0bb"
Sep 14 05:02:02.335620 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:02.335597    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad86ca06-7981-4a47-9a2a-90717d8c565b" path="/var/lib/kubelet/pods/ad86ca06-7981-4a47-9a2a-90717d8c565b/volumes"
Sep 14 05:02:03.015086 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:03.015047    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f14475-59bb95458-z5mxz"]
Sep 14 05:02:03.015392 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:03.015379    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edd0f06e-d2a0-4435-a79d-f245fc2c0960" containerName="revoke-keys"
Sep 14 05:02:03.015438 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:03.015394    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="edd0f06e-d2a0-4435-a79d-f245fc2c0960" containerName="revoke-keys"
Sep 14 05:02:03.015438 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:03.015414    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30b99dbe-ea2e-475a-9607-7598a8d3ca01" containerName="test-curl-7545-ba9b29"
Sep 14 05:02:03.015438 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:03.015420    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="30b99dbe-ea2e-475a-9607-7598a8d3ca01" containerName="test-curl-7545-ba9b29"
Sep 14 05:02:03.015534 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:03.015472    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="30b99dbe-ea2e-475a-9607-7598a8d3ca01" containerName="test-curl-7545-ba9b29"
Sep 14 05:02:03.015534 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:03.015482    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="edd0f06e-d2a0-4435-a79d-f245fc2c0960" containerName="revoke-keys"
Sep 14 05:02:03.019701 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:03.019683    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-f14475-59bb95458-z5mxz"
Sep 14 05:02:03.021891 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:03.021871    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-f14475-dockercfg-bwxfp\""
Sep 14 05:02:03.021960 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:03.021892    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-f14475\""
Sep 14 05:02:03.027559 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:03.027535    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f14475-59bb95458-z5mxz"]
Sep 14 05:02:03.055811 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:03.055779    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-f14475-857d5b4f79-crdsn"]
Sep 14 05:02:03.059193 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:03.059175    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-f14475-857d5b4f79-crdsn"
Sep 14 05:02:03.067733 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:03.067709    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-f14475-857d5b4f79-crdsn"]
Sep 14 05:02:03.095993 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:03.095962    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nvtq\" (UniqueName: \"kubernetes.io/projected/cc6a0970-e07a-4a2a-9232-57b8d402b0bb-kube-api-access-2nvtq\") pod \"payload-pre-processing-e2e-autores-f14475-59bb95458-z5mxz\" (UID: \"cc6a0970-e07a-4a2a-9232-57b8d402b0bb\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f14475-59bb95458-z5mxz"
Sep 14 05:02:03.096118 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:03.096020    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc6a0970-e07a-4a2a-9232-57b8d402b0bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-f14475-59bb95458-z5mxz\" (UID: \"cc6a0970-e07a-4a2a-9232-57b8d402b0bb\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f14475-59bb95458-z5mxz"
Sep 14 05:02:03.096118 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:03.096082    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5fa6bb8-e10f-41af-9b37-962fad6aef8e-plugins-config-volume\") pod \"payload-processing-e2e-autores-f14475-857d5b4f79-crdsn\" (UID: \"a5fa6bb8-e10f-41af-9b37-962fad6aef8e\") " pod="openshift-ingress/payload-processing-e2e-autores-f14475-857d5b4f79-crdsn"
Sep 14 05:02:03.096207 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:03.096123    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j48dw\" (UniqueName: \"kubernetes.io/projected/a5fa6bb8-e10f-41af-9b37-962fad6aef8e-kube-api-access-j48dw\") pod \"payload-processing-e2e-autores-f14475-857d5b4f79-crdsn\" (UID: \"a5fa6bb8-e10f-41af-9b37-962fad6aef8e\") " pod="openshift-ingress/payload-processing-e2e-autores-f14475-857d5b4f79-crdsn"
Sep 14 05:02:03.197331 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:03.197290    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nvtq\" (UniqueName: \"kubernetes.io/projected/cc6a0970-e07a-4a2a-9232-57b8d402b0bb-kube-api-access-2nvtq\") pod \"payload-pre-processing-e2e-autores-f14475-59bb95458-z5mxz\" (UID: \"cc6a0970-e07a-4a2a-9232-57b8d402b0bb\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f14475-59bb95458-z5mxz"
Sep 14 05:02:03.197507 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:03.197353    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc6a0970-e07a-4a2a-9232-57b8d402b0bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-f14475-59bb95458-z5mxz\" (UID: \"cc6a0970-e07a-4a2a-9232-57b8d402b0bb\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f14475-59bb95458-z5mxz"
Sep 14 05:02:03.197507 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:03.197376    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5fa6bb8-e10f-41af-9b37-962fad6aef8e-plugins-config-volume\") pod \"payload-processing-e2e-autores-f14475-857d5b4f79-crdsn\" (UID: \"a5fa6bb8-e10f-41af-9b37-962fad6aef8e\") " pod="openshift-ingress/payload-processing-e2e-autores-f14475-857d5b4f79-crdsn"
Sep 14 05:02:03.197507 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:03.197394    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j48dw\" (UniqueName: \"kubernetes.io/projected/a5fa6bb8-e10f-41af-9b37-962fad6aef8e-kube-api-access-j48dw\") pod \"payload-processing-e2e-autores-f14475-857d5b4f79-crdsn\" (UID: \"a5fa6bb8-e10f-41af-9b37-962fad6aef8e\") " pod="openshift-ingress/payload-processing-e2e-autores-f14475-857d5b4f79-crdsn"
Sep 14 05:02:03.198117 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:03.198095    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc6a0970-e07a-4a2a-9232-57b8d402b0bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-f14475-59bb95458-z5mxz\" (UID: \"cc6a0970-e07a-4a2a-9232-57b8d402b0bb\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f14475-59bb95458-z5mxz"
Sep 14 05:02:03.198179 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:03.198117    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5fa6bb8-e10f-41af-9b37-962fad6aef8e-plugins-config-volume\") pod \"payload-processing-e2e-autores-f14475-857d5b4f79-crdsn\" (UID: \"a5fa6bb8-e10f-41af-9b37-962fad6aef8e\") " pod="openshift-ingress/payload-processing-e2e-autores-f14475-857d5b4f79-crdsn"
Sep 14 05:02:03.205050 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:03.205026    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j48dw\" (UniqueName: \"kubernetes.io/projected/a5fa6bb8-e10f-41af-9b37-962fad6aef8e-kube-api-access-j48dw\") pod \"payload-processing-e2e-autores-f14475-857d5b4f79-crdsn\" (UID: \"a5fa6bb8-e10f-41af-9b37-962fad6aef8e\") " pod="openshift-ingress/payload-processing-e2e-autores-f14475-857d5b4f79-crdsn"
Sep 14 05:02:03.205144 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:03.205026    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nvtq\" (UniqueName: \"kubernetes.io/projected/cc6a0970-e07a-4a2a-9232-57b8d402b0bb-kube-api-access-2nvtq\") pod \"payload-pre-processing-e2e-autores-f14475-59bb95458-z5mxz\" (UID: \"cc6a0970-e07a-4a2a-9232-57b8d402b0bb\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f14475-59bb95458-z5mxz"
Sep 14 05:02:03.332079 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:03.332056    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-f14475-59bb95458-z5mxz"
Sep 14 05:02:03.370008 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:03.369214    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-f14475-857d5b4f79-crdsn"
Sep 14 05:02:03.491183 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:03.491159    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7545-ba9b29"
Sep 14 05:02:03.601724 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:03.601691    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvhtd\" (UniqueName: \"kubernetes.io/projected/30b99dbe-ea2e-475a-9607-7598a8d3ca01-kube-api-access-nvhtd\") pod \"30b99dbe-ea2e-475a-9607-7598a8d3ca01\" (UID: \"30b99dbe-ea2e-475a-9607-7598a8d3ca01\") "
Sep 14 05:02:03.603585 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:03.603564    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30b99dbe-ea2e-475a-9607-7598a8d3ca01-kube-api-access-nvhtd" (OuterVolumeSpecName: "kube-api-access-nvhtd") pod "30b99dbe-ea2e-475a-9607-7598a8d3ca01" (UID: "30b99dbe-ea2e-475a-9607-7598a8d3ca01"). InnerVolumeSpecName "kube-api-access-nvhtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:02:03.608670 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:03.608647    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f14475-59bb95458-z5mxz"]
Sep 14 05:02:03.609812 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:02:03.609788    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6a0970_e07a_4a2a_9232_57b8d402b0bb.slice/crio-0c510eebd6a42d3c361dfd363476ac170aab085aef5a22eccd608dd569a986d9 WatchSource:0}: Error finding container 0c510eebd6a42d3c361dfd363476ac170aab085aef5a22eccd608dd569a986d9: Status 404 returned error can't find the container with id 0c510eebd6a42d3c361dfd363476ac170aab085aef5a22eccd608dd569a986d9
Sep 14 05:02:03.654945 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:03.654923    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-f14475-857d5b4f79-crdsn"]
Sep 14 05:02:03.656279 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:02:03.656256    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5fa6bb8_e10f_41af_9b37_962fad6aef8e.slice/crio-692004580bcc195c42296643418aa9c670303647520a43a66e6040bd512815df WatchSource:0}: Error finding container 692004580bcc195c42296643418aa9c670303647520a43a66e6040bd512815df: Status 404 returned error can't find the container with id 692004580bcc195c42296643418aa9c670303647520a43a66e6040bd512815df
Sep 14 05:02:03.702340 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:03.702315    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvhtd\" (UniqueName: \"kubernetes.io/projected/30b99dbe-ea2e-475a-9607-7598a8d3ca01-kube-api-access-nvhtd\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:02:04.315132 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:04.315094    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-f14475-857d5b4f79-crdsn" event={"ID":"a5fa6bb8-e10f-41af-9b37-962fad6aef8e","Type":"ContainerStarted","Data":"98da3ea55890a050c05816f0a4ab9ba5bc4a802101db5a2aba02ccc11153ef10"}
Sep 14 05:02:04.315297 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:04.315146    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-f14475-857d5b4f79-crdsn" event={"ID":"a5fa6bb8-e10f-41af-9b37-962fad6aef8e","Type":"ContainerStarted","Data":"692004580bcc195c42296643418aa9c670303647520a43a66e6040bd512815df"}
Sep 14 05:02:04.315297 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:04.315198    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-f14475-857d5b4f79-crdsn"
Sep 14 05:02:04.316426 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:04.316401    2565 generic.go:356] "Generic (PLEG): container finished" podID="30b99dbe-ea2e-475a-9607-7598a8d3ca01" containerID="f494897c3f84ab3e9197ec2eddcc860973f4630f929109a7fdf70aaaa11bd33f" exitCode=137
Sep 14 05:02:04.316530 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:04.316439    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7545-ba9b29"
Sep 14 05:02:04.316530 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:04.316479    2565 scope.go:117] "RemoveContainer" containerID="f494897c3f84ab3e9197ec2eddcc860973f4630f929109a7fdf70aaaa11bd33f"
Sep 14 05:02:04.318111 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:04.318085    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-f14475-59bb95458-z5mxz" event={"ID":"cc6a0970-e07a-4a2a-9232-57b8d402b0bb","Type":"ContainerStarted","Data":"39447f692fb5245276fb446983dd3aa3f1f7c88486526fb128d3701c38449104"}
Sep 14 05:02:04.318213 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:04.318133    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-f14475-59bb95458-z5mxz" event={"ID":"cc6a0970-e07a-4a2a-9232-57b8d402b0bb","Type":"ContainerStarted","Data":"0c510eebd6a42d3c361dfd363476ac170aab085aef5a22eccd608dd569a986d9"}
Sep 14 05:02:04.318213 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:04.318165    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-f14475-59bb95458-z5mxz"
Sep 14 05:02:04.325801 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:04.325780    2565 scope.go:117] "RemoveContainer" containerID="f494897c3f84ab3e9197ec2eddcc860973f4630f929109a7fdf70aaaa11bd33f"
Sep 14 05:02:04.326118 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:02:04.326093    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f494897c3f84ab3e9197ec2eddcc860973f4630f929109a7fdf70aaaa11bd33f\": container with ID starting with f494897c3f84ab3e9197ec2eddcc860973f4630f929109a7fdf70aaaa11bd33f not found: ID does not exist" containerID="f494897c3f84ab3e9197ec2eddcc860973f4630f929109a7fdf70aaaa11bd33f"
Sep 14 05:02:04.326193 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:04.326126    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f494897c3f84ab3e9197ec2eddcc860973f4630f929109a7fdf70aaaa11bd33f"} err="failed to get container status \"f494897c3f84ab3e9197ec2eddcc860973f4630f929109a7fdf70aaaa11bd33f\": rpc error: code = NotFound desc = could not find container \"f494897c3f84ab3e9197ec2eddcc860973f4630f929109a7fdf70aaaa11bd33f\": container with ID starting with f494897c3f84ab3e9197ec2eddcc860973f4630f929109a7fdf70aaaa11bd33f not found: ID does not exist"
Sep 14 05:02:04.335159 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:04.335108    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-f14475-857d5b4f79-crdsn" podStartSLOduration=1.3350921279999999 podStartE2EDuration="1.335092128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:02:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:02:04.331249583 +0000 UTC m=+922.507021132" watchObservedRunningTime="2026-09-14 05:02:04.335092128 +0000 UTC m=+922.510863689"
Sep 14 05:02:04.341188 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:02:04.341161    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7545-ba9b29\" is forbidden: User \"system:node:ip-10-0-133-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-99.ec2.internal' and this object" podUID="30b99dbe-ea2e-475a-9607-7598a8d3ca01" pod="opendatahub/test-curl-7545-ba9b29"
Sep 14 05:02:04.342649 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:04.342628    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30b99dbe-ea2e-475a-9607-7598a8d3ca01" path="/var/lib/kubelet/pods/30b99dbe-ea2e-475a-9607-7598a8d3ca01/volumes"
Sep 14 05:02:04.363274 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:04.363228    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-f14475-59bb95458-z5mxz" podStartSLOduration=2.363211425 podStartE2EDuration="2.363211425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:02:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:02:04.359385376 +0000 UTC m=+922.535156934" watchObservedRunningTime="2026-09-14 05:02:04.363211425 +0000 UTC m=+922.538982984"
Sep 14 05:02:07.363422 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:07.363389    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-654e8bde-f5c84745-5s7k2"]
Sep 14 05:02:07.363831 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:07.363657    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-654e8bde-f5c84745-5s7k2" podUID="9d2bd35b-18c9-4944-b616-65a8ef81f4d8" containerName="payload-processing" containerID="cri-o://e39812e2ad01daf8ccb944b2a011279e37220851ce2b7141904c2ffb54b8fb8f" gracePeriod=30
Sep 14 05:02:07.368262 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:07.368234    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-654e8bde-5c9d48dbf4-mqjqq"]
Sep 14 05:02:07.371498 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:07.368479    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-654e8bde-5c9d48dbf4-mqjqq" podUID="e78c59a2-18da-4216-8d59-f267b844cf72" containerName="payload-pre-processing" containerID="cri-o://1d32565df3964e832a89cc4d3ad17e89097db63bc4843ccdae1f67a9775e69c2" gracePeriod=30
Sep 14 05:02:07.528486 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:07.528366    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-654e8bde-bvll5"]
Sep 14 05:02:07.530803 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:07.530750    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-654e8bde-bvll5"]
Sep 14 05:02:07.628808 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:07.628781    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-654e8bde-f5c84745-5s7k2"
Sep 14 05:02:07.651261 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:07.651230    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-654e8bde-5c9d48dbf4-mqjqq"
Sep 14 05:02:07.742264 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:07.742230    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jc2nv\" (UniqueName: \"kubernetes.io/projected/9d2bd35b-18c9-4944-b616-65a8ef81f4d8-kube-api-access-jc2nv\") pod \"9d2bd35b-18c9-4944-b616-65a8ef81f4d8\" (UID: \"9d2bd35b-18c9-4944-b616-65a8ef81f4d8\") "
Sep 14 05:02:07.742428 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:07.742285    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ftdn\" (UniqueName: \"kubernetes.io/projected/e78c59a2-18da-4216-8d59-f267b844cf72-kube-api-access-8ftdn\") pod \"e78c59a2-18da-4216-8d59-f267b844cf72\" (UID: \"e78c59a2-18da-4216-8d59-f267b844cf72\") "
Sep 14 05:02:07.742428 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:07.742367    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e78c59a2-18da-4216-8d59-f267b844cf72-plugins-config-volume\") pod \"e78c59a2-18da-4216-8d59-f267b844cf72\" (UID: \"e78c59a2-18da-4216-8d59-f267b844cf72\") "
Sep 14 05:02:07.742428 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:07.742408    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d2bd35b-18c9-4944-b616-65a8ef81f4d8-plugins-config-volume\") pod \"9d2bd35b-18c9-4944-b616-65a8ef81f4d8\" (UID: \"9d2bd35b-18c9-4944-b616-65a8ef81f4d8\") "
Sep 14 05:02:07.742787 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:07.742735    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e78c59a2-18da-4216-8d59-f267b844cf72-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e78c59a2-18da-4216-8d59-f267b844cf72" (UID: "e78c59a2-18da-4216-8d59-f267b844cf72"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 05:02:07.742920 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:07.742894    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d2bd35b-18c9-4944-b616-65a8ef81f4d8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9d2bd35b-18c9-4944-b616-65a8ef81f4d8" (UID: "9d2bd35b-18c9-4944-b616-65a8ef81f4d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 05:02:07.744545 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:07.744517    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e78c59a2-18da-4216-8d59-f267b844cf72-kube-api-access-8ftdn" (OuterVolumeSpecName: "kube-api-access-8ftdn") pod "e78c59a2-18da-4216-8d59-f267b844cf72" (UID: "e78c59a2-18da-4216-8d59-f267b844cf72"). InnerVolumeSpecName "kube-api-access-8ftdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:02:07.744638 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:07.744551    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d2bd35b-18c9-4944-b616-65a8ef81f4d8-kube-api-access-jc2nv" (OuterVolumeSpecName: "kube-api-access-jc2nv") pod "9d2bd35b-18c9-4944-b616-65a8ef81f4d8" (UID: "9d2bd35b-18c9-4944-b616-65a8ef81f4d8"). InnerVolumeSpecName "kube-api-access-jc2nv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:02:07.843323 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:07.843292    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e78c59a2-18da-4216-8d59-f267b844cf72-plugins-config-volume\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:02:07.843323 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:07.843318    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d2bd35b-18c9-4944-b616-65a8ef81f4d8-plugins-config-volume\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:02:07.843323 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:07.843329    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc2nv\" (UniqueName: \"kubernetes.io/projected/9d2bd35b-18c9-4944-b616-65a8ef81f4d8-kube-api-access-jc2nv\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:02:07.843527 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:07.843337    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ftdn\" (UniqueName: \"kubernetes.io/projected/e78c59a2-18da-4216-8d59-f267b844cf72-kube-api-access-8ftdn\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:02:08.276131 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:08.276101    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e7fdd6-56c7655575-4ltrp"
Sep 14 05:02:08.335830 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:08.335796    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edd0f06e-d2a0-4435-a79d-f245fc2c0960" path="/var/lib/kubelet/pods/edd0f06e-d2a0-4435-a79d-f245fc2c0960/volumes"
Sep 14 05:02:08.342493 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:08.342462    2565 generic.go:356] "Generic (PLEG): container finished" podID="9d2bd35b-18c9-4944-b616-65a8ef81f4d8" containerID="e39812e2ad01daf8ccb944b2a011279e37220851ce2b7141904c2ffb54b8fb8f" exitCode=0
Sep 14 05:02:08.342646 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:08.342514    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-654e8bde-f5c84745-5s7k2"
Sep 14 05:02:08.342646 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:08.342539    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-654e8bde-f5c84745-5s7k2" event={"ID":"9d2bd35b-18c9-4944-b616-65a8ef81f4d8","Type":"ContainerDied","Data":"e39812e2ad01daf8ccb944b2a011279e37220851ce2b7141904c2ffb54b8fb8f"}
Sep 14 05:02:08.342646 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:08.342597    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-654e8bde-f5c84745-5s7k2" event={"ID":"9d2bd35b-18c9-4944-b616-65a8ef81f4d8","Type":"ContainerDied","Data":"6ec45984430ad0e2a06df2f96749735023f5389fa3059d7261f0c83921f7062a"}
Sep 14 05:02:08.342646 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:08.342628    2565 scope.go:117] "RemoveContainer" containerID="e39812e2ad01daf8ccb944b2a011279e37220851ce2b7141904c2ffb54b8fb8f"
Sep 14 05:02:08.345848 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:08.345610    2565 generic.go:356] "Generic (PLEG): container finished" podID="e78c59a2-18da-4216-8d59-f267b844cf72" containerID="1d32565df3964e832a89cc4d3ad17e89097db63bc4843ccdae1f67a9775e69c2" exitCode=0
Sep 14 05:02:08.345848 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:08.345651    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-654e8bde-5c9d48dbf4-mqjqq"
Sep 14 05:02:08.345848 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:08.345711    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-654e8bde-5c9d48dbf4-mqjqq" event={"ID":"e78c59a2-18da-4216-8d59-f267b844cf72","Type":"ContainerDied","Data":"1d32565df3964e832a89cc4d3ad17e89097db63bc4843ccdae1f67a9775e69c2"}
Sep 14 05:02:08.345848 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:08.345765    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-654e8bde-5c9d48dbf4-mqjqq" event={"ID":"e78c59a2-18da-4216-8d59-f267b844cf72","Type":"ContainerDied","Data":"5399dd69db3afcc5cb897fefcdd6516becfd064ac8cb6cbf08918b51f4a1942a"}
Sep 14 05:02:08.354680 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:08.354663    2565 scope.go:117] "RemoveContainer" containerID="e39812e2ad01daf8ccb944b2a011279e37220851ce2b7141904c2ffb54b8fb8f"
Sep 14 05:02:08.354983 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:02:08.354954    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e39812e2ad01daf8ccb944b2a011279e37220851ce2b7141904c2ffb54b8fb8f\": container with ID starting with e39812e2ad01daf8ccb944b2a011279e37220851ce2b7141904c2ffb54b8fb8f not found: ID does not exist" containerID="e39812e2ad01daf8ccb944b2a011279e37220851ce2b7141904c2ffb54b8fb8f"
Sep 14 05:02:08.355107 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:08.354980    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e39812e2ad01daf8ccb944b2a011279e37220851ce2b7141904c2ffb54b8fb8f"} err="failed to get container status \"e39812e2ad01daf8ccb944b2a011279e37220851ce2b7141904c2ffb54b8fb8f\": rpc error: code = NotFound desc = could not find container \"e39812e2ad01daf8ccb944b2a011279e37220851ce2b7141904c2ffb54b8fb8f\": container with ID starting with e39812e2ad01daf8ccb944b2a011279e37220851ce2b7141904c2ffb54b8fb8f not found: ID does not exist"
Sep 14 05:02:08.355107 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:08.355000    2565 scope.go:117] "RemoveContainer" containerID="1d32565df3964e832a89cc4d3ad17e89097db63bc4843ccdae1f67a9775e69c2"
Sep 14 05:02:08.364646 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:08.364622    2565 scope.go:117] "RemoveContainer" containerID="1d32565df3964e832a89cc4d3ad17e89097db63bc4843ccdae1f67a9775e69c2"
Sep 14 05:02:08.364984 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:02:08.364947    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d32565df3964e832a89cc4d3ad17e89097db63bc4843ccdae1f67a9775e69c2\": container with ID starting with 1d32565df3964e832a89cc4d3ad17e89097db63bc4843ccdae1f67a9775e69c2 not found: ID does not exist" containerID="1d32565df3964e832a89cc4d3ad17e89097db63bc4843ccdae1f67a9775e69c2"
Sep 14 05:02:08.365036 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:08.364973    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d32565df3964e832a89cc4d3ad17e89097db63bc4843ccdae1f67a9775e69c2"} err="failed to get container status \"1d32565df3964e832a89cc4d3ad17e89097db63bc4843ccdae1f67a9775e69c2\": rpc error: code = NotFound desc = could not find container \"1d32565df3964e832a89cc4d3ad17e89097db63bc4843ccdae1f67a9775e69c2\": container with ID starting with 1d32565df3964e832a89cc4d3ad17e89097db63bc4843ccdae1f67a9775e69c2 not found: ID does not exist"
Sep 14 05:02:08.365459 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:08.365440    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-654e8bde-5c9d48dbf4-mqjqq"]
Sep 14 05:02:08.367833 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:08.367814    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-654e8bde-5c9d48dbf4-mqjqq"]
Sep 14 05:02:08.376791 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:08.376749    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-654e8bde-f5c84745-5s7k2"]
Sep 14 05:02:08.380445 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:08.380423    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-654e8bde-f5c84745-5s7k2"]
Sep 14 05:02:09.129441 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:09.129407    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:02:10.334182 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:10.334148    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d2bd35b-18c9-4944-b616-65a8ef81f4d8" path="/var/lib/kubelet/pods/9d2bd35b-18c9-4944-b616-65a8ef81f4d8/volumes"
Sep 14 05:02:10.334544 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:10.334499    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e78c59a2-18da-4216-8d59-f267b844cf72" path="/var/lib/kubelet/pods/e78c59a2-18da-4216-8d59-f267b844cf72/volumes"
Sep 14 05:02:13.483969 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:13.483938    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-60e333a9-79654cfdfd-mfmjv"]
Sep 14 05:02:13.484331 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:13.484308    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d2bd35b-18c9-4944-b616-65a8ef81f4d8" containerName="payload-processing"
Sep 14 05:02:13.484331 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:13.484321    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d2bd35b-18c9-4944-b616-65a8ef81f4d8" containerName="payload-processing"
Sep 14 05:02:13.484410 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:13.484336    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e78c59a2-18da-4216-8d59-f267b844cf72" containerName="payload-pre-processing"
Sep 14 05:02:13.484410 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:13.484343    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e78c59a2-18da-4216-8d59-f267b844cf72" containerName="payload-pre-processing"
Sep 14 05:02:13.484410 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:13.484407    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d2bd35b-18c9-4944-b616-65a8ef81f4d8" containerName="payload-processing"
Sep 14 05:02:13.484499 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:13.484419    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e78c59a2-18da-4216-8d59-f267b844cf72" containerName="payload-pre-processing"
Sep 14 05:02:13.488635 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:13.488620    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-60e333a9-79654cfdfd-mfmjv"
Sep 14 05:02:13.490744 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:13.490725    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-60e333a9-dockercfg-vnr6s\""
Sep 14 05:02:13.490844 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:13.490743    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-60e333a9\""
Sep 14 05:02:13.497245 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:13.497223    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-60e333a9-79654cfdfd-mfmjv"]
Sep 14 05:02:13.514637 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:13.514613    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-60e333a9-575c6488c8-2h2wc"]
Sep 14 05:02:13.517864 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:13.517848    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-60e333a9-575c6488c8-2h2wc"
Sep 14 05:02:13.527255 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:13.527232    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-60e333a9-575c6488c8-2h2wc"]
Sep 14 05:02:13.598855 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:13.598823    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57313b1d-6d32-4378-971b-4b830d0ed8bd-plugins-config-volume\") pod \"payload-processing-e2e-migrate-60e333a9-575c6488c8-2h2wc\" (UID: \"57313b1d-6d32-4378-971b-4b830d0ed8bd\") " pod="openshift-ingress/payload-processing-e2e-migrate-60e333a9-575c6488c8-2h2wc"
Sep 14 05:02:13.599009 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:13.598863    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/050a107d-6c44-4628-9a6d-1d593d38b7dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-60e333a9-79654cfdfd-mfmjv\" (UID: \"050a107d-6c44-4628-9a6d-1d593d38b7dd\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-60e333a9-79654cfdfd-mfmjv"
Sep 14 05:02:13.599009 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:13.598892    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nq4s\" (UniqueName: \"kubernetes.io/projected/57313b1d-6d32-4378-971b-4b830d0ed8bd-kube-api-access-8nq4s\") pod \"payload-processing-e2e-migrate-60e333a9-575c6488c8-2h2wc\" (UID: \"57313b1d-6d32-4378-971b-4b830d0ed8bd\") " pod="openshift-ingress/payload-processing-e2e-migrate-60e333a9-575c6488c8-2h2wc"
Sep 14 05:02:13.599009 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:13.598924    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfgdn\" (UniqueName: \"kubernetes.io/projected/050a107d-6c44-4628-9a6d-1d593d38b7dd-kube-api-access-lfgdn\") pod \"payload-pre-processing-e2e-migrate-60e333a9-79654cfdfd-mfmjv\" (UID: \"050a107d-6c44-4628-9a6d-1d593d38b7dd\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-60e333a9-79654cfdfd-mfmjv"
Sep 14 05:02:13.700175 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:13.700127    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57313b1d-6d32-4378-971b-4b830d0ed8bd-plugins-config-volume\") pod \"payload-processing-e2e-migrate-60e333a9-575c6488c8-2h2wc\" (UID: \"57313b1d-6d32-4378-971b-4b830d0ed8bd\") " pod="openshift-ingress/payload-processing-e2e-migrate-60e333a9-575c6488c8-2h2wc"
Sep 14 05:02:13.700175 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:13.700183    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/050a107d-6c44-4628-9a6d-1d593d38b7dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-60e333a9-79654cfdfd-mfmjv\" (UID: \"050a107d-6c44-4628-9a6d-1d593d38b7dd\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-60e333a9-79654cfdfd-mfmjv"
Sep 14 05:02:13.700427 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:13.700219    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nq4s\" (UniqueName: \"kubernetes.io/projected/57313b1d-6d32-4378-971b-4b830d0ed8bd-kube-api-access-8nq4s\") pod \"payload-processing-e2e-migrate-60e333a9-575c6488c8-2h2wc\" (UID: \"57313b1d-6d32-4378-971b-4b830d0ed8bd\") " pod="openshift-ingress/payload-processing-e2e-migrate-60e333a9-575c6488c8-2h2wc"
Sep 14 05:02:13.700427 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:13.700244    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfgdn\" (UniqueName: \"kubernetes.io/projected/050a107d-6c44-4628-9a6d-1d593d38b7dd-kube-api-access-lfgdn\") pod \"payload-pre-processing-e2e-migrate-60e333a9-79654cfdfd-mfmjv\" (UID: \"050a107d-6c44-4628-9a6d-1d593d38b7dd\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-60e333a9-79654cfdfd-mfmjv"
Sep 14 05:02:13.700927 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:13.700907    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/050a107d-6c44-4628-9a6d-1d593d38b7dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-60e333a9-79654cfdfd-mfmjv\" (UID: \"050a107d-6c44-4628-9a6d-1d593d38b7dd\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-60e333a9-79654cfdfd-mfmjv"
Sep 14 05:02:13.701054 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:13.701037    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57313b1d-6d32-4378-971b-4b830d0ed8bd-plugins-config-volume\") pod \"payload-processing-e2e-migrate-60e333a9-575c6488c8-2h2wc\" (UID: \"57313b1d-6d32-4378-971b-4b830d0ed8bd\") " pod="openshift-ingress/payload-processing-e2e-migrate-60e333a9-575c6488c8-2h2wc"
Sep 14 05:02:13.708687 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:13.708665    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfgdn\" (UniqueName: \"kubernetes.io/projected/050a107d-6c44-4628-9a6d-1d593d38b7dd-kube-api-access-lfgdn\") pod \"payload-pre-processing-e2e-migrate-60e333a9-79654cfdfd-mfmjv\" (UID: \"050a107d-6c44-4628-9a6d-1d593d38b7dd\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-60e333a9-79654cfdfd-mfmjv"
Sep 14 05:02:13.710565 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:13.710543    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nq4s\" (UniqueName: \"kubernetes.io/projected/57313b1d-6d32-4378-971b-4b830d0ed8bd-kube-api-access-8nq4s\") pod \"payload-processing-e2e-migrate-60e333a9-575c6488c8-2h2wc\" (UID: \"57313b1d-6d32-4378-971b-4b830d0ed8bd\") " pod="openshift-ingress/payload-processing-e2e-migrate-60e333a9-575c6488c8-2h2wc"
Sep 14 05:02:13.799272 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:13.799190    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-60e333a9-79654cfdfd-mfmjv"
Sep 14 05:02:13.827853 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:13.827825    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-60e333a9-575c6488c8-2h2wc"
Sep 14 05:02:14.098086 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:14.098060    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-60e333a9-79654cfdfd-mfmjv"]
Sep 14 05:02:14.100181 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:02:14.100143    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod050a107d_6c44_4628_9a6d_1d593d38b7dd.slice/crio-84c7c2bdcac1a7aac85e1c992ad1708d507c37af2a676fa6d377ea42108bfb79 WatchSource:0}: Error finding container 84c7c2bdcac1a7aac85e1c992ad1708d507c37af2a676fa6d377ea42108bfb79: Status 404 returned error can't find the container with id 84c7c2bdcac1a7aac85e1c992ad1708d507c37af2a676fa6d377ea42108bfb79
Sep 14 05:02:14.126493 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:14.126470    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-60e333a9-575c6488c8-2h2wc"]
Sep 14 05:02:14.383624 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:14.383519    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-60e333a9-575c6488c8-2h2wc" event={"ID":"57313b1d-6d32-4378-971b-4b830d0ed8bd","Type":"ContainerStarted","Data":"5149d5de3e175ad159d08767cda29f7b4a74efd333c1edf4c583f5e48108125e"}
Sep 14 05:02:14.383624 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:14.383583    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-60e333a9-575c6488c8-2h2wc" event={"ID":"57313b1d-6d32-4378-971b-4b830d0ed8bd","Type":"ContainerStarted","Data":"e0f59fba44fedd20880e2b4def1ce3f8fe6fb1d70ac9a903e0d3220e0152cafa"}
Sep 14 05:02:14.383881 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:14.383657    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-60e333a9-575c6488c8-2h2wc"
Sep 14 05:02:14.385560 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:14.385538    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-60e333a9-79654cfdfd-mfmjv" event={"ID":"050a107d-6c44-4628-9a6d-1d593d38b7dd","Type":"ContainerStarted","Data":"7ab01af8e2bb78b97aa29e766155f5085647dea433c7796558f47349a6479fca"}
Sep 14 05:02:14.385668 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:14.385574    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-60e333a9-79654cfdfd-mfmjv" event={"ID":"050a107d-6c44-4628-9a6d-1d593d38b7dd","Type":"ContainerStarted","Data":"84c7c2bdcac1a7aac85e1c992ad1708d507c37af2a676fa6d377ea42108bfb79"}
Sep 14 05:02:14.385668 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:14.385640    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-60e333a9-79654cfdfd-mfmjv"
Sep 14 05:02:14.400925 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:14.400881    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-60e333a9-575c6488c8-2h2wc" podStartSLOduration=1.400868628 podStartE2EDuration="1.400868628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:02:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:02:14.39895902 +0000 UTC m=+932.574730592" watchObservedRunningTime="2026-09-14 05:02:14.400868628 +0000 UTC m=+932.576640213"
Sep 14 05:02:14.416663 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:14.416613    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-60e333a9-79654cfdfd-mfmjv" podStartSLOduration=1.416600223 podStartE2EDuration="1.416600223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:02:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:02:14.415136757 +0000 UTC m=+932.590908318" watchObservedRunningTime="2026-09-14 05:02:14.416600223 +0000 UTC m=+932.592371782"
Sep 14 05:02:15.327317 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:15.327283    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-f14475-857d5b4f79-crdsn"
Sep 14 05:02:15.327795 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:15.327442    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-f14475-59bb95458-z5mxz"
Sep 14 05:02:19.419925 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:19.419892    2565 generic.go:356] "Generic (PLEG): container finished" podID="38ea6dfe-3dce-404e-988c-8b3d8f912061" containerID="ab07c93f774034c727d587b22c01c02ec9cc6d1f897e8c3fbb65960d1fc1fd4f" exitCode=0
Sep 14 05:02:19.420346 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:19.419964    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9" event={"ID":"38ea6dfe-3dce-404e-988c-8b3d8f912061","Type":"ContainerDied","Data":"ab07c93f774034c727d587b22c01c02ec9cc6d1f897e8c3fbb65960d1fc1fd4f"}
Sep 14 05:02:19.420427 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:19.420411    2565 scope.go:117] "RemoveContainer" containerID="ab07c93f774034c727d587b22c01c02ec9cc6d1f897e8c3fbb65960d1fc1fd4f"
Sep 14 05:02:19.421483 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:19.421454    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:02:19.421550 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:19.421537    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:02:19.421591 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:19.421571    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:02:19.643048 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:19.642965    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:02:20.427324 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:20.427286    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9" event={"ID":"38ea6dfe-3dce-404e-988c-8b3d8f912061","Type":"ContainerStarted","Data":"bbb9cbc907fba4b0cc609308f526156a19da61b2b51b47a1326345db0bc7322a"}
Sep 14 05:02:20.818530 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:20.818433    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"
Sep 14 05:02:20.818690 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:20.818606    2565 patch_prober.go:28] interesting pod/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 14 05:02:20.818690 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:20.818648    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9" podUID="38ea6dfe-3dce-404e-988c-8b3d8f912061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 14 05:02:21.819151 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:21.819110    2565 patch_prober.go:28] interesting pod/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 14 05:02:21.819605 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:21.819172    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9" podUID="38ea6dfe-3dce-404e-988c-8b3d8f912061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 14 05:02:22.818955 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:22.818912    2565 patch_prober.go:28] interesting pod/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 14 05:02:22.819156 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:22.818975    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9" podUID="38ea6dfe-3dce-404e-988c-8b3d8f912061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 14 05:02:23.818495 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:23.818454    2565 patch_prober.go:28] interesting pod/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 14 05:02:23.818954 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:23.818510    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9" podUID="38ea6dfe-3dce-404e-988c-8b3d8f912061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 14 05:02:24.818539 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:24.818501    2565 patch_prober.go:28] interesting pod/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 14 05:02:24.818998 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:24.818555    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9" podUID="38ea6dfe-3dce-404e-988c-8b3d8f912061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 14 05:02:25.398811 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:25.398774    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-60e333a9-575c6488c8-2h2wc"
Sep 14 05:02:25.399019 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:25.398961    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-60e333a9-79654cfdfd-mfmjv"
Sep 14 05:02:25.822941 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:25.822867    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"
Sep 14 05:02:25.823307 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:25.823218    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"
Sep 14 05:02:25.823819 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:25.823802    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"
Sep 14 05:02:36.308180 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:36.308136    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60e333a9-mvwvt"]
Sep 14 05:02:36.316308 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:36.316283    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60e333a9-mvwvt"
Sep 14 05:02:36.318594 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:36.318572    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k6hpm\""
Sep 14 05:02:36.318767 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:36.318572    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 05:02:36.318864 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:36.318846    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60e333a9-mvwvt"]
Sep 14 05:02:36.396585 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:36.396553    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74a4080d-109b-41ca-9fc3-095719691224-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-60e333a9-mvwvt\" (UID: \"74a4080d-109b-41ca-9fc3-095719691224\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60e333a9-mvwvt"
Sep 14 05:02:36.497899 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:36.497869    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74a4080d-109b-41ca-9fc3-095719691224-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-60e333a9-mvwvt\" (UID: \"74a4080d-109b-41ca-9fc3-095719691224\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60e333a9-mvwvt"
Sep 14 05:02:36.498462 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:36.498441    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74a4080d-109b-41ca-9fc3-095719691224-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-60e333a9-mvwvt\" (UID: \"74a4080d-109b-41ca-9fc3-095719691224\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60e333a9-mvwvt"
Sep 14 05:02:36.628366 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:36.628341    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60e333a9-mvwvt"
Sep 14 05:02:36.896829 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:36.896804    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60e333a9-mvwvt"]
Sep 14 05:02:36.898672 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:02:36.898646    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74a4080d_109b_41ca_9fc3_095719691224.slice/crio-634f90c8885a792671c587991f3d16bc9946424afbdfd00231ddd18c10e6e838 WatchSource:0}: Error finding container 634f90c8885a792671c587991f3d16bc9946424afbdfd00231ddd18c10e6e838: Status 404 returned error can't find the container with id 634f90c8885a792671c587991f3d16bc9946424afbdfd00231ddd18c10e6e838
Sep 14 05:02:37.548050 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:37.548022    2565 generic.go:356] "Generic (PLEG): container finished" podID="74a4080d-109b-41ca-9fc3-095719691224" containerID="5b17bb081a98e6baee6565187be7daaf2f39fdd56a141ca0b071ebb78f134e07" exitCode=0
Sep 14 05:02:37.548386 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:37.548103    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60e333a9-mvwvt" event={"ID":"74a4080d-109b-41ca-9fc3-095719691224","Type":"ContainerDied","Data":"5b17bb081a98e6baee6565187be7daaf2f39fdd56a141ca0b071ebb78f134e07"}
Sep 14 05:02:37.548386 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:37.548173    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60e333a9-mvwvt" event={"ID":"74a4080d-109b-41ca-9fc3-095719691224","Type":"ContainerStarted","Data":"634f90c8885a792671c587991f3d16bc9946424afbdfd00231ddd18c10e6e838"}
Sep 14 05:02:38.691329 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:38.691309    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60e333a9-mvwvt"
Sep 14 05:02:38.817337 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:38.817269    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74a4080d-109b-41ca-9fc3-095719691224-maas-api-service-ca\") pod \"74a4080d-109b-41ca-9fc3-095719691224\" (UID: \"74a4080d-109b-41ca-9fc3-095719691224\") "
Sep 14 05:02:38.817601 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:38.817580    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74a4080d-109b-41ca-9fc3-095719691224-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "74a4080d-109b-41ca-9fc3-095719691224" (UID: "74a4080d-109b-41ca-9fc3-095719691224"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 05:02:38.917837 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:38.917810    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74a4080d-109b-41ca-9fc3-095719691224-maas-api-service-ca\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:02:39.560108 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:39.560080    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60e333a9-mvwvt"
Sep 14 05:02:39.560283 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:39.560115    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60e333a9-mvwvt" event={"ID":"74a4080d-109b-41ca-9fc3-095719691224","Type":"ContainerDied","Data":"634f90c8885a792671c587991f3d16bc9946424afbdfd00231ddd18c10e6e838"}
Sep 14 05:02:39.560283 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:39.560162    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="634f90c8885a792671c587991f3d16bc9946424afbdfd00231ddd18c10e6e838"
Sep 14 05:02:43.264395 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:43.264359    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-60e333a9-575c6488c8-2h2wc"]
Sep 14 05:02:43.264877 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:43.264569    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-60e333a9-575c6488c8-2h2wc" podUID="57313b1d-6d32-4378-971b-4b830d0ed8bd" containerName="payload-processing" containerID="cri-o://5149d5de3e175ad159d08767cda29f7b4a74efd333c1edf4c583f5e48108125e" gracePeriod=30
Sep 14 05:02:43.276645 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:43.276616    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-60e333a9-79654cfdfd-mfmjv"]
Sep 14 05:02:43.276877 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:43.276853    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-60e333a9-79654cfdfd-mfmjv" podUID="050a107d-6c44-4628-9a6d-1d593d38b7dd" containerName="payload-pre-processing" containerID="cri-o://7ab01af8e2bb78b97aa29e766155f5085647dea433c7796558f47349a6479fca" gracePeriod=30
Sep 14 05:02:43.427385 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:43.427347    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60e333a9-mvwvt"]
Sep 14 05:02:43.431126 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:43.431099    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60e333a9-mvwvt"]
Sep 14 05:02:43.530868 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:43.530801    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-60e333a9-575c6488c8-2h2wc"
Sep 14 05:02:43.562945 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:43.562921    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-60e333a9-79654cfdfd-mfmjv"
Sep 14 05:02:43.584477 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:43.584448    2565 generic.go:356] "Generic (PLEG): container finished" podID="050a107d-6c44-4628-9a6d-1d593d38b7dd" containerID="7ab01af8e2bb78b97aa29e766155f5085647dea433c7796558f47349a6479fca" exitCode=0
Sep 14 05:02:43.584626 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:43.584498    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-60e333a9-79654cfdfd-mfmjv"
Sep 14 05:02:43.584626 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:43.584548    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-60e333a9-79654cfdfd-mfmjv" event={"ID":"050a107d-6c44-4628-9a6d-1d593d38b7dd","Type":"ContainerDied","Data":"7ab01af8e2bb78b97aa29e766155f5085647dea433c7796558f47349a6479fca"}
Sep 14 05:02:43.584626 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:43.584622    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-60e333a9-79654cfdfd-mfmjv" event={"ID":"050a107d-6c44-4628-9a6d-1d593d38b7dd","Type":"ContainerDied","Data":"84c7c2bdcac1a7aac85e1c992ad1708d507c37af2a676fa6d377ea42108bfb79"}
Sep 14 05:02:43.584822 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:43.584670    2565 scope.go:117] "RemoveContainer" containerID="7ab01af8e2bb78b97aa29e766155f5085647dea433c7796558f47349a6479fca"
Sep 14 05:02:43.586238 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:43.586212    2565 generic.go:356] "Generic (PLEG): container finished" podID="57313b1d-6d32-4378-971b-4b830d0ed8bd" containerID="5149d5de3e175ad159d08767cda29f7b4a74efd333c1edf4c583f5e48108125e" exitCode=0
Sep 14 05:02:43.586343 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:43.586261    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-60e333a9-575c6488c8-2h2wc"
Sep 14 05:02:43.586343 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:43.586289    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-60e333a9-575c6488c8-2h2wc" event={"ID":"57313b1d-6d32-4378-971b-4b830d0ed8bd","Type":"ContainerDied","Data":"5149d5de3e175ad159d08767cda29f7b4a74efd333c1edf4c583f5e48108125e"}
Sep 14 05:02:43.586343 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:43.586328    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-60e333a9-575c6488c8-2h2wc" event={"ID":"57313b1d-6d32-4378-971b-4b830d0ed8bd","Type":"ContainerDied","Data":"e0f59fba44fedd20880e2b4def1ce3f8fe6fb1d70ac9a903e0d3220e0152cafa"}
Sep 14 05:02:43.593531 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:43.593505    2565 scope.go:117] "RemoveContainer" containerID="7ab01af8e2bb78b97aa29e766155f5085647dea433c7796558f47349a6479fca"
Sep 14 05:02:43.593823 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:02:43.593801    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ab01af8e2bb78b97aa29e766155f5085647dea433c7796558f47349a6479fca\": container with ID starting with 7ab01af8e2bb78b97aa29e766155f5085647dea433c7796558f47349a6479fca not found: ID does not exist" containerID="7ab01af8e2bb78b97aa29e766155f5085647dea433c7796558f47349a6479fca"
Sep 14 05:02:43.593874 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:43.593838    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ab01af8e2bb78b97aa29e766155f5085647dea433c7796558f47349a6479fca"} err="failed to get container status \"7ab01af8e2bb78b97aa29e766155f5085647dea433c7796558f47349a6479fca\": rpc error: code = NotFound desc = could not find container \"7ab01af8e2bb78b97aa29e766155f5085647dea433c7796558f47349a6479fca\": container with ID starting with 7ab01af8e2bb78b97aa29e766155f5085647dea433c7796558f47349a6479fca not found: ID does not exist"
Sep 14 05:02:43.593874 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:43.593861    2565 scope.go:117] "RemoveContainer" containerID="5149d5de3e175ad159d08767cda29f7b4a74efd333c1edf4c583f5e48108125e"
Sep 14 05:02:43.602289 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:43.602274    2565 scope.go:117] "RemoveContainer" containerID="5149d5de3e175ad159d08767cda29f7b4a74efd333c1edf4c583f5e48108125e"
Sep 14 05:02:43.604590 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:02:43.604570    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5149d5de3e175ad159d08767cda29f7b4a74efd333c1edf4c583f5e48108125e\": container with ID starting with 5149d5de3e175ad159d08767cda29f7b4a74efd333c1edf4c583f5e48108125e not found: ID does not exist" containerID="5149d5de3e175ad159d08767cda29f7b4a74efd333c1edf4c583f5e48108125e"
Sep 14 05:02:43.604671 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:43.604602    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5149d5de3e175ad159d08767cda29f7b4a74efd333c1edf4c583f5e48108125e"} err="failed to get container status \"5149d5de3e175ad159d08767cda29f7b4a74efd333c1edf4c583f5e48108125e\": rpc error: code = NotFound desc = could not find container \"5149d5de3e175ad159d08767cda29f7b4a74efd333c1edf4c583f5e48108125e\": container with ID starting with 5149d5de3e175ad159d08767cda29f7b4a74efd333c1edf4c583f5e48108125e not found: ID does not exist"
Sep 14 05:02:43.660931 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:43.660904    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/050a107d-6c44-4628-9a6d-1d593d38b7dd-plugins-config-volume\") pod \"050a107d-6c44-4628-9a6d-1d593d38b7dd\" (UID: \"050a107d-6c44-4628-9a6d-1d593d38b7dd\") "
Sep 14 05:02:43.661062 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:43.660948    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfgdn\" (UniqueName: \"kubernetes.io/projected/050a107d-6c44-4628-9a6d-1d593d38b7dd-kube-api-access-lfgdn\") pod \"050a107d-6c44-4628-9a6d-1d593d38b7dd\" (UID: \"050a107d-6c44-4628-9a6d-1d593d38b7dd\") "
Sep 14 05:02:43.661062 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:43.660970    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57313b1d-6d32-4378-971b-4b830d0ed8bd-plugins-config-volume\") pod \"57313b1d-6d32-4378-971b-4b830d0ed8bd\" (UID: \"57313b1d-6d32-4378-971b-4b830d0ed8bd\") "
Sep 14 05:02:43.661062 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:43.661059    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8nq4s\" (UniqueName: \"kubernetes.io/projected/57313b1d-6d32-4378-971b-4b830d0ed8bd-kube-api-access-8nq4s\") pod \"57313b1d-6d32-4378-971b-4b830d0ed8bd\" (UID: \"57313b1d-6d32-4378-971b-4b830d0ed8bd\") "
Sep 14 05:02:43.661315 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:43.661293    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/050a107d-6c44-4628-9a6d-1d593d38b7dd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "050a107d-6c44-4628-9a6d-1d593d38b7dd" (UID: "050a107d-6c44-4628-9a6d-1d593d38b7dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 05:02:43.661388 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:43.661358    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57313b1d-6d32-4378-971b-4b830d0ed8bd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "57313b1d-6d32-4378-971b-4b830d0ed8bd" (UID: "57313b1d-6d32-4378-971b-4b830d0ed8bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 05:02:43.663095 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:43.663070    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/050a107d-6c44-4628-9a6d-1d593d38b7dd-kube-api-access-lfgdn" (OuterVolumeSpecName: "kube-api-access-lfgdn") pod "050a107d-6c44-4628-9a6d-1d593d38b7dd" (UID: "050a107d-6c44-4628-9a6d-1d593d38b7dd"). InnerVolumeSpecName "kube-api-access-lfgdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:02:43.663318 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:43.663293    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57313b1d-6d32-4378-971b-4b830d0ed8bd-kube-api-access-8nq4s" (OuterVolumeSpecName: "kube-api-access-8nq4s") pod "57313b1d-6d32-4378-971b-4b830d0ed8bd" (UID: "57313b1d-6d32-4378-971b-4b830d0ed8bd"). InnerVolumeSpecName "kube-api-access-8nq4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:02:43.762151 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:43.762121    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfgdn\" (UniqueName: \"kubernetes.io/projected/050a107d-6c44-4628-9a6d-1d593d38b7dd-kube-api-access-lfgdn\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:02:43.762151 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:43.762147    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57313b1d-6d32-4378-971b-4b830d0ed8bd-plugins-config-volume\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:02:43.762151 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:43.762157    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nq4s\" (UniqueName: \"kubernetes.io/projected/57313b1d-6d32-4378-971b-4b830d0ed8bd-kube-api-access-8nq4s\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:02:43.762379 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:43.762166    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/050a107d-6c44-4628-9a6d-1d593d38b7dd-plugins-config-volume\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:02:43.912117 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:43.912081    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-60e333a9-79654cfdfd-mfmjv"]
Sep 14 05:02:43.916972 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:43.916946    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-60e333a9-79654cfdfd-mfmjv"]
Sep 14 05:02:43.925606 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:43.925580    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-60e333a9-575c6488c8-2h2wc"]
Sep 14 05:02:43.929214 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:43.929192    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-60e333a9-575c6488c8-2h2wc"]
Sep 14 05:02:44.319711 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.319681    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"]
Sep 14 05:02:44.320081 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.320066    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="050a107d-6c44-4628-9a6d-1d593d38b7dd" containerName="payload-pre-processing"
Sep 14 05:02:44.320123 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.320082    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="050a107d-6c44-4628-9a6d-1d593d38b7dd" containerName="payload-pre-processing"
Sep 14 05:02:44.320123 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.320099    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74a4080d-109b-41ca-9fc3-095719691224" containerName="revoke-keys"
Sep 14 05:02:44.320123 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.320105    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="74a4080d-109b-41ca-9fc3-095719691224" containerName="revoke-keys"
Sep 14 05:02:44.320123 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.320113    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57313b1d-6d32-4378-971b-4b830d0ed8bd" containerName="payload-processing"
Sep 14 05:02:44.320123 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.320118    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="57313b1d-6d32-4378-971b-4b830d0ed8bd" containerName="payload-processing"
Sep 14 05:02:44.320282 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.320181    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="74a4080d-109b-41ca-9fc3-095719691224" containerName="revoke-keys"
Sep 14 05:02:44.320282 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.320189    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="57313b1d-6d32-4378-971b-4b830d0ed8bd" containerName="payload-processing"
Sep 14 05:02:44.320282 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.320198    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="050a107d-6c44-4628-9a6d-1d593d38b7dd" containerName="payload-pre-processing"
Sep 14 05:02:44.323191 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.323174    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"
Sep 14 05:02:44.325433 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.325414    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8f579f00-openshift-default-dockercfg-sbkn8\""
Sep 14 05:02:44.336665 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.336640    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="050a107d-6c44-4628-9a6d-1d593d38b7dd" path="/var/lib/kubelet/pods/050a107d-6c44-4628-9a6d-1d593d38b7dd/volumes"
Sep 14 05:02:44.337106 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.337082    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57313b1d-6d32-4378-971b-4b830d0ed8bd" path="/var/lib/kubelet/pods/57313b1d-6d32-4378-971b-4b830d0ed8bd/volumes"
Sep 14 05:02:44.337613 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.337517    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74a4080d-109b-41ca-9fc3-095719691224" path="/var/lib/kubelet/pods/74a4080d-109b-41ca-9fc3-095719691224/volumes"
Sep 14 05:02:44.337867 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.337851    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"]
Sep 14 05:02:44.467899 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.467867    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d8f7ab-d2ab-4112-8498-07b238c0b047-credential-socket\") pod \"e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l\" (UID: \"81d8f7ab-d2ab-4112-8498-07b238c0b047\") " pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"
Sep 14 05:02:44.468073 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.467909    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81d8f7ab-d2ab-4112-8498-07b238c0b047-workload-certs\") pod \"e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l\" (UID: \"81d8f7ab-d2ab-4112-8498-07b238c0b047\") " pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"
Sep 14 05:02:44.468073 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.467954    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81d8f7ab-d2ab-4112-8498-07b238c0b047-istio-envoy\") pod \"e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l\" (UID: \"81d8f7ab-d2ab-4112-8498-07b238c0b047\") " pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"
Sep 14 05:02:44.468073 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.467984    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81d8f7ab-d2ab-4112-8498-07b238c0b047-istio-data\") pod \"e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l\" (UID: \"81d8f7ab-d2ab-4112-8498-07b238c0b047\") " pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"
Sep 14 05:02:44.468073 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.468000    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d8f7ab-d2ab-4112-8498-07b238c0b047-workload-socket\") pod \"e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l\" (UID: \"81d8f7ab-d2ab-4112-8498-07b238c0b047\") " pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"
Sep 14 05:02:44.468073 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.468017    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81d8f7ab-d2ab-4112-8498-07b238c0b047-istio-podinfo\") pod \"e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l\" (UID: \"81d8f7ab-d2ab-4112-8498-07b238c0b047\") " pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"
Sep 14 05:02:44.468073 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.468042    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81d8f7ab-d2ab-4112-8498-07b238c0b047-istio-token\") pod \"e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l\" (UID: \"81d8f7ab-d2ab-4112-8498-07b238c0b047\") " pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"
Sep 14 05:02:44.468073 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.468069    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81d8f7ab-d2ab-4112-8498-07b238c0b047-istiod-ca-cert\") pod \"e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l\" (UID: \"81d8f7ab-d2ab-4112-8498-07b238c0b047\") " pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"
Sep 14 05:02:44.468351 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.468107    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gh2x\" (UniqueName: \"kubernetes.io/projected/81d8f7ab-d2ab-4112-8498-07b238c0b047-kube-api-access-2gh2x\") pod \"e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l\" (UID: \"81d8f7ab-d2ab-4112-8498-07b238c0b047\") " pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"
Sep 14 05:02:44.568735 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.568641    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81d8f7ab-d2ab-4112-8498-07b238c0b047-istio-envoy\") pod \"e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l\" (UID: \"81d8f7ab-d2ab-4112-8498-07b238c0b047\") " pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"
Sep 14 05:02:44.568735 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.568705    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81d8f7ab-d2ab-4112-8498-07b238c0b047-istio-data\") pod \"e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l\" (UID: \"81d8f7ab-d2ab-4112-8498-07b238c0b047\") " pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"
Sep 14 05:02:44.568735 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.568733    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d8f7ab-d2ab-4112-8498-07b238c0b047-workload-socket\") pod \"e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l\" (UID: \"81d8f7ab-d2ab-4112-8498-07b238c0b047\") " pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"
Sep 14 05:02:44.568998 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.568783    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81d8f7ab-d2ab-4112-8498-07b238c0b047-istio-podinfo\") pod \"e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l\" (UID: \"81d8f7ab-d2ab-4112-8498-07b238c0b047\") " pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"
Sep 14 05:02:44.568998 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.568808    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81d8f7ab-d2ab-4112-8498-07b238c0b047-istio-token\") pod \"e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l\" (UID: \"81d8f7ab-d2ab-4112-8498-07b238c0b047\") " pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"
Sep 14 05:02:44.568998 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.568834    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81d8f7ab-d2ab-4112-8498-07b238c0b047-istiod-ca-cert\") pod \"e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l\" (UID: \"81d8f7ab-d2ab-4112-8498-07b238c0b047\") " pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"
Sep 14 05:02:44.568998 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.568884    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gh2x\" (UniqueName: \"kubernetes.io/projected/81d8f7ab-d2ab-4112-8498-07b238c0b047-kube-api-access-2gh2x\") pod \"e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l\" (UID: \"81d8f7ab-d2ab-4112-8498-07b238c0b047\") " pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"
Sep 14 05:02:44.568998 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.568952    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d8f7ab-d2ab-4112-8498-07b238c0b047-credential-socket\") pod \"e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l\" (UID: \"81d8f7ab-d2ab-4112-8498-07b238c0b047\") " pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"
Sep 14 05:02:44.568998 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.568989    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81d8f7ab-d2ab-4112-8498-07b238c0b047-workload-certs\") pod \"e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l\" (UID: \"81d8f7ab-d2ab-4112-8498-07b238c0b047\") " pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"
Sep 14 05:02:44.569296 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.569098    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81d8f7ab-d2ab-4112-8498-07b238c0b047-istio-data\") pod \"e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l\" (UID: \"81d8f7ab-d2ab-4112-8498-07b238c0b047\") " pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"
Sep 14 05:02:44.569386 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.569363    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d8f7ab-d2ab-4112-8498-07b238c0b047-credential-socket\") pod \"e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l\" (UID: \"81d8f7ab-d2ab-4112-8498-07b238c0b047\") " pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"
Sep 14 05:02:44.569468 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.569429    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d8f7ab-d2ab-4112-8498-07b238c0b047-workload-socket\") pod \"e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l\" (UID: \"81d8f7ab-d2ab-4112-8498-07b238c0b047\") " pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"
Sep 14 05:02:44.569790 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.569769    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81d8f7ab-d2ab-4112-8498-07b238c0b047-istiod-ca-cert\") pod \"e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l\" (UID: \"81d8f7ab-d2ab-4112-8498-07b238c0b047\") " pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"
Sep 14 05:02:44.570059 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.570038    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81d8f7ab-d2ab-4112-8498-07b238c0b047-workload-certs\") pod \"e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l\" (UID: \"81d8f7ab-d2ab-4112-8498-07b238c0b047\") " pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"
Sep 14 05:02:44.571233 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.571212    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81d8f7ab-d2ab-4112-8498-07b238c0b047-istio-envoy\") pod \"e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l\" (UID: \"81d8f7ab-d2ab-4112-8498-07b238c0b047\") " pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"
Sep 14 05:02:44.571467 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.571449    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81d8f7ab-d2ab-4112-8498-07b238c0b047-istio-podinfo\") pod \"e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l\" (UID: \"81d8f7ab-d2ab-4112-8498-07b238c0b047\") " pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"
Sep 14 05:02:44.576170 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.576141    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81d8f7ab-d2ab-4112-8498-07b238c0b047-istio-token\") pod \"e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l\" (UID: \"81d8f7ab-d2ab-4112-8498-07b238c0b047\") " pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"
Sep 14 05:02:44.576328 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.576311    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gh2x\" (UniqueName: \"kubernetes.io/projected/81d8f7ab-d2ab-4112-8498-07b238c0b047-kube-api-access-2gh2x\") pod \"e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l\" (UID: \"81d8f7ab-d2ab-4112-8498-07b238c0b047\") " pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"
Sep 14 05:02:44.637279 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.637252    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"
Sep 14 05:02:44.913272 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.913245    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"]
Sep 14 05:02:44.914412 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:02:44.914387    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81d8f7ab_d2ab_4112_8498_07b238c0b047.slice/crio-52d81ca5240fcaa8f3197be5f3cb653798ed024d972c2972c53f610f3df3f2c2 WatchSource:0}: Error finding container 52d81ca5240fcaa8f3197be5f3cb653798ed024d972c2972c53f610f3df3f2c2: Status 404 returned error can't find the container with id 52d81ca5240fcaa8f3197be5f3cb653798ed024d972c2972c53f610f3df3f2c2
Sep 14 05:02:44.916646 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.916616    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:02:44.916694 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.916681    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:02:44.916729 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:44.916712    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:02:45.601254 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:45.601216    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l" event={"ID":"81d8f7ab-d2ab-4112-8498-07b238c0b047","Type":"ContainerStarted","Data":"cd809b355f20109f6fbeafd704ca5c4685201c6175bfc9785de6b2ff0a36e96a"}
Sep 14 05:02:45.601636 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:45.601276    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l" event={"ID":"81d8f7ab-d2ab-4112-8498-07b238c0b047","Type":"ContainerStarted","Data":"52d81ca5240fcaa8f3197be5f3cb653798ed024d972c2972c53f610f3df3f2c2"}
Sep 14 05:02:45.622600 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:45.622550    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l" podStartSLOduration=1.62253526 podStartE2EDuration="1.62253526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:02:45.620441968 +0000 UTC m=+963.796213528" watchObservedRunningTime="2026-09-14 05:02:45.62253526 +0000 UTC m=+963.798306819"
Sep 14 05:02:45.638181 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:45.638155    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"
Sep 14 05:02:45.643461 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:45.643438    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"
Sep 14 05:02:46.606839 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:46.606806    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"
Sep 14 05:02:46.607895 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:46.607875    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"
Sep 14 05:02:49.555014 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:49.554980    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8f579f00-68c4db85cc-6z8zf"]
Sep 14 05:02:49.558562 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:49.558540    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8f579f00-68c4db85cc-6z8zf"
Sep 14 05:02:49.560905 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:49.560876    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8f579f00\""
Sep 14 05:02:49.560992 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:49.560915    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8f579f00-dockercfg-mklc2\""
Sep 14 05:02:49.569476 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:49.569452    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8f579f00-68c4db85cc-6z8zf"]
Sep 14 05:02:49.606492 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:49.606463    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdnfq\" (UniqueName: \"kubernetes.io/projected/0c1539a4-e686-404e-a81e-9ea85366b64a-kube-api-access-rdnfq\") pod \"payload-processing-e2e-ait-8f579f00-68c4db85cc-6z8zf\" (UID: \"0c1539a4-e686-404e-a81e-9ea85366b64a\") " pod="openshift-ingress/payload-processing-e2e-ait-8f579f00-68c4db85cc-6z8zf"
Sep 14 05:02:49.606609 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:49.606538    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c1539a4-e686-404e-a81e-9ea85366b64a-plugins-config-volume\") pod \"payload-processing-e2e-ait-8f579f00-68c4db85cc-6z8zf\" (UID: \"0c1539a4-e686-404e-a81e-9ea85366b64a\") " pod="openshift-ingress/payload-processing-e2e-ait-8f579f00-68c4db85cc-6z8zf"
Sep 14 05:02:49.707069 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:49.707035    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c1539a4-e686-404e-a81e-9ea85366b64a-plugins-config-volume\") pod \"payload-processing-e2e-ait-8f579f00-68c4db85cc-6z8zf\" (UID: \"0c1539a4-e686-404e-a81e-9ea85366b64a\") " pod="openshift-ingress/payload-processing-e2e-ait-8f579f00-68c4db85cc-6z8zf"
Sep 14 05:02:49.707222 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:49.707119    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdnfq\" (UniqueName: \"kubernetes.io/projected/0c1539a4-e686-404e-a81e-9ea85366b64a-kube-api-access-rdnfq\") pod \"payload-processing-e2e-ait-8f579f00-68c4db85cc-6z8zf\" (UID: \"0c1539a4-e686-404e-a81e-9ea85366b64a\") " pod="openshift-ingress/payload-processing-e2e-ait-8f579f00-68c4db85cc-6z8zf"
Sep 14 05:02:49.707705 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:49.707686    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c1539a4-e686-404e-a81e-9ea85366b64a-plugins-config-volume\") pod \"payload-processing-e2e-ait-8f579f00-68c4db85cc-6z8zf\" (UID: \"0c1539a4-e686-404e-a81e-9ea85366b64a\") " pod="openshift-ingress/payload-processing-e2e-ait-8f579f00-68c4db85cc-6z8zf"
Sep 14 05:02:49.715160 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:49.715127    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdnfq\" (UniqueName: \"kubernetes.io/projected/0c1539a4-e686-404e-a81e-9ea85366b64a-kube-api-access-rdnfq\") pod \"payload-processing-e2e-ait-8f579f00-68c4db85cc-6z8zf\" (UID: \"0c1539a4-e686-404e-a81e-9ea85366b64a\") " pod="openshift-ingress/payload-processing-e2e-ait-8f579f00-68c4db85cc-6z8zf"
Sep 14 05:02:49.731629 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:49.731606    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:02:49.869149 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:49.869121    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8f579f00-68c4db85cc-6z8zf"
Sep 14 05:02:50.140917 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:50.140893    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8f579f00-68c4db85cc-6z8zf"]
Sep 14 05:02:50.142706 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:02:50.142680    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c1539a4_e686_404e_a81e_9ea85366b64a.slice/crio-a7bae95ccdb1cb73d6214cce2dc2f9f64052fa3378d6b5d9f43a808372a9bbbe WatchSource:0}: Error finding container a7bae95ccdb1cb73d6214cce2dc2f9f64052fa3378d6b5d9f43a808372a9bbbe: Status 404 returned error can't find the container with id a7bae95ccdb1cb73d6214cce2dc2f9f64052fa3378d6b5d9f43a808372a9bbbe
Sep 14 05:02:50.637224 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:50.637184    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8f579f00-68c4db85cc-6z8zf" event={"ID":"0c1539a4-e686-404e-a81e-9ea85366b64a","Type":"ContainerStarted","Data":"7d5b9f09286e0228e6aad3e8bbecaf5ae2cbeff3f9e461a60b120e446c6ec28b"}
Sep 14 05:02:50.637649 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:50.637238    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8f579f00-68c4db85cc-6z8zf" event={"ID":"0c1539a4-e686-404e-a81e-9ea85366b64a","Type":"ContainerStarted","Data":"a7bae95ccdb1cb73d6214cce2dc2f9f64052fa3378d6b5d9f43a808372a9bbbe"}
Sep 14 05:02:50.637649 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:50.637258    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-8f579f00-68c4db85cc-6z8zf"
Sep 14 05:02:50.657987 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:02:50.657934    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-8f579f00-68c4db85cc-6z8zf" podStartSLOduration=1.657918706 podStartE2EDuration="1.657918706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:02:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:02:50.654896665 +0000 UTC m=+968.830668236" watchObservedRunningTime="2026-09-14 05:02:50.657918706 +0000 UTC m=+968.833690295"
Sep 14 05:03:01.645039 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:01.644980    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-8f579f00-68c4db85cc-6z8zf"
Sep 14 05:03:04.427728 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:04.427690    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:03:06.808056 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:06.808024    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-e08402/simulator-w4-e08402-kserve-5f5788cc6-cxtfd"]
Sep 14 05:03:06.812557 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:06.812541    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e08402/simulator-w4-e08402-kserve-5f5788cc6-cxtfd"
Sep 14 05:03:06.815969 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:06.815945    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-e08402\"/\"kube-root-ca.crt\""
Sep 14 05:03:06.816259 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:06.816234    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e08402\"/\"default-dockercfg-26l6j\""
Sep 14 05:03:06.816365 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:06.816319    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-e08402\"/\"openshift-service-ca.crt\""
Sep 14 05:03:06.816365 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:06.816334    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e08402\"/\"simulator-w4-e08402-kserve-self-signed-certs\""
Sep 14 05:03:06.822271 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:06.822249    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e08402/simulator-w4-e08402-kserve-5f5788cc6-cxtfd"]
Sep 14 05:03:06.862528 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:06.862496    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqrhl\" (UniqueName: \"kubernetes.io/projected/d925517d-719f-41bb-ae99-09e538af6104-kube-api-access-nqrhl\") pod \"simulator-w4-e08402-kserve-5f5788cc6-cxtfd\" (UID: \"d925517d-719f-41bb-ae99-09e538af6104\") " pod="e2e-models-e2e-worker-w4-e08402/simulator-w4-e08402-kserve-5f5788cc6-cxtfd"
Sep 14 05:03:06.862528 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:06.862531    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d925517d-719f-41bb-ae99-09e538af6104-home\") pod \"simulator-w4-e08402-kserve-5f5788cc6-cxtfd\" (UID: \"d925517d-719f-41bb-ae99-09e538af6104\") " pod="e2e-models-e2e-worker-w4-e08402/simulator-w4-e08402-kserve-5f5788cc6-cxtfd"
Sep 14 05:03:06.862709 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:06.862630    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d925517d-719f-41bb-ae99-09e538af6104-tmp-dir\") pod \"simulator-w4-e08402-kserve-5f5788cc6-cxtfd\" (UID: \"d925517d-719f-41bb-ae99-09e538af6104\") " pod="e2e-models-e2e-worker-w4-e08402/simulator-w4-e08402-kserve-5f5788cc6-cxtfd"
Sep 14 05:03:06.862709 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:06.862668    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d925517d-719f-41bb-ae99-09e538af6104-dshm\") pod \"simulator-w4-e08402-kserve-5f5788cc6-cxtfd\" (UID: \"d925517d-719f-41bb-ae99-09e538af6104\") " pod="e2e-models-e2e-worker-w4-e08402/simulator-w4-e08402-kserve-5f5788cc6-cxtfd"
Sep 14 05:03:06.862709 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:06.862688    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d925517d-719f-41bb-ae99-09e538af6104-tls-certs\") pod \"simulator-w4-e08402-kserve-5f5788cc6-cxtfd\" (UID: \"d925517d-719f-41bb-ae99-09e538af6104\") " pod="e2e-models-e2e-worker-w4-e08402/simulator-w4-e08402-kserve-5f5788cc6-cxtfd"
Sep 14 05:03:06.862842 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:06.862709    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d925517d-719f-41bb-ae99-09e538af6104-model-cache\") pod \"simulator-w4-e08402-kserve-5f5788cc6-cxtfd\" (UID: \"d925517d-719f-41bb-ae99-09e538af6104\") " pod="e2e-models-e2e-worker-w4-e08402/simulator-w4-e08402-kserve-5f5788cc6-cxtfd"
Sep 14 05:03:06.963337 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:06.963306    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d925517d-719f-41bb-ae99-09e538af6104-tmp-dir\") pod \"simulator-w4-e08402-kserve-5f5788cc6-cxtfd\" (UID: \"d925517d-719f-41bb-ae99-09e538af6104\") " pod="e2e-models-e2e-worker-w4-e08402/simulator-w4-e08402-kserve-5f5788cc6-cxtfd"
Sep 14 05:03:06.963595 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:06.963346    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d925517d-719f-41bb-ae99-09e538af6104-dshm\") pod \"simulator-w4-e08402-kserve-5f5788cc6-cxtfd\" (UID: \"d925517d-719f-41bb-ae99-09e538af6104\") " pod="e2e-models-e2e-worker-w4-e08402/simulator-w4-e08402-kserve-5f5788cc6-cxtfd"
Sep 14 05:03:06.963595 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:06.963372    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d925517d-719f-41bb-ae99-09e538af6104-tls-certs\") pod \"simulator-w4-e08402-kserve-5f5788cc6-cxtfd\" (UID: \"d925517d-719f-41bb-ae99-09e538af6104\") " pod="e2e-models-e2e-worker-w4-e08402/simulator-w4-e08402-kserve-5f5788cc6-cxtfd"
Sep 14 05:03:06.963595 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:06.963402    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d925517d-719f-41bb-ae99-09e538af6104-model-cache\") pod \"simulator-w4-e08402-kserve-5f5788cc6-cxtfd\" (UID: \"d925517d-719f-41bb-ae99-09e538af6104\") " pod="e2e-models-e2e-worker-w4-e08402/simulator-w4-e08402-kserve-5f5788cc6-cxtfd"
Sep 14 05:03:06.963595 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:06.963454    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqrhl\" (UniqueName: \"kubernetes.io/projected/d925517d-719f-41bb-ae99-09e538af6104-kube-api-access-nqrhl\") pod \"simulator-w4-e08402-kserve-5f5788cc6-cxtfd\" (UID: \"d925517d-719f-41bb-ae99-09e538af6104\") " pod="e2e-models-e2e-worker-w4-e08402/simulator-w4-e08402-kserve-5f5788cc6-cxtfd"
Sep 14 05:03:06.963595 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:06.963482    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d925517d-719f-41bb-ae99-09e538af6104-home\") pod \"simulator-w4-e08402-kserve-5f5788cc6-cxtfd\" (UID: \"d925517d-719f-41bb-ae99-09e538af6104\") " pod="e2e-models-e2e-worker-w4-e08402/simulator-w4-e08402-kserve-5f5788cc6-cxtfd"
Sep 14 05:03:06.963911 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:06.963789    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d925517d-719f-41bb-ae99-09e538af6104-tmp-dir\") pod \"simulator-w4-e08402-kserve-5f5788cc6-cxtfd\" (UID: \"d925517d-719f-41bb-ae99-09e538af6104\") " pod="e2e-models-e2e-worker-w4-e08402/simulator-w4-e08402-kserve-5f5788cc6-cxtfd"
Sep 14 05:03:06.963977 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:06.963939    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d925517d-719f-41bb-ae99-09e538af6104-model-cache\") pod \"simulator-w4-e08402-kserve-5f5788cc6-cxtfd\" (UID: \"d925517d-719f-41bb-ae99-09e538af6104\") " pod="e2e-models-e2e-worker-w4-e08402/simulator-w4-e08402-kserve-5f5788cc6-cxtfd"
Sep 14 05:03:06.964028 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:06.963973    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d925517d-719f-41bb-ae99-09e538af6104-home\") pod \"simulator-w4-e08402-kserve-5f5788cc6-cxtfd\" (UID: \"d925517d-719f-41bb-ae99-09e538af6104\") " pod="e2e-models-e2e-worker-w4-e08402/simulator-w4-e08402-kserve-5f5788cc6-cxtfd"
Sep 14 05:03:06.965942 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:06.965922    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d925517d-719f-41bb-ae99-09e538af6104-tls-certs\") pod \"simulator-w4-e08402-kserve-5f5788cc6-cxtfd\" (UID: \"d925517d-719f-41bb-ae99-09e538af6104\") " pod="e2e-models-e2e-worker-w4-e08402/simulator-w4-e08402-kserve-5f5788cc6-cxtfd"
Sep 14 05:03:06.966086 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:06.966069    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d925517d-719f-41bb-ae99-09e538af6104-dshm\") pod \"simulator-w4-e08402-kserve-5f5788cc6-cxtfd\" (UID: \"d925517d-719f-41bb-ae99-09e538af6104\") " pod="e2e-models-e2e-worker-w4-e08402/simulator-w4-e08402-kserve-5f5788cc6-cxtfd"
Sep 14 05:03:06.973682 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:06.973662    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqrhl\" (UniqueName: \"kubernetes.io/projected/d925517d-719f-41bb-ae99-09e538af6104-kube-api-access-nqrhl\") pod \"simulator-w4-e08402-kserve-5f5788cc6-cxtfd\" (UID: \"d925517d-719f-41bb-ae99-09e538af6104\") " pod="e2e-models-e2e-worker-w4-e08402/simulator-w4-e08402-kserve-5f5788cc6-cxtfd"
Sep 14 05:03:07.123332 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:07.123294    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e08402/simulator-w4-e08402-kserve-5f5788cc6-cxtfd"
Sep 14 05:03:07.404346 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:07.404319    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e08402/simulator-w4-e08402-kserve-5f5788cc6-cxtfd"]
Sep 14 05:03:07.405868 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:03:07.405834    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd925517d_719f_41bb_ae99_09e538af6104.slice/crio-edce1b8c01c0d656afb2761e0e8fc62d77c90e611a7f904c0269ff45027cc20e WatchSource:0}: Error finding container edce1b8c01c0d656afb2761e0e8fc62d77c90e611a7f904c0269ff45027cc20e: Status 404 returned error can't find the container with id edce1b8c01c0d656afb2761e0e8fc62d77c90e611a7f904c0269ff45027cc20e
Sep 14 05:03:07.730206 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:07.730164    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e08402/simulator-w4-e08402-kserve-5f5788cc6-cxtfd" event={"ID":"d925517d-719f-41bb-ae99-09e538af6104","Type":"ContainerStarted","Data":"168b33a1b881ffafadb62f50332bbf0975d67d698732ff74b81e43ddc8ed7091"}
Sep 14 05:03:07.730206 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:07.730219    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e08402/simulator-w4-e08402-kserve-5f5788cc6-cxtfd" event={"ID":"d925517d-719f-41bb-ae99-09e538af6104","Type":"ContainerStarted","Data":"edce1b8c01c0d656afb2761e0e8fc62d77c90e611a7f904c0269ff45027cc20e"}
Sep 14 05:03:07.747837 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:07.747787    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-e08402/simulator-w4-e08402-kserve-5f5788cc6-cxtfd" podStartSLOduration=1.747770781 podStartE2EDuration="1.747770781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:03:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:03:07.746027975 +0000 UTC m=+985.921799533" watchObservedRunningTime="2026-09-14 05:03:07.747770781 +0000 UTC m=+985.923542331"
Sep 14 05:03:08.736600 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:08.736564    2565 generic.go:356] "Generic (PLEG): container finished" podID="81d8f7ab-d2ab-4112-8498-07b238c0b047" containerID="cd809b355f20109f6fbeafd704ca5c4685201c6175bfc9785de6b2ff0a36e96a" exitCode=0
Sep 14 05:03:08.737069 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:08.736633    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l" event={"ID":"81d8f7ab-d2ab-4112-8498-07b238c0b047","Type":"ContainerDied","Data":"cd809b355f20109f6fbeafd704ca5c4685201c6175bfc9785de6b2ff0a36e96a"}
Sep 14 05:03:08.737069 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:08.736988    2565 scope.go:117] "RemoveContainer" containerID="cd809b355f20109f6fbeafd704ca5c4685201c6175bfc9785de6b2ff0a36e96a"
Sep 14 05:03:08.738010 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:08.737982    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:03:08.738107 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:08.738043    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:03:08.738107 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:08.738076    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:03:09.749483 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:09.749448    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l" event={"ID":"81d8f7ab-d2ab-4112-8498-07b238c0b047","Type":"ContainerStarted","Data":"98d1931301335684853e8ee930546d27b6ef0b23d13b868f207dcf0327f57cab"}
Sep 14 05:03:10.637586 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:10.637552    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"
Sep 14 05:03:10.637826 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:10.637798    2565 patch_prober.go:28] interesting pod/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 14 05:03:10.638057 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:10.638029    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l" podUID="81d8f7ab-d2ab-4112-8498-07b238c0b047" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 14 05:03:11.638561 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:11.638521    2565 patch_prober.go:28] interesting pod/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 14 05:03:11.639351 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:11.639320    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l" podUID="81d8f7ab-d2ab-4112-8498-07b238c0b047" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 14 05:03:12.638613 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:12.638579    2565 patch_prober.go:28] interesting pod/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 14 05:03:12.639217 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:12.639187    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l" podUID="81d8f7ab-d2ab-4112-8498-07b238c0b047" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 14 05:03:13.638091 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:13.638046    2565 patch_prober.go:28] interesting pod/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 14 05:03:13.638292 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:13.638117    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l" podUID="81d8f7ab-d2ab-4112-8498-07b238c0b047" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 14 05:03:14.638299 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:14.638264    2565 patch_prober.go:28] interesting pod/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 14 05:03:14.638672 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:14.638319    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l" podUID="81d8f7ab-d2ab-4112-8498-07b238c0b047" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 14 05:03:15.134918 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:15.134883    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:03:15.642542 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:15.642508    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"
Sep 14 05:03:15.643034 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:15.642879    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"
Sep 14 05:03:15.643480 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:15.643459    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"
Sep 14 05:03:17.123630 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:17.123592    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e08402/simulator-w4-e08402-kserve-5f5788cc6-cxtfd"
Sep 14 05:03:17.124091 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:17.123651    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-e08402/simulator-w4-e08402-kserve-5f5788cc6-cxtfd"
Sep 14 05:03:17.136309 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:17.136285    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-e08402/simulator-w4-e08402-kserve-5f5788cc6-cxtfd"
Sep 14 05:03:17.809985 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:17.809956    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e08402/simulator-w4-e08402-kserve-5f5788cc6-cxtfd"
Sep 14 05:03:29.054116 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:29.054076    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8f579f00-68c4db85cc-6z8zf"]
Sep 14 05:03:29.054801 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:29.054726    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-8f579f00-68c4db85cc-6z8zf" podUID="0c1539a4-e686-404e-a81e-9ea85366b64a" containerName="payload-processing" containerID="cri-o://7d5b9f09286e0228e6aad3e8bbecaf5ae2cbeff3f9e461a60b120e446c6ec28b" gracePeriod=30
Sep 14 05:03:29.312364 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:29.312297    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8f579f00-68c4db85cc-6z8zf"
Sep 14 05:03:29.469733 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:29.469688    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c1539a4-e686-404e-a81e-9ea85366b64a-plugins-config-volume\") pod \"0c1539a4-e686-404e-a81e-9ea85366b64a\" (UID: \"0c1539a4-e686-404e-a81e-9ea85366b64a\") "
Sep 14 05:03:29.469924 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:29.469818    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdnfq\" (UniqueName: \"kubernetes.io/projected/0c1539a4-e686-404e-a81e-9ea85366b64a-kube-api-access-rdnfq\") pod \"0c1539a4-e686-404e-a81e-9ea85366b64a\" (UID: \"0c1539a4-e686-404e-a81e-9ea85366b64a\") "
Sep 14 05:03:29.470070 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:29.470047    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c1539a4-e686-404e-a81e-9ea85366b64a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0c1539a4-e686-404e-a81e-9ea85366b64a" (UID: "0c1539a4-e686-404e-a81e-9ea85366b64a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 05:03:29.471843 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:29.471809    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c1539a4-e686-404e-a81e-9ea85366b64a-kube-api-access-rdnfq" (OuterVolumeSpecName: "kube-api-access-rdnfq") pod "0c1539a4-e686-404e-a81e-9ea85366b64a" (UID: "0c1539a4-e686-404e-a81e-9ea85366b64a"). InnerVolumeSpecName "kube-api-access-rdnfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:03:29.571061 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:29.570979    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c1539a4-e686-404e-a81e-9ea85366b64a-plugins-config-volume\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:03:29.571061 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:29.571017    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdnfq\" (UniqueName: \"kubernetes.io/projected/0c1539a4-e686-404e-a81e-9ea85366b64a-kube-api-access-rdnfq\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:03:29.862396 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:29.862359    2565 generic.go:356] "Generic (PLEG): container finished" podID="0c1539a4-e686-404e-a81e-9ea85366b64a" containerID="7d5b9f09286e0228e6aad3e8bbecaf5ae2cbeff3f9e461a60b120e446c6ec28b" exitCode=0
Sep 14 05:03:29.862558 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:29.862399    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8f579f00-68c4db85cc-6z8zf"
Sep 14 05:03:29.862558 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:29.862431    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8f579f00-68c4db85cc-6z8zf" event={"ID":"0c1539a4-e686-404e-a81e-9ea85366b64a","Type":"ContainerDied","Data":"7d5b9f09286e0228e6aad3e8bbecaf5ae2cbeff3f9e461a60b120e446c6ec28b"}
Sep 14 05:03:29.862558 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:29.862490    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8f579f00-68c4db85cc-6z8zf" event={"ID":"0c1539a4-e686-404e-a81e-9ea85366b64a","Type":"ContainerDied","Data":"a7bae95ccdb1cb73d6214cce2dc2f9f64052fa3378d6b5d9f43a808372a9bbbe"}
Sep 14 05:03:29.862558 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:29.862512    2565 scope.go:117] "RemoveContainer" containerID="7d5b9f09286e0228e6aad3e8bbecaf5ae2cbeff3f9e461a60b120e446c6ec28b"
Sep 14 05:03:29.871343 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:29.871321    2565 scope.go:117] "RemoveContainer" containerID="7d5b9f09286e0228e6aad3e8bbecaf5ae2cbeff3f9e461a60b120e446c6ec28b"
Sep 14 05:03:29.871628 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:03:29.871609    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d5b9f09286e0228e6aad3e8bbecaf5ae2cbeff3f9e461a60b120e446c6ec28b\": container with ID starting with 7d5b9f09286e0228e6aad3e8bbecaf5ae2cbeff3f9e461a60b120e446c6ec28b not found: ID does not exist" containerID="7d5b9f09286e0228e6aad3e8bbecaf5ae2cbeff3f9e461a60b120e446c6ec28b"
Sep 14 05:03:29.871684 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:29.871637    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d5b9f09286e0228e6aad3e8bbecaf5ae2cbeff3f9e461a60b120e446c6ec28b"} err="failed to get container status \"7d5b9f09286e0228e6aad3e8bbecaf5ae2cbeff3f9e461a60b120e446c6ec28b\": rpc error: code = NotFound desc = could not find container \"7d5b9f09286e0228e6aad3e8bbecaf5ae2cbeff3f9e461a60b120e446c6ec28b\": container with ID starting with 7d5b9f09286e0228e6aad3e8bbecaf5ae2cbeff3f9e461a60b120e446c6ec28b not found: ID does not exist"
Sep 14 05:03:29.883995 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:29.883964    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8f579f00-68c4db85cc-6z8zf"]
Sep 14 05:03:29.889288 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:29.889270    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8f579f00-68c4db85cc-6z8zf"]
Sep 14 05:03:30.334205 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:30.334180    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c1539a4-e686-404e-a81e-9ea85366b64a" path="/var/lib/kubelet/pods/0c1539a4-e686-404e-a81e-9ea85366b64a/volumes"
Sep 14 05:03:30.436654 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:30.436618    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"]
Sep 14 05:03:30.436889 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:30.436862    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l" podUID="81d8f7ab-d2ab-4112-8498-07b238c0b047" containerName="istio-proxy" containerID="cri-o://98d1931301335684853e8ee930546d27b6ef0b23d13b868f207dcf0327f57cab" gracePeriod=30
Sep 14 05:03:30.645607 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:30.645521    2565 patch_prober.go:28] interesting pod/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 14 05:03:30.645607 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:30.645584    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l" podUID="81d8f7ab-d2ab-4112-8498-07b238c0b047" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 14 05:03:35.679026 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:35.679005    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"
Sep 14 05:03:35.728470 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:35.728404    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:03:35.827455 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:35.827421    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d8f7ab-d2ab-4112-8498-07b238c0b047-credential-socket\") pod \"81d8f7ab-d2ab-4112-8498-07b238c0b047\" (UID: \"81d8f7ab-d2ab-4112-8498-07b238c0b047\") "
Sep 14 05:03:35.827634 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:35.827464    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81d8f7ab-d2ab-4112-8498-07b238c0b047-istiod-ca-cert\") pod \"81d8f7ab-d2ab-4112-8498-07b238c0b047\" (UID: \"81d8f7ab-d2ab-4112-8498-07b238c0b047\") "
Sep 14 05:03:35.827634 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:35.827501    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81d8f7ab-d2ab-4112-8498-07b238c0b047-workload-certs\") pod \"81d8f7ab-d2ab-4112-8498-07b238c0b047\" (UID: \"81d8f7ab-d2ab-4112-8498-07b238c0b047\") "
Sep 14 05:03:35.827634 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:35.827526    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81d8f7ab-d2ab-4112-8498-07b238c0b047-istio-envoy\") pod \"81d8f7ab-d2ab-4112-8498-07b238c0b047\" (UID: \"81d8f7ab-d2ab-4112-8498-07b238c0b047\") "
Sep 14 05:03:35.827634 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:35.827566    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gh2x\" (UniqueName: \"kubernetes.io/projected/81d8f7ab-d2ab-4112-8498-07b238c0b047-kube-api-access-2gh2x\") pod \"81d8f7ab-d2ab-4112-8498-07b238c0b047\" (UID: \"81d8f7ab-d2ab-4112-8498-07b238c0b047\") "
Sep 14 05:03:35.827634 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:35.827595    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81d8f7ab-d2ab-4112-8498-07b238c0b047-istio-podinfo\") pod \"81d8f7ab-d2ab-4112-8498-07b238c0b047\" (UID: \"81d8f7ab-d2ab-4112-8498-07b238c0b047\") "
Sep 14 05:03:35.827930 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:35.827649    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d8f7ab-d2ab-4112-8498-07b238c0b047-workload-socket\") pod \"81d8f7ab-d2ab-4112-8498-07b238c0b047\" (UID: \"81d8f7ab-d2ab-4112-8498-07b238c0b047\") "
Sep 14 05:03:35.827930 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:35.827702    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81d8f7ab-d2ab-4112-8498-07b238c0b047-istio-token\") pod \"81d8f7ab-d2ab-4112-8498-07b238c0b047\" (UID: \"81d8f7ab-d2ab-4112-8498-07b238c0b047\") "
Sep 14 05:03:35.827930 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:35.827736    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81d8f7ab-d2ab-4112-8498-07b238c0b047-istio-data\") pod \"81d8f7ab-d2ab-4112-8498-07b238c0b047\" (UID: \"81d8f7ab-d2ab-4112-8498-07b238c0b047\") "
Sep 14 05:03:35.827930 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:35.827750    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d8f7ab-d2ab-4112-8498-07b238c0b047-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "81d8f7ab-d2ab-4112-8498-07b238c0b047" (UID: "81d8f7ab-d2ab-4112-8498-07b238c0b047"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:03:35.827930 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:35.827854    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d8f7ab-d2ab-4112-8498-07b238c0b047-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "81d8f7ab-d2ab-4112-8498-07b238c0b047" (UID: "81d8f7ab-d2ab-4112-8498-07b238c0b047"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:03:35.827930 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:35.827910    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81d8f7ab-d2ab-4112-8498-07b238c0b047-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "81d8f7ab-d2ab-4112-8498-07b238c0b047" (UID: "81d8f7ab-d2ab-4112-8498-07b238c0b047"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 05:03:35.828233 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:35.827994    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d8f7ab-d2ab-4112-8498-07b238c0b047-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "81d8f7ab-d2ab-4112-8498-07b238c0b047" (UID: "81d8f7ab-d2ab-4112-8498-07b238c0b047"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:03:35.828233 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:35.828059    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d8f7ab-d2ab-4112-8498-07b238c0b047-istio-data" (OuterVolumeSpecName: "istio-data") pod "81d8f7ab-d2ab-4112-8498-07b238c0b047" (UID: "81d8f7ab-d2ab-4112-8498-07b238c0b047"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:03:35.828233 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:35.828097    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81d8f7ab-d2ab-4112-8498-07b238c0b047-workload-certs\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:03:35.828233 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:35.828113    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d8f7ab-d2ab-4112-8498-07b238c0b047-workload-socket\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:03:35.828233 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:35.828122    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d8f7ab-d2ab-4112-8498-07b238c0b047-credential-socket\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:03:35.828233 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:35.828131    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81d8f7ab-d2ab-4112-8498-07b238c0b047-istiod-ca-cert\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:03:35.829933 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:35.829905    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81d8f7ab-d2ab-4112-8498-07b238c0b047-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "81d8f7ab-d2ab-4112-8498-07b238c0b047" (UID: "81d8f7ab-d2ab-4112-8498-07b238c0b047"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 05:03:35.829933 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:35.829922    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81d8f7ab-d2ab-4112-8498-07b238c0b047-kube-api-access-2gh2x" (OuterVolumeSpecName: "kube-api-access-2gh2x") pod "81d8f7ab-d2ab-4112-8498-07b238c0b047" (UID: "81d8f7ab-d2ab-4112-8498-07b238c0b047"). InnerVolumeSpecName "kube-api-access-2gh2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:03:35.830197 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:35.830175    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81d8f7ab-d2ab-4112-8498-07b238c0b047-istio-token" (OuterVolumeSpecName: "istio-token") pod "81d8f7ab-d2ab-4112-8498-07b238c0b047" (UID: "81d8f7ab-d2ab-4112-8498-07b238c0b047"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:03:35.830197 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:35.830187    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d8f7ab-d2ab-4112-8498-07b238c0b047-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "81d8f7ab-d2ab-4112-8498-07b238c0b047" (UID: "81d8f7ab-d2ab-4112-8498-07b238c0b047"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:03:35.897247 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:35.897214    2565 generic.go:356] "Generic (PLEG): container finished" podID="81d8f7ab-d2ab-4112-8498-07b238c0b047" containerID="98d1931301335684853e8ee930546d27b6ef0b23d13b868f207dcf0327f57cab" exitCode=0
Sep 14 05:03:35.897392 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:35.897255    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l" event={"ID":"81d8f7ab-d2ab-4112-8498-07b238c0b047","Type":"ContainerDied","Data":"98d1931301335684853e8ee930546d27b6ef0b23d13b868f207dcf0327f57cab"}
Sep 14 05:03:35.897392 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:35.897317    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l" event={"ID":"81d8f7ab-d2ab-4112-8498-07b238c0b047","Type":"ContainerDied","Data":"52d81ca5240fcaa8f3197be5f3cb653798ed024d972c2972c53f610f3df3f2c2"}
Sep 14 05:03:35.897392 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:35.897332    2565 scope.go:117] "RemoveContainer" containerID="98d1931301335684853e8ee930546d27b6ef0b23d13b868f207dcf0327f57cab"
Sep 14 05:03:35.897392 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:35.897264    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"
Sep 14 05:03:35.906649 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:35.906631    2565 scope.go:117] "RemoveContainer" containerID="cd809b355f20109f6fbeafd704ca5c4685201c6175bfc9785de6b2ff0a36e96a"
Sep 14 05:03:35.914566 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:35.914550    2565 scope.go:117] "RemoveContainer" containerID="98d1931301335684853e8ee930546d27b6ef0b23d13b868f207dcf0327f57cab"
Sep 14 05:03:35.914957 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:03:35.914938    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98d1931301335684853e8ee930546d27b6ef0b23d13b868f207dcf0327f57cab\": container with ID starting with 98d1931301335684853e8ee930546d27b6ef0b23d13b868f207dcf0327f57cab not found: ID does not exist" containerID="98d1931301335684853e8ee930546d27b6ef0b23d13b868f207dcf0327f57cab"
Sep 14 05:03:35.915032 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:35.914963    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98d1931301335684853e8ee930546d27b6ef0b23d13b868f207dcf0327f57cab"} err="failed to get container status \"98d1931301335684853e8ee930546d27b6ef0b23d13b868f207dcf0327f57cab\": rpc error: code = NotFound desc = could not find container \"98d1931301335684853e8ee930546d27b6ef0b23d13b868f207dcf0327f57cab\": container with ID starting with 98d1931301335684853e8ee930546d27b6ef0b23d13b868f207dcf0327f57cab not found: ID does not exist"
Sep 14 05:03:35.915032 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:35.914978    2565 scope.go:117] "RemoveContainer" containerID="cd809b355f20109f6fbeafd704ca5c4685201c6175bfc9785de6b2ff0a36e96a"
Sep 14 05:03:35.915226 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:03:35.915206    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd809b355f20109f6fbeafd704ca5c4685201c6175bfc9785de6b2ff0a36e96a\": container with ID starting with cd809b355f20109f6fbeafd704ca5c4685201c6175bfc9785de6b2ff0a36e96a not found: ID does not exist" containerID="cd809b355f20109f6fbeafd704ca5c4685201c6175bfc9785de6b2ff0a36e96a"
Sep 14 05:03:35.915268 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:35.915232    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd809b355f20109f6fbeafd704ca5c4685201c6175bfc9785de6b2ff0a36e96a"} err="failed to get container status \"cd809b355f20109f6fbeafd704ca5c4685201c6175bfc9785de6b2ff0a36e96a\": rpc error: code = NotFound desc = could not find container \"cd809b355f20109f6fbeafd704ca5c4685201c6175bfc9785de6b2ff0a36e96a\": container with ID starting with cd809b355f20109f6fbeafd704ca5c4685201c6175bfc9785de6b2ff0a36e96a not found: ID does not exist"
Sep 14 05:03:35.919871 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:35.919849    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"]
Sep 14 05:03:35.923737 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:35.923711    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8f579f00-openshift-default-5d5dcf9d4b-ndv5l"]
Sep 14 05:03:35.928893 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:35.928870    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gh2x\" (UniqueName: \"kubernetes.io/projected/81d8f7ab-d2ab-4112-8498-07b238c0b047-kube-api-access-2gh2x\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:03:35.928984 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:35.928896    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81d8f7ab-d2ab-4112-8498-07b238c0b047-istio-podinfo\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:03:35.928984 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:35.928912    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81d8f7ab-d2ab-4112-8498-07b238c0b047-istio-token\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:03:35.928984 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:35.928925    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81d8f7ab-d2ab-4112-8498-07b238c0b047-istio-data\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:03:35.928984 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:35.928937    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81d8f7ab-d2ab-4112-8498-07b238c0b047-istio-envoy\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:03:36.339406 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:36.339370    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81d8f7ab-d2ab-4112-8498-07b238c0b047" path="/var/lib/kubelet/pods/81d8f7ab-d2ab-4112-8498-07b238c0b047/volumes"
Sep 14 05:03:39.151191 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:39.151155    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2a2a3973-5985c67478-fjrj9"]
Sep 14 05:03:39.151577 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:39.151516    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81d8f7ab-d2ab-4112-8498-07b238c0b047" containerName="istio-proxy"
Sep 14 05:03:39.151577 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:39.151529    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="81d8f7ab-d2ab-4112-8498-07b238c0b047" containerName="istio-proxy"
Sep 14 05:03:39.151577 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:39.151541    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c1539a4-e686-404e-a81e-9ea85366b64a" containerName="payload-processing"
Sep 14 05:03:39.151577 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:39.151547    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c1539a4-e686-404e-a81e-9ea85366b64a" containerName="payload-processing"
Sep 14 05:03:39.151781 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:39.151625    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c1539a4-e686-404e-a81e-9ea85366b64a" containerName="payload-processing"
Sep 14 05:03:39.151781 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:39.151635    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="81d8f7ab-d2ab-4112-8498-07b238c0b047" containerName="istio-proxy"
Sep 14 05:03:39.151781 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:39.151643    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="81d8f7ab-d2ab-4112-8498-07b238c0b047" containerName="istio-proxy"
Sep 14 05:03:39.156217 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:39.156198    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2a2a3973-5985c67478-fjrj9"
Sep 14 05:03:39.158583 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:39.158561    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-2a2a3973-dockercfg-9lr2q\""
Sep 14 05:03:39.158659 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:39.158567    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-2a2a3973\""
Sep 14 05:03:39.164929 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:39.164908    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2a2a3973-5985c67478-fjrj9"]
Sep 14 05:03:39.167123 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:39.167102    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2a2a3973-cff7fb645-wmr6k"]
Sep 14 05:03:39.167443 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:39.167429    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81d8f7ab-d2ab-4112-8498-07b238c0b047" containerName="istio-proxy"
Sep 14 05:03:39.167443 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:39.167442    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="81d8f7ab-d2ab-4112-8498-07b238c0b047" containerName="istio-proxy"
Sep 14 05:03:39.170425 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:39.170404    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2a2a3973-cff7fb645-wmr6k"
Sep 14 05:03:39.181929 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:39.181908    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2a2a3973-cff7fb645-wmr6k"]
Sep 14 05:03:39.254186 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:39.254151    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/400be53b-0dcb-42b9-ab74-b17631119076-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-2a2a3973-5985c67478-fjrj9\" (UID: \"400be53b-0dcb-42b9-ab74-b17631119076\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2a2a3973-5985c67478-fjrj9"
Sep 14 05:03:39.254387 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:39.254232    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvp2m\" (UniqueName: \"kubernetes.io/projected/400be53b-0dcb-42b9-ab74-b17631119076-kube-api-access-bvp2m\") pod \"payload-pre-processing-e2e-derive-2a2a3973-5985c67478-fjrj9\" (UID: \"400be53b-0dcb-42b9-ab74-b17631119076\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2a2a3973-5985c67478-fjrj9"
Sep 14 05:03:39.354922 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:39.354890    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwcbq\" (UniqueName: \"kubernetes.io/projected/e4cfc41e-5904-48c4-97b8-2a390ef6e197-kube-api-access-vwcbq\") pod \"payload-processing-e2e-derive-2a2a3973-cff7fb645-wmr6k\" (UID: \"e4cfc41e-5904-48c4-97b8-2a390ef6e197\") " pod="openshift-ingress/payload-processing-e2e-derive-2a2a3973-cff7fb645-wmr6k"
Sep 14 05:03:39.355087 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:39.354936    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvp2m\" (UniqueName: \"kubernetes.io/projected/400be53b-0dcb-42b9-ab74-b17631119076-kube-api-access-bvp2m\") pod \"payload-pre-processing-e2e-derive-2a2a3973-5985c67478-fjrj9\" (UID: \"400be53b-0dcb-42b9-ab74-b17631119076\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2a2a3973-5985c67478-fjrj9"
Sep 14 05:03:39.355087 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:39.355061    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4cfc41e-5904-48c4-97b8-2a390ef6e197-plugins-config-volume\") pod \"payload-processing-e2e-derive-2a2a3973-cff7fb645-wmr6k\" (UID: \"e4cfc41e-5904-48c4-97b8-2a390ef6e197\") " pod="openshift-ingress/payload-processing-e2e-derive-2a2a3973-cff7fb645-wmr6k"
Sep 14 05:03:39.355184 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:39.355103    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/400be53b-0dcb-42b9-ab74-b17631119076-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-2a2a3973-5985c67478-fjrj9\" (UID: \"400be53b-0dcb-42b9-ab74-b17631119076\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2a2a3973-5985c67478-fjrj9"
Sep 14 05:03:39.355734 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:39.355716    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/400be53b-0dcb-42b9-ab74-b17631119076-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-2a2a3973-5985c67478-fjrj9\" (UID: \"400be53b-0dcb-42b9-ab74-b17631119076\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2a2a3973-5985c67478-fjrj9"
Sep 14 05:03:39.362889 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:39.362858    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvp2m\" (UniqueName: \"kubernetes.io/projected/400be53b-0dcb-42b9-ab74-b17631119076-kube-api-access-bvp2m\") pod \"payload-pre-processing-e2e-derive-2a2a3973-5985c67478-fjrj9\" (UID: \"400be53b-0dcb-42b9-ab74-b17631119076\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2a2a3973-5985c67478-fjrj9"
Sep 14 05:03:39.455876 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:39.455797    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwcbq\" (UniqueName: \"kubernetes.io/projected/e4cfc41e-5904-48c4-97b8-2a390ef6e197-kube-api-access-vwcbq\") pod \"payload-processing-e2e-derive-2a2a3973-cff7fb645-wmr6k\" (UID: \"e4cfc41e-5904-48c4-97b8-2a390ef6e197\") " pod="openshift-ingress/payload-processing-e2e-derive-2a2a3973-cff7fb645-wmr6k"
Sep 14 05:03:39.455876 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:39.455845    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4cfc41e-5904-48c4-97b8-2a390ef6e197-plugins-config-volume\") pod \"payload-processing-e2e-derive-2a2a3973-cff7fb645-wmr6k\" (UID: \"e4cfc41e-5904-48c4-97b8-2a390ef6e197\") " pod="openshift-ingress/payload-processing-e2e-derive-2a2a3973-cff7fb645-wmr6k"
Sep 14 05:03:39.456458 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:39.456439    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4cfc41e-5904-48c4-97b8-2a390ef6e197-plugins-config-volume\") pod \"payload-processing-e2e-derive-2a2a3973-cff7fb645-wmr6k\" (UID: \"e4cfc41e-5904-48c4-97b8-2a390ef6e197\") " pod="openshift-ingress/payload-processing-e2e-derive-2a2a3973-cff7fb645-wmr6k"
Sep 14 05:03:39.463583 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:39.463557    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwcbq\" (UniqueName: \"kubernetes.io/projected/e4cfc41e-5904-48c4-97b8-2a390ef6e197-kube-api-access-vwcbq\") pod \"payload-processing-e2e-derive-2a2a3973-cff7fb645-wmr6k\" (UID: \"e4cfc41e-5904-48c4-97b8-2a390ef6e197\") " pod="openshift-ingress/payload-processing-e2e-derive-2a2a3973-cff7fb645-wmr6k"
Sep 14 05:03:39.467382 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:39.467361    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2a2a3973-5985c67478-fjrj9"
Sep 14 05:03:39.479648 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:39.479628    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2a2a3973-cff7fb645-wmr6k"
Sep 14 05:03:39.740814 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:39.740788    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2a2a3973-5985c67478-fjrj9"]
Sep 14 05:03:39.741809 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:03:39.741751    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod400be53b_0dcb_42b9_ab74_b17631119076.slice/crio-3ccd20d9071139ab0b7531831c780eb68f47a2a67a22488336da036ee038f548 WatchSource:0}: Error finding container 3ccd20d9071139ab0b7531831c780eb68f47a2a67a22488336da036ee038f548: Status 404 returned error can't find the container with id 3ccd20d9071139ab0b7531831c780eb68f47a2a67a22488336da036ee038f548
Sep 14 05:03:39.763106 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:39.763083    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2a2a3973-cff7fb645-wmr6k"]
Sep 14 05:03:39.772909 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:03:39.772885    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4cfc41e_5904_48c4_97b8_2a390ef6e197.slice/crio-29d76f915c164298546ec268e68f6faac18c864070926f692f67187b4cf51900 WatchSource:0}: Error finding container 29d76f915c164298546ec268e68f6faac18c864070926f692f67187b4cf51900: Status 404 returned error can't find the container with id 29d76f915c164298546ec268e68f6faac18c864070926f692f67187b4cf51900
Sep 14 05:03:39.923364 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:39.923311    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2a2a3973-5985c67478-fjrj9" event={"ID":"400be53b-0dcb-42b9-ab74-b17631119076","Type":"ContainerStarted","Data":"f5e63fea49136bd46c2c83b914b184e27a02a4132c1bb22fe245b9bfdcb832bb"}
Sep 14 05:03:39.923560 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:39.923375    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2a2a3973-5985c67478-fjrj9" event={"ID":"400be53b-0dcb-42b9-ab74-b17631119076","Type":"ContainerStarted","Data":"3ccd20d9071139ab0b7531831c780eb68f47a2a67a22488336da036ee038f548"}
Sep 14 05:03:39.923560 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:39.923405    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-2a2a3973-5985c67478-fjrj9"
Sep 14 05:03:39.925048 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:39.925026    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2a2a3973-cff7fb645-wmr6k" event={"ID":"e4cfc41e-5904-48c4-97b8-2a390ef6e197","Type":"ContainerStarted","Data":"3061eeba104767a753608d7a7c43e3b53b28f8a61c881b67ceadc81d99a57ace"}
Sep 14 05:03:39.925162 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:39.925062    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2a2a3973-cff7fb645-wmr6k" event={"ID":"e4cfc41e-5904-48c4-97b8-2a390ef6e197","Type":"ContainerStarted","Data":"29d76f915c164298546ec268e68f6faac18c864070926f692f67187b4cf51900"}
Sep 14 05:03:39.925162 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:39.925128    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-2a2a3973-cff7fb645-wmr6k"
Sep 14 05:03:39.940815 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:39.940767    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-2a2a3973-5985c67478-fjrj9" podStartSLOduration=0.940733273 podStartE2EDuration="940.733273ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:03:39.939320723 +0000 UTC m=+1018.115092281" watchObservedRunningTime="2026-09-14 05:03:39.940733273 +0000 UTC m=+1018.116504833"
Sep 14 05:03:39.956430 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:39.956371    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-2a2a3973-cff7fb645-wmr6k" podStartSLOduration=0.956352124 podStartE2EDuration="956.352124ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:03:39.954075112 +0000 UTC m=+1018.129846674" watchObservedRunningTime="2026-09-14 05:03:39.956352124 +0000 UTC m=+1018.132123688"
Sep 14 05:03:50.932932 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:50.932897    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-2a2a3973-5985c67478-fjrj9"
Sep 14 05:03:50.933301 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:50.933056    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-2a2a3973-cff7fb645-wmr6k"
Sep 14 05:03:52.128579 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:52.128545    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:03:58.034516 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:58.034481    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-c366d3/simulator-w0-c366d3-kserve-9db75fd8d-45mgs"]
Sep 14 05:03:58.039140 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:58.039121    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c366d3/simulator-w0-c366d3-kserve-9db75fd8d-45mgs"
Sep 14 05:03:58.041600 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:58.041577    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-c366d3\"/\"kube-root-ca.crt\""
Sep 14 05:03:58.042779 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:58.042672    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-c366d3\"/\"openshift-service-ca.crt\""
Sep 14 05:03:58.042914 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:58.042744    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c366d3\"/\"simulator-w0-c366d3-kserve-self-signed-certs\""
Sep 14 05:03:58.042914 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:58.042902    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c366d3\"/\"default-dockercfg-n8djp\""
Sep 14 05:03:58.045449 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:58.045427    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c366d3/simulator-w0-c366d3-kserve-9db75fd8d-45mgs"]
Sep 14 05:03:58.109705 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:58.109671    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/471b3028-0304-4295-b5a7-b03ebb701d80-tls-certs\") pod \"simulator-w0-c366d3-kserve-9db75fd8d-45mgs\" (UID: \"471b3028-0304-4295-b5a7-b03ebb701d80\") " pod="e2e-models-e2e-worker-w0-c366d3/simulator-w0-c366d3-kserve-9db75fd8d-45mgs"
Sep 14 05:03:58.109861 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:58.109733    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/471b3028-0304-4295-b5a7-b03ebb701d80-model-cache\") pod \"simulator-w0-c366d3-kserve-9db75fd8d-45mgs\" (UID: \"471b3028-0304-4295-b5a7-b03ebb701d80\") " pod="e2e-models-e2e-worker-w0-c366d3/simulator-w0-c366d3-kserve-9db75fd8d-45mgs"
Sep 14 05:03:58.109923 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:58.109849    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4vds\" (UniqueName: \"kubernetes.io/projected/471b3028-0304-4295-b5a7-b03ebb701d80-kube-api-access-h4vds\") pod \"simulator-w0-c366d3-kserve-9db75fd8d-45mgs\" (UID: \"471b3028-0304-4295-b5a7-b03ebb701d80\") " pod="e2e-models-e2e-worker-w0-c366d3/simulator-w0-c366d3-kserve-9db75fd8d-45mgs"
Sep 14 05:03:58.109923 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:58.109884    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/471b3028-0304-4295-b5a7-b03ebb701d80-home\") pod \"simulator-w0-c366d3-kserve-9db75fd8d-45mgs\" (UID: \"471b3028-0304-4295-b5a7-b03ebb701d80\") " pod="e2e-models-e2e-worker-w0-c366d3/simulator-w0-c366d3-kserve-9db75fd8d-45mgs"
Sep 14 05:03:58.109923 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:58.109904    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/471b3028-0304-4295-b5a7-b03ebb701d80-dshm\") pod \"simulator-w0-c366d3-kserve-9db75fd8d-45mgs\" (UID: \"471b3028-0304-4295-b5a7-b03ebb701d80\") " pod="e2e-models-e2e-worker-w0-c366d3/simulator-w0-c366d3-kserve-9db75fd8d-45mgs"
Sep 14 05:03:58.110022 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:58.109926    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/471b3028-0304-4295-b5a7-b03ebb701d80-tmp-dir\") pod \"simulator-w0-c366d3-kserve-9db75fd8d-45mgs\" (UID: \"471b3028-0304-4295-b5a7-b03ebb701d80\") " pod="e2e-models-e2e-worker-w0-c366d3/simulator-w0-c366d3-kserve-9db75fd8d-45mgs"
Sep 14 05:03:58.211158 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:58.211131    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4vds\" (UniqueName: \"kubernetes.io/projected/471b3028-0304-4295-b5a7-b03ebb701d80-kube-api-access-h4vds\") pod \"simulator-w0-c366d3-kserve-9db75fd8d-45mgs\" (UID: \"471b3028-0304-4295-b5a7-b03ebb701d80\") " pod="e2e-models-e2e-worker-w0-c366d3/simulator-w0-c366d3-kserve-9db75fd8d-45mgs"
Sep 14 05:03:58.211344 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:58.211165    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/471b3028-0304-4295-b5a7-b03ebb701d80-home\") pod \"simulator-w0-c366d3-kserve-9db75fd8d-45mgs\" (UID: \"471b3028-0304-4295-b5a7-b03ebb701d80\") " pod="e2e-models-e2e-worker-w0-c366d3/simulator-w0-c366d3-kserve-9db75fd8d-45mgs"
Sep 14 05:03:58.211344 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:58.211185    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/471b3028-0304-4295-b5a7-b03ebb701d80-dshm\") pod \"simulator-w0-c366d3-kserve-9db75fd8d-45mgs\" (UID: \"471b3028-0304-4295-b5a7-b03ebb701d80\") " pod="e2e-models-e2e-worker-w0-c366d3/simulator-w0-c366d3-kserve-9db75fd8d-45mgs"
Sep 14 05:03:58.211344 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:58.211218    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/471b3028-0304-4295-b5a7-b03ebb701d80-tmp-dir\") pod \"simulator-w0-c366d3-kserve-9db75fd8d-45mgs\" (UID: \"471b3028-0304-4295-b5a7-b03ebb701d80\") " pod="e2e-models-e2e-worker-w0-c366d3/simulator-w0-c366d3-kserve-9db75fd8d-45mgs"
Sep 14 05:03:58.211344 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:58.211256    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/471b3028-0304-4295-b5a7-b03ebb701d80-tls-certs\") pod \"simulator-w0-c366d3-kserve-9db75fd8d-45mgs\" (UID: \"471b3028-0304-4295-b5a7-b03ebb701d80\") " pod="e2e-models-e2e-worker-w0-c366d3/simulator-w0-c366d3-kserve-9db75fd8d-45mgs"
Sep 14 05:03:58.211344 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:58.211331    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/471b3028-0304-4295-b5a7-b03ebb701d80-model-cache\") pod \"simulator-w0-c366d3-kserve-9db75fd8d-45mgs\" (UID: \"471b3028-0304-4295-b5a7-b03ebb701d80\") " pod="e2e-models-e2e-worker-w0-c366d3/simulator-w0-c366d3-kserve-9db75fd8d-45mgs"
Sep 14 05:03:58.211617 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:58.211589    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/471b3028-0304-4295-b5a7-b03ebb701d80-home\") pod \"simulator-w0-c366d3-kserve-9db75fd8d-45mgs\" (UID: \"471b3028-0304-4295-b5a7-b03ebb701d80\") " pod="e2e-models-e2e-worker-w0-c366d3/simulator-w0-c366d3-kserve-9db75fd8d-45mgs"
Sep 14 05:03:58.211681 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:58.211623    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/471b3028-0304-4295-b5a7-b03ebb701d80-tmp-dir\") pod \"simulator-w0-c366d3-kserve-9db75fd8d-45mgs\" (UID: \"471b3028-0304-4295-b5a7-b03ebb701d80\") " pod="e2e-models-e2e-worker-w0-c366d3/simulator-w0-c366d3-kserve-9db75fd8d-45mgs"
Sep 14 05:03:58.211737 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:58.211702    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/471b3028-0304-4295-b5a7-b03ebb701d80-model-cache\") pod \"simulator-w0-c366d3-kserve-9db75fd8d-45mgs\" (UID: \"471b3028-0304-4295-b5a7-b03ebb701d80\") " pod="e2e-models-e2e-worker-w0-c366d3/simulator-w0-c366d3-kserve-9db75fd8d-45mgs"
Sep 14 05:03:58.213430 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:58.213410    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/471b3028-0304-4295-b5a7-b03ebb701d80-dshm\") pod \"simulator-w0-c366d3-kserve-9db75fd8d-45mgs\" (UID: \"471b3028-0304-4295-b5a7-b03ebb701d80\") " pod="e2e-models-e2e-worker-w0-c366d3/simulator-w0-c366d3-kserve-9db75fd8d-45mgs"
Sep 14 05:03:58.213796 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:58.213776    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/471b3028-0304-4295-b5a7-b03ebb701d80-tls-certs\") pod \"simulator-w0-c366d3-kserve-9db75fd8d-45mgs\" (UID: \"471b3028-0304-4295-b5a7-b03ebb701d80\") " pod="e2e-models-e2e-worker-w0-c366d3/simulator-w0-c366d3-kserve-9db75fd8d-45mgs"
Sep 14 05:03:58.220149 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:58.220126    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4vds\" (UniqueName: \"kubernetes.io/projected/471b3028-0304-4295-b5a7-b03ebb701d80-kube-api-access-h4vds\") pod \"simulator-w0-c366d3-kserve-9db75fd8d-45mgs\" (UID: \"471b3028-0304-4295-b5a7-b03ebb701d80\") " pod="e2e-models-e2e-worker-w0-c366d3/simulator-w0-c366d3-kserve-9db75fd8d-45mgs"
Sep 14 05:03:58.348971 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:58.348940    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c366d3/simulator-w0-c366d3-kserve-9db75fd8d-45mgs"
Sep 14 05:03:58.621085 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:58.621056    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c366d3/simulator-w0-c366d3-kserve-9db75fd8d-45mgs"]
Sep 14 05:03:58.622267 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:03:58.622236    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod471b3028_0304_4295_b5a7_b03ebb701d80.slice/crio-712efa2a1e031b824b3d598dbe433a5c745c35502b05762a2812717174a38199 WatchSource:0}: Error finding container 712efa2a1e031b824b3d598dbe433a5c745c35502b05762a2812717174a38199: Status 404 returned error can't find the container with id 712efa2a1e031b824b3d598dbe433a5c745c35502b05762a2812717174a38199
Sep 14 05:03:59.028930 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:59.028838    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c366d3/simulator-w0-c366d3-kserve-9db75fd8d-45mgs" event={"ID":"471b3028-0304-4295-b5a7-b03ebb701d80","Type":"ContainerStarted","Data":"5364942a03fb0509237af33d551c97987012d2954a1c2fbfada3787ba88b0f19"}
Sep 14 05:03:59.028930 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:59.028882    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c366d3/simulator-w0-c366d3-kserve-9db75fd8d-45mgs" event={"ID":"471b3028-0304-4295-b5a7-b03ebb701d80","Type":"ContainerStarted","Data":"712efa2a1e031b824b3d598dbe433a5c745c35502b05762a2812717174a38199"}
Sep 14 05:03:59.046906 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:03:59.046857    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-c366d3/simulator-w0-c366d3-kserve-9db75fd8d-45mgs" podStartSLOduration=1.046841716 podStartE2EDuration="1.046841716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:03:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:03:59.045713512 +0000 UTC m=+1037.221485072" watchObservedRunningTime="2026-09-14 05:03:59.046841716 +0000 UTC m=+1037.222613271"
Sep 14 05:04:05.538042 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:05.537999    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:04:08.349746 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:08.349715    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-c366d3/simulator-w0-c366d3-kserve-9db75fd8d-45mgs"
Sep 14 05:04:08.350202 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:08.349773    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c366d3/simulator-w0-c366d3-kserve-9db75fd8d-45mgs"
Sep 14 05:04:08.362366 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:08.362335    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-c366d3/simulator-w0-c366d3-kserve-9db75fd8d-45mgs"
Sep 14 05:04:09.092996 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:09.092968    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c366d3/simulator-w0-c366d3-kserve-9db75fd8d-45mgs"
Sep 14 05:04:15.744457 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:15.744425    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2a2a3973-cff7fb645-wmr6k"]
Sep 14 05:04:15.744956 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:15.744679    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-2a2a3973-cff7fb645-wmr6k" podUID="e4cfc41e-5904-48c4-97b8-2a390ef6e197" containerName="payload-processing" containerID="cri-o://3061eeba104767a753608d7a7c43e3b53b28f8a61c881b67ceadc81d99a57ace" gracePeriod=30
Sep 14 05:04:15.751052 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:15.750914    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2a2a3973-5985c67478-fjrj9"]
Sep 14 05:04:15.751131 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:15.751108    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-2a2a3973-5985c67478-fjrj9" podUID="400be53b-0dcb-42b9-ab74-b17631119076" containerName="payload-pre-processing" containerID="cri-o://f5e63fea49136bd46c2c83b914b184e27a02a4132c1bb22fe245b9bfdcb832bb" gracePeriod=30
Sep 14 05:04:16.005879 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.005823    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2a2a3973-cff7fb645-wmr6k"
Sep 14 05:04:16.040891 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.040870    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2a2a3973-5985c67478-fjrj9"
Sep 14 05:04:16.059371 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.059335    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4cfc41e-5904-48c4-97b8-2a390ef6e197-plugins-config-volume\") pod \"e4cfc41e-5904-48c4-97b8-2a390ef6e197\" (UID: \"e4cfc41e-5904-48c4-97b8-2a390ef6e197\") "
Sep 14 05:04:16.059371 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.059372    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwcbq\" (UniqueName: \"kubernetes.io/projected/e4cfc41e-5904-48c4-97b8-2a390ef6e197-kube-api-access-vwcbq\") pod \"e4cfc41e-5904-48c4-97b8-2a390ef6e197\" (UID: \"e4cfc41e-5904-48c4-97b8-2a390ef6e197\") "
Sep 14 05:04:16.059810 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.059723    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4cfc41e-5904-48c4-97b8-2a390ef6e197-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e4cfc41e-5904-48c4-97b8-2a390ef6e197" (UID: "e4cfc41e-5904-48c4-97b8-2a390ef6e197"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 05:04:16.062162 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.062134    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4cfc41e-5904-48c4-97b8-2a390ef6e197-kube-api-access-vwcbq" (OuterVolumeSpecName: "kube-api-access-vwcbq") pod "e4cfc41e-5904-48c4-97b8-2a390ef6e197" (UID: "e4cfc41e-5904-48c4-97b8-2a390ef6e197"). InnerVolumeSpecName "kube-api-access-vwcbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:04:16.120021 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.119995    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-f14475/autores-model-f14475-kserve-855786fd99-b5k8p"]
Sep 14 05:04:16.120209 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.120189    2565 generic.go:356] "Generic (PLEG): container finished" podID="400be53b-0dcb-42b9-ab74-b17631119076" containerID="f5e63fea49136bd46c2c83b914b184e27a02a4132c1bb22fe245b9bfdcb832bb" exitCode=0
Sep 14 05:04:16.120314 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.120299    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2a2a3973-5985c67478-fjrj9"
Sep 14 05:04:16.120409 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.120392    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4cfc41e-5904-48c4-97b8-2a390ef6e197" containerName="payload-processing"
Sep 14 05:04:16.120473 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.120412    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4cfc41e-5904-48c4-97b8-2a390ef6e197" containerName="payload-processing"
Sep 14 05:04:16.120473 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.120431    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="400be53b-0dcb-42b9-ab74-b17631119076" containerName="payload-pre-processing"
Sep 14 05:04:16.120473 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.120438    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="400be53b-0dcb-42b9-ab74-b17631119076" containerName="payload-pre-processing"
Sep 14 05:04:16.120623 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.120526    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4cfc41e-5904-48c4-97b8-2a390ef6e197" containerName="payload-processing"
Sep 14 05:04:16.120623 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.120541    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="400be53b-0dcb-42b9-ab74-b17631119076" containerName="payload-pre-processing"
Sep 14 05:04:16.121700 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.121678    2565 generic.go:356] "Generic (PLEG): container finished" podID="e4cfc41e-5904-48c4-97b8-2a390ef6e197" containerID="3061eeba104767a753608d7a7c43e3b53b28f8a61c881b67ceadc81d99a57ace" exitCode=0
Sep 14 05:04:16.121814 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.121793    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2a2a3973-cff7fb645-wmr6k"
Sep 14 05:04:16.123634 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.123613    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2a2a3973-5985c67478-fjrj9" event={"ID":"400be53b-0dcb-42b9-ab74-b17631119076","Type":"ContainerDied","Data":"f5e63fea49136bd46c2c83b914b184e27a02a4132c1bb22fe245b9bfdcb832bb"}
Sep 14 05:04:16.123679 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.123657    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2a2a3973-5985c67478-fjrj9" event={"ID":"400be53b-0dcb-42b9-ab74-b17631119076","Type":"ContainerDied","Data":"3ccd20d9071139ab0b7531831c780eb68f47a2a67a22488336da036ee038f548"}
Sep 14 05:04:16.123720 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.123683    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2a2a3973-cff7fb645-wmr6k" event={"ID":"e4cfc41e-5904-48c4-97b8-2a390ef6e197","Type":"ContainerDied","Data":"3061eeba104767a753608d7a7c43e3b53b28f8a61c881b67ceadc81d99a57ace"}
Sep 14 05:04:16.123720 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.123708    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-f14475/autores-model-f14475-kserve-855786fd99-b5k8p"
Sep 14 05:04:16.123808 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.123721    2565 scope.go:117] "RemoveContainer" containerID="f5e63fea49136bd46c2c83b914b184e27a02a4132c1bb22fe245b9bfdcb832bb"
Sep 14 05:04:16.123808 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.123708    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2a2a3973-cff7fb645-wmr6k" event={"ID":"e4cfc41e-5904-48c4-97b8-2a390ef6e197","Type":"ContainerDied","Data":"29d76f915c164298546ec268e68f6faac18c864070926f692f67187b4cf51900"}
Sep 14 05:04:16.126055 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.126032    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-f14475\"/\"kube-root-ca.crt\""
Sep 14 05:04:16.126168 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.126076    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-f14475\"/\"openshift-service-ca.crt\""
Sep 14 05:04:16.126168 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.126100    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-f14475\"/\"default-dockercfg-mbg6c\""
Sep 14 05:04:16.126168 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.126106    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-f14475\"/\"autores-model-f14475-kserve-self-signed-certs\""
Sep 14 05:04:16.133632 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.133148    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-f14475/autores-model-f14475-kserve-855786fd99-b5k8p"]
Sep 14 05:04:16.136920 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.136901    2565 scope.go:117] "RemoveContainer" containerID="f5e63fea49136bd46c2c83b914b184e27a02a4132c1bb22fe245b9bfdcb832bb"
Sep 14 05:04:16.137674 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:04:16.137651    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5e63fea49136bd46c2c83b914b184e27a02a4132c1bb22fe245b9bfdcb832bb\": container with ID starting with f5e63fea49136bd46c2c83b914b184e27a02a4132c1bb22fe245b9bfdcb832bb not found: ID does not exist" containerID="f5e63fea49136bd46c2c83b914b184e27a02a4132c1bb22fe245b9bfdcb832bb"
Sep 14 05:04:16.138587 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.138556    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5e63fea49136bd46c2c83b914b184e27a02a4132c1bb22fe245b9bfdcb832bb"} err="failed to get container status \"f5e63fea49136bd46c2c83b914b184e27a02a4132c1bb22fe245b9bfdcb832bb\": rpc error: code = NotFound desc = could not find container \"f5e63fea49136bd46c2c83b914b184e27a02a4132c1bb22fe245b9bfdcb832bb\": container with ID starting with f5e63fea49136bd46c2c83b914b184e27a02a4132c1bb22fe245b9bfdcb832bb not found: ID does not exist"
Sep 14 05:04:16.138692 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.138673    2565 scope.go:117] "RemoveContainer" containerID="3061eeba104767a753608d7a7c43e3b53b28f8a61c881b67ceadc81d99a57ace"
Sep 14 05:04:16.151585 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.151562    2565 scope.go:117] "RemoveContainer" containerID="3061eeba104767a753608d7a7c43e3b53b28f8a61c881b67ceadc81d99a57ace"
Sep 14 05:04:16.151989 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:04:16.151960    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3061eeba104767a753608d7a7c43e3b53b28f8a61c881b67ceadc81d99a57ace\": container with ID starting with 3061eeba104767a753608d7a7c43e3b53b28f8a61c881b67ceadc81d99a57ace not found: ID does not exist" containerID="3061eeba104767a753608d7a7c43e3b53b28f8a61c881b67ceadc81d99a57ace"
Sep 14 05:04:16.152097 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.151999    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3061eeba104767a753608d7a7c43e3b53b28f8a61c881b67ceadc81d99a57ace"} err="failed to get container status \"3061eeba104767a753608d7a7c43e3b53b28f8a61c881b67ceadc81d99a57ace\": rpc error: code = NotFound desc = could not find container \"3061eeba104767a753608d7a7c43e3b53b28f8a61c881b67ceadc81d99a57ace\": container with ID starting with 3061eeba104767a753608d7a7c43e3b53b28f8a61c881b67ceadc81d99a57ace not found: ID does not exist"
Sep 14 05:04:16.160183 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.160152    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvp2m\" (UniqueName: \"kubernetes.io/projected/400be53b-0dcb-42b9-ab74-b17631119076-kube-api-access-bvp2m\") pod \"400be53b-0dcb-42b9-ab74-b17631119076\" (UID: \"400be53b-0dcb-42b9-ab74-b17631119076\") "
Sep 14 05:04:16.160322 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.160305    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/400be53b-0dcb-42b9-ab74-b17631119076-plugins-config-volume\") pod \"400be53b-0dcb-42b9-ab74-b17631119076\" (UID: \"400be53b-0dcb-42b9-ab74-b17631119076\") "
Sep 14 05:04:16.160568 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.160550    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4cfc41e-5904-48c4-97b8-2a390ef6e197-plugins-config-volume\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:04:16.160641 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.160575    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwcbq\" (UniqueName: \"kubernetes.io/projected/e4cfc41e-5904-48c4-97b8-2a390ef6e197-kube-api-access-vwcbq\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:04:16.160687 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.160633    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/400be53b-0dcb-42b9-ab74-b17631119076-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "400be53b-0dcb-42b9-ab74-b17631119076" (UID: "400be53b-0dcb-42b9-ab74-b17631119076"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 05:04:16.162111 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.162087    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/400be53b-0dcb-42b9-ab74-b17631119076-kube-api-access-bvp2m" (OuterVolumeSpecName: "kube-api-access-bvp2m") pod "400be53b-0dcb-42b9-ab74-b17631119076" (UID: "400be53b-0dcb-42b9-ab74-b17631119076"). InnerVolumeSpecName "kube-api-access-bvp2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:04:16.169271 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.169246    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2a2a3973-cff7fb645-wmr6k"]
Sep 14 05:04:16.171316 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.171294    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2a2a3973-cff7fb645-wmr6k"]
Sep 14 05:04:16.261138 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.261060    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bba8a0f2-6895-4749-8098-19f9154aca6f-dshm\") pod \"autores-model-f14475-kserve-855786fd99-b5k8p\" (UID: \"bba8a0f2-6895-4749-8098-19f9154aca6f\") " pod="ai-tenant-e2e-autores-f14475/autores-model-f14475-kserve-855786fd99-b5k8p"
Sep 14 05:04:16.261138 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.261104    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bba8a0f2-6895-4749-8098-19f9154aca6f-model-cache\") pod \"autores-model-f14475-kserve-855786fd99-b5k8p\" (UID: \"bba8a0f2-6895-4749-8098-19f9154aca6f\") " pod="ai-tenant-e2e-autores-f14475/autores-model-f14475-kserve-855786fd99-b5k8p"
Sep 14 05:04:16.261138 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.261131    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bba8a0f2-6895-4749-8098-19f9154aca6f-tls-certs\") pod \"autores-model-f14475-kserve-855786fd99-b5k8p\" (UID: \"bba8a0f2-6895-4749-8098-19f9154aca6f\") " pod="ai-tenant-e2e-autores-f14475/autores-model-f14475-kserve-855786fd99-b5k8p"
Sep 14 05:04:16.261338 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.261164    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bba8a0f2-6895-4749-8098-19f9154aca6f-tmp-dir\") pod \"autores-model-f14475-kserve-855786fd99-b5k8p\" (UID: \"bba8a0f2-6895-4749-8098-19f9154aca6f\") " pod="ai-tenant-e2e-autores-f14475/autores-model-f14475-kserve-855786fd99-b5k8p"
Sep 14 05:04:16.261338 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.261188    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bba8a0f2-6895-4749-8098-19f9154aca6f-home\") pod \"autores-model-f14475-kserve-855786fd99-b5k8p\" (UID: \"bba8a0f2-6895-4749-8098-19f9154aca6f\") " pod="ai-tenant-e2e-autores-f14475/autores-model-f14475-kserve-855786fd99-b5k8p"
Sep 14 05:04:16.261338 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.261224    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcdbs\" (UniqueName: \"kubernetes.io/projected/bba8a0f2-6895-4749-8098-19f9154aca6f-kube-api-access-jcdbs\") pod \"autores-model-f14475-kserve-855786fd99-b5k8p\" (UID: \"bba8a0f2-6895-4749-8098-19f9154aca6f\") " pod="ai-tenant-e2e-autores-f14475/autores-model-f14475-kserve-855786fd99-b5k8p"
Sep 14 05:04:16.261338 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.261304    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvp2m\" (UniqueName: \"kubernetes.io/projected/400be53b-0dcb-42b9-ab74-b17631119076-kube-api-access-bvp2m\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:04:16.261338 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.261327    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/400be53b-0dcb-42b9-ab74-b17631119076-plugins-config-volume\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:04:16.335225 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.335186    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4cfc41e-5904-48c4-97b8-2a390ef6e197" path="/var/lib/kubelet/pods/e4cfc41e-5904-48c4-97b8-2a390ef6e197/volumes"
Sep 14 05:04:16.362179 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.362146    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bba8a0f2-6895-4749-8098-19f9154aca6f-dshm\") pod \"autores-model-f14475-kserve-855786fd99-b5k8p\" (UID: \"bba8a0f2-6895-4749-8098-19f9154aca6f\") " pod="ai-tenant-e2e-autores-f14475/autores-model-f14475-kserve-855786fd99-b5k8p"
Sep 14 05:04:16.362290 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.362199    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bba8a0f2-6895-4749-8098-19f9154aca6f-model-cache\") pod \"autores-model-f14475-kserve-855786fd99-b5k8p\" (UID: \"bba8a0f2-6895-4749-8098-19f9154aca6f\") " pod="ai-tenant-e2e-autores-f14475/autores-model-f14475-kserve-855786fd99-b5k8p"
Sep 14 05:04:16.362290 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.362224    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bba8a0f2-6895-4749-8098-19f9154aca6f-tls-certs\") pod \"autores-model-f14475-kserve-855786fd99-b5k8p\" (UID: \"bba8a0f2-6895-4749-8098-19f9154aca6f\") " pod="ai-tenant-e2e-autores-f14475/autores-model-f14475-kserve-855786fd99-b5k8p"
Sep 14 05:04:16.362290 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.362246    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bba8a0f2-6895-4749-8098-19f9154aca6f-tmp-dir\") pod \"autores-model-f14475-kserve-855786fd99-b5k8p\" (UID: \"bba8a0f2-6895-4749-8098-19f9154aca6f\") " pod="ai-tenant-e2e-autores-f14475/autores-model-f14475-kserve-855786fd99-b5k8p"
Sep 14 05:04:16.362290 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.362262    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bba8a0f2-6895-4749-8098-19f9154aca6f-home\") pod \"autores-model-f14475-kserve-855786fd99-b5k8p\" (UID: \"bba8a0f2-6895-4749-8098-19f9154aca6f\") " pod="ai-tenant-e2e-autores-f14475/autores-model-f14475-kserve-855786fd99-b5k8p"
Sep 14 05:04:16.362499 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.362299    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcdbs\" (UniqueName: \"kubernetes.io/projected/bba8a0f2-6895-4749-8098-19f9154aca6f-kube-api-access-jcdbs\") pod \"autores-model-f14475-kserve-855786fd99-b5k8p\" (UID: \"bba8a0f2-6895-4749-8098-19f9154aca6f\") " pod="ai-tenant-e2e-autores-f14475/autores-model-f14475-kserve-855786fd99-b5k8p"
Sep 14 05:04:16.362652 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.362632    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bba8a0f2-6895-4749-8098-19f9154aca6f-tmp-dir\") pod \"autores-model-f14475-kserve-855786fd99-b5k8p\" (UID: \"bba8a0f2-6895-4749-8098-19f9154aca6f\") " pod="ai-tenant-e2e-autores-f14475/autores-model-f14475-kserve-855786fd99-b5k8p"
Sep 14 05:04:16.362720 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.362659    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bba8a0f2-6895-4749-8098-19f9154aca6f-model-cache\") pod \"autores-model-f14475-kserve-855786fd99-b5k8p\" (UID: \"bba8a0f2-6895-4749-8098-19f9154aca6f\") " pod="ai-tenant-e2e-autores-f14475/autores-model-f14475-kserve-855786fd99-b5k8p"
Sep 14 05:04:16.362720 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.362703    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bba8a0f2-6895-4749-8098-19f9154aca6f-home\") pod \"autores-model-f14475-kserve-855786fd99-b5k8p\" (UID: \"bba8a0f2-6895-4749-8098-19f9154aca6f\") " pod="ai-tenant-e2e-autores-f14475/autores-model-f14475-kserve-855786fd99-b5k8p"
Sep 14 05:04:16.364455 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.364439    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bba8a0f2-6895-4749-8098-19f9154aca6f-dshm\") pod \"autores-model-f14475-kserve-855786fd99-b5k8p\" (UID: \"bba8a0f2-6895-4749-8098-19f9154aca6f\") " pod="ai-tenant-e2e-autores-f14475/autores-model-f14475-kserve-855786fd99-b5k8p"
Sep 14 05:04:16.364619 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.364602    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bba8a0f2-6895-4749-8098-19f9154aca6f-tls-certs\") pod \"autores-model-f14475-kserve-855786fd99-b5k8p\" (UID: \"bba8a0f2-6895-4749-8098-19f9154aca6f\") " pod="ai-tenant-e2e-autores-f14475/autores-model-f14475-kserve-855786fd99-b5k8p"
Sep 14 05:04:16.369846 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.369825    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcdbs\" (UniqueName: \"kubernetes.io/projected/bba8a0f2-6895-4749-8098-19f9154aca6f-kube-api-access-jcdbs\") pod \"autores-model-f14475-kserve-855786fd99-b5k8p\" (UID: \"bba8a0f2-6895-4749-8098-19f9154aca6f\") " pod="ai-tenant-e2e-autores-f14475/autores-model-f14475-kserve-855786fd99-b5k8p"
Sep 14 05:04:16.436054 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.436022    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2a2a3973-5985c67478-fjrj9"]
Sep 14 05:04:16.438213 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.438188    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2a2a3973-5985c67478-fjrj9"]
Sep 14 05:04:16.447675 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.447652    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-f14475/autores-model-f14475-kserve-855786fd99-b5k8p"
Sep 14 05:04:16.725248 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:16.725225    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-f14475/autores-model-f14475-kserve-855786fd99-b5k8p"]
Sep 14 05:04:16.726447 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:04:16.726421    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbba8a0f2_6895_4749_8098_19f9154aca6f.slice/crio-7fad7111cc176aa07dc6a88600651acf76f39c74bb758f6b8dd3711251554844 WatchSource:0}: Error finding container 7fad7111cc176aa07dc6a88600651acf76f39c74bb758f6b8dd3711251554844: Status 404 returned error can't find the container with id 7fad7111cc176aa07dc6a88600651acf76f39c74bb758f6b8dd3711251554844
Sep 14 05:04:17.132707 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:17.132670    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-f14475/autores-model-f14475-kserve-855786fd99-b5k8p" event={"ID":"bba8a0f2-6895-4749-8098-19f9154aca6f","Type":"ContainerStarted","Data":"db4e70d7057542fbae79b6bedf673bb7d5e43613b6a25c042fff0dca1757e5c6"}
Sep 14 05:04:17.133197 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:17.132721    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-f14475/autores-model-f14475-kserve-855786fd99-b5k8p" event={"ID":"bba8a0f2-6895-4749-8098-19f9154aca6f","Type":"ContainerStarted","Data":"7fad7111cc176aa07dc6a88600651acf76f39c74bb758f6b8dd3711251554844"}
Sep 14 05:04:17.159590 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:17.159537    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-f14475/autores-model-f14475-kserve-855786fd99-b5k8p" podStartSLOduration=1.159521881 podStartE2EDuration="1.159521881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:04:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:04:17.157892469 +0000 UTC m=+1055.333664028" watchObservedRunningTime="2026-09-14 05:04:17.159521881 +0000 UTC m=+1055.335293441"
Sep 14 05:04:18.334923 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:18.334894    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="400be53b-0dcb-42b9-ab74-b17631119076" path="/var/lib/kubelet/pods/400be53b-0dcb-42b9-ab74-b17631119076/volumes"
Sep 14 05:04:22.526380 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:22.526338    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:04:26.448477 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:26.448442    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-f14475/autores-model-f14475-kserve-855786fd99-b5k8p"
Sep 14 05:04:26.448995 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:26.448506    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-f14475/autores-model-f14475-kserve-855786fd99-b5k8p"
Sep 14 05:04:26.461006 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:26.460982    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-f14475/autores-model-f14475-kserve-855786fd99-b5k8p"
Sep 14 05:04:27.203032 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:27.203006    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-f14475/autores-model-f14475-kserve-855786fd99-b5k8p"
Sep 14 05:04:32.909691 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:32.909658    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-667664/test-model-667664-kserve-96bfc7c9f-ktdrk"]
Sep 14 05:04:32.914333 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:32.914307    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-667664/test-model-667664-kserve-96bfc7c9f-ktdrk"
Sep 14 05:04:32.917766 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:32.917721    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-667664\"/\"openshift-service-ca.crt\""
Sep 14 05:04:32.917873 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:32.917779    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-667664\"/\"default-dockercfg-hrnln\""
Sep 14 05:04:32.917873 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:32.917805    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-667664\"/\"test-model-667664-kserve-self-signed-certs\""
Sep 14 05:04:32.917873 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:32.917858    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-667664\"/\"kube-root-ca.crt\""
Sep 14 05:04:32.924645 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:32.924624    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-667664/test-model-667664-kserve-96bfc7c9f-ktdrk"]
Sep 14 05:04:33.009073 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:33.009045    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/956bf287-009d-4eb1-be24-78c2cc343a76-tls-certs\") pod \"test-model-667664-kserve-96bfc7c9f-ktdrk\" (UID: \"956bf287-009d-4eb1-be24-78c2cc343a76\") " pod="ai-tenant-e2e-inf-a-667664/test-model-667664-kserve-96bfc7c9f-ktdrk"
Sep 14 05:04:33.009073 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:33.009076    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9qdk\" (UniqueName: \"kubernetes.io/projected/956bf287-009d-4eb1-be24-78c2cc343a76-kube-api-access-p9qdk\") pod \"test-model-667664-kserve-96bfc7c9f-ktdrk\" (UID: \"956bf287-009d-4eb1-be24-78c2cc343a76\") " pod="ai-tenant-e2e-inf-a-667664/test-model-667664-kserve-96bfc7c9f-ktdrk"
Sep 14 05:04:33.009264 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:33.009099    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/956bf287-009d-4eb1-be24-78c2cc343a76-model-cache\") pod \"test-model-667664-kserve-96bfc7c9f-ktdrk\" (UID: \"956bf287-009d-4eb1-be24-78c2cc343a76\") " pod="ai-tenant-e2e-inf-a-667664/test-model-667664-kserve-96bfc7c9f-ktdrk"
Sep 14 05:04:33.009264 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:33.009234    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/956bf287-009d-4eb1-be24-78c2cc343a76-home\") pod \"test-model-667664-kserve-96bfc7c9f-ktdrk\" (UID: \"956bf287-009d-4eb1-be24-78c2cc343a76\") " pod="ai-tenant-e2e-inf-a-667664/test-model-667664-kserve-96bfc7c9f-ktdrk"
Sep 14 05:04:33.009336 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:33.009269    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/956bf287-009d-4eb1-be24-78c2cc343a76-dshm\") pod \"test-model-667664-kserve-96bfc7c9f-ktdrk\" (UID: \"956bf287-009d-4eb1-be24-78c2cc343a76\") " pod="ai-tenant-e2e-inf-a-667664/test-model-667664-kserve-96bfc7c9f-ktdrk"
Sep 14 05:04:33.009336 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:33.009290    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/956bf287-009d-4eb1-be24-78c2cc343a76-tmp-dir\") pod \"test-model-667664-kserve-96bfc7c9f-ktdrk\" (UID: \"956bf287-009d-4eb1-be24-78c2cc343a76\") " pod="ai-tenant-e2e-inf-a-667664/test-model-667664-kserve-96bfc7c9f-ktdrk"
Sep 14 05:04:33.109971 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:33.109942    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/956bf287-009d-4eb1-be24-78c2cc343a76-tls-certs\") pod \"test-model-667664-kserve-96bfc7c9f-ktdrk\" (UID: \"956bf287-009d-4eb1-be24-78c2cc343a76\") " pod="ai-tenant-e2e-inf-a-667664/test-model-667664-kserve-96bfc7c9f-ktdrk"
Sep 14 05:04:33.109971 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:33.109976    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9qdk\" (UniqueName: \"kubernetes.io/projected/956bf287-009d-4eb1-be24-78c2cc343a76-kube-api-access-p9qdk\") pod \"test-model-667664-kserve-96bfc7c9f-ktdrk\" (UID: \"956bf287-009d-4eb1-be24-78c2cc343a76\") " pod="ai-tenant-e2e-inf-a-667664/test-model-667664-kserve-96bfc7c9f-ktdrk"
Sep 14 05:04:33.110212 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:33.109994    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/956bf287-009d-4eb1-be24-78c2cc343a76-model-cache\") pod \"test-model-667664-kserve-96bfc7c9f-ktdrk\" (UID: \"956bf287-009d-4eb1-be24-78c2cc343a76\") " pod="ai-tenant-e2e-inf-a-667664/test-model-667664-kserve-96bfc7c9f-ktdrk"
Sep 14 05:04:33.110212 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:33.110052    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/956bf287-009d-4eb1-be24-78c2cc343a76-home\") pod \"test-model-667664-kserve-96bfc7c9f-ktdrk\" (UID: \"956bf287-009d-4eb1-be24-78c2cc343a76\") " pod="ai-tenant-e2e-inf-a-667664/test-model-667664-kserve-96bfc7c9f-ktdrk"
Sep 14 05:04:33.110212 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:33.110071    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/956bf287-009d-4eb1-be24-78c2cc343a76-dshm\") pod \"test-model-667664-kserve-96bfc7c9f-ktdrk\" (UID: \"956bf287-009d-4eb1-be24-78c2cc343a76\") " pod="ai-tenant-e2e-inf-a-667664/test-model-667664-kserve-96bfc7c9f-ktdrk"
Sep 14 05:04:33.110212 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:33.110086    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/956bf287-009d-4eb1-be24-78c2cc343a76-tmp-dir\") pod \"test-model-667664-kserve-96bfc7c9f-ktdrk\" (UID: \"956bf287-009d-4eb1-be24-78c2cc343a76\") " pod="ai-tenant-e2e-inf-a-667664/test-model-667664-kserve-96bfc7c9f-ktdrk"
Sep 14 05:04:33.110504 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:33.110475    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/956bf287-009d-4eb1-be24-78c2cc343a76-home\") pod \"test-model-667664-kserve-96bfc7c9f-ktdrk\" (UID: \"956bf287-009d-4eb1-be24-78c2cc343a76\") " pod="ai-tenant-e2e-inf-a-667664/test-model-667664-kserve-96bfc7c9f-ktdrk"
Sep 14 05:04:33.110574 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:33.110480    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/956bf287-009d-4eb1-be24-78c2cc343a76-tmp-dir\") pod \"test-model-667664-kserve-96bfc7c9f-ktdrk\" (UID: \"956bf287-009d-4eb1-be24-78c2cc343a76\") " pod="ai-tenant-e2e-inf-a-667664/test-model-667664-kserve-96bfc7c9f-ktdrk"
Sep 14 05:04:33.110574 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:33.110547    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/956bf287-009d-4eb1-be24-78c2cc343a76-model-cache\") pod \"test-model-667664-kserve-96bfc7c9f-ktdrk\" (UID: \"956bf287-009d-4eb1-be24-78c2cc343a76\") " pod="ai-tenant-e2e-inf-a-667664/test-model-667664-kserve-96bfc7c9f-ktdrk"
Sep 14 05:04:33.112382 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:33.112361    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/956bf287-009d-4eb1-be24-78c2cc343a76-dshm\") pod \"test-model-667664-kserve-96bfc7c9f-ktdrk\" (UID: \"956bf287-009d-4eb1-be24-78c2cc343a76\") " pod="ai-tenant-e2e-inf-a-667664/test-model-667664-kserve-96bfc7c9f-ktdrk"
Sep 14 05:04:33.112533 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:33.112517    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/956bf287-009d-4eb1-be24-78c2cc343a76-tls-certs\") pod \"test-model-667664-kserve-96bfc7c9f-ktdrk\" (UID: \"956bf287-009d-4eb1-be24-78c2cc343a76\") " pod="ai-tenant-e2e-inf-a-667664/test-model-667664-kserve-96bfc7c9f-ktdrk"
Sep 14 05:04:33.118648 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:33.118627    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9qdk\" (UniqueName: \"kubernetes.io/projected/956bf287-009d-4eb1-be24-78c2cc343a76-kube-api-access-p9qdk\") pod \"test-model-667664-kserve-96bfc7c9f-ktdrk\" (UID: \"956bf287-009d-4eb1-be24-78c2cc343a76\") " pod="ai-tenant-e2e-inf-a-667664/test-model-667664-kserve-96bfc7c9f-ktdrk"
Sep 14 05:04:33.224178 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:33.224103    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-667664/test-model-667664-kserve-96bfc7c9f-ktdrk"
Sep 14 05:04:33.495210 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:33.495183    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-667664/test-model-667664-kserve-96bfc7c9f-ktdrk"]
Sep 14 05:04:33.496962 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:04:33.496930    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod956bf287_009d_4eb1_be24_78c2cc343a76.slice/crio-2d80337243f7581244780534056584e45ecdfba8a5b655431bf1fee78a201033 WatchSource:0}: Error finding container 2d80337243f7581244780534056584e45ecdfba8a5b655431bf1fee78a201033: Status 404 returned error can't find the container with id 2d80337243f7581244780534056584e45ecdfba8a5b655431bf1fee78a201033
Sep 14 05:04:34.233801 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:34.233733    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-667664/test-model-667664-kserve-96bfc7c9f-ktdrk" event={"ID":"956bf287-009d-4eb1-be24-78c2cc343a76","Type":"ContainerStarted","Data":"b4a500de9ff7a8b4e750207abdad0b470d475d5bd5d5dcde4b82fc5cf33ec993"}
Sep 14 05:04:34.233801 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:34.233806    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-667664/test-model-667664-kserve-96bfc7c9f-ktdrk" event={"ID":"956bf287-009d-4eb1-be24-78c2cc343a76","Type":"ContainerStarted","Data":"2d80337243f7581244780534056584e45ecdfba8a5b655431bf1fee78a201033"}
Sep 14 05:04:34.252131 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:34.252083    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-667664/test-model-667664-kserve-96bfc7c9f-ktdrk" podStartSLOduration=2.252068528 podStartE2EDuration="2.252068528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:04:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:04:34.250355169 +0000 UTC m=+1072.426126730" watchObservedRunningTime="2026-09-14 05:04:34.252068528 +0000 UTC m=+1072.427840087"
Sep 14 05:04:41.127868 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:41.127786    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:04:43.224249 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:43.224211    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-667664/test-model-667664-kserve-96bfc7c9f-ktdrk"
Sep 14 05:04:43.224626 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:43.224264    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-667664/test-model-667664-kserve-96bfc7c9f-ktdrk"
Sep 14 05:04:43.236409 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:43.236379    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-667664/test-model-667664-kserve-96bfc7c9f-ktdrk"
Sep 14 05:04:43.299385 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:04:43.299360    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-667664/test-model-667664-kserve-96bfc7c9f-ktdrk"
Sep 14 05:05:01.436064 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:01.436023    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-f14475-857d5b4f79-crdsn"]
Sep 14 05:05:01.436668 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:01.436267    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-f14475-857d5b4f79-crdsn" podUID="a5fa6bb8-e10f-41af-9b37-962fad6aef8e" containerName="payload-processing" containerID="cri-o://98da3ea55890a050c05816f0a4ab9ba5bc4a802101db5a2aba02ccc11153ef10" gracePeriod=30
Sep 14 05:05:01.445156 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:01.444152    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f14475-59bb95458-z5mxz"]
Sep 14 05:05:01.445156 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:01.444377    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-f14475-59bb95458-z5mxz" podUID="cc6a0970-e07a-4a2a-9232-57b8d402b0bb" containerName="payload-pre-processing" containerID="cri-o://39447f692fb5245276fb446983dd3aa3f1f7c88486526fb128d3701c38449104" gracePeriod=30
Sep 14 05:05:01.714617 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:01.714594    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-f14475-857d5b4f79-crdsn"
Sep 14 05:05:01.717139 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:01.717119    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-f14475-59bb95458-z5mxz"
Sep 14 05:05:01.754833 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:01.754799    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5fa6bb8-e10f-41af-9b37-962fad6aef8e-plugins-config-volume\") pod \"a5fa6bb8-e10f-41af-9b37-962fad6aef8e\" (UID: \"a5fa6bb8-e10f-41af-9b37-962fad6aef8e\") "
Sep 14 05:05:01.755011 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:01.754907    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j48dw\" (UniqueName: \"kubernetes.io/projected/a5fa6bb8-e10f-41af-9b37-962fad6aef8e-kube-api-access-j48dw\") pod \"a5fa6bb8-e10f-41af-9b37-962fad6aef8e\" (UID: \"a5fa6bb8-e10f-41af-9b37-962fad6aef8e\") "
Sep 14 05:05:01.755218 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:01.755196    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5fa6bb8-e10f-41af-9b37-962fad6aef8e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a5fa6bb8-e10f-41af-9b37-962fad6aef8e" (UID: "a5fa6bb8-e10f-41af-9b37-962fad6aef8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 05:05:01.757073 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:01.757043    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5fa6bb8-e10f-41af-9b37-962fad6aef8e-kube-api-access-j48dw" (OuterVolumeSpecName: "kube-api-access-j48dw") pod "a5fa6bb8-e10f-41af-9b37-962fad6aef8e" (UID: "a5fa6bb8-e10f-41af-9b37-962fad6aef8e"). InnerVolumeSpecName "kube-api-access-j48dw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:05:01.856252 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:01.856212    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2nvtq\" (UniqueName: \"kubernetes.io/projected/cc6a0970-e07a-4a2a-9232-57b8d402b0bb-kube-api-access-2nvtq\") pod \"cc6a0970-e07a-4a2a-9232-57b8d402b0bb\" (UID: \"cc6a0970-e07a-4a2a-9232-57b8d402b0bb\") "
Sep 14 05:05:01.856425 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:01.856328    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc6a0970-e07a-4a2a-9232-57b8d402b0bb-plugins-config-volume\") pod \"cc6a0970-e07a-4a2a-9232-57b8d402b0bb\" (UID: \"cc6a0970-e07a-4a2a-9232-57b8d402b0bb\") "
Sep 14 05:05:01.856501 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:01.856486    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5fa6bb8-e10f-41af-9b37-962fad6aef8e-plugins-config-volume\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:05:01.856541 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:01.856503    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j48dw\" (UniqueName: \"kubernetes.io/projected/a5fa6bb8-e10f-41af-9b37-962fad6aef8e-kube-api-access-j48dw\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:05:01.856720 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:01.856688    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc6a0970-e07a-4a2a-9232-57b8d402b0bb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cc6a0970-e07a-4a2a-9232-57b8d402b0bb" (UID: "cc6a0970-e07a-4a2a-9232-57b8d402b0bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 05:05:01.858337 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:01.858313    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc6a0970-e07a-4a2a-9232-57b8d402b0bb-kube-api-access-2nvtq" (OuterVolumeSpecName: "kube-api-access-2nvtq") pod "cc6a0970-e07a-4a2a-9232-57b8d402b0bb" (UID: "cc6a0970-e07a-4a2a-9232-57b8d402b0bb"). InnerVolumeSpecName "kube-api-access-2nvtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:05:01.957972 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:01.957876    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nvtq\" (UniqueName: \"kubernetes.io/projected/cc6a0970-e07a-4a2a-9232-57b8d402b0bb-kube-api-access-2nvtq\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:05:01.957972 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:01.957911    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc6a0970-e07a-4a2a-9232-57b8d402b0bb-plugins-config-volume\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:05:02.391394 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:02.391361    2565 generic.go:356] "Generic (PLEG): container finished" podID="a5fa6bb8-e10f-41af-9b37-962fad6aef8e" containerID="98da3ea55890a050c05816f0a4ab9ba5bc4a802101db5a2aba02ccc11153ef10" exitCode=0
Sep 14 05:05:02.391555 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:02.391407    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-f14475-857d5b4f79-crdsn"
Sep 14 05:05:02.391555 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:02.391447    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-f14475-857d5b4f79-crdsn" event={"ID":"a5fa6bb8-e10f-41af-9b37-962fad6aef8e","Type":"ContainerDied","Data":"98da3ea55890a050c05816f0a4ab9ba5bc4a802101db5a2aba02ccc11153ef10"}
Sep 14 05:05:02.391555 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:02.391505    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-f14475-857d5b4f79-crdsn" event={"ID":"a5fa6bb8-e10f-41af-9b37-962fad6aef8e","Type":"ContainerDied","Data":"692004580bcc195c42296643418aa9c670303647520a43a66e6040bd512815df"}
Sep 14 05:05:02.391555 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:02.391538    2565 scope.go:117] "RemoveContainer" containerID="98da3ea55890a050c05816f0a4ab9ba5bc4a802101db5a2aba02ccc11153ef10"
Sep 14 05:05:02.392886 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:02.392860    2565 generic.go:356] "Generic (PLEG): container finished" podID="cc6a0970-e07a-4a2a-9232-57b8d402b0bb" containerID="39447f692fb5245276fb446983dd3aa3f1f7c88486526fb128d3701c38449104" exitCode=0
Sep 14 05:05:02.392989 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:02.392887    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-f14475-59bb95458-z5mxz" event={"ID":"cc6a0970-e07a-4a2a-9232-57b8d402b0bb","Type":"ContainerDied","Data":"39447f692fb5245276fb446983dd3aa3f1f7c88486526fb128d3701c38449104"}
Sep 14 05:05:02.392989 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:02.392908    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-f14475-59bb95458-z5mxz"
Sep 14 05:05:02.392989 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:02.392928    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-f14475-59bb95458-z5mxz" event={"ID":"cc6a0970-e07a-4a2a-9232-57b8d402b0bb","Type":"ContainerDied","Data":"0c510eebd6a42d3c361dfd363476ac170aab085aef5a22eccd608dd569a986d9"}
Sep 14 05:05:02.400603 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:02.400586    2565 scope.go:117] "RemoveContainer" containerID="98da3ea55890a050c05816f0a4ab9ba5bc4a802101db5a2aba02ccc11153ef10"
Sep 14 05:05:02.400907 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:05:02.400887    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98da3ea55890a050c05816f0a4ab9ba5bc4a802101db5a2aba02ccc11153ef10\": container with ID starting with 98da3ea55890a050c05816f0a4ab9ba5bc4a802101db5a2aba02ccc11153ef10 not found: ID does not exist" containerID="98da3ea55890a050c05816f0a4ab9ba5bc4a802101db5a2aba02ccc11153ef10"
Sep 14 05:05:02.400963 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:02.400915    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98da3ea55890a050c05816f0a4ab9ba5bc4a802101db5a2aba02ccc11153ef10"} err="failed to get container status \"98da3ea55890a050c05816f0a4ab9ba5bc4a802101db5a2aba02ccc11153ef10\": rpc error: code = NotFound desc = could not find container \"98da3ea55890a050c05816f0a4ab9ba5bc4a802101db5a2aba02ccc11153ef10\": container with ID starting with 98da3ea55890a050c05816f0a4ab9ba5bc4a802101db5a2aba02ccc11153ef10 not found: ID does not exist"
Sep 14 05:05:02.400963 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:02.400931    2565 scope.go:117] "RemoveContainer" containerID="39447f692fb5245276fb446983dd3aa3f1f7c88486526fb128d3701c38449104"
Sep 14 05:05:02.408941 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:02.408923    2565 scope.go:117] "RemoveContainer" containerID="39447f692fb5245276fb446983dd3aa3f1f7c88486526fb128d3701c38449104"
Sep 14 05:05:02.409193 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:05:02.409173    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39447f692fb5245276fb446983dd3aa3f1f7c88486526fb128d3701c38449104\": container with ID starting with 39447f692fb5245276fb446983dd3aa3f1f7c88486526fb128d3701c38449104 not found: ID does not exist" containerID="39447f692fb5245276fb446983dd3aa3f1f7c88486526fb128d3701c38449104"
Sep 14 05:05:02.409240 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:02.409201    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39447f692fb5245276fb446983dd3aa3f1f7c88486526fb128d3701c38449104"} err="failed to get container status \"39447f692fb5245276fb446983dd3aa3f1f7c88486526fb128d3701c38449104\": rpc error: code = NotFound desc = could not find container \"39447f692fb5245276fb446983dd3aa3f1f7c88486526fb128d3701c38449104\": container with ID starting with 39447f692fb5245276fb446983dd3aa3f1f7c88486526fb128d3701c38449104 not found: ID does not exist"
Sep 14 05:05:02.409280 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:02.409259    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f14475-59bb95458-z5mxz"]
Sep 14 05:05:02.414349 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:02.414326    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f14475-59bb95458-z5mxz"]
Sep 14 05:05:02.427377 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:02.427349    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-f14475-857d5b4f79-crdsn"]
Sep 14 05:05:02.430312 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:02.430292    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-f14475-857d5b4f79-crdsn"]
Sep 14 05:05:04.334894 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:04.334858    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5fa6bb8-e10f-41af-9b37-962fad6aef8e" path="/var/lib/kubelet/pods/a5fa6bb8-e10f-41af-9b37-962fad6aef8e/volumes"
Sep 14 05:05:04.335242 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:04.335226    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc6a0970-e07a-4a2a-9232-57b8d402b0bb" path="/var/lib/kubelet/pods/cc6a0970-e07a-4a2a-9232-57b8d402b0bb/volumes"
Sep 14 05:05:05.190309 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:05.190275    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c366d3/simulator-w0-c366d3-kserve-9db75fd8d-45mgs"]
Sep 14 05:05:07.170434 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:07.170399    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-f14475/autores-model-f14475-kserve-855786fd99-b5k8p"]
Sep 14 05:05:07.722171 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:07.722133    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:05:11.180900 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:11.180865    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c366d3-668d8688df-dtphc"]
Sep 14 05:05:11.181345 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:11.181128    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-c366d3-668d8688df-dtphc" podUID="5feb03be-ccbd-4401-8d8e-95b8942e9270" containerName="payload-processing" containerID="cri-o://fe3673f24c33cf6a2e98521d830455b9b56b6ee4c3388db97f9cd5d8e5aa9219" gracePeriod=30
Sep 14 05:05:11.430636 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:11.430612    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c366d3-668d8688df-dtphc"
Sep 14 05:05:11.444798 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:11.444721    2565 generic.go:356] "Generic (PLEG): container finished" podID="5feb03be-ccbd-4401-8d8e-95b8942e9270" containerID="fe3673f24c33cf6a2e98521d830455b9b56b6ee4c3388db97f9cd5d8e5aa9219" exitCode=0
Sep 14 05:05:11.444798 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:11.444789    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c366d3-668d8688df-dtphc"
Sep 14 05:05:11.444945 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:11.444789    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c366d3-668d8688df-dtphc" event={"ID":"5feb03be-ccbd-4401-8d8e-95b8942e9270","Type":"ContainerDied","Data":"fe3673f24c33cf6a2e98521d830455b9b56b6ee4c3388db97f9cd5d8e5aa9219"}
Sep 14 05:05:11.444945 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:11.444906    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c366d3-668d8688df-dtphc" event={"ID":"5feb03be-ccbd-4401-8d8e-95b8942e9270","Type":"ContainerDied","Data":"4e0779c11500013ca09552f8b5aa6d27a4f906e562a4010e3759363002e735fb"}
Sep 14 05:05:11.444945 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:11.444933    2565 scope.go:117] "RemoveContainer" containerID="fe3673f24c33cf6a2e98521d830455b9b56b6ee4c3388db97f9cd5d8e5aa9219"
Sep 14 05:05:11.456143 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:11.456119    2565 scope.go:117] "RemoveContainer" containerID="fe3673f24c33cf6a2e98521d830455b9b56b6ee4c3388db97f9cd5d8e5aa9219"
Sep 14 05:05:11.456535 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:05:11.456515    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe3673f24c33cf6a2e98521d830455b9b56b6ee4c3388db97f9cd5d8e5aa9219\": container with ID starting with fe3673f24c33cf6a2e98521d830455b9b56b6ee4c3388db97f9cd5d8e5aa9219 not found: ID does not exist" containerID="fe3673f24c33cf6a2e98521d830455b9b56b6ee4c3388db97f9cd5d8e5aa9219"
Sep 14 05:05:11.456643 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:11.456543    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe3673f24c33cf6a2e98521d830455b9b56b6ee4c3388db97f9cd5d8e5aa9219"} err="failed to get container status \"fe3673f24c33cf6a2e98521d830455b9b56b6ee4c3388db97f9cd5d8e5aa9219\": rpc error: code = NotFound desc = could not find container \"fe3673f24c33cf6a2e98521d830455b9b56b6ee4c3388db97f9cd5d8e5aa9219\": container with ID starting with fe3673f24c33cf6a2e98521d830455b9b56b6ee4c3388db97f9cd5d8e5aa9219 not found: ID does not exist"
Sep 14 05:05:11.540938 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:11.540910    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57w26\" (UniqueName: \"kubernetes.io/projected/5feb03be-ccbd-4401-8d8e-95b8942e9270-kube-api-access-57w26\") pod \"5feb03be-ccbd-4401-8d8e-95b8942e9270\" (UID: \"5feb03be-ccbd-4401-8d8e-95b8942e9270\") "
Sep 14 05:05:11.541095 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:11.540957    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5feb03be-ccbd-4401-8d8e-95b8942e9270-plugins-config-volume\") pod \"5feb03be-ccbd-4401-8d8e-95b8942e9270\" (UID: \"5feb03be-ccbd-4401-8d8e-95b8942e9270\") "
Sep 14 05:05:11.541338 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:11.541301    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5feb03be-ccbd-4401-8d8e-95b8942e9270-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5feb03be-ccbd-4401-8d8e-95b8942e9270" (UID: "5feb03be-ccbd-4401-8d8e-95b8942e9270"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 05:05:11.543013 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:11.542987    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5feb03be-ccbd-4401-8d8e-95b8942e9270-kube-api-access-57w26" (OuterVolumeSpecName: "kube-api-access-57w26") pod "5feb03be-ccbd-4401-8d8e-95b8942e9270" (UID: "5feb03be-ccbd-4401-8d8e-95b8942e9270"). InnerVolumeSpecName "kube-api-access-57w26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:05:11.641695 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:11.641659    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5feb03be-ccbd-4401-8d8e-95b8942e9270-plugins-config-volume\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:05:11.641695 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:11.641698    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57w26\" (UniqueName: \"kubernetes.io/projected/5feb03be-ccbd-4401-8d8e-95b8942e9270-kube-api-access-57w26\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:05:11.766237 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:11.766204    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c366d3-668d8688df-dtphc"]
Sep 14 05:05:11.770028 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:11.769995    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c366d3-668d8688df-dtphc"]
Sep 14 05:05:12.335263 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:12.335233    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5feb03be-ccbd-4401-8d8e-95b8942e9270" path="/var/lib/kubelet/pods/5feb03be-ccbd-4401-8d8e-95b8942e9270/volumes"
Sep 14 05:05:20.209175 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:20.209129    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-c366d3/simulator-w0-c366d3-kserve-9db75fd8d-45mgs" podUID="471b3028-0304-4295-b5a7-b03ebb701d80" containerName="main" containerID="cri-o://5364942a03fb0509237af33d551c97987012d2954a1c2fbfada3787ba88b0f19" gracePeriod=45
Sep 14 05:05:20.468658 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:20.468599    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c366d3/simulator-w0-c366d3-kserve-9db75fd8d-45mgs"
Sep 14 05:05:20.496002 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:20.495970    2565 generic.go:356] "Generic (PLEG): container finished" podID="471b3028-0304-4295-b5a7-b03ebb701d80" containerID="5364942a03fb0509237af33d551c97987012d2954a1c2fbfada3787ba88b0f19" exitCode=0
Sep 14 05:05:20.496169 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:20.496031    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c366d3/simulator-w0-c366d3-kserve-9db75fd8d-45mgs"
Sep 14 05:05:20.496169 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:20.496072    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c366d3/simulator-w0-c366d3-kserve-9db75fd8d-45mgs" event={"ID":"471b3028-0304-4295-b5a7-b03ebb701d80","Type":"ContainerDied","Data":"5364942a03fb0509237af33d551c97987012d2954a1c2fbfada3787ba88b0f19"}
Sep 14 05:05:20.496169 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:20.496101    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c366d3/simulator-w0-c366d3-kserve-9db75fd8d-45mgs" event={"ID":"471b3028-0304-4295-b5a7-b03ebb701d80","Type":"ContainerDied","Data":"712efa2a1e031b824b3d598dbe433a5c745c35502b05762a2812717174a38199"}
Sep 14 05:05:20.496169 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:20.496123    2565 scope.go:117] "RemoveContainer" containerID="5364942a03fb0509237af33d551c97987012d2954a1c2fbfada3787ba88b0f19"
Sep 14 05:05:20.504703 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:20.504672    2565 scope.go:117] "RemoveContainer" containerID="5364942a03fb0509237af33d551c97987012d2954a1c2fbfada3787ba88b0f19"
Sep 14 05:05:20.505090 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:05:20.505066    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5364942a03fb0509237af33d551c97987012d2954a1c2fbfada3787ba88b0f19\": container with ID starting with 5364942a03fb0509237af33d551c97987012d2954a1c2fbfada3787ba88b0f19 not found: ID does not exist" containerID="5364942a03fb0509237af33d551c97987012d2954a1c2fbfada3787ba88b0f19"
Sep 14 05:05:20.505222 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:20.505100    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5364942a03fb0509237af33d551c97987012d2954a1c2fbfada3787ba88b0f19"} err="failed to get container status \"5364942a03fb0509237af33d551c97987012d2954a1c2fbfada3787ba88b0f19\": rpc error: code = NotFound desc = could not find container \"5364942a03fb0509237af33d551c97987012d2954a1c2fbfada3787ba88b0f19\": container with ID starting with 5364942a03fb0509237af33d551c97987012d2954a1c2fbfada3787ba88b0f19 not found: ID does not exist"
Sep 14 05:05:20.612294 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:20.612263    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/471b3028-0304-4295-b5a7-b03ebb701d80-model-cache\") pod \"471b3028-0304-4295-b5a7-b03ebb701d80\" (UID: \"471b3028-0304-4295-b5a7-b03ebb701d80\") "
Sep 14 05:05:20.612466 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:20.612312    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4vds\" (UniqueName: \"kubernetes.io/projected/471b3028-0304-4295-b5a7-b03ebb701d80-kube-api-access-h4vds\") pod \"471b3028-0304-4295-b5a7-b03ebb701d80\" (UID: \"471b3028-0304-4295-b5a7-b03ebb701d80\") "
Sep 14 05:05:20.612466 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:20.612333    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/471b3028-0304-4295-b5a7-b03ebb701d80-tmp-dir\") pod \"471b3028-0304-4295-b5a7-b03ebb701d80\" (UID: \"471b3028-0304-4295-b5a7-b03ebb701d80\") "
Sep 14 05:05:20.612466 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:20.612377    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/471b3028-0304-4295-b5a7-b03ebb701d80-dshm\") pod \"471b3028-0304-4295-b5a7-b03ebb701d80\" (UID: \"471b3028-0304-4295-b5a7-b03ebb701d80\") "
Sep 14 05:05:20.612638 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:20.612496    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/471b3028-0304-4295-b5a7-b03ebb701d80-home\") pod \"471b3028-0304-4295-b5a7-b03ebb701d80\" (UID: \"471b3028-0304-4295-b5a7-b03ebb701d80\") "
Sep 14 05:05:20.612638 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:20.612532    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/471b3028-0304-4295-b5a7-b03ebb701d80-tls-certs\") pod \"471b3028-0304-4295-b5a7-b03ebb701d80\" (UID: \"471b3028-0304-4295-b5a7-b03ebb701d80\") "
Sep 14 05:05:20.612638 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:20.612581    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/471b3028-0304-4295-b5a7-b03ebb701d80-model-cache" (OuterVolumeSpecName: "model-cache") pod "471b3028-0304-4295-b5a7-b03ebb701d80" (UID: "471b3028-0304-4295-b5a7-b03ebb701d80"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:05:20.612638 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:20.612597    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/471b3028-0304-4295-b5a7-b03ebb701d80-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "471b3028-0304-4295-b5a7-b03ebb701d80" (UID: "471b3028-0304-4295-b5a7-b03ebb701d80"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:05:20.612875 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:20.612806    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/471b3028-0304-4295-b5a7-b03ebb701d80-model-cache\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:05:20.612875 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:20.612804    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/471b3028-0304-4295-b5a7-b03ebb701d80-home" (OuterVolumeSpecName: "home") pod "471b3028-0304-4295-b5a7-b03ebb701d80" (UID: "471b3028-0304-4295-b5a7-b03ebb701d80"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:05:20.612875 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:20.612819    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/471b3028-0304-4295-b5a7-b03ebb701d80-tmp-dir\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:05:20.614618 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:20.614586    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/471b3028-0304-4295-b5a7-b03ebb701d80-kube-api-access-h4vds" (OuterVolumeSpecName: "kube-api-access-h4vds") pod "471b3028-0304-4295-b5a7-b03ebb701d80" (UID: "471b3028-0304-4295-b5a7-b03ebb701d80"). InnerVolumeSpecName "kube-api-access-h4vds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:05:20.614618 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:20.614584    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/471b3028-0304-4295-b5a7-b03ebb701d80-dshm" (OuterVolumeSpecName: "dshm") pod "471b3028-0304-4295-b5a7-b03ebb701d80" (UID: "471b3028-0304-4295-b5a7-b03ebb701d80"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:05:20.614791 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:20.614614    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/471b3028-0304-4295-b5a7-b03ebb701d80-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "471b3028-0304-4295-b5a7-b03ebb701d80" (UID: "471b3028-0304-4295-b5a7-b03ebb701d80"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 05:05:20.714012 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:20.713979    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/471b3028-0304-4295-b5a7-b03ebb701d80-home\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:05:20.714012 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:20.714006    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/471b3028-0304-4295-b5a7-b03ebb701d80-tls-certs\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:05:20.714012 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:20.714019    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4vds\" (UniqueName: \"kubernetes.io/projected/471b3028-0304-4295-b5a7-b03ebb701d80-kube-api-access-h4vds\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:05:20.714226 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:20.714034    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/471b3028-0304-4295-b5a7-b03ebb701d80-dshm\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:05:20.824711 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:20.824684    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c366d3/simulator-w0-c366d3-kserve-9db75fd8d-45mgs"]
Sep 14 05:05:20.828539 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:20.828507    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-c366d3/simulator-w0-c366d3-kserve-9db75fd8d-45mgs"]
Sep 14 05:05:22.189513 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:22.189449    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-f14475/autores-model-f14475-kserve-855786fd99-b5k8p" podUID="bba8a0f2-6895-4749-8098-19f9154aca6f" containerName="main" containerID="cri-o://db4e70d7057542fbae79b6bedf673bb7d5e43613b6a25c042fff0dca1757e5c6" gracePeriod=45
Sep 14 05:05:22.336322 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:22.336286    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="471b3028-0304-4295-b5a7-b03ebb701d80" path="/var/lib/kubelet/pods/471b3028-0304-4295-b5a7-b03ebb701d80/volumes"
Sep 14 05:05:22.446889 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:22.446836    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-f14475/autores-model-f14475-kserve-855786fd99-b5k8p"
Sep 14 05:05:22.507745 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:22.507714    2565 generic.go:356] "Generic (PLEG): container finished" podID="bba8a0f2-6895-4749-8098-19f9154aca6f" containerID="db4e70d7057542fbae79b6bedf673bb7d5e43613b6a25c042fff0dca1757e5c6" exitCode=0
Sep 14 05:05:22.507916 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:22.507767    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-f14475/autores-model-f14475-kserve-855786fd99-b5k8p"
Sep 14 05:05:22.507916 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:22.507783    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-f14475/autores-model-f14475-kserve-855786fd99-b5k8p" event={"ID":"bba8a0f2-6895-4749-8098-19f9154aca6f","Type":"ContainerDied","Data":"db4e70d7057542fbae79b6bedf673bb7d5e43613b6a25c042fff0dca1757e5c6"}
Sep 14 05:05:22.507916 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:22.507814    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-f14475/autores-model-f14475-kserve-855786fd99-b5k8p" event={"ID":"bba8a0f2-6895-4749-8098-19f9154aca6f","Type":"ContainerDied","Data":"7fad7111cc176aa07dc6a88600651acf76f39c74bb758f6b8dd3711251554844"}
Sep 14 05:05:22.507916 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:22.507830    2565 scope.go:117] "RemoveContainer" containerID="db4e70d7057542fbae79b6bedf673bb7d5e43613b6a25c042fff0dca1757e5c6"
Sep 14 05:05:22.516833 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:22.516791    2565 scope.go:117] "RemoveContainer" containerID="db4e70d7057542fbae79b6bedf673bb7d5e43613b6a25c042fff0dca1757e5c6"
Sep 14 05:05:22.517078 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:05:22.517058    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db4e70d7057542fbae79b6bedf673bb7d5e43613b6a25c042fff0dca1757e5c6\": container with ID starting with db4e70d7057542fbae79b6bedf673bb7d5e43613b6a25c042fff0dca1757e5c6 not found: ID does not exist" containerID="db4e70d7057542fbae79b6bedf673bb7d5e43613b6a25c042fff0dca1757e5c6"
Sep 14 05:05:22.517134 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:22.517086    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db4e70d7057542fbae79b6bedf673bb7d5e43613b6a25c042fff0dca1757e5c6"} err="failed to get container status \"db4e70d7057542fbae79b6bedf673bb7d5e43613b6a25c042fff0dca1757e5c6\": rpc error: code = NotFound desc = could not find container \"db4e70d7057542fbae79b6bedf673bb7d5e43613b6a25c042fff0dca1757e5c6\": container with ID starting with db4e70d7057542fbae79b6bedf673bb7d5e43613b6a25c042fff0dca1757e5c6 not found: ID does not exist"
Sep 14 05:05:22.530915 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:22.530894    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bba8a0f2-6895-4749-8098-19f9154aca6f-tls-certs\") pod \"bba8a0f2-6895-4749-8098-19f9154aca6f\" (UID: \"bba8a0f2-6895-4749-8098-19f9154aca6f\") "
Sep 14 05:05:22.531007 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:22.530926    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bba8a0f2-6895-4749-8098-19f9154aca6f-tmp-dir\") pod \"bba8a0f2-6895-4749-8098-19f9154aca6f\" (UID: \"bba8a0f2-6895-4749-8098-19f9154aca6f\") "
Sep 14 05:05:22.531007 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:22.530978    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcdbs\" (UniqueName: \"kubernetes.io/projected/bba8a0f2-6895-4749-8098-19f9154aca6f-kube-api-access-jcdbs\") pod \"bba8a0f2-6895-4749-8098-19f9154aca6f\" (UID: \"bba8a0f2-6895-4749-8098-19f9154aca6f\") "
Sep 14 05:05:22.531007 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:22.531001    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bba8a0f2-6895-4749-8098-19f9154aca6f-dshm\") pod \"bba8a0f2-6895-4749-8098-19f9154aca6f\" (UID: \"bba8a0f2-6895-4749-8098-19f9154aca6f\") "
Sep 14 05:05:22.531177 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:22.531042    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bba8a0f2-6895-4749-8098-19f9154aca6f-model-cache\") pod \"bba8a0f2-6895-4749-8098-19f9154aca6f\" (UID: \"bba8a0f2-6895-4749-8098-19f9154aca6f\") "
Sep 14 05:05:22.531177 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:22.531130    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bba8a0f2-6895-4749-8098-19f9154aca6f-home\") pod \"bba8a0f2-6895-4749-8098-19f9154aca6f\" (UID: \"bba8a0f2-6895-4749-8098-19f9154aca6f\") "
Sep 14 05:05:22.531311 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:22.531258    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba8a0f2-6895-4749-8098-19f9154aca6f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bba8a0f2-6895-4749-8098-19f9154aca6f" (UID: "bba8a0f2-6895-4749-8098-19f9154aca6f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:05:22.531373 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:22.531331    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba8a0f2-6895-4749-8098-19f9154aca6f-model-cache" (OuterVolumeSpecName: "model-cache") pod "bba8a0f2-6895-4749-8098-19f9154aca6f" (UID: "bba8a0f2-6895-4749-8098-19f9154aca6f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:05:22.531441 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:22.531424    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bba8a0f2-6895-4749-8098-19f9154aca6f-model-cache\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:05:22.531503 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:22.531444    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bba8a0f2-6895-4749-8098-19f9154aca6f-tmp-dir\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:05:22.531503 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:22.531470    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba8a0f2-6895-4749-8098-19f9154aca6f-home" (OuterVolumeSpecName: "home") pod "bba8a0f2-6895-4749-8098-19f9154aca6f" (UID: "bba8a0f2-6895-4749-8098-19f9154aca6f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:05:22.533169 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:22.533148    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bba8a0f2-6895-4749-8098-19f9154aca6f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bba8a0f2-6895-4749-8098-19f9154aca6f" (UID: "bba8a0f2-6895-4749-8098-19f9154aca6f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 05:05:22.533260 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:22.533181    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba8a0f2-6895-4749-8098-19f9154aca6f-dshm" (OuterVolumeSpecName: "dshm") pod "bba8a0f2-6895-4749-8098-19f9154aca6f" (UID: "bba8a0f2-6895-4749-8098-19f9154aca6f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:05:22.533260 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:22.533181    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bba8a0f2-6895-4749-8098-19f9154aca6f-kube-api-access-jcdbs" (OuterVolumeSpecName: "kube-api-access-jcdbs") pod "bba8a0f2-6895-4749-8098-19f9154aca6f" (UID: "bba8a0f2-6895-4749-8098-19f9154aca6f"). InnerVolumeSpecName "kube-api-access-jcdbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:05:22.632238 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:22.632205    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bba8a0f2-6895-4749-8098-19f9154aca6f-home\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:05:22.632238 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:22.632233    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bba8a0f2-6895-4749-8098-19f9154aca6f-tls-certs\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:05:22.632238 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:22.632245    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcdbs\" (UniqueName: \"kubernetes.io/projected/bba8a0f2-6895-4749-8098-19f9154aca6f-kube-api-access-jcdbs\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:05:22.632463 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:22.632261    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bba8a0f2-6895-4749-8098-19f9154aca6f-dshm\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:05:22.829992 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:22.829964    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-f14475/autores-model-f14475-kserve-855786fd99-b5k8p"]
Sep 14 05:05:22.833228 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:22.833203    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-f14475/autores-model-f14475-kserve-855786fd99-b5k8p"]
Sep 14 05:05:24.333801 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:24.333768    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bba8a0f2-6895-4749-8098-19f9154aca6f" path="/var/lib/kubelet/pods/bba8a0f2-6895-4749-8098-19f9154aca6f/volumes"
Sep 14 05:05:25.622845 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:25.622806    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:05:43.331816 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:43.331777    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:05:44.736882 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:44.736848    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"]
Sep 14 05:05:44.737304 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:44.737056    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9" podUID="38ea6dfe-3dce-404e-988c-8b3d8f912061" containerName="istio-proxy" containerID="cri-o://bbb9cbc907fba4b0cc609308f526156a19da61b2b51b47a1326345db0bc7322a" gracePeriod=30
Sep 14 05:05:45.529139 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.529107    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"]
Sep 14 05:05:45.529447 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.529433    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5feb03be-ccbd-4401-8d8e-95b8942e9270" containerName="payload-processing"
Sep 14 05:05:45.529520 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.529447    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5feb03be-ccbd-4401-8d8e-95b8942e9270" containerName="payload-processing"
Sep 14 05:05:45.529520 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.529458    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc6a0970-e07a-4a2a-9232-57b8d402b0bb" containerName="payload-pre-processing"
Sep 14 05:05:45.529520 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.529464    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc6a0970-e07a-4a2a-9232-57b8d402b0bb" containerName="payload-pre-processing"
Sep 14 05:05:45.529520 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.529475    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5fa6bb8-e10f-41af-9b37-962fad6aef8e" containerName="payload-processing"
Sep 14 05:05:45.529520 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.529481    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5fa6bb8-e10f-41af-9b37-962fad6aef8e" containerName="payload-processing"
Sep 14 05:05:45.529520 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.529487    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="471b3028-0304-4295-b5a7-b03ebb701d80" containerName="main"
Sep 14 05:05:45.529520 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.529493    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="471b3028-0304-4295-b5a7-b03ebb701d80" containerName="main"
Sep 14 05:05:45.529520 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.529505    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bba8a0f2-6895-4749-8098-19f9154aca6f" containerName="main"
Sep 14 05:05:45.529520 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.529510    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bba8a0f2-6895-4749-8098-19f9154aca6f" containerName="main"
Sep 14 05:05:45.529982 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.529559    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5fa6bb8-e10f-41af-9b37-962fad6aef8e" containerName="payload-processing"
Sep 14 05:05:45.529982 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.529566    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5feb03be-ccbd-4401-8d8e-95b8942e9270" containerName="payload-processing"
Sep 14 05:05:45.529982 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.529574    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="471b3028-0304-4295-b5a7-b03ebb701d80" containerName="main"
Sep 14 05:05:45.529982 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.529580    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc6a0970-e07a-4a2a-9232-57b8d402b0bb" containerName="payload-pre-processing"
Sep 14 05:05:45.529982 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.529586    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bba8a0f2-6895-4749-8098-19f9154aca6f" containerName="main"
Sep 14 05:05:45.533866 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.533846    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"
Sep 14 05:05:45.536178 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.536158    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-9529a1-openshift-default-dockercfg-cbfw4\""
Sep 14 05:05:45.550474 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.550439    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"]
Sep 14 05:05:45.725183 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.725144    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58465fe-e115-4ba7-be72-4d2b17957d90-credential-socket\") pod \"e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr\" (UID: \"f58465fe-e115-4ba7-be72-4d2b17957d90\") " pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"
Sep 14 05:05:45.725354 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.725212    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f58465fe-e115-4ba7-be72-4d2b17957d90-istio-podinfo\") pod \"e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr\" (UID: \"f58465fe-e115-4ba7-be72-4d2b17957d90\") " pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"
Sep 14 05:05:45.725354 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.725259    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f58465fe-e115-4ba7-be72-4d2b17957d90-istio-envoy\") pod \"e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr\" (UID: \"f58465fe-e115-4ba7-be72-4d2b17957d90\") " pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"
Sep 14 05:05:45.725354 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.725284    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f58465fe-e115-4ba7-be72-4d2b17957d90-workload-certs\") pod \"e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr\" (UID: \"f58465fe-e115-4ba7-be72-4d2b17957d90\") " pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"
Sep 14 05:05:45.725354 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.725299    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f58465fe-e115-4ba7-be72-4d2b17957d90-istio-data\") pod \"e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr\" (UID: \"f58465fe-e115-4ba7-be72-4d2b17957d90\") " pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"
Sep 14 05:05:45.725354 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.725318    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f58465fe-e115-4ba7-be72-4d2b17957d90-istiod-ca-cert\") pod \"e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr\" (UID: \"f58465fe-e115-4ba7-be72-4d2b17957d90\") " pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"
Sep 14 05:05:45.725354 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.725333    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nm56\" (UniqueName: \"kubernetes.io/projected/f58465fe-e115-4ba7-be72-4d2b17957d90-kube-api-access-2nm56\") pod \"e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr\" (UID: \"f58465fe-e115-4ba7-be72-4d2b17957d90\") " pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"
Sep 14 05:05:45.725649 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.725372    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58465fe-e115-4ba7-be72-4d2b17957d90-workload-socket\") pod \"e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr\" (UID: \"f58465fe-e115-4ba7-be72-4d2b17957d90\") " pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"
Sep 14 05:05:45.725649 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.725493    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f58465fe-e115-4ba7-be72-4d2b17957d90-istio-token\") pod \"e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr\" (UID: \"f58465fe-e115-4ba7-be72-4d2b17957d90\") " pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"
Sep 14 05:05:45.826526 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.826434    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58465fe-e115-4ba7-be72-4d2b17957d90-credential-socket\") pod \"e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr\" (UID: \"f58465fe-e115-4ba7-be72-4d2b17957d90\") " pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"
Sep 14 05:05:45.826526 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.826490    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f58465fe-e115-4ba7-be72-4d2b17957d90-istio-podinfo\") pod \"e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr\" (UID: \"f58465fe-e115-4ba7-be72-4d2b17957d90\") " pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"
Sep 14 05:05:45.827014 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.826615    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f58465fe-e115-4ba7-be72-4d2b17957d90-istio-envoy\") pod \"e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr\" (UID: \"f58465fe-e115-4ba7-be72-4d2b17957d90\") " pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"
Sep 14 05:05:45.827014 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.826656    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f58465fe-e115-4ba7-be72-4d2b17957d90-workload-certs\") pod \"e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr\" (UID: \"f58465fe-e115-4ba7-be72-4d2b17957d90\") " pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"
Sep 14 05:05:45.827014 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.826685    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f58465fe-e115-4ba7-be72-4d2b17957d90-istio-data\") pod \"e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr\" (UID: \"f58465fe-e115-4ba7-be72-4d2b17957d90\") " pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"
Sep 14 05:05:45.827014 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.826717    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f58465fe-e115-4ba7-be72-4d2b17957d90-istiod-ca-cert\") pod \"e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr\" (UID: \"f58465fe-e115-4ba7-be72-4d2b17957d90\") " pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"
Sep 14 05:05:45.827014 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.826742    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nm56\" (UniqueName: \"kubernetes.io/projected/f58465fe-e115-4ba7-be72-4d2b17957d90-kube-api-access-2nm56\") pod \"e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr\" (UID: \"f58465fe-e115-4ba7-be72-4d2b17957d90\") " pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"
Sep 14 05:05:45.827014 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.826792    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58465fe-e115-4ba7-be72-4d2b17957d90-workload-socket\") pod \"e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr\" (UID: \"f58465fe-e115-4ba7-be72-4d2b17957d90\") " pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"
Sep 14 05:05:45.827014 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.826832    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f58465fe-e115-4ba7-be72-4d2b17957d90-istio-token\") pod \"e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr\" (UID: \"f58465fe-e115-4ba7-be72-4d2b17957d90\") " pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"
Sep 14 05:05:45.827014 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.826917    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58465fe-e115-4ba7-be72-4d2b17957d90-credential-socket\") pod \"e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr\" (UID: \"f58465fe-e115-4ba7-be72-4d2b17957d90\") " pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"
Sep 14 05:05:45.827397 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.827374    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f58465fe-e115-4ba7-be72-4d2b17957d90-istio-data\") pod \"e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr\" (UID: \"f58465fe-e115-4ba7-be72-4d2b17957d90\") " pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"
Sep 14 05:05:45.827478 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.827433    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f58465fe-e115-4ba7-be72-4d2b17957d90-workload-certs\") pod \"e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr\" (UID: \"f58465fe-e115-4ba7-be72-4d2b17957d90\") " pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"
Sep 14 05:05:45.827478 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.827440    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58465fe-e115-4ba7-be72-4d2b17957d90-workload-socket\") pod \"e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr\" (UID: \"f58465fe-e115-4ba7-be72-4d2b17957d90\") " pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"
Sep 14 05:05:45.827905 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.827880    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f58465fe-e115-4ba7-be72-4d2b17957d90-istiod-ca-cert\") pod \"e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr\" (UID: \"f58465fe-e115-4ba7-be72-4d2b17957d90\") " pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"
Sep 14 05:05:45.829359 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.829337    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f58465fe-e115-4ba7-be72-4d2b17957d90-istio-envoy\") pod \"e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr\" (UID: \"f58465fe-e115-4ba7-be72-4d2b17957d90\") " pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"
Sep 14 05:05:45.829473 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.829457    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f58465fe-e115-4ba7-be72-4d2b17957d90-istio-podinfo\") pod \"e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr\" (UID: \"f58465fe-e115-4ba7-be72-4d2b17957d90\") " pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"
Sep 14 05:05:45.834643 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.834616    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f58465fe-e115-4ba7-be72-4d2b17957d90-istio-token\") pod \"e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr\" (UID: \"f58465fe-e115-4ba7-be72-4d2b17957d90\") " pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"
Sep 14 05:05:45.835492 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.835466    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nm56\" (UniqueName: \"kubernetes.io/projected/f58465fe-e115-4ba7-be72-4d2b17957d90-kube-api-access-2nm56\") pod \"e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr\" (UID: \"f58465fe-e115-4ba7-be72-4d2b17957d90\") " pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"
Sep 14 05:05:45.847225 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:45.847204    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"
Sep 14 05:05:46.120586 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:46.120560    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"]
Sep 14 05:05:46.121989 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:05:46.121963    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf58465fe_e115_4ba7_be72_4d2b17957d90.slice/crio-3fd6759af74bfb68fe3cea0747088facf1293630f1e542f5fc5241f97bb0f4da WatchSource:0}: Error finding container 3fd6759af74bfb68fe3cea0747088facf1293630f1e542f5fc5241f97bb0f4da: Status 404 returned error can't find the container with id 3fd6759af74bfb68fe3cea0747088facf1293630f1e542f5fc5241f97bb0f4da
Sep 14 05:05:46.124202 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:46.124165    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:05:46.124303 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:46.124236    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:05:46.124303 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:46.124272    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:05:46.636907 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:46.636874    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr" event={"ID":"f58465fe-e115-4ba7-be72-4d2b17957d90","Type":"ContainerStarted","Data":"41a0873129427c2f709d6f4e0947024e1355eb227ef25bcfc81aeac870fe056c"}
Sep 14 05:05:46.637085 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:46.636928    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr" event={"ID":"f58465fe-e115-4ba7-be72-4d2b17957d90","Type":"ContainerStarted","Data":"3fd6759af74bfb68fe3cea0747088facf1293630f1e542f5fc5241f97bb0f4da"}
Sep 14 05:05:46.657784 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:46.657719    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr" podStartSLOduration=1.657703639 podStartE2EDuration="1.657703639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:05:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:05:46.656605257 +0000 UTC m=+1144.832376814" watchObservedRunningTime="2026-09-14 05:05:46.657703639 +0000 UTC m=+1144.833475268"
Sep 14 05:05:46.848003 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:46.847976    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"
Sep 14 05:05:47.853469 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:47.853437    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"
Sep 14 05:05:48.647920 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:48.647893    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"
Sep 14 05:05:48.648991 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:48.648971    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"
Sep 14 05:05:49.980787 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:49.980743    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"
Sep 14 05:05:50.061367 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:50.061280    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38ea6dfe-3dce-404e-988c-8b3d8f912061-istio-podinfo\") pod \"38ea6dfe-3dce-404e-988c-8b3d8f912061\" (UID: \"38ea6dfe-3dce-404e-988c-8b3d8f912061\") "
Sep 14 05:05:50.061367 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:50.061355    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38ea6dfe-3dce-404e-988c-8b3d8f912061-istio-data\") pod \"38ea6dfe-3dce-404e-988c-8b3d8f912061\" (UID: \"38ea6dfe-3dce-404e-988c-8b3d8f912061\") "
Sep 14 05:05:50.061593 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:50.061376    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38ea6dfe-3dce-404e-988c-8b3d8f912061-istio-envoy\") pod \"38ea6dfe-3dce-404e-988c-8b3d8f912061\" (UID: \"38ea6dfe-3dce-404e-988c-8b3d8f912061\") "
Sep 14 05:05:50.061593 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:50.061405    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ea6dfe-3dce-404e-988c-8b3d8f912061-workload-socket\") pod \"38ea6dfe-3dce-404e-988c-8b3d8f912061\" (UID: \"38ea6dfe-3dce-404e-988c-8b3d8f912061\") "
Sep 14 05:05:50.061593 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:50.061424    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38ea6dfe-3dce-404e-988c-8b3d8f912061-workload-certs\") pod \"38ea6dfe-3dce-404e-988c-8b3d8f912061\" (UID: \"38ea6dfe-3dce-404e-988c-8b3d8f912061\") "
Sep 14 05:05:50.061593 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:50.061445    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ea6dfe-3dce-404e-988c-8b3d8f912061-credential-socket\") pod \"38ea6dfe-3dce-404e-988c-8b3d8f912061\" (UID: \"38ea6dfe-3dce-404e-988c-8b3d8f912061\") "
Sep 14 05:05:50.061593 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:50.061476    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38ea6dfe-3dce-404e-988c-8b3d8f912061-istio-token\") pod \"38ea6dfe-3dce-404e-988c-8b3d8f912061\" (UID: \"38ea6dfe-3dce-404e-988c-8b3d8f912061\") "
Sep 14 05:05:50.061593 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:50.061565    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xlbwk\" (UniqueName: \"kubernetes.io/projected/38ea6dfe-3dce-404e-988c-8b3d8f912061-kube-api-access-xlbwk\") pod \"38ea6dfe-3dce-404e-988c-8b3d8f912061\" (UID: \"38ea6dfe-3dce-404e-988c-8b3d8f912061\") "
Sep 14 05:05:50.061963 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:50.061943    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38ea6dfe-3dce-404e-988c-8b3d8f912061-istiod-ca-cert\") pod \"38ea6dfe-3dce-404e-988c-8b3d8f912061\" (UID: \"38ea6dfe-3dce-404e-988c-8b3d8f912061\") "
Sep 14 05:05:50.062024 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:50.061733    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ea6dfe-3dce-404e-988c-8b3d8f912061-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "38ea6dfe-3dce-404e-988c-8b3d8f912061" (UID: "38ea6dfe-3dce-404e-988c-8b3d8f912061"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:05:50.062024 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:50.061751    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ea6dfe-3dce-404e-988c-8b3d8f912061-istio-data" (OuterVolumeSpecName: "istio-data") pod "38ea6dfe-3dce-404e-988c-8b3d8f912061" (UID: "38ea6dfe-3dce-404e-988c-8b3d8f912061"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:05:50.062133 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:50.061795    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ea6dfe-3dce-404e-988c-8b3d8f912061-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "38ea6dfe-3dce-404e-988c-8b3d8f912061" (UID: "38ea6dfe-3dce-404e-988c-8b3d8f912061"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:05:50.062133 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:50.061808    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ea6dfe-3dce-404e-988c-8b3d8f912061-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "38ea6dfe-3dce-404e-988c-8b3d8f912061" (UID: "38ea6dfe-3dce-404e-988c-8b3d8f912061"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:05:50.062320 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:50.062240    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ea6dfe-3dce-404e-988c-8b3d8f912061-workload-socket\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:05:50.062494 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:50.062473    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38ea6dfe-3dce-404e-988c-8b3d8f912061-workload-certs\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:05:50.062604 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:50.062498    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ea6dfe-3dce-404e-988c-8b3d8f912061-credential-socket\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:05:50.062604 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:50.062514    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38ea6dfe-3dce-404e-988c-8b3d8f912061-istio-data\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:05:50.062604 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:50.062271    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38ea6dfe-3dce-404e-988c-8b3d8f912061-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "38ea6dfe-3dce-404e-988c-8b3d8f912061" (UID: "38ea6dfe-3dce-404e-988c-8b3d8f912061"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 05:05:50.063549 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:50.063525    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38ea6dfe-3dce-404e-988c-8b3d8f912061-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "38ea6dfe-3dce-404e-988c-8b3d8f912061" (UID: "38ea6dfe-3dce-404e-988c-8b3d8f912061"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 05:05:50.063641 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:50.063595    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ea6dfe-3dce-404e-988c-8b3d8f912061-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "38ea6dfe-3dce-404e-988c-8b3d8f912061" (UID: "38ea6dfe-3dce-404e-988c-8b3d8f912061"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:05:50.063740 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:50.063720    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38ea6dfe-3dce-404e-988c-8b3d8f912061-istio-token" (OuterVolumeSpecName: "istio-token") pod "38ea6dfe-3dce-404e-988c-8b3d8f912061" (UID: "38ea6dfe-3dce-404e-988c-8b3d8f912061"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:05:50.063820 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:50.063800    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38ea6dfe-3dce-404e-988c-8b3d8f912061-kube-api-access-xlbwk" (OuterVolumeSpecName: "kube-api-access-xlbwk") pod "38ea6dfe-3dce-404e-988c-8b3d8f912061" (UID: "38ea6dfe-3dce-404e-988c-8b3d8f912061"). InnerVolumeSpecName "kube-api-access-xlbwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:05:50.163530 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:50.163489    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38ea6dfe-3dce-404e-988c-8b3d8f912061-istiod-ca-cert\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:05:50.163530 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:50.163520    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38ea6dfe-3dce-404e-988c-8b3d8f912061-istio-podinfo\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:05:50.163530 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:50.163528    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38ea6dfe-3dce-404e-988c-8b3d8f912061-istio-envoy\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:05:50.163530 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:50.163537    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38ea6dfe-3dce-404e-988c-8b3d8f912061-istio-token\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:05:50.163835 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:50.163546    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlbwk\" (UniqueName: \"kubernetes.io/projected/38ea6dfe-3dce-404e-988c-8b3d8f912061-kube-api-access-xlbwk\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:05:50.665599 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:50.665505    2565 generic.go:356] "Generic (PLEG): container finished" podID="38ea6dfe-3dce-404e-988c-8b3d8f912061" containerID="bbb9cbc907fba4b0cc609308f526156a19da61b2b51b47a1326345db0bc7322a" exitCode=0
Sep 14 05:05:50.665599 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:50.665551    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"
Sep 14 05:05:50.665599 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:50.665547    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9" event={"ID":"38ea6dfe-3dce-404e-988c-8b3d8f912061","Type":"ContainerDied","Data":"bbb9cbc907fba4b0cc609308f526156a19da61b2b51b47a1326345db0bc7322a"}
Sep 14 05:05:50.666079 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:50.665624    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9" event={"ID":"38ea6dfe-3dce-404e-988c-8b3d8f912061","Type":"ContainerDied","Data":"e8c3869a8af64b72e37685016c37439bbcc84912339d004fb1aded14a7236cfa"}
Sep 14 05:05:50.666079 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:50.665646    2565 scope.go:117] "RemoveContainer" containerID="bbb9cbc907fba4b0cc609308f526156a19da61b2b51b47a1326345db0bc7322a"
Sep 14 05:05:50.674704 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:50.674681    2565 scope.go:117] "RemoveContainer" containerID="ab07c93f774034c727d587b22c01c02ec9cc6d1f897e8c3fbb65960d1fc1fd4f"
Sep 14 05:05:50.683340 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:50.683324    2565 scope.go:117] "RemoveContainer" containerID="bbb9cbc907fba4b0cc609308f526156a19da61b2b51b47a1326345db0bc7322a"
Sep 14 05:05:50.683484 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:50.683462    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"]
Sep 14 05:05:50.683584 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:05:50.683565    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbb9cbc907fba4b0cc609308f526156a19da61b2b51b47a1326345db0bc7322a\": container with ID starting with bbb9cbc907fba4b0cc609308f526156a19da61b2b51b47a1326345db0bc7322a not found: ID does not exist" containerID="bbb9cbc907fba4b0cc609308f526156a19da61b2b51b47a1326345db0bc7322a"
Sep 14 05:05:50.683629 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:50.683594    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbb9cbc907fba4b0cc609308f526156a19da61b2b51b47a1326345db0bc7322a"} err="failed to get container status \"bbb9cbc907fba4b0cc609308f526156a19da61b2b51b47a1326345db0bc7322a\": rpc error: code = NotFound desc = could not find container \"bbb9cbc907fba4b0cc609308f526156a19da61b2b51b47a1326345db0bc7322a\": container with ID starting with bbb9cbc907fba4b0cc609308f526156a19da61b2b51b47a1326345db0bc7322a not found: ID does not exist"
Sep 14 05:05:50.683629 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:50.683612    2565 scope.go:117] "RemoveContainer" containerID="ab07c93f774034c727d587b22c01c02ec9cc6d1f897e8c3fbb65960d1fc1fd4f"
Sep 14 05:05:50.683871 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:05:50.683854    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab07c93f774034c727d587b22c01c02ec9cc6d1f897e8c3fbb65960d1fc1fd4f\": container with ID starting with ab07c93f774034c727d587b22c01c02ec9cc6d1f897e8c3fbb65960d1fc1fd4f not found: ID does not exist" containerID="ab07c93f774034c727d587b22c01c02ec9cc6d1f897e8c3fbb65960d1fc1fd4f"
Sep 14 05:05:50.683923 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:50.683875    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab07c93f774034c727d587b22c01c02ec9cc6d1f897e8c3fbb65960d1fc1fd4f"} err="failed to get container status \"ab07c93f774034c727d587b22c01c02ec9cc6d1f897e8c3fbb65960d1fc1fd4f\": rpc error: code = NotFound desc = could not find container \"ab07c93f774034c727d587b22c01c02ec9cc6d1f897e8c3fbb65960d1fc1fd4f\": container with ID starting with ab07c93f774034c727d587b22c01c02ec9cc6d1f897e8c3fbb65960d1fc1fd4f not found: ID does not exist"
Sep 14 05:05:50.688292 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:50.688270    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-f14475-openshift-default-96796fcc6-bfzx9"]
Sep 14 05:05:51.866112 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:51.866081    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-9529a1-7f565fd978-mtdwz"]
Sep 14 05:05:51.866469 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:51.866407    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38ea6dfe-3dce-404e-988c-8b3d8f912061" containerName="istio-proxy"
Sep 14 05:05:51.866469 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:51.866419    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="38ea6dfe-3dce-404e-988c-8b3d8f912061" containerName="istio-proxy"
Sep 14 05:05:51.866469 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:51.866439    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38ea6dfe-3dce-404e-988c-8b3d8f912061" containerName="istio-proxy"
Sep 14 05:05:51.866469 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:51.866445    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="38ea6dfe-3dce-404e-988c-8b3d8f912061" containerName="istio-proxy"
Sep 14 05:05:51.866601 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:51.866501    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="38ea6dfe-3dce-404e-988c-8b3d8f912061" containerName="istio-proxy"
Sep 14 05:05:51.866601 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:51.866510    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="38ea6dfe-3dce-404e-988c-8b3d8f912061" containerName="istio-proxy"
Sep 14 05:05:51.870567 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:51.870540    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9529a1-7f565fd978-mtdwz"
Sep 14 05:05:51.872932 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:51.872910    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-9529a1-dockercfg-9dl2r\""
Sep 14 05:05:51.872932 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:51.872921    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-9529a1\""
Sep 14 05:05:51.877262 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:51.877206    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmn9h\" (UniqueName: \"kubernetes.io/projected/31e7684e-18d5-48a0-91a3-a186a552d733-kube-api-access-bmn9h\") pod \"payload-pre-processing-e2e-explicit-9529a1-7f565fd978-mtdwz\" (UID: \"31e7684e-18d5-48a0-91a3-a186a552d733\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-9529a1-7f565fd978-mtdwz"
Sep 14 05:05:51.877375 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:51.877273    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31e7684e-18d5-48a0-91a3-a186a552d733-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-9529a1-7f565fd978-mtdwz\" (UID: \"31e7684e-18d5-48a0-91a3-a186a552d733\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-9529a1-7f565fd978-mtdwz"
Sep 14 05:05:51.878559 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:51.878538    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-9529a1-7f565fd978-mtdwz"]
Sep 14 05:05:51.960287 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:51.960246    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-9529a1-6f7495cf77-p6vbq"]
Sep 14 05:05:51.963664 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:51.963647    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-9529a1-6f7495cf77-p6vbq"
Sep 14 05:05:51.974527 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:51.974501    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-9529a1-6f7495cf77-p6vbq"]
Sep 14 05:05:51.978107 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:51.978084    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bslbz\" (UniqueName: \"kubernetes.io/projected/1957684e-22df-458a-b66b-ab24e745d37d-kube-api-access-bslbz\") pod \"payload-processing-e2e-explicit-9529a1-6f7495cf77-p6vbq\" (UID: \"1957684e-22df-458a-b66b-ab24e745d37d\") " pod="openshift-ingress/payload-processing-e2e-explicit-9529a1-6f7495cf77-p6vbq"
Sep 14 05:05:51.978228 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:51.978130    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmn9h\" (UniqueName: \"kubernetes.io/projected/31e7684e-18d5-48a0-91a3-a186a552d733-kube-api-access-bmn9h\") pod \"payload-pre-processing-e2e-explicit-9529a1-7f565fd978-mtdwz\" (UID: \"31e7684e-18d5-48a0-91a3-a186a552d733\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-9529a1-7f565fd978-mtdwz"
Sep 14 05:05:51.978228 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:51.978208    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31e7684e-18d5-48a0-91a3-a186a552d733-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-9529a1-7f565fd978-mtdwz\" (UID: \"31e7684e-18d5-48a0-91a3-a186a552d733\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-9529a1-7f565fd978-mtdwz"
Sep 14 05:05:51.978332 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:51.978277    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1957684e-22df-458a-b66b-ab24e745d37d-plugins-config-volume\") pod \"payload-processing-e2e-explicit-9529a1-6f7495cf77-p6vbq\" (UID: \"1957684e-22df-458a-b66b-ab24e745d37d\") " pod="openshift-ingress/payload-processing-e2e-explicit-9529a1-6f7495cf77-p6vbq"
Sep 14 05:05:51.979111 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:51.979090    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31e7684e-18d5-48a0-91a3-a186a552d733-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-9529a1-7f565fd978-mtdwz\" (UID: \"31e7684e-18d5-48a0-91a3-a186a552d733\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-9529a1-7f565fd978-mtdwz"
Sep 14 05:05:51.985519 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:51.985494    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmn9h\" (UniqueName: \"kubernetes.io/projected/31e7684e-18d5-48a0-91a3-a186a552d733-kube-api-access-bmn9h\") pod \"payload-pre-processing-e2e-explicit-9529a1-7f565fd978-mtdwz\" (UID: \"31e7684e-18d5-48a0-91a3-a186a552d733\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-9529a1-7f565fd978-mtdwz"
Sep 14 05:05:52.079170 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:52.079133    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1957684e-22df-458a-b66b-ab24e745d37d-plugins-config-volume\") pod \"payload-processing-e2e-explicit-9529a1-6f7495cf77-p6vbq\" (UID: \"1957684e-22df-458a-b66b-ab24e745d37d\") " pod="openshift-ingress/payload-processing-e2e-explicit-9529a1-6f7495cf77-p6vbq"
Sep 14 05:05:52.079327 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:52.079196    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bslbz\" (UniqueName: \"kubernetes.io/projected/1957684e-22df-458a-b66b-ab24e745d37d-kube-api-access-bslbz\") pod \"payload-processing-e2e-explicit-9529a1-6f7495cf77-p6vbq\" (UID: \"1957684e-22df-458a-b66b-ab24e745d37d\") " pod="openshift-ingress/payload-processing-e2e-explicit-9529a1-6f7495cf77-p6vbq"
Sep 14 05:05:52.079926 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:52.079905    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1957684e-22df-458a-b66b-ab24e745d37d-plugins-config-volume\") pod \"payload-processing-e2e-explicit-9529a1-6f7495cf77-p6vbq\" (UID: \"1957684e-22df-458a-b66b-ab24e745d37d\") " pod="openshift-ingress/payload-processing-e2e-explicit-9529a1-6f7495cf77-p6vbq"
Sep 14 05:05:52.086708 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:52.086685    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bslbz\" (UniqueName: \"kubernetes.io/projected/1957684e-22df-458a-b66b-ab24e745d37d-kube-api-access-bslbz\") pod \"payload-processing-e2e-explicit-9529a1-6f7495cf77-p6vbq\" (UID: \"1957684e-22df-458a-b66b-ab24e745d37d\") " pod="openshift-ingress/payload-processing-e2e-explicit-9529a1-6f7495cf77-p6vbq"
Sep 14 05:05:52.180914 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:52.180822    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9529a1-7f565fd978-mtdwz"
Sep 14 05:05:52.274268 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:52.274236    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-9529a1-6f7495cf77-p6vbq"
Sep 14 05:05:52.337474 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:52.337401    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38ea6dfe-3dce-404e-988c-8b3d8f912061" path="/var/lib/kubelet/pods/38ea6dfe-3dce-404e-988c-8b3d8f912061/volumes"
Sep 14 05:05:52.451974 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:52.451950    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-9529a1-7f565fd978-mtdwz"]
Sep 14 05:05:52.453409 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:05:52.453388    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31e7684e_18d5_48a0_91a3_a186a552d733.slice/crio-ed008f716ab3b6053ed74be5d4bfb9355a7cc9a9ed1d14d7c0ded0a11728d9ce WatchSource:0}: Error finding container ed008f716ab3b6053ed74be5d4bfb9355a7cc9a9ed1d14d7c0ded0a11728d9ce: Status 404 returned error can't find the container with id ed008f716ab3b6053ed74be5d4bfb9355a7cc9a9ed1d14d7c0ded0a11728d9ce
Sep 14 05:05:52.545341 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:52.545308    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-9529a1-6f7495cf77-p6vbq"]
Sep 14 05:05:52.545837 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:05:52.545808    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1957684e_22df_458a_b66b_ab24e745d37d.slice/crio-6c0ebe81894d22cbd2d004d14d933da10c8963711ae4f66d9bf4b344c27a967c WatchSource:0}: Error finding container 6c0ebe81894d22cbd2d004d14d933da10c8963711ae4f66d9bf4b344c27a967c: Status 404 returned error can't find the container with id 6c0ebe81894d22cbd2d004d14d933da10c8963711ae4f66d9bf4b344c27a967c
Sep 14 05:05:52.680691 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:52.680656    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-9529a1-6f7495cf77-p6vbq" event={"ID":"1957684e-22df-458a-b66b-ab24e745d37d","Type":"ContainerStarted","Data":"d45f822abcda3a2cb2cbb425fdbdba1b0bdd09553d2510cce14bf883dd8ced25"}
Sep 14 05:05:52.680881 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:52.680711    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-9529a1-6f7495cf77-p6vbq" event={"ID":"1957684e-22df-458a-b66b-ab24e745d37d","Type":"ContainerStarted","Data":"6c0ebe81894d22cbd2d004d14d933da10c8963711ae4f66d9bf4b344c27a967c"}
Sep 14 05:05:52.680881 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:52.680772    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-9529a1-6f7495cf77-p6vbq"
Sep 14 05:05:52.682395 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:52.682371    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9529a1-7f565fd978-mtdwz" event={"ID":"31e7684e-18d5-48a0-91a3-a186a552d733","Type":"ContainerStarted","Data":"eae36b4a44b8c098707638fb420a21091bf52e0063043fe37464a3a706a71eeb"}
Sep 14 05:05:52.682506 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:52.682409    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9529a1-7f565fd978-mtdwz" event={"ID":"31e7684e-18d5-48a0-91a3-a186a552d733","Type":"ContainerStarted","Data":"ed008f716ab3b6053ed74be5d4bfb9355a7cc9a9ed1d14d7c0ded0a11728d9ce"}
Sep 14 05:05:52.682506 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:52.682458    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9529a1-7f565fd978-mtdwz"
Sep 14 05:05:52.697644 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:52.697592    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-9529a1-6f7495cf77-p6vbq" podStartSLOduration=1.697573526 podStartE2EDuration="1.697573526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:05:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:05:52.696124694 +0000 UTC m=+1150.871896253" watchObservedRunningTime="2026-09-14 05:05:52.697573526 +0000 UTC m=+1150.873345087"
Sep 14 05:05:52.713649 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:52.713548    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9529a1-7f565fd978-mtdwz" podStartSLOduration=1.7135298460000001 podStartE2EDuration="1.713529846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:05:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:05:52.713007541 +0000 UTC m=+1150.888779099" watchObservedRunningTime="2026-09-14 05:05:52.713529846 +0000 UTC m=+1150.889301406"
Sep 14 05:05:52.814675 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:52.814638    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-e08402/premium-w4-e08402-kserve-d7dff54c7-fddfc"]
Sep 14 05:05:52.818118 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:52.818101    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e08402/premium-w4-e08402-kserve-d7dff54c7-fddfc"
Sep 14 05:05:52.820507 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:52.820485    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e08402\"/\"premium-w4-e08402-kserve-self-signed-certs\""
Sep 14 05:05:52.827456 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:52.827434    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e08402/premium-w4-e08402-kserve-d7dff54c7-fddfc"]
Sep 14 05:05:52.889441 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:52.889412    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10e39a9c-3c5c-4e22-a788-a09735193dcd-tls-certs\") pod \"premium-w4-e08402-kserve-d7dff54c7-fddfc\" (UID: \"10e39a9c-3c5c-4e22-a788-a09735193dcd\") " pod="e2e-models-e2e-worker-w4-e08402/premium-w4-e08402-kserve-d7dff54c7-fddfc"
Sep 14 05:05:52.889441 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:52.889451    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmxkj\" (UniqueName: \"kubernetes.io/projected/10e39a9c-3c5c-4e22-a788-a09735193dcd-kube-api-access-gmxkj\") pod \"premium-w4-e08402-kserve-d7dff54c7-fddfc\" (UID: \"10e39a9c-3c5c-4e22-a788-a09735193dcd\") " pod="e2e-models-e2e-worker-w4-e08402/premium-w4-e08402-kserve-d7dff54c7-fddfc"
Sep 14 05:05:52.889936 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:52.889522    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10e39a9c-3c5c-4e22-a788-a09735193dcd-model-cache\") pod \"premium-w4-e08402-kserve-d7dff54c7-fddfc\" (UID: \"10e39a9c-3c5c-4e22-a788-a09735193dcd\") " pod="e2e-models-e2e-worker-w4-e08402/premium-w4-e08402-kserve-d7dff54c7-fddfc"
Sep 14 05:05:52.889936 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:52.889578    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10e39a9c-3c5c-4e22-a788-a09735193dcd-tmp-dir\") pod \"premium-w4-e08402-kserve-d7dff54c7-fddfc\" (UID: \"10e39a9c-3c5c-4e22-a788-a09735193dcd\") " pod="e2e-models-e2e-worker-w4-e08402/premium-w4-e08402-kserve-d7dff54c7-fddfc"
Sep 14 05:05:52.889936 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:52.889619    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10e39a9c-3c5c-4e22-a788-a09735193dcd-dshm\") pod \"premium-w4-e08402-kserve-d7dff54c7-fddfc\" (UID: \"10e39a9c-3c5c-4e22-a788-a09735193dcd\") " pod="e2e-models-e2e-worker-w4-e08402/premium-w4-e08402-kserve-d7dff54c7-fddfc"
Sep 14 05:05:52.889936 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:52.889679    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10e39a9c-3c5c-4e22-a788-a09735193dcd-home\") pod \"premium-w4-e08402-kserve-d7dff54c7-fddfc\" (UID: \"10e39a9c-3c5c-4e22-a788-a09735193dcd\") " pod="e2e-models-e2e-worker-w4-e08402/premium-w4-e08402-kserve-d7dff54c7-fddfc"
Sep 14 05:05:52.990646 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:52.990556    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10e39a9c-3c5c-4e22-a788-a09735193dcd-home\") pod \"premium-w4-e08402-kserve-d7dff54c7-fddfc\" (UID: \"10e39a9c-3c5c-4e22-a788-a09735193dcd\") " pod="e2e-models-e2e-worker-w4-e08402/premium-w4-e08402-kserve-d7dff54c7-fddfc"
Sep 14 05:05:52.990646 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:52.990614    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10e39a9c-3c5c-4e22-a788-a09735193dcd-tls-certs\") pod \"premium-w4-e08402-kserve-d7dff54c7-fddfc\" (UID: \"10e39a9c-3c5c-4e22-a788-a09735193dcd\") " pod="e2e-models-e2e-worker-w4-e08402/premium-w4-e08402-kserve-d7dff54c7-fddfc"
Sep 14 05:05:52.990646 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:52.990639    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmxkj\" (UniqueName: \"kubernetes.io/projected/10e39a9c-3c5c-4e22-a788-a09735193dcd-kube-api-access-gmxkj\") pod \"premium-w4-e08402-kserve-d7dff54c7-fddfc\" (UID: \"10e39a9c-3c5c-4e22-a788-a09735193dcd\") " pod="e2e-models-e2e-worker-w4-e08402/premium-w4-e08402-kserve-d7dff54c7-fddfc"
Sep 14 05:05:52.990948 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:52.990772    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10e39a9c-3c5c-4e22-a788-a09735193dcd-model-cache\") pod \"premium-w4-e08402-kserve-d7dff54c7-fddfc\" (UID: \"10e39a9c-3c5c-4e22-a788-a09735193dcd\") " pod="e2e-models-e2e-worker-w4-e08402/premium-w4-e08402-kserve-d7dff54c7-fddfc"
Sep 14 05:05:52.990948 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:52.990848    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10e39a9c-3c5c-4e22-a788-a09735193dcd-tmp-dir\") pod \"premium-w4-e08402-kserve-d7dff54c7-fddfc\" (UID: \"10e39a9c-3c5c-4e22-a788-a09735193dcd\") " pod="e2e-models-e2e-worker-w4-e08402/premium-w4-e08402-kserve-d7dff54c7-fddfc"
Sep 14 05:05:52.990948 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:52.990894    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10e39a9c-3c5c-4e22-a788-a09735193dcd-dshm\") pod \"premium-w4-e08402-kserve-d7dff54c7-fddfc\" (UID: \"10e39a9c-3c5c-4e22-a788-a09735193dcd\") " pod="e2e-models-e2e-worker-w4-e08402/premium-w4-e08402-kserve-d7dff54c7-fddfc"
Sep 14 05:05:52.991136 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:52.991102    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10e39a9c-3c5c-4e22-a788-a09735193dcd-model-cache\") pod \"premium-w4-e08402-kserve-d7dff54c7-fddfc\" (UID: \"10e39a9c-3c5c-4e22-a788-a09735193dcd\") " pod="e2e-models-e2e-worker-w4-e08402/premium-w4-e08402-kserve-d7dff54c7-fddfc"
Sep 14 05:05:52.991230 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:52.991201    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10e39a9c-3c5c-4e22-a788-a09735193dcd-tmp-dir\") pod \"premium-w4-e08402-kserve-d7dff54c7-fddfc\" (UID: \"10e39a9c-3c5c-4e22-a788-a09735193dcd\") " pod="e2e-models-e2e-worker-w4-e08402/premium-w4-e08402-kserve-d7dff54c7-fddfc"
Sep 14 05:05:52.991355 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:52.991270    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10e39a9c-3c5c-4e22-a788-a09735193dcd-home\") pod \"premium-w4-e08402-kserve-d7dff54c7-fddfc\" (UID: \"10e39a9c-3c5c-4e22-a788-a09735193dcd\") " pod="e2e-models-e2e-worker-w4-e08402/premium-w4-e08402-kserve-d7dff54c7-fddfc"
Sep 14 05:05:52.993097 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:52.993074    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10e39a9c-3c5c-4e22-a788-a09735193dcd-dshm\") pod \"premium-w4-e08402-kserve-d7dff54c7-fddfc\" (UID: \"10e39a9c-3c5c-4e22-a788-a09735193dcd\") " pod="e2e-models-e2e-worker-w4-e08402/premium-w4-e08402-kserve-d7dff54c7-fddfc"
Sep 14 05:05:52.993210 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:52.993192    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10e39a9c-3c5c-4e22-a788-a09735193dcd-tls-certs\") pod \"premium-w4-e08402-kserve-d7dff54c7-fddfc\" (UID: \"10e39a9c-3c5c-4e22-a788-a09735193dcd\") " pod="e2e-models-e2e-worker-w4-e08402/premium-w4-e08402-kserve-d7dff54c7-fddfc"
Sep 14 05:05:52.999011 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:52.998976    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmxkj\" (UniqueName: \"kubernetes.io/projected/10e39a9c-3c5c-4e22-a788-a09735193dcd-kube-api-access-gmxkj\") pod \"premium-w4-e08402-kserve-d7dff54c7-fddfc\" (UID: \"10e39a9c-3c5c-4e22-a788-a09735193dcd\") " pod="e2e-models-e2e-worker-w4-e08402/premium-w4-e08402-kserve-d7dff54c7-fddfc"
Sep 14 05:05:53.127529 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:53.127488    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e08402/premium-w4-e08402-kserve-d7dff54c7-fddfc"
Sep 14 05:05:53.409461 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:53.409432    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e08402/premium-w4-e08402-kserve-d7dff54c7-fddfc"]
Sep 14 05:05:53.410699 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:05:53.410677    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10e39a9c_3c5c_4e22_a788_a09735193dcd.slice/crio-fcb19d063a2bd6fe8ffc7d0b706b4763563158509f006038496a1bcfb1c7a004 WatchSource:0}: Error finding container fcb19d063a2bd6fe8ffc7d0b706b4763563158509f006038496a1bcfb1c7a004: Status 404 returned error can't find the container with id fcb19d063a2bd6fe8ffc7d0b706b4763563158509f006038496a1bcfb1c7a004
Sep 14 05:05:53.690661 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:53.690565    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e08402/premium-w4-e08402-kserve-d7dff54c7-fddfc" event={"ID":"10e39a9c-3c5c-4e22-a788-a09735193dcd","Type":"ContainerStarted","Data":"8a62d9d2e0bc976d488a655872314e3614404a8d9ba2f927b0df5da6b0ca6ed6"}
Sep 14 05:05:53.690661 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:53.690615    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e08402/premium-w4-e08402-kserve-d7dff54c7-fddfc" event={"ID":"10e39a9c-3c5c-4e22-a788-a09735193dcd","Type":"ContainerStarted","Data":"fcb19d063a2bd6fe8ffc7d0b706b4763563158509f006038496a1bcfb1c7a004"}
Sep 14 05:05:53.707776 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:53.707718    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-e08402/premium-w4-e08402-kserve-d7dff54c7-fddfc" podStartSLOduration=1.707703333 podStartE2EDuration="1.707703333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:05:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:05:53.706479723 +0000 UTC m=+1151.882251282" watchObservedRunningTime="2026-09-14 05:05:53.707703333 +0000 UTC m=+1151.883474891"
Sep 14 05:05:54.089483 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:54.089451    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"]
Sep 14 05:05:54.093441 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:54.093424    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"
Sep 14 05:05:54.095879 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:54.095855    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-4ee4bdd9-openshift-default-dockercfg-z8x9s\""
Sep 14 05:05:54.104501 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:54.104475    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"]
Sep 14 05:05:54.202574 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:54.202543    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c551cf85-0a5a-42a1-af5f-66fcd43e5672-workload-socket\") pod \"e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s\" (UID: \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\") " pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"
Sep 14 05:05:54.202770 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:54.202584    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c551cf85-0a5a-42a1-af5f-66fcd43e5672-istio-data\") pod \"e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s\" (UID: \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\") " pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"
Sep 14 05:05:54.202770 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:54.202606    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c551cf85-0a5a-42a1-af5f-66fcd43e5672-istio-podinfo\") pod \"e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s\" (UID: \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\") " pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"
Sep 14 05:05:54.202770 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:54.202631    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c551cf85-0a5a-42a1-af5f-66fcd43e5672-credential-socket\") pod \"e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s\" (UID: \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\") " pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"
Sep 14 05:05:54.202770 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:54.202740    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c551cf85-0a5a-42a1-af5f-66fcd43e5672-istio-envoy\") pod \"e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s\" (UID: \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\") " pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"
Sep 14 05:05:54.202915 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:54.202804    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c551cf85-0a5a-42a1-af5f-66fcd43e5672-workload-certs\") pod \"e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s\" (UID: \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\") " pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"
Sep 14 05:05:54.202915 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:54.202831    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b7ts\" (UniqueName: \"kubernetes.io/projected/c551cf85-0a5a-42a1-af5f-66fcd43e5672-kube-api-access-9b7ts\") pod \"e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s\" (UID: \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\") " pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"
Sep 14 05:05:54.202915 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:54.202876    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c551cf85-0a5a-42a1-af5f-66fcd43e5672-istio-token\") pod \"e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s\" (UID: \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\") " pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"
Sep 14 05:05:54.202915 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:54.202902    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c551cf85-0a5a-42a1-af5f-66fcd43e5672-istiod-ca-cert\") pod \"e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s\" (UID: \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\") " pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"
Sep 14 05:05:54.303503 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:54.303471    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c551cf85-0a5a-42a1-af5f-66fcd43e5672-istio-token\") pod \"e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s\" (UID: \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\") " pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"
Sep 14 05:05:54.303723 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:54.303510    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c551cf85-0a5a-42a1-af5f-66fcd43e5672-istiod-ca-cert\") pod \"e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s\" (UID: \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\") " pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"
Sep 14 05:05:54.303723 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:54.303550    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c551cf85-0a5a-42a1-af5f-66fcd43e5672-workload-socket\") pod \"e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s\" (UID: \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\") " pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"
Sep 14 05:05:54.303723 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:54.303584    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c551cf85-0a5a-42a1-af5f-66fcd43e5672-istio-data\") pod \"e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s\" (UID: \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\") " pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"
Sep 14 05:05:54.303723 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:54.303607    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c551cf85-0a5a-42a1-af5f-66fcd43e5672-istio-podinfo\") pod \"e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s\" (UID: \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\") " pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"
Sep 14 05:05:54.303723 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:54.303628    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c551cf85-0a5a-42a1-af5f-66fcd43e5672-credential-socket\") pod \"e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s\" (UID: \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\") " pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"
Sep 14 05:05:54.303723 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:54.303663    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c551cf85-0a5a-42a1-af5f-66fcd43e5672-istio-envoy\") pod \"e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s\" (UID: \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\") " pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"
Sep 14 05:05:54.303723 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:54.303706    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c551cf85-0a5a-42a1-af5f-66fcd43e5672-workload-certs\") pod \"e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s\" (UID: \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\") " pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"
Sep 14 05:05:54.304192 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:54.303729    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b7ts\" (UniqueName: \"kubernetes.io/projected/c551cf85-0a5a-42a1-af5f-66fcd43e5672-kube-api-access-9b7ts\") pod \"e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s\" (UID: \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\") " pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"
Sep 14 05:05:54.304192 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:54.304056    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c551cf85-0a5a-42a1-af5f-66fcd43e5672-workload-socket\") pod \"e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s\" (UID: \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\") " pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"
Sep 14 05:05:54.304317 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:54.304248    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c551cf85-0a5a-42a1-af5f-66fcd43e5672-credential-socket\") pod \"e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s\" (UID: \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\") " pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"
Sep 14 05:05:54.304317 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:54.304282    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c551cf85-0a5a-42a1-af5f-66fcd43e5672-workload-certs\") pod \"e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s\" (UID: \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\") " pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"
Sep 14 05:05:54.304404 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:54.304371    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c551cf85-0a5a-42a1-af5f-66fcd43e5672-istio-data\") pod \"e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s\" (UID: \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\") " pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"
Sep 14 05:05:54.304465 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:54.304443    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c551cf85-0a5a-42a1-af5f-66fcd43e5672-istiod-ca-cert\") pod \"e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s\" (UID: \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\") " pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"
Sep 14 05:05:54.306037 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:54.306018    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c551cf85-0a5a-42a1-af5f-66fcd43e5672-istio-envoy\") pod \"e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s\" (UID: \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\") " pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"
Sep 14 05:05:54.306241 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:54.306225    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c551cf85-0a5a-42a1-af5f-66fcd43e5672-istio-podinfo\") pod \"e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s\" (UID: \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\") " pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"
Sep 14 05:05:54.311482 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:54.311459    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b7ts\" (UniqueName: \"kubernetes.io/projected/c551cf85-0a5a-42a1-af5f-66fcd43e5672-kube-api-access-9b7ts\") pod \"e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s\" (UID: \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\") " pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"
Sep 14 05:05:54.311631 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:54.311612    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c551cf85-0a5a-42a1-af5f-66fcd43e5672-istio-token\") pod \"e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s\" (UID: \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\") " pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"
Sep 14 05:05:54.404883 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:54.404810    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"
Sep 14 05:05:54.810296 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:05:54.810261    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc551cf85_0a5a_42a1_af5f_66fcd43e5672.slice/crio-eda9fa63b688920f3f2faf5eae77c785d0bf72d408ddf713fc583ab395547262 WatchSource:0}: Error finding container eda9fa63b688920f3f2faf5eae77c785d0bf72d408ddf713fc583ab395547262: Status 404 returned error can't find the container with id eda9fa63b688920f3f2faf5eae77c785d0bf72d408ddf713fc583ab395547262
Sep 14 05:05:54.812632 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:54.812595    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:05:54.812746 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:54.812683    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:05:54.812746 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:54.812728    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:05:54.816642 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:54.816621    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"]
Sep 14 05:05:55.703445 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:55.703404    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s" event={"ID":"c551cf85-0a5a-42a1-af5f-66fcd43e5672","Type":"ContainerStarted","Data":"614caf088415cec45511b8744f1df21bea6be9b11c3aa9c65fd17fdd78590f82"}
Sep 14 05:05:55.703846 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:55.703462    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s" event={"ID":"c551cf85-0a5a-42a1-af5f-66fcd43e5672","Type":"ContainerStarted","Data":"eda9fa63b688920f3f2faf5eae77c785d0bf72d408ddf713fc583ab395547262"}
Sep 14 05:05:55.730776 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:55.730709    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s" podStartSLOduration=1.730693013 podStartE2EDuration="1.730693013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:05:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:05:55.729413908 +0000 UTC m=+1153.905185467" watchObservedRunningTime="2026-09-14 05:05:55.730693013 +0000 UTC m=+1153.906464572"
Sep 14 05:05:56.405729 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:56.405692    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"
Sep 14 05:05:56.410830 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:56.410804    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"
Sep 14 05:05:56.714325 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:56.714237    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"
Sep 14 05:05:56.715258 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:05:56.715235    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"
Sep 14 05:06:00.035130 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:00.035091    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:06:03.128504 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:03.128466    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e08402/premium-w4-e08402-kserve-d7dff54c7-fddfc"
Sep 14 05:06:03.128957 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:03.128546    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-e08402/premium-w4-e08402-kserve-d7dff54c7-fddfc"
Sep 14 05:06:03.166143 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:03.166107    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-e08402/premium-w4-e08402-kserve-d7dff54c7-fddfc"
Sep 14 05:06:03.692517 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:03.692487    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9529a1-7f565fd978-mtdwz"
Sep 14 05:06:03.693285 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:03.693258    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-9529a1-6f7495cf77-p6vbq"
Sep 14 05:06:03.784494 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:03.784462    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e08402/premium-w4-e08402-kserve-d7dff54c7-fddfc"
Sep 14 05:06:10.812614 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:10.812509    2565 generic.go:356] "Generic (PLEG): container finished" podID="f58465fe-e115-4ba7-be72-4d2b17957d90" containerID="41a0873129427c2f709d6f4e0947024e1355eb227ef25bcfc81aeac870fe056c" exitCode=0
Sep 14 05:06:10.812614 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:10.812587    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr" event={"ID":"f58465fe-e115-4ba7-be72-4d2b17957d90","Type":"ContainerDied","Data":"41a0873129427c2f709d6f4e0947024e1355eb227ef25bcfc81aeac870fe056c"}
Sep 14 05:06:10.813057 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:10.813005    2565 scope.go:117] "RemoveContainer" containerID="41a0873129427c2f709d6f4e0947024e1355eb227ef25bcfc81aeac870fe056c"
Sep 14 05:06:10.814112 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:10.814086    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:06:10.814174 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:10.814148    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:06:10.814213 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:10.814179    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:06:11.820098 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:11.820070    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr" event={"ID":"f58465fe-e115-4ba7-be72-4d2b17957d90","Type":"ContainerStarted","Data":"9e0b265fb8ddcc8a1a5f70dd83374f941ffad9e7a7d460a5c980d8d72bc26d1c"}
Sep 14 05:06:11.847815 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:11.847787    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"
Sep 14 05:06:11.848033 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:11.848010    2565 patch_prober.go:28] interesting pod/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 14 05:06:11.848093 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:11.848067    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr" podUID="f58465fe-e115-4ba7-be72-4d2b17957d90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 14 05:06:12.848733 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:12.848697    2565 patch_prober.go:28] interesting pod/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 14 05:06:12.849371 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:12.849337    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr" podUID="f58465fe-e115-4ba7-be72-4d2b17957d90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 14 05:06:13.847856 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:13.847810    2565 patch_prober.go:28] interesting pod/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 14 05:06:13.848176 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:13.847881    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr" podUID="f58465fe-e115-4ba7-be72-4d2b17957d90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 14 05:06:14.847910 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:14.847861    2565 patch_prober.go:28] interesting pod/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 14 05:06:14.848443 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:14.847947    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr" podUID="f58465fe-e115-4ba7-be72-4d2b17957d90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 14 05:06:15.848681 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:15.848641    2565 patch_prober.go:28] interesting pod/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 14 05:06:15.849636 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:15.849247    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr" podUID="f58465fe-e115-4ba7-be72-4d2b17957d90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 14 05:06:16.848126 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:16.848084    2565 patch_prober.go:28] interesting pod/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 14 05:06:16.848350 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:16.848176    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr" podUID="f58465fe-e115-4ba7-be72-4d2b17957d90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 14 05:06:17.848078 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:17.848040    2565 patch_prober.go:28] interesting pod/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 14 05:06:17.849073 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:17.848659    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr" podUID="f58465fe-e115-4ba7-be72-4d2b17957d90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 14 05:06:18.648066 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:18.648019    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"
Sep 14 05:06:18.854051 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:18.854020    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"
Sep 14 05:06:18.874942 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:18.874909    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"
Sep 14 05:06:23.823573 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:23.823542    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:06:23.898928 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:23.898896    2565 generic.go:356] "Generic (PLEG): container finished" podID="c551cf85-0a5a-42a1-af5f-66fcd43e5672" containerID="614caf088415cec45511b8744f1df21bea6be9b11c3aa9c65fd17fdd78590f82" exitCode=0
Sep 14 05:06:23.899086 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:23.898974    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s" event={"ID":"c551cf85-0a5a-42a1-af5f-66fcd43e5672","Type":"ContainerDied","Data":"614caf088415cec45511b8744f1df21bea6be9b11c3aa9c65fd17fdd78590f82"}
Sep 14 05:06:23.899353 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:23.899338    2565 scope.go:117] "RemoveContainer" containerID="614caf088415cec45511b8744f1df21bea6be9b11c3aa9c65fd17fdd78590f82"
Sep 14 05:06:23.900395 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:23.900368    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:06:23.900457 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:23.900430    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:06:23.900496 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:23.900475    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:06:24.905922 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:24.905889    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s" event={"ID":"c551cf85-0a5a-42a1-af5f-66fcd43e5672","Type":"ContainerStarted","Data":"8540fd2b5365bc139acbc782a25be0b0bfe66147fc45683b7fe6af50896a1fe8"}
Sep 14 05:06:25.405491 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:25.405457    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"
Sep 14 05:06:25.406382 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:25.406357    2565 patch_prober.go:28] interesting pod/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 14 05:06:25.406474 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:25.406416    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s" podUID="c551cf85-0a5a-42a1-af5f-66fcd43e5672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 14 05:06:26.406170 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:26.406135    2565 patch_prober.go:28] interesting pod/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 14 05:06:26.406640 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:26.406197    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s" podUID="c551cf85-0a5a-42a1-af5f-66fcd43e5672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 14 05:06:26.715260 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:26.715173    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"
Sep 14 05:06:27.405572 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:27.405532    2565 patch_prober.go:28] interesting pod/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 14 05:06:27.405794 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:27.405590    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s" podUID="c551cf85-0a5a-42a1-af5f-66fcd43e5672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 14 05:06:28.405222 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:28.405188    2565 patch_prober.go:28] interesting pod/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 14 05:06:28.405598 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:28.405240    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s" podUID="c551cf85-0a5a-42a1-af5f-66fcd43e5672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 14 05:06:29.406269 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:29.406229    2565 patch_prober.go:28] interesting pod/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 14 05:06:29.406660 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:29.406281    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s" podUID="c551cf85-0a5a-42a1-af5f-66fcd43e5672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 14 05:06:30.410109 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:30.410079    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"
Sep 14 05:06:30.411084 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:30.411065    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"
Sep 14 05:06:37.717682 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:37.717635    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-e7fdd6/test-model-e7fdd6-kserve-78dc566997-7ssq4"]
Sep 14 05:06:37.720659 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:37.720640    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e7fdd6/test-model-e7fdd6-kserve-78dc566997-7ssq4"
Sep 14 05:06:37.723921 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:37.723901    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e7fdd6\"/\"kube-root-ca.crt\""
Sep 14 05:06:37.724055 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:37.723969    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e7fdd6\"/\"openshift-service-ca.crt\""
Sep 14 05:06:37.724055 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:37.724046    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e7fdd6\"/\"default-dockercfg-vhtjz\""
Sep 14 05:06:37.724170 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:37.724066    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e7fdd6\"/\"test-model-e7fdd6-kserve-self-signed-certs\""
Sep 14 05:06:37.729832 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:37.729809    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e7fdd6/test-model-e7fdd6-kserve-78dc566997-7ssq4"]
Sep 14 05:06:37.902463 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:37.902434    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81dac50a-cd9d-4762-907b-878715836a9c-tmp-dir\") pod \"test-model-e7fdd6-kserve-78dc566997-7ssq4\" (UID: \"81dac50a-cd9d-4762-907b-878715836a9c\") " pod="ai-tenant-e2e-inf-b-e7fdd6/test-model-e7fdd6-kserve-78dc566997-7ssq4"
Sep 14 05:06:37.902613 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:37.902482    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmtjb\" (UniqueName: \"kubernetes.io/projected/81dac50a-cd9d-4762-907b-878715836a9c-kube-api-access-vmtjb\") pod \"test-model-e7fdd6-kserve-78dc566997-7ssq4\" (UID: \"81dac50a-cd9d-4762-907b-878715836a9c\") " pod="ai-tenant-e2e-inf-b-e7fdd6/test-model-e7fdd6-kserve-78dc566997-7ssq4"
Sep 14 05:06:37.902613 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:37.902504    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81dac50a-cd9d-4762-907b-878715836a9c-home\") pod \"test-model-e7fdd6-kserve-78dc566997-7ssq4\" (UID: \"81dac50a-cd9d-4762-907b-878715836a9c\") " pod="ai-tenant-e2e-inf-b-e7fdd6/test-model-e7fdd6-kserve-78dc566997-7ssq4"
Sep 14 05:06:37.902613 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:37.902581    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81dac50a-cd9d-4762-907b-878715836a9c-tls-certs\") pod \"test-model-e7fdd6-kserve-78dc566997-7ssq4\" (UID: \"81dac50a-cd9d-4762-907b-878715836a9c\") " pod="ai-tenant-e2e-inf-b-e7fdd6/test-model-e7fdd6-kserve-78dc566997-7ssq4"
Sep 14 05:06:37.902613 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:37.902608    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81dac50a-cd9d-4762-907b-878715836a9c-dshm\") pod \"test-model-e7fdd6-kserve-78dc566997-7ssq4\" (UID: \"81dac50a-cd9d-4762-907b-878715836a9c\") " pod="ai-tenant-e2e-inf-b-e7fdd6/test-model-e7fdd6-kserve-78dc566997-7ssq4"
Sep 14 05:06:37.902790 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:37.902708    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81dac50a-cd9d-4762-907b-878715836a9c-model-cache\") pod \"test-model-e7fdd6-kserve-78dc566997-7ssq4\" (UID: \"81dac50a-cd9d-4762-907b-878715836a9c\") " pod="ai-tenant-e2e-inf-b-e7fdd6/test-model-e7fdd6-kserve-78dc566997-7ssq4"
Sep 14 05:06:38.003889 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:38.003806    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81dac50a-cd9d-4762-907b-878715836a9c-model-cache\") pod \"test-model-e7fdd6-kserve-78dc566997-7ssq4\" (UID: \"81dac50a-cd9d-4762-907b-878715836a9c\") " pod="ai-tenant-e2e-inf-b-e7fdd6/test-model-e7fdd6-kserve-78dc566997-7ssq4"
Sep 14 05:06:38.003889 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:38.003847    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81dac50a-cd9d-4762-907b-878715836a9c-tmp-dir\") pod \"test-model-e7fdd6-kserve-78dc566997-7ssq4\" (UID: \"81dac50a-cd9d-4762-907b-878715836a9c\") " pod="ai-tenant-e2e-inf-b-e7fdd6/test-model-e7fdd6-kserve-78dc566997-7ssq4"
Sep 14 05:06:38.004062 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:38.003973    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmtjb\" (UniqueName: \"kubernetes.io/projected/81dac50a-cd9d-4762-907b-878715836a9c-kube-api-access-vmtjb\") pod \"test-model-e7fdd6-kserve-78dc566997-7ssq4\" (UID: \"81dac50a-cd9d-4762-907b-878715836a9c\") " pod="ai-tenant-e2e-inf-b-e7fdd6/test-model-e7fdd6-kserve-78dc566997-7ssq4"
Sep 14 05:06:38.004062 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:38.004009    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81dac50a-cd9d-4762-907b-878715836a9c-home\") pod \"test-model-e7fdd6-kserve-78dc566997-7ssq4\" (UID: \"81dac50a-cd9d-4762-907b-878715836a9c\") " pod="ai-tenant-e2e-inf-b-e7fdd6/test-model-e7fdd6-kserve-78dc566997-7ssq4"
Sep 14 05:06:38.004062 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:38.004039    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81dac50a-cd9d-4762-907b-878715836a9c-tls-certs\") pod \"test-model-e7fdd6-kserve-78dc566997-7ssq4\" (UID: \"81dac50a-cd9d-4762-907b-878715836a9c\") " pod="ai-tenant-e2e-inf-b-e7fdd6/test-model-e7fdd6-kserve-78dc566997-7ssq4"
Sep 14 05:06:38.004062 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:38.004056    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81dac50a-cd9d-4762-907b-878715836a9c-dshm\") pod \"test-model-e7fdd6-kserve-78dc566997-7ssq4\" (UID: \"81dac50a-cd9d-4762-907b-878715836a9c\") " pod="ai-tenant-e2e-inf-b-e7fdd6/test-model-e7fdd6-kserve-78dc566997-7ssq4"
Sep 14 05:06:38.004246 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:38.004189    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81dac50a-cd9d-4762-907b-878715836a9c-model-cache\") pod \"test-model-e7fdd6-kserve-78dc566997-7ssq4\" (UID: \"81dac50a-cd9d-4762-907b-878715836a9c\") " pod="ai-tenant-e2e-inf-b-e7fdd6/test-model-e7fdd6-kserve-78dc566997-7ssq4"
Sep 14 05:06:38.004246 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:38.004228    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81dac50a-cd9d-4762-907b-878715836a9c-tmp-dir\") pod \"test-model-e7fdd6-kserve-78dc566997-7ssq4\" (UID: \"81dac50a-cd9d-4762-907b-878715836a9c\") " pod="ai-tenant-e2e-inf-b-e7fdd6/test-model-e7fdd6-kserve-78dc566997-7ssq4"
Sep 14 05:06:38.004366 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:38.004309    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81dac50a-cd9d-4762-907b-878715836a9c-home\") pod \"test-model-e7fdd6-kserve-78dc566997-7ssq4\" (UID: \"81dac50a-cd9d-4762-907b-878715836a9c\") " pod="ai-tenant-e2e-inf-b-e7fdd6/test-model-e7fdd6-kserve-78dc566997-7ssq4"
Sep 14 05:06:38.006317 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:38.006293    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81dac50a-cd9d-4762-907b-878715836a9c-dshm\") pod \"test-model-e7fdd6-kserve-78dc566997-7ssq4\" (UID: \"81dac50a-cd9d-4762-907b-878715836a9c\") " pod="ai-tenant-e2e-inf-b-e7fdd6/test-model-e7fdd6-kserve-78dc566997-7ssq4"
Sep 14 05:06:38.006442 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:38.006421    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81dac50a-cd9d-4762-907b-878715836a9c-tls-certs\") pod \"test-model-e7fdd6-kserve-78dc566997-7ssq4\" (UID: \"81dac50a-cd9d-4762-907b-878715836a9c\") " pod="ai-tenant-e2e-inf-b-e7fdd6/test-model-e7fdd6-kserve-78dc566997-7ssq4"
Sep 14 05:06:38.011057 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:38.011036    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmtjb\" (UniqueName: \"kubernetes.io/projected/81dac50a-cd9d-4762-907b-878715836a9c-kube-api-access-vmtjb\") pod \"test-model-e7fdd6-kserve-78dc566997-7ssq4\" (UID: \"81dac50a-cd9d-4762-907b-878715836a9c\") " pod="ai-tenant-e2e-inf-b-e7fdd6/test-model-e7fdd6-kserve-78dc566997-7ssq4"
Sep 14 05:06:38.032044 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:38.032026    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e7fdd6/test-model-e7fdd6-kserve-78dc566997-7ssq4"
Sep 14 05:06:38.303319 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:38.303284    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e7fdd6/test-model-e7fdd6-kserve-78dc566997-7ssq4"]
Sep 14 05:06:38.306112 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:06:38.306081    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81dac50a_cd9d_4762_907b_878715836a9c.slice/crio-c7e19f11f1bd7949866c0afe943e8205374e86dae5b5b040721b2b5a6ffb3249 WatchSource:0}: Error finding container c7e19f11f1bd7949866c0afe943e8205374e86dae5b5b040721b2b5a6ffb3249: Status 404 returned error can't find the container with id c7e19f11f1bd7949866c0afe943e8205374e86dae5b5b040721b2b5a6ffb3249
Sep 14 05:06:38.988824 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:38.988785    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e7fdd6/test-model-e7fdd6-kserve-78dc566997-7ssq4" event={"ID":"81dac50a-cd9d-4762-907b-878715836a9c","Type":"ContainerStarted","Data":"a9e37c2b8fc6aab77fe729031734c627b1ea6592aef49da3b5927ee48ab585f6"}
Sep 14 05:06:38.989209 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:38.988837    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e7fdd6/test-model-e7fdd6-kserve-78dc566997-7ssq4" event={"ID":"81dac50a-cd9d-4762-907b-878715836a9c","Type":"ContainerStarted","Data":"c7e19f11f1bd7949866c0afe943e8205374e86dae5b5b040721b2b5a6ffb3249"}
Sep 14 05:06:39.009786 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:39.005518    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-e7fdd6/test-model-e7fdd6-kserve-78dc566997-7ssq4" podStartSLOduration=2.005498505 podStartE2EDuration="2.005498505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:06:39.003668462 +0000 UTC m=+1197.179440023" watchObservedRunningTime="2026-09-14 05:06:39.005498505 +0000 UTC m=+1197.181270065"
Sep 14 05:06:42.357186 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:42.357154    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kndxq_e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae/ovn-acl-logging/0.log"
Sep 14 05:06:42.359245 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:42.359223    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kndxq_e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae/ovn-acl-logging/0.log"
Sep 14 05:06:45.726464 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:45.726425    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:06:48.033137 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:48.033091    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-e7fdd6/test-model-e7fdd6-kserve-78dc566997-7ssq4"
Sep 14 05:06:48.033137 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:48.033144    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-e7fdd6/test-model-e7fdd6-kserve-78dc566997-7ssq4"
Sep 14 05:06:48.045921 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:48.045896    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-e7fdd6/test-model-e7fdd6-kserve-78dc566997-7ssq4"
Sep 14 05:06:49.053477 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:49.053445    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-e7fdd6/test-model-e7fdd6-kserve-78dc566997-7ssq4"
Sep 14 05:06:55.628433 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:55.628402    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"]
Sep 14 05:06:55.628905 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:06:55.628625    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s" podUID="c551cf85-0a5a-42a1-af5f-66fcd43e5672" containerName="istio-proxy" containerID="cri-o://8540fd2b5365bc139acbc782a25be0b0bfe66147fc45683b7fe6af50896a1fe8" gracePeriod=30
Sep 14 05:07:00.413292 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:00.413253    2565 patch_prober.go:28] interesting pod/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 14 05:07:00.413655 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:00.413315    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s" podUID="c551cf85-0a5a-42a1-af5f-66fcd43e5672" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 14 05:07:00.876667 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:00.876645    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"
Sep 14 05:07:00.995779 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:00.995681    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c551cf85-0a5a-42a1-af5f-66fcd43e5672-istiod-ca-cert\") pod \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\" (UID: \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\") "
Sep 14 05:07:00.995779 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:00.995730    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c551cf85-0a5a-42a1-af5f-66fcd43e5672-credential-socket\") pod \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\" (UID: \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\") "
Sep 14 05:07:00.995779 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:00.995777    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c551cf85-0a5a-42a1-af5f-66fcd43e5672-istio-envoy\") pod \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\" (UID: \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\") "
Sep 14 05:07:00.996115 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:00.995795    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c551cf85-0a5a-42a1-af5f-66fcd43e5672-workload-certs\") pod \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\" (UID: \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\") "
Sep 14 05:07:00.996115 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:00.995818    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9b7ts\" (UniqueName: \"kubernetes.io/projected/c551cf85-0a5a-42a1-af5f-66fcd43e5672-kube-api-access-9b7ts\") pod \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\" (UID: \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\") "
Sep 14 05:07:00.996115 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:00.995844    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c551cf85-0a5a-42a1-af5f-66fcd43e5672-workload-socket\") pod \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\" (UID: \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\") "
Sep 14 05:07:00.996115 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:00.995866    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c551cf85-0a5a-42a1-af5f-66fcd43e5672-istio-podinfo\") pod \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\" (UID: \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\") "
Sep 14 05:07:00.996115 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:00.995916    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c551cf85-0a5a-42a1-af5f-66fcd43e5672-istio-data\") pod \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\" (UID: \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\") "
Sep 14 05:07:00.996115 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:00.995949    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c551cf85-0a5a-42a1-af5f-66fcd43e5672-istio-token\") pod \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\" (UID: \"c551cf85-0a5a-42a1-af5f-66fcd43e5672\") "
Sep 14 05:07:00.996435 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:00.996148    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c551cf85-0a5a-42a1-af5f-66fcd43e5672-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c551cf85-0a5a-42a1-af5f-66fcd43e5672" (UID: "c551cf85-0a5a-42a1-af5f-66fcd43e5672"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 05:07:00.996435 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:00.996188    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c551cf85-0a5a-42a1-af5f-66fcd43e5672-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c551cf85-0a5a-42a1-af5f-66fcd43e5672" (UID: "c551cf85-0a5a-42a1-af5f-66fcd43e5672"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:07:00.996435 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:00.996205    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c551cf85-0a5a-42a1-af5f-66fcd43e5672-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c551cf85-0a5a-42a1-af5f-66fcd43e5672" (UID: "c551cf85-0a5a-42a1-af5f-66fcd43e5672"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:07:00.996435 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:00.996278    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c551cf85-0a5a-42a1-af5f-66fcd43e5672-istiod-ca-cert\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:07:00.996435 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:00.996300    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c551cf85-0a5a-42a1-af5f-66fcd43e5672-credential-socket\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:07:00.996435 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:00.996340    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c551cf85-0a5a-42a1-af5f-66fcd43e5672-workload-socket\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:07:00.996435 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:00.996365    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c551cf85-0a5a-42a1-af5f-66fcd43e5672-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c551cf85-0a5a-42a1-af5f-66fcd43e5672" (UID: "c551cf85-0a5a-42a1-af5f-66fcd43e5672"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:07:00.996435 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:00.996430    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c551cf85-0a5a-42a1-af5f-66fcd43e5672-istio-data" (OuterVolumeSpecName: "istio-data") pod "c551cf85-0a5a-42a1-af5f-66fcd43e5672" (UID: "c551cf85-0a5a-42a1-af5f-66fcd43e5672"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:07:00.998109 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:00.998076    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c551cf85-0a5a-42a1-af5f-66fcd43e5672-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c551cf85-0a5a-42a1-af5f-66fcd43e5672" (UID: "c551cf85-0a5a-42a1-af5f-66fcd43e5672"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 05:07:00.998215 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:00.998122    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c551cf85-0a5a-42a1-af5f-66fcd43e5672-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c551cf85-0a5a-42a1-af5f-66fcd43e5672" (UID: "c551cf85-0a5a-42a1-af5f-66fcd43e5672"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:07:00.998215 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:00.998140    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c551cf85-0a5a-42a1-af5f-66fcd43e5672-kube-api-access-9b7ts" (OuterVolumeSpecName: "kube-api-access-9b7ts") pod "c551cf85-0a5a-42a1-af5f-66fcd43e5672" (UID: "c551cf85-0a5a-42a1-af5f-66fcd43e5672"). InnerVolumeSpecName "kube-api-access-9b7ts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:07:00.998292 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:00.998284    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c551cf85-0a5a-42a1-af5f-66fcd43e5672-istio-token" (OuterVolumeSpecName: "istio-token") pod "c551cf85-0a5a-42a1-af5f-66fcd43e5672" (UID: "c551cf85-0a5a-42a1-af5f-66fcd43e5672"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:07:01.097460 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:01.097424    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c551cf85-0a5a-42a1-af5f-66fcd43e5672-istio-podinfo\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:07:01.097460 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:01.097454    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c551cf85-0a5a-42a1-af5f-66fcd43e5672-istio-data\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:07:01.097460 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:01.097468    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c551cf85-0a5a-42a1-af5f-66fcd43e5672-istio-token\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:07:01.097694 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:01.097479    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c551cf85-0a5a-42a1-af5f-66fcd43e5672-istio-envoy\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:07:01.097694 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:01.097488    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c551cf85-0a5a-42a1-af5f-66fcd43e5672-workload-certs\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:07:01.097694 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:01.097497    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b7ts\" (UniqueName: \"kubernetes.io/projected/c551cf85-0a5a-42a1-af5f-66fcd43e5672-kube-api-access-9b7ts\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:07:01.106927 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:01.106899    2565 generic.go:356] "Generic (PLEG): container finished" podID="c551cf85-0a5a-42a1-af5f-66fcd43e5672" containerID="8540fd2b5365bc139acbc782a25be0b0bfe66147fc45683b7fe6af50896a1fe8" exitCode=0
Sep 14 05:07:01.107085 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:01.106956    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"
Sep 14 05:07:01.107085 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:01.106965    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s" event={"ID":"c551cf85-0a5a-42a1-af5f-66fcd43e5672","Type":"ContainerDied","Data":"8540fd2b5365bc139acbc782a25be0b0bfe66147fc45683b7fe6af50896a1fe8"}
Sep 14 05:07:01.107085 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:01.107012    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s" event={"ID":"c551cf85-0a5a-42a1-af5f-66fcd43e5672","Type":"ContainerDied","Data":"eda9fa63b688920f3f2faf5eae77c785d0bf72d408ddf713fc583ab395547262"}
Sep 14 05:07:01.107085 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:01.107037    2565 scope.go:117] "RemoveContainer" containerID="8540fd2b5365bc139acbc782a25be0b0bfe66147fc45683b7fe6af50896a1fe8"
Sep 14 05:07:01.117099 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:01.117082    2565 scope.go:117] "RemoveContainer" containerID="614caf088415cec45511b8744f1df21bea6be9b11c3aa9c65fd17fdd78590f82"
Sep 14 05:07:01.125736 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:01.125712    2565 scope.go:117] "RemoveContainer" containerID="8540fd2b5365bc139acbc782a25be0b0bfe66147fc45683b7fe6af50896a1fe8"
Sep 14 05:07:01.126170 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:07:01.126041    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8540fd2b5365bc139acbc782a25be0b0bfe66147fc45683b7fe6af50896a1fe8\": container with ID starting with 8540fd2b5365bc139acbc782a25be0b0bfe66147fc45683b7fe6af50896a1fe8 not found: ID does not exist" containerID="8540fd2b5365bc139acbc782a25be0b0bfe66147fc45683b7fe6af50896a1fe8"
Sep 14 05:07:01.126170 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:01.126075    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8540fd2b5365bc139acbc782a25be0b0bfe66147fc45683b7fe6af50896a1fe8"} err="failed to get container status \"8540fd2b5365bc139acbc782a25be0b0bfe66147fc45683b7fe6af50896a1fe8\": rpc error: code = NotFound desc = could not find container \"8540fd2b5365bc139acbc782a25be0b0bfe66147fc45683b7fe6af50896a1fe8\": container with ID starting with 8540fd2b5365bc139acbc782a25be0b0bfe66147fc45683b7fe6af50896a1fe8 not found: ID does not exist"
Sep 14 05:07:01.126170 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:01.126133    2565 scope.go:117] "RemoveContainer" containerID="614caf088415cec45511b8744f1df21bea6be9b11c3aa9c65fd17fdd78590f82"
Sep 14 05:07:01.126417 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:07:01.126397    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"614caf088415cec45511b8744f1df21bea6be9b11c3aa9c65fd17fdd78590f82\": container with ID starting with 614caf088415cec45511b8744f1df21bea6be9b11c3aa9c65fd17fdd78590f82 not found: ID does not exist" containerID="614caf088415cec45511b8744f1df21bea6be9b11c3aa9c65fd17fdd78590f82"
Sep 14 05:07:01.126463 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:01.126423    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"614caf088415cec45511b8744f1df21bea6be9b11c3aa9c65fd17fdd78590f82"} err="failed to get container status \"614caf088415cec45511b8744f1df21bea6be9b11c3aa9c65fd17fdd78590f82\": rpc error: code = NotFound desc = could not find container \"614caf088415cec45511b8744f1df21bea6be9b11c3aa9c65fd17fdd78590f82\": container with ID starting with 614caf088415cec45511b8744f1df21bea6be9b11c3aa9c65fd17fdd78590f82 not found: ID does not exist"
Sep 14 05:07:01.128410 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:01.128390    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"]
Sep 14 05:07:01.134343 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:01.134322    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-4ee4bdd9-openshift-default-7fccc644fb-7847s"]
Sep 14 05:07:02.335377 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:02.335342    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c551cf85-0a5a-42a1-af5f-66fcd43e5672" path="/var/lib/kubelet/pods/c551cf85-0a5a-42a1-af5f-66fcd43e5672/volumes"
Sep 14 05:07:05.535959 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:05.535925    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:07:16.723566 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:16.723527    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2eeb50-68db997bf4-s7kpn"]
Sep 14 05:07:16.724034 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:16.723914    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c551cf85-0a5a-42a1-af5f-66fcd43e5672" containerName="istio-proxy"
Sep 14 05:07:16.724034 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:16.723928    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c551cf85-0a5a-42a1-af5f-66fcd43e5672" containerName="istio-proxy"
Sep 14 05:07:16.724034 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:16.724000    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c551cf85-0a5a-42a1-af5f-66fcd43e5672" containerName="istio-proxy"
Sep 14 05:07:16.724034 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:16.724007    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c551cf85-0a5a-42a1-af5f-66fcd43e5672" containerName="istio-proxy"
Sep 14 05:07:16.726123 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:16.726104    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2eeb50-68db997bf4-s7kpn"
Sep 14 05:07:16.728404 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:16.728382    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-2eeb50\""
Sep 14 05:07:16.729204 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:16.728985    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-2eeb50-dockercfg-2wv7k\""
Sep 14 05:07:16.739068 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:16.738814    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2eeb50-68db997bf4-s7kpn"]
Sep 14 05:07:16.832178 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:16.832125    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97875616-7a3a-4ef7-963d-be811d6ddf46-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-2eeb50-68db997bf4-s7kpn\" (UID: \"97875616-7a3a-4ef7-963d-be811d6ddf46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2eeb50-68db997bf4-s7kpn"
Sep 14 05:07:16.832375 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:16.832190    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkvvf\" (UniqueName: \"kubernetes.io/projected/97875616-7a3a-4ef7-963d-be811d6ddf46-kube-api-access-gkvvf\") pod \"payload-pre-processing-e2e-shared-a-w2-2eeb50-68db997bf4-s7kpn\" (UID: \"97875616-7a3a-4ef7-963d-be811d6ddf46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2eeb50-68db997bf4-s7kpn"
Sep 14 05:07:16.933323 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:16.933281    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97875616-7a3a-4ef7-963d-be811d6ddf46-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-2eeb50-68db997bf4-s7kpn\" (UID: \"97875616-7a3a-4ef7-963d-be811d6ddf46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2eeb50-68db997bf4-s7kpn"
Sep 14 05:07:16.933492 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:16.933362    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkvvf\" (UniqueName: \"kubernetes.io/projected/97875616-7a3a-4ef7-963d-be811d6ddf46-kube-api-access-gkvvf\") pod \"payload-pre-processing-e2e-shared-a-w2-2eeb50-68db997bf4-s7kpn\" (UID: \"97875616-7a3a-4ef7-963d-be811d6ddf46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2eeb50-68db997bf4-s7kpn"
Sep 14 05:07:16.934042 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:16.934021    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97875616-7a3a-4ef7-963d-be811d6ddf46-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-2eeb50-68db997bf4-s7kpn\" (UID: \"97875616-7a3a-4ef7-963d-be811d6ddf46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2eeb50-68db997bf4-s7kpn"
Sep 14 05:07:16.940852 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:16.940830    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkvvf\" (UniqueName: \"kubernetes.io/projected/97875616-7a3a-4ef7-963d-be811d6ddf46-kube-api-access-gkvvf\") pod \"payload-pre-processing-e2e-shared-a-w2-2eeb50-68db997bf4-s7kpn\" (UID: \"97875616-7a3a-4ef7-963d-be811d6ddf46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2eeb50-68db997bf4-s7kpn"
Sep 14 05:07:17.042874 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:17.042804    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2eeb50-68db997bf4-s7kpn"
Sep 14 05:07:17.310550 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:17.310517    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2eeb50-68db997bf4-s7kpn"]
Sep 14 05:07:17.311515 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:07:17.311489    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97875616_7a3a_4ef7_963d_be811d6ddf46.slice/crio-2f75f625d760d7818efbc05ad864067f0e6c0775facf2bb87baff37ae6657622 WatchSource:0}: Error finding container 2f75f625d760d7818efbc05ad864067f0e6c0775facf2bb87baff37ae6657622: Status 404 returned error can't find the container with id 2f75f625d760d7818efbc05ad864067f0e6c0775facf2bb87baff37ae6657622
Sep 14 05:07:18.203279 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:18.203238    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2eeb50-68db997bf4-s7kpn" event={"ID":"97875616-7a3a-4ef7-963d-be811d6ddf46","Type":"ContainerStarted","Data":"667ca8ce211b27b279db436c66e458b32ba7ac182c92da64e83c35b2d86042e2"}
Sep 14 05:07:18.203680 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:18.203298    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2eeb50-68db997bf4-s7kpn" event={"ID":"97875616-7a3a-4ef7-963d-be811d6ddf46","Type":"ContainerStarted","Data":"2f75f625d760d7818efbc05ad864067f0e6c0775facf2bb87baff37ae6657622"}
Sep 14 05:07:18.203680 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:18.203325    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2eeb50-68db997bf4-s7kpn"
Sep 14 05:07:18.220816 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:18.220773    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2eeb50-68db997bf4-s7kpn" podStartSLOduration=2.220745092 podStartE2EDuration="2.220745092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:07:18.218826222 +0000 UTC m=+1236.394597773" watchObservedRunningTime="2026-09-14 05:07:18.220745092 +0000 UTC m=+1236.396516696"
Sep 14 05:07:25.929040 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:25.929002    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:07:29.211382 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:29.211348    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2eeb50-68db997bf4-s7kpn"
Sep 14 05:07:46.407602 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:46.407566    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-9529a1/explicit-model-9529a1-kserve-5845b75467-cp2cm"]
Sep 14 05:07:46.408011 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:46.407919    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c551cf85-0a5a-42a1-af5f-66fcd43e5672" containerName="istio-proxy"
Sep 14 05:07:46.408011 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:46.407930    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c551cf85-0a5a-42a1-af5f-66fcd43e5672" containerName="istio-proxy"
Sep 14 05:07:46.410549 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:46.410532    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-9529a1/explicit-model-9529a1-kserve-5845b75467-cp2cm"
Sep 14 05:07:46.413556 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:46.413534    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-9529a1\"/\"explicit-model-9529a1-kserve-self-signed-certs\""
Sep 14 05:07:46.413692 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:46.413539    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-9529a1\"/\"kube-root-ca.crt\""
Sep 14 05:07:46.413780 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:46.413704    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-9529a1\"/\"openshift-service-ca.crt\""
Sep 14 05:07:46.413780 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:46.413727    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-9529a1\"/\"default-dockercfg-w2khj\""
Sep 14 05:07:46.420131 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:46.420107    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-9529a1/explicit-model-9529a1-kserve-5845b75467-cp2cm"]
Sep 14 05:07:46.492938 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:46.492898    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85ff3c1b-3e6f-4aaa-b641-2aca25661fda-tls-certs\") pod \"explicit-model-9529a1-kserve-5845b75467-cp2cm\" (UID: \"85ff3c1b-3e6f-4aaa-b641-2aca25661fda\") " pod="ai-tenant-e2e-explicit-9529a1/explicit-model-9529a1-kserve-5845b75467-cp2cm"
Sep 14 05:07:46.493085 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:46.493047    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c1b-3e6f-4aaa-b641-2aca25661fda-home\") pod \"explicit-model-9529a1-kserve-5845b75467-cp2cm\" (UID: \"85ff3c1b-3e6f-4aaa-b641-2aca25661fda\") " pod="ai-tenant-e2e-explicit-9529a1/explicit-model-9529a1-kserve-5845b75467-cp2cm"
Sep 14 05:07:46.493085 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:46.493081    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c1b-3e6f-4aaa-b641-2aca25661fda-dshm\") pod \"explicit-model-9529a1-kserve-5845b75467-cp2cm\" (UID: \"85ff3c1b-3e6f-4aaa-b641-2aca25661fda\") " pod="ai-tenant-e2e-explicit-9529a1/explicit-model-9529a1-kserve-5845b75467-cp2cm"
Sep 14 05:07:46.493210 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:46.493110    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c1b-3e6f-4aaa-b641-2aca25661fda-model-cache\") pod \"explicit-model-9529a1-kserve-5845b75467-cp2cm\" (UID: \"85ff3c1b-3e6f-4aaa-b641-2aca25661fda\") " pod="ai-tenant-e2e-explicit-9529a1/explicit-model-9529a1-kserve-5845b75467-cp2cm"
Sep 14 05:07:46.493210 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:46.493152    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddm2n\" (UniqueName: \"kubernetes.io/projected/85ff3c1b-3e6f-4aaa-b641-2aca25661fda-kube-api-access-ddm2n\") pod \"explicit-model-9529a1-kserve-5845b75467-cp2cm\" (UID: \"85ff3c1b-3e6f-4aaa-b641-2aca25661fda\") " pod="ai-tenant-e2e-explicit-9529a1/explicit-model-9529a1-kserve-5845b75467-cp2cm"
Sep 14 05:07:46.493210 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:46.493182    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c1b-3e6f-4aaa-b641-2aca25661fda-tmp-dir\") pod \"explicit-model-9529a1-kserve-5845b75467-cp2cm\" (UID: \"85ff3c1b-3e6f-4aaa-b641-2aca25661fda\") " pod="ai-tenant-e2e-explicit-9529a1/explicit-model-9529a1-kserve-5845b75467-cp2cm"
Sep 14 05:07:46.593961 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:46.593928    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c1b-3e6f-4aaa-b641-2aca25661fda-tmp-dir\") pod \"explicit-model-9529a1-kserve-5845b75467-cp2cm\" (UID: \"85ff3c1b-3e6f-4aaa-b641-2aca25661fda\") " pod="ai-tenant-e2e-explicit-9529a1/explicit-model-9529a1-kserve-5845b75467-cp2cm"
Sep 14 05:07:46.594096 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:46.593971    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85ff3c1b-3e6f-4aaa-b641-2aca25661fda-tls-certs\") pod \"explicit-model-9529a1-kserve-5845b75467-cp2cm\" (UID: \"85ff3c1b-3e6f-4aaa-b641-2aca25661fda\") " pod="ai-tenant-e2e-explicit-9529a1/explicit-model-9529a1-kserve-5845b75467-cp2cm"
Sep 14 05:07:46.594096 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:46.594063    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c1b-3e6f-4aaa-b641-2aca25661fda-home\") pod \"explicit-model-9529a1-kserve-5845b75467-cp2cm\" (UID: \"85ff3c1b-3e6f-4aaa-b641-2aca25661fda\") " pod="ai-tenant-e2e-explicit-9529a1/explicit-model-9529a1-kserve-5845b75467-cp2cm"
Sep 14 05:07:46.594229 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:46.594095    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c1b-3e6f-4aaa-b641-2aca25661fda-dshm\") pod \"explicit-model-9529a1-kserve-5845b75467-cp2cm\" (UID: \"85ff3c1b-3e6f-4aaa-b641-2aca25661fda\") " pod="ai-tenant-e2e-explicit-9529a1/explicit-model-9529a1-kserve-5845b75467-cp2cm"
Sep 14 05:07:46.594229 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:46.594128    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c1b-3e6f-4aaa-b641-2aca25661fda-model-cache\") pod \"explicit-model-9529a1-kserve-5845b75467-cp2cm\" (UID: \"85ff3c1b-3e6f-4aaa-b641-2aca25661fda\") " pod="ai-tenant-e2e-explicit-9529a1/explicit-model-9529a1-kserve-5845b75467-cp2cm"
Sep 14 05:07:46.594229 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:46.594172    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddm2n\" (UniqueName: \"kubernetes.io/projected/85ff3c1b-3e6f-4aaa-b641-2aca25661fda-kube-api-access-ddm2n\") pod \"explicit-model-9529a1-kserve-5845b75467-cp2cm\" (UID: \"85ff3c1b-3e6f-4aaa-b641-2aca25661fda\") " pod="ai-tenant-e2e-explicit-9529a1/explicit-model-9529a1-kserve-5845b75467-cp2cm"
Sep 14 05:07:46.594393 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:46.594366    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c1b-3e6f-4aaa-b641-2aca25661fda-tmp-dir\") pod \"explicit-model-9529a1-kserve-5845b75467-cp2cm\" (UID: \"85ff3c1b-3e6f-4aaa-b641-2aca25661fda\") " pod="ai-tenant-e2e-explicit-9529a1/explicit-model-9529a1-kserve-5845b75467-cp2cm"
Sep 14 05:07:46.594457 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:46.594423    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c1b-3e6f-4aaa-b641-2aca25661fda-home\") pod \"explicit-model-9529a1-kserve-5845b75467-cp2cm\" (UID: \"85ff3c1b-3e6f-4aaa-b641-2aca25661fda\") " pod="ai-tenant-e2e-explicit-9529a1/explicit-model-9529a1-kserve-5845b75467-cp2cm"
Sep 14 05:07:46.594457 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:46.594443    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c1b-3e6f-4aaa-b641-2aca25661fda-model-cache\") pod \"explicit-model-9529a1-kserve-5845b75467-cp2cm\" (UID: \"85ff3c1b-3e6f-4aaa-b641-2aca25661fda\") " pod="ai-tenant-e2e-explicit-9529a1/explicit-model-9529a1-kserve-5845b75467-cp2cm"
Sep 14 05:07:46.596282 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:46.596262    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c1b-3e6f-4aaa-b641-2aca25661fda-dshm\") pod \"explicit-model-9529a1-kserve-5845b75467-cp2cm\" (UID: \"85ff3c1b-3e6f-4aaa-b641-2aca25661fda\") " pod="ai-tenant-e2e-explicit-9529a1/explicit-model-9529a1-kserve-5845b75467-cp2cm"
Sep 14 05:07:46.596398 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:46.596382    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85ff3c1b-3e6f-4aaa-b641-2aca25661fda-tls-certs\") pod \"explicit-model-9529a1-kserve-5845b75467-cp2cm\" (UID: \"85ff3c1b-3e6f-4aaa-b641-2aca25661fda\") " pod="ai-tenant-e2e-explicit-9529a1/explicit-model-9529a1-kserve-5845b75467-cp2cm"
Sep 14 05:07:46.606375 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:46.606355    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddm2n\" (UniqueName: \"kubernetes.io/projected/85ff3c1b-3e6f-4aaa-b641-2aca25661fda-kube-api-access-ddm2n\") pod \"explicit-model-9529a1-kserve-5845b75467-cp2cm\" (UID: \"85ff3c1b-3e6f-4aaa-b641-2aca25661fda\") " pod="ai-tenant-e2e-explicit-9529a1/explicit-model-9529a1-kserve-5845b75467-cp2cm"
Sep 14 05:07:46.721015 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:46.720944    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-9529a1/explicit-model-9529a1-kserve-5845b75467-cp2cm"
Sep 14 05:07:46.995310 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:46.995277    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-9529a1/explicit-model-9529a1-kserve-5845b75467-cp2cm"]
Sep 14 05:07:46.997112 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:07:46.997089    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85ff3c1b_3e6f_4aaa_b641_2aca25661fda.slice/crio-bc185ca8bd88c48b9ccb13afe4890ddddec6607647ed8c56bbb879d035c35190 WatchSource:0}: Error finding container bc185ca8bd88c48b9ccb13afe4890ddddec6607647ed8c56bbb879d035c35190: Status 404 returned error can't find the container with id bc185ca8bd88c48b9ccb13afe4890ddddec6607647ed8c56bbb879d035c35190
Sep 14 05:07:47.359818 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:47.359783    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-9529a1/explicit-model-9529a1-kserve-5845b75467-cp2cm" event={"ID":"85ff3c1b-3e6f-4aaa-b641-2aca25661fda","Type":"ContainerStarted","Data":"9030c9c7134be58abf0c095eace3b51b35d1b40fc4c2d68da310926273f56f0f"}
Sep 14 05:07:47.359995 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:47.359830    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-9529a1/explicit-model-9529a1-kserve-5845b75467-cp2cm" event={"ID":"85ff3c1b-3e6f-4aaa-b641-2aca25661fda","Type":"ContainerStarted","Data":"bc185ca8bd88c48b9ccb13afe4890ddddec6607647ed8c56bbb879d035c35190"}
Sep 14 05:07:47.381283 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:47.381240    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-9529a1/explicit-model-9529a1-kserve-5845b75467-cp2cm" podStartSLOduration=1.381225752 podStartE2EDuration="1.381225752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:07:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:07:47.378898564 +0000 UTC m=+1265.554670124" watchObservedRunningTime="2026-09-14 05:07:47.381225752 +0000 UTC m=+1265.556997311"
Sep 14 05:07:51.485815 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:51.485776    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a8c46f-8585cb8f4-vjtwx"]
Sep 14 05:07:51.489677 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:51.489655    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a8c46f-8585cb8f4-vjtwx"
Sep 14 05:07:51.492061 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:51.492038    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-a8c46f-dockercfg-55qzd\""
Sep 14 05:07:51.492177 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:51.492046    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-a8c46f\""
Sep 14 05:07:51.500143 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:51.500118    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a8c46f-8585cb8f4-vjtwx"]
Sep 14 05:07:51.522965 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:51.522932    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a8c46f-746b7b7d67-844g5"]
Sep 14 05:07:51.526208 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:51.526185    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a8c46f-746b7b7d67-844g5"
Sep 14 05:07:51.536924 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:51.536801    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a8c46f-746b7b7d67-844g5"]
Sep 14 05:07:51.541629 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:51.541604    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc1256d8-060b-45b4-a613-5b57ea866bc9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-a8c46f-8585cb8f4-vjtwx\" (UID: \"fc1256d8-060b-45b4-a613-5b57ea866bc9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a8c46f-8585cb8f4-vjtwx"
Sep 14 05:07:51.541799 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:51.541635    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsnwp\" (UniqueName: \"kubernetes.io/projected/fc1256d8-060b-45b4-a613-5b57ea866bc9-kube-api-access-qsnwp\") pod \"payload-pre-processing-e2e-shared-b-w2-a8c46f-8585cb8f4-vjtwx\" (UID: \"fc1256d8-060b-45b4-a613-5b57ea866bc9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a8c46f-8585cb8f4-vjtwx"
Sep 14 05:07:51.642714 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:51.642677    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/993a22f2-b2c6-465b-b468-ce054475e45b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-a8c46f-746b7b7d67-844g5\" (UID: \"993a22f2-b2c6-465b-b468-ce054475e45b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a8c46f-746b7b7d67-844g5"
Sep 14 05:07:51.642909 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:51.642728    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc7fw\" (UniqueName: \"kubernetes.io/projected/993a22f2-b2c6-465b-b468-ce054475e45b-kube-api-access-lc7fw\") pod \"payload-processing-e2e-shared-b-w2-a8c46f-746b7b7d67-844g5\" (UID: \"993a22f2-b2c6-465b-b468-ce054475e45b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a8c46f-746b7b7d67-844g5"
Sep 14 05:07:51.642909 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:51.642777    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc1256d8-060b-45b4-a613-5b57ea866bc9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-a8c46f-8585cb8f4-vjtwx\" (UID: \"fc1256d8-060b-45b4-a613-5b57ea866bc9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a8c46f-8585cb8f4-vjtwx"
Sep 14 05:07:51.642909 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:51.642802    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsnwp\" (UniqueName: \"kubernetes.io/projected/fc1256d8-060b-45b4-a613-5b57ea866bc9-kube-api-access-qsnwp\") pod \"payload-pre-processing-e2e-shared-b-w2-a8c46f-8585cb8f4-vjtwx\" (UID: \"fc1256d8-060b-45b4-a613-5b57ea866bc9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a8c46f-8585cb8f4-vjtwx"
Sep 14 05:07:51.643456 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:51.643436    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc1256d8-060b-45b4-a613-5b57ea866bc9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-a8c46f-8585cb8f4-vjtwx\" (UID: \"fc1256d8-060b-45b4-a613-5b57ea866bc9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a8c46f-8585cb8f4-vjtwx"
Sep 14 05:07:51.650742 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:51.650714    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsnwp\" (UniqueName: \"kubernetes.io/projected/fc1256d8-060b-45b4-a613-5b57ea866bc9-kube-api-access-qsnwp\") pod \"payload-pre-processing-e2e-shared-b-w2-a8c46f-8585cb8f4-vjtwx\" (UID: \"fc1256d8-060b-45b4-a613-5b57ea866bc9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a8c46f-8585cb8f4-vjtwx"
Sep 14 05:07:51.744208 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:51.744129    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/993a22f2-b2c6-465b-b468-ce054475e45b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-a8c46f-746b7b7d67-844g5\" (UID: \"993a22f2-b2c6-465b-b468-ce054475e45b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a8c46f-746b7b7d67-844g5"
Sep 14 05:07:51.744208 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:51.744173    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc7fw\" (UniqueName: \"kubernetes.io/projected/993a22f2-b2c6-465b-b468-ce054475e45b-kube-api-access-lc7fw\") pod \"payload-processing-e2e-shared-b-w2-a8c46f-746b7b7d67-844g5\" (UID: \"993a22f2-b2c6-465b-b468-ce054475e45b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a8c46f-746b7b7d67-844g5"
Sep 14 05:07:51.744845 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:51.744824    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/993a22f2-b2c6-465b-b468-ce054475e45b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-a8c46f-746b7b7d67-844g5\" (UID: \"993a22f2-b2c6-465b-b468-ce054475e45b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a8c46f-746b7b7d67-844g5"
Sep 14 05:07:51.751705 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:51.751680    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc7fw\" (UniqueName: \"kubernetes.io/projected/993a22f2-b2c6-465b-b468-ce054475e45b-kube-api-access-lc7fw\") pod \"payload-processing-e2e-shared-b-w2-a8c46f-746b7b7d67-844g5\" (UID: \"993a22f2-b2c6-465b-b468-ce054475e45b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a8c46f-746b7b7d67-844g5"
Sep 14 05:07:51.802274 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:51.802240    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a8c46f-8585cb8f4-vjtwx"
Sep 14 05:07:51.838658 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:51.838618    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a8c46f-746b7b7d67-844g5"
Sep 14 05:07:52.077312 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:52.077280    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a8c46f-8585cb8f4-vjtwx"]
Sep 14 05:07:52.078531 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:07:52.078508    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc1256d8_060b_45b4_a613_5b57ea866bc9.slice/crio-d20cecff66725e7855a8b823543670225801cf72fbb4a074356da243e7d91280 WatchSource:0}: Error finding container d20cecff66725e7855a8b823543670225801cf72fbb4a074356da243e7d91280: Status 404 returned error can't find the container with id d20cecff66725e7855a8b823543670225801cf72fbb4a074356da243e7d91280
Sep 14 05:07:52.125896 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:52.125864    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a8c46f-746b7b7d67-844g5"]
Sep 14 05:07:52.129903 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:07:52.129878    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod993a22f2_b2c6_465b_b468_ce054475e45b.slice/crio-d056a4dfc78438f65ed71693f3f94150fac812bc1b88252ab0f58124208bde0a WatchSource:0}: Error finding container d056a4dfc78438f65ed71693f3f94150fac812bc1b88252ab0f58124208bde0a: Status 404 returned error can't find the container with id d056a4dfc78438f65ed71693f3f94150fac812bc1b88252ab0f58124208bde0a
Sep 14 05:07:52.390566 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:52.390460    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a8c46f-746b7b7d67-844g5" event={"ID":"993a22f2-b2c6-465b-b468-ce054475e45b","Type":"ContainerStarted","Data":"85af6f68fff5ec45dc7c39029f0d011f73b1f04bbf43018424678dad8137eaf6"}
Sep 14 05:07:52.390566 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:52.390529    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a8c46f-746b7b7d67-844g5" event={"ID":"993a22f2-b2c6-465b-b468-ce054475e45b","Type":"ContainerStarted","Data":"d056a4dfc78438f65ed71693f3f94150fac812bc1b88252ab0f58124208bde0a"}
Sep 14 05:07:52.390872 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:52.390653    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a8c46f-746b7b7d67-844g5"
Sep 14 05:07:52.392539 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:52.392513    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a8c46f-8585cb8f4-vjtwx" event={"ID":"fc1256d8-060b-45b4-a613-5b57ea866bc9","Type":"ContainerStarted","Data":"8f5085adadb13bcee960f3221962f4413f2e64cc38cb9a50e66a41736b3c0336"}
Sep 14 05:07:52.392646 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:52.392561    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a8c46f-8585cb8f4-vjtwx" event={"ID":"fc1256d8-060b-45b4-a613-5b57ea866bc9","Type":"ContainerStarted","Data":"d20cecff66725e7855a8b823543670225801cf72fbb4a074356da243e7d91280"}
Sep 14 05:07:52.392646 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:52.392591    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a8c46f-8585cb8f4-vjtwx"
Sep 14 05:07:52.409705 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:52.409654    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a8c46f-746b7b7d67-844g5" podStartSLOduration=1.409632669 podStartE2EDuration="1.409632669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:07:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:07:52.408978556 +0000 UTC m=+1270.584750140" watchObservedRunningTime="2026-09-14 05:07:52.409632669 +0000 UTC m=+1270.585404274"
Sep 14 05:07:52.425124 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:52.425064    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a8c46f-8585cb8f4-vjtwx" podStartSLOduration=1.4250477830000001 podStartE2EDuration="1.425047783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:07:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:07:52.422653138 +0000 UTC m=+1270.598424699" watchObservedRunningTime="2026-09-14 05:07:52.425047783 +0000 UTC m=+1270.600819341"
Sep 14 05:07:52.443369 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:52.443337    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:07:56.721394 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:56.721353    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-9529a1/explicit-model-9529a1-kserve-5845b75467-cp2cm"
Sep 14 05:07:56.722022 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:56.721412    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-9529a1/explicit-model-9529a1-kserve-5845b75467-cp2cm"
Sep 14 05:07:56.733714 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:56.733691    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-9529a1/explicit-model-9529a1-kserve-5845b75467-cp2cm"
Sep 14 05:07:57.438804 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:07:57.438778    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-9529a1/explicit-model-9529a1-kserve-5845b75467-cp2cm"
Sep 14 05:08:03.402332 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:03.402301    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a8c46f-746b7b7d67-844g5"
Sep 14 05:08:03.402676 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:03.402383    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a8c46f-8585cb8f4-vjtwx"
Sep 14 05:08:13.716712 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:13.716676    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-9529a1/explicit-model-9529a1-kserve-5845b75467-cp2cm"]
Sep 14 05:08:19.830283 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:19.830204    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:08:24.578390 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:24.578357    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-9529a1-6f7495cf77-p6vbq"]
Sep 14 05:08:24.579036 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:24.578561    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-9529a1-6f7495cf77-p6vbq" podUID="1957684e-22df-458a-b66b-ab24e745d37d" containerName="payload-processing" containerID="cri-o://d45f822abcda3a2cb2cbb425fdbdba1b0bdd09553d2510cce14bf883dd8ced25" gracePeriod=30
Sep 14 05:08:24.587280 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:24.587251    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-9529a1-7f565fd978-mtdwz"]
Sep 14 05:08:24.587636 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:24.587605    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9529a1-7f565fd978-mtdwz" podUID="31e7684e-18d5-48a0-91a3-a186a552d733" containerName="payload-pre-processing" containerID="cri-o://eae36b4a44b8c098707638fb420a21091bf52e0063043fe37464a3a706a71eeb" gracePeriod=30
Sep 14 05:08:24.864828 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:24.864804    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-9529a1-6f7495cf77-p6vbq"
Sep 14 05:08:24.867688 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:24.867668    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9529a1-7f565fd978-mtdwz"
Sep 14 05:08:24.938859 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:24.938827    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bslbz\" (UniqueName: \"kubernetes.io/projected/1957684e-22df-458a-b66b-ab24e745d37d-kube-api-access-bslbz\") pod \"1957684e-22df-458a-b66b-ab24e745d37d\" (UID: \"1957684e-22df-458a-b66b-ab24e745d37d\") "
Sep 14 05:08:24.939005 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:24.938874    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31e7684e-18d5-48a0-91a3-a186a552d733-plugins-config-volume\") pod \"31e7684e-18d5-48a0-91a3-a186a552d733\" (UID: \"31e7684e-18d5-48a0-91a3-a186a552d733\") "
Sep 14 05:08:24.939005 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:24.938911    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1957684e-22df-458a-b66b-ab24e745d37d-plugins-config-volume\") pod \"1957684e-22df-458a-b66b-ab24e745d37d\" (UID: \"1957684e-22df-458a-b66b-ab24e745d37d\") "
Sep 14 05:08:24.939005 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:24.938992    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmn9h\" (UniqueName: \"kubernetes.io/projected/31e7684e-18d5-48a0-91a3-a186a552d733-kube-api-access-bmn9h\") pod \"31e7684e-18d5-48a0-91a3-a186a552d733\" (UID: \"31e7684e-18d5-48a0-91a3-a186a552d733\") "
Sep 14 05:08:24.939316 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:24.939284    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31e7684e-18d5-48a0-91a3-a186a552d733-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "31e7684e-18d5-48a0-91a3-a186a552d733" (UID: "31e7684e-18d5-48a0-91a3-a186a552d733"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 05:08:24.939429 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:24.939353    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1957684e-22df-458a-b66b-ab24e745d37d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1957684e-22df-458a-b66b-ab24e745d37d" (UID: "1957684e-22df-458a-b66b-ab24e745d37d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 05:08:24.941084 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:24.941063    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1957684e-22df-458a-b66b-ab24e745d37d-kube-api-access-bslbz" (OuterVolumeSpecName: "kube-api-access-bslbz") pod "1957684e-22df-458a-b66b-ab24e745d37d" (UID: "1957684e-22df-458a-b66b-ab24e745d37d"). InnerVolumeSpecName "kube-api-access-bslbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:08:24.941154 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:24.941068    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31e7684e-18d5-48a0-91a3-a186a552d733-kube-api-access-bmn9h" (OuterVolumeSpecName: "kube-api-access-bmn9h") pod "31e7684e-18d5-48a0-91a3-a186a552d733" (UID: "31e7684e-18d5-48a0-91a3-a186a552d733"). InnerVolumeSpecName "kube-api-access-bmn9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:08:25.039786 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:25.039711    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmn9h\" (UniqueName: \"kubernetes.io/projected/31e7684e-18d5-48a0-91a3-a186a552d733-kube-api-access-bmn9h\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:08:25.039786 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:25.039778    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bslbz\" (UniqueName: \"kubernetes.io/projected/1957684e-22df-458a-b66b-ab24e745d37d-kube-api-access-bslbz\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:08:25.039786 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:25.039789    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31e7684e-18d5-48a0-91a3-a186a552d733-plugins-config-volume\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:08:25.039786 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:25.039797    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1957684e-22df-458a-b66b-ab24e745d37d-plugins-config-volume\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:08:25.581209 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:25.581175    2565 generic.go:356] "Generic (PLEG): container finished" podID="1957684e-22df-458a-b66b-ab24e745d37d" containerID="d45f822abcda3a2cb2cbb425fdbdba1b0bdd09553d2510cce14bf883dd8ced25" exitCode=0
Sep 14 05:08:25.581683 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:25.581224    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-9529a1-6f7495cf77-p6vbq" event={"ID":"1957684e-22df-458a-b66b-ab24e745d37d","Type":"ContainerDied","Data":"d45f822abcda3a2cb2cbb425fdbdba1b0bdd09553d2510cce14bf883dd8ced25"}
Sep 14 05:08:25.581683 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:25.581251    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-9529a1-6f7495cf77-p6vbq"
Sep 14 05:08:25.581683 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:25.581290    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-9529a1-6f7495cf77-p6vbq" event={"ID":"1957684e-22df-458a-b66b-ab24e745d37d","Type":"ContainerDied","Data":"6c0ebe81894d22cbd2d004d14d933da10c8963711ae4f66d9bf4b344c27a967c"}
Sep 14 05:08:25.581683 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:25.581317    2565 scope.go:117] "RemoveContainer" containerID="d45f822abcda3a2cb2cbb425fdbdba1b0bdd09553d2510cce14bf883dd8ced25"
Sep 14 05:08:25.582687 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:25.582643    2565 generic.go:356] "Generic (PLEG): container finished" podID="31e7684e-18d5-48a0-91a3-a186a552d733" containerID="eae36b4a44b8c098707638fb420a21091bf52e0063043fe37464a3a706a71eeb" exitCode=0
Sep 14 05:08:25.582687 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:25.582670    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9529a1-7f565fd978-mtdwz" event={"ID":"31e7684e-18d5-48a0-91a3-a186a552d733","Type":"ContainerDied","Data":"eae36b4a44b8c098707638fb420a21091bf52e0063043fe37464a3a706a71eeb"}
Sep 14 05:08:25.582821 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:25.582704    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9529a1-7f565fd978-mtdwz"
Sep 14 05:08:25.582821 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:25.582711    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9529a1-7f565fd978-mtdwz" event={"ID":"31e7684e-18d5-48a0-91a3-a186a552d733","Type":"ContainerDied","Data":"ed008f716ab3b6053ed74be5d4bfb9355a7cc9a9ed1d14d7c0ded0a11728d9ce"}
Sep 14 05:08:25.592213 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:25.592193    2565 scope.go:117] "RemoveContainer" containerID="d45f822abcda3a2cb2cbb425fdbdba1b0bdd09553d2510cce14bf883dd8ced25"
Sep 14 05:08:25.592500 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:08:25.592482    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d45f822abcda3a2cb2cbb425fdbdba1b0bdd09553d2510cce14bf883dd8ced25\": container with ID starting with d45f822abcda3a2cb2cbb425fdbdba1b0bdd09553d2510cce14bf883dd8ced25 not found: ID does not exist" containerID="d45f822abcda3a2cb2cbb425fdbdba1b0bdd09553d2510cce14bf883dd8ced25"
Sep 14 05:08:25.592555 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:25.592508    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d45f822abcda3a2cb2cbb425fdbdba1b0bdd09553d2510cce14bf883dd8ced25"} err="failed to get container status \"d45f822abcda3a2cb2cbb425fdbdba1b0bdd09553d2510cce14bf883dd8ced25\": rpc error: code = NotFound desc = could not find container \"d45f822abcda3a2cb2cbb425fdbdba1b0bdd09553d2510cce14bf883dd8ced25\": container with ID starting with d45f822abcda3a2cb2cbb425fdbdba1b0bdd09553d2510cce14bf883dd8ced25 not found: ID does not exist"
Sep 14 05:08:25.592555 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:25.592525    2565 scope.go:117] "RemoveContainer" containerID="eae36b4a44b8c098707638fb420a21091bf52e0063043fe37464a3a706a71eeb"
Sep 14 05:08:25.602816 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:25.602801    2565 scope.go:117] "RemoveContainer" containerID="eae36b4a44b8c098707638fb420a21091bf52e0063043fe37464a3a706a71eeb"
Sep 14 05:08:25.603136 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:08:25.603115    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eae36b4a44b8c098707638fb420a21091bf52e0063043fe37464a3a706a71eeb\": container with ID starting with eae36b4a44b8c098707638fb420a21091bf52e0063043fe37464a3a706a71eeb not found: ID does not exist" containerID="eae36b4a44b8c098707638fb420a21091bf52e0063043fe37464a3a706a71eeb"
Sep 14 05:08:25.603203 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:25.603142    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eae36b4a44b8c098707638fb420a21091bf52e0063043fe37464a3a706a71eeb"} err="failed to get container status \"eae36b4a44b8c098707638fb420a21091bf52e0063043fe37464a3a706a71eeb\": rpc error: code = NotFound desc = could not find container \"eae36b4a44b8c098707638fb420a21091bf52e0063043fe37464a3a706a71eeb\": container with ID starting with eae36b4a44b8c098707638fb420a21091bf52e0063043fe37464a3a706a71eeb not found: ID does not exist"
Sep 14 05:08:25.613244 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:25.613222    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-9529a1-7f565fd978-mtdwz"]
Sep 14 05:08:25.616189 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:25.616169    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-9529a1-7f565fd978-mtdwz"]
Sep 14 05:08:25.624491 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:25.624472    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-9529a1-6f7495cf77-p6vbq"]
Sep 14 05:08:25.627877 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:25.627857    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-9529a1-6f7495cf77-p6vbq"]
Sep 14 05:08:26.335488 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:26.335457    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1957684e-22df-458a-b66b-ab24e745d37d" path="/var/lib/kubelet/pods/1957684e-22df-458a-b66b-ab24e745d37d/volumes"
Sep 14 05:08:26.335838 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:26.335824    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31e7684e-18d5-48a0-91a3-a186a552d733" path="/var/lib/kubelet/pods/31e7684e-18d5-48a0-91a3-a186a552d733/volumes"
Sep 14 05:08:28.736203 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:28.736156    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-9529a1/explicit-model-9529a1-kserve-5845b75467-cp2cm" podUID="85ff3c1b-3e6f-4aaa-b641-2aca25661fda" containerName="main" containerID="cri-o://9030c9c7134be58abf0c095eace3b51b35d1b40fc4c2d68da310926273f56f0f" gracePeriod=45
Sep 14 05:08:28.985721 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:28.985698    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-9529a1/explicit-model-9529a1-kserve-5845b75467-cp2cm"
Sep 14 05:08:29.076929 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.075036    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85ff3c1b-3e6f-4aaa-b641-2aca25661fda-tls-certs\") pod \"85ff3c1b-3e6f-4aaa-b641-2aca25661fda\" (UID: \"85ff3c1b-3e6f-4aaa-b641-2aca25661fda\") "
Sep 14 05:08:29.076929 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.075085    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddm2n\" (UniqueName: \"kubernetes.io/projected/85ff3c1b-3e6f-4aaa-b641-2aca25661fda-kube-api-access-ddm2n\") pod \"85ff3c1b-3e6f-4aaa-b641-2aca25661fda\" (UID: \"85ff3c1b-3e6f-4aaa-b641-2aca25661fda\") "
Sep 14 05:08:29.076929 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.075131    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c1b-3e6f-4aaa-b641-2aca25661fda-dshm\") pod \"85ff3c1b-3e6f-4aaa-b641-2aca25661fda\" (UID: \"85ff3c1b-3e6f-4aaa-b641-2aca25661fda\") "
Sep 14 05:08:29.076929 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.075167    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c1b-3e6f-4aaa-b641-2aca25661fda-tmp-dir\") pod \"85ff3c1b-3e6f-4aaa-b641-2aca25661fda\" (UID: \"85ff3c1b-3e6f-4aaa-b641-2aca25661fda\") "
Sep 14 05:08:29.076929 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.075191    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c1b-3e6f-4aaa-b641-2aca25661fda-home\") pod \"85ff3c1b-3e6f-4aaa-b641-2aca25661fda\" (UID: \"85ff3c1b-3e6f-4aaa-b641-2aca25661fda\") "
Sep 14 05:08:29.076929 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.075254    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c1b-3e6f-4aaa-b641-2aca25661fda-model-cache\") pod \"85ff3c1b-3e6f-4aaa-b641-2aca25661fda\" (UID: \"85ff3c1b-3e6f-4aaa-b641-2aca25661fda\") "
Sep 14 05:08:29.076929 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.075909    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85ff3c1b-3e6f-4aaa-b641-2aca25661fda-model-cache" (OuterVolumeSpecName: "model-cache") pod "85ff3c1b-3e6f-4aaa-b641-2aca25661fda" (UID: "85ff3c1b-3e6f-4aaa-b641-2aca25661fda"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:08:29.078650 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.078616    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85ff3c1b-3e6f-4aaa-b641-2aca25661fda-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "85ff3c1b-3e6f-4aaa-b641-2aca25661fda" (UID: "85ff3c1b-3e6f-4aaa-b641-2aca25661fda"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 05:08:29.081047 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.081018    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85ff3c1b-3e6f-4aaa-b641-2aca25661fda-kube-api-access-ddm2n" (OuterVolumeSpecName: "kube-api-access-ddm2n") pod "85ff3c1b-3e6f-4aaa-b641-2aca25661fda" (UID: "85ff3c1b-3e6f-4aaa-b641-2aca25661fda"). InnerVolumeSpecName "kube-api-access-ddm2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:08:29.081228 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.081201    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85ff3c1b-3e6f-4aaa-b641-2aca25661fda-home" (OuterVolumeSpecName: "home") pod "85ff3c1b-3e6f-4aaa-b641-2aca25661fda" (UID: "85ff3c1b-3e6f-4aaa-b641-2aca25661fda"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:08:29.081307 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.081178    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85ff3c1b-3e6f-4aaa-b641-2aca25661fda-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "85ff3c1b-3e6f-4aaa-b641-2aca25661fda" (UID: "85ff3c1b-3e6f-4aaa-b641-2aca25661fda"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:08:29.083596 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.083494    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85ff3c1b-3e6f-4aaa-b641-2aca25661fda-dshm" (OuterVolumeSpecName: "dshm") pod "85ff3c1b-3e6f-4aaa-b641-2aca25661fda" (UID: "85ff3c1b-3e6f-4aaa-b641-2aca25661fda"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:08:29.176144 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.176109    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85ff3c1b-3e6f-4aaa-b641-2aca25661fda-tls-certs\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:08:29.176308 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.176159    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddm2n\" (UniqueName: \"kubernetes.io/projected/85ff3c1b-3e6f-4aaa-b641-2aca25661fda-kube-api-access-ddm2n\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:08:29.176308 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.176169    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c1b-3e6f-4aaa-b641-2aca25661fda-dshm\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:08:29.176308 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.176179    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c1b-3e6f-4aaa-b641-2aca25661fda-tmp-dir\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:08:29.176308 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.176187    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c1b-3e6f-4aaa-b641-2aca25661fda-home\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:08:29.176308 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.176195    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c1b-3e6f-4aaa-b641-2aca25661fda-model-cache\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:08:29.339541 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.339506    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"]
Sep 14 05:08:29.339922 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.339907    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85ff3c1b-3e6f-4aaa-b641-2aca25661fda" containerName="main"
Sep 14 05:08:29.339970 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.339923    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="85ff3c1b-3e6f-4aaa-b641-2aca25661fda" containerName="main"
Sep 14 05:08:29.339970 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.339932    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31e7684e-18d5-48a0-91a3-a186a552d733" containerName="payload-pre-processing"
Sep 14 05:08:29.339970 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.339938    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="31e7684e-18d5-48a0-91a3-a186a552d733" containerName="payload-pre-processing"
Sep 14 05:08:29.339970 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.339959    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1957684e-22df-458a-b66b-ab24e745d37d" containerName="payload-processing"
Sep 14 05:08:29.339970 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.339965    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1957684e-22df-458a-b66b-ab24e745d37d" containerName="payload-processing"
Sep 14 05:08:29.340150 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.340017    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1957684e-22df-458a-b66b-ab24e745d37d" containerName="payload-processing"
Sep 14 05:08:29.340150 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.340027    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="31e7684e-18d5-48a0-91a3-a186a552d733" containerName="payload-pre-processing"
Sep 14 05:08:29.340150 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.340035    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="85ff3c1b-3e6f-4aaa-b641-2aca25661fda" containerName="main"
Sep 14 05:08:29.343160 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.343144    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"
Sep 14 05:08:29.345428 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.345410    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c7c6a48c-openshift-default-dockercfg-56m96\""
Sep 14 05:08:29.354906 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.354881    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"]
Sep 14 05:08:29.377652 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.377621    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8753cc9c-d346-441f-ba35-725feb7818d3-istio-token\") pod \"e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd\" (UID: \"8753cc9c-d346-441f-ba35-725feb7818d3\") " pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"
Sep 14 05:08:29.377791 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.377736    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8753cc9c-d346-441f-ba35-725feb7818d3-istio-envoy\") pod \"e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd\" (UID: \"8753cc9c-d346-441f-ba35-725feb7818d3\") " pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"
Sep 14 05:08:29.377866 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.377820    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8753cc9c-d346-441f-ba35-725feb7818d3-istio-data\") pod \"e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd\" (UID: \"8753cc9c-d346-441f-ba35-725feb7818d3\") " pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"
Sep 14 05:08:29.377866 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.377848    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8753cc9c-d346-441f-ba35-725feb7818d3-istio-podinfo\") pod \"e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd\" (UID: \"8753cc9c-d346-441f-ba35-725feb7818d3\") " pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"
Sep 14 05:08:29.377974 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.377950    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8753cc9c-d346-441f-ba35-725feb7818d3-istiod-ca-cert\") pod \"e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd\" (UID: \"8753cc9c-d346-441f-ba35-725feb7818d3\") " pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"
Sep 14 05:08:29.378049 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.378035    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8753cc9c-d346-441f-ba35-725feb7818d3-workload-socket\") pod \"e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd\" (UID: \"8753cc9c-d346-441f-ba35-725feb7818d3\") " pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"
Sep 14 05:08:29.378085 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.378065    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8753cc9c-d346-441f-ba35-725feb7818d3-credential-socket\") pod \"e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd\" (UID: \"8753cc9c-d346-441f-ba35-725feb7818d3\") " pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"
Sep 14 05:08:29.378121 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.378088    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8753cc9c-d346-441f-ba35-725feb7818d3-workload-certs\") pod \"e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd\" (UID: \"8753cc9c-d346-441f-ba35-725feb7818d3\") " pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"
Sep 14 05:08:29.378153 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.378123    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp824\" (UniqueName: \"kubernetes.io/projected/8753cc9c-d346-441f-ba35-725feb7818d3-kube-api-access-cp824\") pod \"e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd\" (UID: \"8753cc9c-d346-441f-ba35-725feb7818d3\") " pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"
Sep 14 05:08:29.479310 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.479276    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8753cc9c-d346-441f-ba35-725feb7818d3-workload-socket\") pod \"e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd\" (UID: \"8753cc9c-d346-441f-ba35-725feb7818d3\") " pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"
Sep 14 05:08:29.479310 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.479310    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8753cc9c-d346-441f-ba35-725feb7818d3-credential-socket\") pod \"e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd\" (UID: \"8753cc9c-d346-441f-ba35-725feb7818d3\") " pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"
Sep 14 05:08:29.479515 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.479325    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8753cc9c-d346-441f-ba35-725feb7818d3-workload-certs\") pod \"e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd\" (UID: \"8753cc9c-d346-441f-ba35-725feb7818d3\") " pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"
Sep 14 05:08:29.479515 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.479342    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp824\" (UniqueName: \"kubernetes.io/projected/8753cc9c-d346-441f-ba35-725feb7818d3-kube-api-access-cp824\") pod \"e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd\" (UID: \"8753cc9c-d346-441f-ba35-725feb7818d3\") " pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"
Sep 14 05:08:29.479515 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.479466    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8753cc9c-d346-441f-ba35-725feb7818d3-istio-token\") pod \"e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd\" (UID: \"8753cc9c-d346-441f-ba35-725feb7818d3\") " pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"
Sep 14 05:08:29.479681 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.479547    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8753cc9c-d346-441f-ba35-725feb7818d3-istio-envoy\") pod \"e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd\" (UID: \"8753cc9c-d346-441f-ba35-725feb7818d3\") " pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"
Sep 14 05:08:29.479681 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.479608    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8753cc9c-d346-441f-ba35-725feb7818d3-istio-data\") pod \"e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd\" (UID: \"8753cc9c-d346-441f-ba35-725feb7818d3\") " pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"
Sep 14 05:08:29.479681 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.479659    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8753cc9c-d346-441f-ba35-725feb7818d3-workload-socket\") pod \"e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd\" (UID: \"8753cc9c-d346-441f-ba35-725feb7818d3\") " pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"
Sep 14 05:08:29.479875 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.479711    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8753cc9c-d346-441f-ba35-725feb7818d3-istio-podinfo\") pod \"e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd\" (UID: \"8753cc9c-d346-441f-ba35-725feb7818d3\") " pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"
Sep 14 05:08:29.479875 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.479798    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8753cc9c-d346-441f-ba35-725feb7818d3-istiod-ca-cert\") pod \"e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd\" (UID: \"8753cc9c-d346-441f-ba35-725feb7818d3\") " pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"
Sep 14 05:08:29.479999 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.479978    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8753cc9c-d346-441f-ba35-725feb7818d3-istio-data\") pod \"e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd\" (UID: \"8753cc9c-d346-441f-ba35-725feb7818d3\") " pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"
Sep 14 05:08:29.480065 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.480021    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8753cc9c-d346-441f-ba35-725feb7818d3-credential-socket\") pod \"e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd\" (UID: \"8753cc9c-d346-441f-ba35-725feb7818d3\") " pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"
Sep 14 05:08:29.480129 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.480058    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8753cc9c-d346-441f-ba35-725feb7818d3-workload-certs\") pod \"e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd\" (UID: \"8753cc9c-d346-441f-ba35-725feb7818d3\") " pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"
Sep 14 05:08:29.480389 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.480369    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8753cc9c-d346-441f-ba35-725feb7818d3-istiod-ca-cert\") pod \"e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd\" (UID: \"8753cc9c-d346-441f-ba35-725feb7818d3\") " pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"
Sep 14 05:08:29.481795 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.481774    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8753cc9c-d346-441f-ba35-725feb7818d3-istio-envoy\") pod \"e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd\" (UID: \"8753cc9c-d346-441f-ba35-725feb7818d3\") " pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"
Sep 14 05:08:29.482014 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.481997    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8753cc9c-d346-441f-ba35-725feb7818d3-istio-podinfo\") pod \"e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd\" (UID: \"8753cc9c-d346-441f-ba35-725feb7818d3\") " pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"
Sep 14 05:08:29.487070 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.487047    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp824\" (UniqueName: \"kubernetes.io/projected/8753cc9c-d346-441f-ba35-725feb7818d3-kube-api-access-cp824\") pod \"e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd\" (UID: \"8753cc9c-d346-441f-ba35-725feb7818d3\") " pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"
Sep 14 05:08:29.487442 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.487425    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8753cc9c-d346-441f-ba35-725feb7818d3-istio-token\") pod \"e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd\" (UID: \"8753cc9c-d346-441f-ba35-725feb7818d3\") " pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"
Sep 14 05:08:29.606524 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.606437    2565 generic.go:356] "Generic (PLEG): container finished" podID="85ff3c1b-3e6f-4aaa-b641-2aca25661fda" containerID="9030c9c7134be58abf0c095eace3b51b35d1b40fc4c2d68da310926273f56f0f" exitCode=0
Sep 14 05:08:29.606524 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.606511    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-9529a1/explicit-model-9529a1-kserve-5845b75467-cp2cm"
Sep 14 05:08:29.606739 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.606532    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-9529a1/explicit-model-9529a1-kserve-5845b75467-cp2cm" event={"ID":"85ff3c1b-3e6f-4aaa-b641-2aca25661fda","Type":"ContainerDied","Data":"9030c9c7134be58abf0c095eace3b51b35d1b40fc4c2d68da310926273f56f0f"}
Sep 14 05:08:29.606739 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.606584    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-9529a1/explicit-model-9529a1-kserve-5845b75467-cp2cm" event={"ID":"85ff3c1b-3e6f-4aaa-b641-2aca25661fda","Type":"ContainerDied","Data":"bc185ca8bd88c48b9ccb13afe4890ddddec6607647ed8c56bbb879d035c35190"}
Sep 14 05:08:29.606739 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.606606    2565 scope.go:117] "RemoveContainer" containerID="9030c9c7134be58abf0c095eace3b51b35d1b40fc4c2d68da310926273f56f0f"
Sep 14 05:08:29.616158 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.616143    2565 scope.go:117] "RemoveContainer" containerID="9030c9c7134be58abf0c095eace3b51b35d1b40fc4c2d68da310926273f56f0f"
Sep 14 05:08:29.616442 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:08:29.616424    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9030c9c7134be58abf0c095eace3b51b35d1b40fc4c2d68da310926273f56f0f\": container with ID starting with 9030c9c7134be58abf0c095eace3b51b35d1b40fc4c2d68da310926273f56f0f not found: ID does not exist" containerID="9030c9c7134be58abf0c095eace3b51b35d1b40fc4c2d68da310926273f56f0f"
Sep 14 05:08:29.616505 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.616450    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9030c9c7134be58abf0c095eace3b51b35d1b40fc4c2d68da310926273f56f0f"} err="failed to get container status \"9030c9c7134be58abf0c095eace3b51b35d1b40fc4c2d68da310926273f56f0f\": rpc error: code = NotFound desc = could not find container \"9030c9c7134be58abf0c095eace3b51b35d1b40fc4c2d68da310926273f56f0f\": container with ID starting with 9030c9c7134be58abf0c095eace3b51b35d1b40fc4c2d68da310926273f56f0f not found: ID does not exist"
Sep 14 05:08:29.627625 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.627601    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-9529a1/explicit-model-9529a1-kserve-5845b75467-cp2cm"]
Sep 14 05:08:29.630059 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.630040    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-9529a1/explicit-model-9529a1-kserve-5845b75467-cp2cm"]
Sep 14 05:08:29.654216 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.654194    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"
Sep 14 05:08:29.935011 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.934961    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:08:29.935372 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.935057    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:08:29.935372 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.935096    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:08:29.935855 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:29.935820    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"]
Sep 14 05:08:30.335824 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:30.335798    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85ff3c1b-3e6f-4aaa-b641-2aca25661fda" path="/var/lib/kubelet/pods/85ff3c1b-3e6f-4aaa-b641-2aca25661fda/volumes"
Sep 14 05:08:30.613401 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:30.613305    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd" event={"ID":"8753cc9c-d346-441f-ba35-725feb7818d3","Type":"ContainerStarted","Data":"82c716a1ab19896fc333b5d9e7ed94e74d6b1dca45aeb3919a5a67286586c2e8"}
Sep 14 05:08:30.613401 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:30.613365    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd" event={"ID":"8753cc9c-d346-441f-ba35-725feb7818d3","Type":"ContainerStarted","Data":"845e00c63970f55bfb2b94f81e95f2c372c193100444392ad0cb23d9f030935b"}
Sep 14 05:08:30.634023 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:30.633971    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd" podStartSLOduration=1.633955973 podStartE2EDuration="1.633955973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:08:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:08:30.632140144 +0000 UTC m=+1308.807911714" watchObservedRunningTime="2026-09-14 05:08:30.633955973 +0000 UTC m=+1308.809727531"
Sep 14 05:08:30.654846 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:30.654818    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"
Sep 14 05:08:31.435530 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:31.435498    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"]
Sep 14 05:08:31.436005 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:31.435702    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr" podUID="f58465fe-e115-4ba7-be72-4d2b17957d90" containerName="istio-proxy" containerID="cri-o://9e0b265fb8ddcc8a1a5f70dd83374f941ffad9e7a7d460a5c980d8d72bc26d1c" gracePeriod=30
Sep 14 05:08:31.660283 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:31.660255    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"
Sep 14 05:08:32.250703 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:32.250671    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh"]
Sep 14 05:08:32.254442 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:32.254424    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh"
Sep 14 05:08:32.256592 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:32.256573    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-255ce3-openshift-default-dockercfg-z6cg9\""
Sep 14 05:08:32.266062 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:32.266038    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh"]
Sep 14 05:08:32.305126 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:32.305097    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/595352d9-0e7d-41a9-a318-7ad8bc720e26-istiod-ca-cert\") pod \"e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh\" (UID: \"595352d9-0e7d-41a9-a318-7ad8bc720e26\") " pod="openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh"
Sep 14 05:08:32.305318 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:32.305133    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpml4\" (UniqueName: \"kubernetes.io/projected/595352d9-0e7d-41a9-a318-7ad8bc720e26-kube-api-access-lpml4\") pod \"e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh\" (UID: \"595352d9-0e7d-41a9-a318-7ad8bc720e26\") " pod="openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh"
Sep 14 05:08:32.305318 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:32.305179    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/595352d9-0e7d-41a9-a318-7ad8bc720e26-workload-certs\") pod \"e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh\" (UID: \"595352d9-0e7d-41a9-a318-7ad8bc720e26\") " pod="openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh"
Sep 14 05:08:32.305318 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:32.305199    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/595352d9-0e7d-41a9-a318-7ad8bc720e26-istio-envoy\") pod \"e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh\" (UID: \"595352d9-0e7d-41a9-a318-7ad8bc720e26\") " pod="openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh"
Sep 14 05:08:32.305318 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:32.305217    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/595352d9-0e7d-41a9-a318-7ad8bc720e26-credential-socket\") pod \"e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh\" (UID: \"595352d9-0e7d-41a9-a318-7ad8bc720e26\") " pod="openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh"
Sep 14 05:08:32.305318 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:32.305297    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/595352d9-0e7d-41a9-a318-7ad8bc720e26-istio-token\") pod \"e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh\" (UID: \"595352d9-0e7d-41a9-a318-7ad8bc720e26\") " pod="openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh"
Sep 14 05:08:32.305587 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:32.305331    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/595352d9-0e7d-41a9-a318-7ad8bc720e26-istio-podinfo\") pod \"e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh\" (UID: \"595352d9-0e7d-41a9-a318-7ad8bc720e26\") " pod="openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh"
Sep 14 05:08:32.305587 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:32.305359    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/595352d9-0e7d-41a9-a318-7ad8bc720e26-workload-socket\") pod \"e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh\" (UID: \"595352d9-0e7d-41a9-a318-7ad8bc720e26\") " pod="openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh"
Sep 14 05:08:32.305587 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:32.305421    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/595352d9-0e7d-41a9-a318-7ad8bc720e26-istio-data\") pod \"e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh\" (UID: \"595352d9-0e7d-41a9-a318-7ad8bc720e26\") " pod="openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh"
Sep 14 05:08:32.406466 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:32.406431    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/595352d9-0e7d-41a9-a318-7ad8bc720e26-workload-certs\") pod \"e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh\" (UID: \"595352d9-0e7d-41a9-a318-7ad8bc720e26\") " pod="openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh"
Sep 14 05:08:32.406466 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:32.406466    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/595352d9-0e7d-41a9-a318-7ad8bc720e26-istio-envoy\") pod \"e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh\" (UID: \"595352d9-0e7d-41a9-a318-7ad8bc720e26\") " pod="openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh"
Sep 14 05:08:32.406697 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:32.406485    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/595352d9-0e7d-41a9-a318-7ad8bc720e26-credential-socket\") pod \"e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh\" (UID: \"595352d9-0e7d-41a9-a318-7ad8bc720e26\") " pod="openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh"
Sep 14 05:08:32.406697 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:32.406508    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/595352d9-0e7d-41a9-a318-7ad8bc720e26-istio-token\") pod \"e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh\" (UID: \"595352d9-0e7d-41a9-a318-7ad8bc720e26\") " pod="openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh"
Sep 14 05:08:32.406697 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:32.406536    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/595352d9-0e7d-41a9-a318-7ad8bc720e26-istio-podinfo\") pod \"e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh\" (UID: \"595352d9-0e7d-41a9-a318-7ad8bc720e26\") " pod="openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh"
Sep 14 05:08:32.406697 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:32.406562    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/595352d9-0e7d-41a9-a318-7ad8bc720e26-workload-socket\") pod \"e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh\" (UID: \"595352d9-0e7d-41a9-a318-7ad8bc720e26\") " pod="openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh"
Sep 14 05:08:32.406697 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:32.406591    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/595352d9-0e7d-41a9-a318-7ad8bc720e26-istio-data\") pod \"e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh\" (UID: \"595352d9-0e7d-41a9-a318-7ad8bc720e26\") " pod="openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh"
Sep 14 05:08:32.406697 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:32.406671    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/595352d9-0e7d-41a9-a318-7ad8bc720e26-istiod-ca-cert\") pod \"e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh\" (UID: \"595352d9-0e7d-41a9-a318-7ad8bc720e26\") " pod="openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh"
Sep 14 05:08:32.407047 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:32.406708    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpml4\" (UniqueName: \"kubernetes.io/projected/595352d9-0e7d-41a9-a318-7ad8bc720e26-kube-api-access-lpml4\") pod \"e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh\" (UID: \"595352d9-0e7d-41a9-a318-7ad8bc720e26\") " pod="openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh"
Sep 14 05:08:32.407047 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:32.406910    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/595352d9-0e7d-41a9-a318-7ad8bc720e26-workload-certs\") pod \"e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh\" (UID: \"595352d9-0e7d-41a9-a318-7ad8bc720e26\") " pod="openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh"
Sep 14 05:08:32.407234 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:32.407206    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/595352d9-0e7d-41a9-a318-7ad8bc720e26-credential-socket\") pod \"e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh\" (UID: \"595352d9-0e7d-41a9-a318-7ad8bc720e26\") " pod="openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh"
Sep 14 05:08:32.407358 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:32.407288    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/595352d9-0e7d-41a9-a318-7ad8bc720e26-istio-data\") pod \"e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh\" (UID: \"595352d9-0e7d-41a9-a318-7ad8bc720e26\") " pod="openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh"
Sep 14 05:08:32.407358 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:32.407299    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/595352d9-0e7d-41a9-a318-7ad8bc720e26-workload-socket\") pod \"e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh\" (UID: \"595352d9-0e7d-41a9-a318-7ad8bc720e26\") " pod="openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh"
Sep 14 05:08:32.407542 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:32.407519    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/595352d9-0e7d-41a9-a318-7ad8bc720e26-istiod-ca-cert\") pod \"e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh\" (UID: \"595352d9-0e7d-41a9-a318-7ad8bc720e26\") " pod="openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh"
Sep 14 05:08:32.408895 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:32.408875    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/595352d9-0e7d-41a9-a318-7ad8bc720e26-istio-envoy\") pod \"e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh\" (UID: \"595352d9-0e7d-41a9-a318-7ad8bc720e26\") " pod="openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh"
Sep 14 05:08:32.409105 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:32.409088    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/595352d9-0e7d-41a9-a318-7ad8bc720e26-istio-podinfo\") pod \"e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh\" (UID: \"595352d9-0e7d-41a9-a318-7ad8bc720e26\") " pod="openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh"
Sep 14 05:08:32.419008 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:32.418987    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/595352d9-0e7d-41a9-a318-7ad8bc720e26-istio-token\") pod \"e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh\" (UID: \"595352d9-0e7d-41a9-a318-7ad8bc720e26\") " pod="openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh"
Sep 14 05:08:32.419202 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:32.419185    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpml4\" (UniqueName: \"kubernetes.io/projected/595352d9-0e7d-41a9-a318-7ad8bc720e26-kube-api-access-lpml4\") pod \"e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh\" (UID: \"595352d9-0e7d-41a9-a318-7ad8bc720e26\") " pod="openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh"
Sep 14 05:08:32.566421 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:32.566361    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh"
Sep 14 05:08:32.629820 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:32.629790    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"
Sep 14 05:08:32.631290 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:32.631062    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"
Sep 14 05:08:32.844494 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:32.844464    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh"]
Sep 14 05:08:32.845638 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:08:32.845614    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod595352d9_0e7d_41a9_a318_7ad8bc720e26.slice/crio-2706bc9f67f9b35f85b0b94cf8aeb7c1a55f80da67bbd7492fe7727aa767b861 WatchSource:0}: Error finding container 2706bc9f67f9b35f85b0b94cf8aeb7c1a55f80da67bbd7492fe7727aa767b861: Status 404 returned error can't find the container with id 2706bc9f67f9b35f85b0b94cf8aeb7c1a55f80da67bbd7492fe7727aa767b861
Sep 14 05:08:32.847924 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:32.847891    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:08:32.847984 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:32.847956    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:08:32.848020 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:32.848000    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:08:33.636868 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:33.636827    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh" event={"ID":"595352d9-0e7d-41a9-a318-7ad8bc720e26","Type":"ContainerStarted","Data":"204df6b0d42a11a2a29018601a725f28cd2cedeb14a3998d16b5b539db6f389a"}
Sep 14 05:08:33.637424 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:33.636902    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh" event={"ID":"595352d9-0e7d-41a9-a318-7ad8bc720e26","Type":"ContainerStarted","Data":"2706bc9f67f9b35f85b0b94cf8aeb7c1a55f80da67bbd7492fe7727aa767b861"}
Sep 14 05:08:33.668426 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:33.668368    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh" podStartSLOduration=1.668349472 podStartE2EDuration="1.668349472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:08:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:08:33.665312417 +0000 UTC m=+1311.841084012" watchObservedRunningTime="2026-09-14 05:08:33.668349472 +0000 UTC m=+1311.844121031"
Sep 14 05:08:33.869914 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:33.869875    2565 patch_prober.go:28] interesting pod/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 14 05:08:33.870073 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:33.869947    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr" podUID="f58465fe-e115-4ba7-be72-4d2b17957d90" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 14 05:08:34.567109 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:34.567071    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh"
Sep 14 05:08:34.572498 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:34.572471    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh"
Sep 14 05:08:34.642893 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:34.642865    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh"
Sep 14 05:08:34.643628 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:34.643611    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh"
Sep 14 05:08:36.661708 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:36.661677    2565 generic.go:356] "Generic (PLEG): container finished" podID="f58465fe-e115-4ba7-be72-4d2b17957d90" containerID="9e0b265fb8ddcc8a1a5f70dd83374f941ffad9e7a7d460a5c980d8d72bc26d1c" exitCode=0
Sep 14 05:08:36.662121 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:36.661747    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr" event={"ID":"f58465fe-e115-4ba7-be72-4d2b17957d90","Type":"ContainerDied","Data":"9e0b265fb8ddcc8a1a5f70dd83374f941ffad9e7a7d460a5c980d8d72bc26d1c"}
Sep 14 05:08:36.662121 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:36.661866    2565 scope.go:117] "RemoveContainer" containerID="41a0873129427c2f709d6f4e0947024e1355eb227ef25bcfc81aeac870fe056c"
Sep 14 05:08:36.683100 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:36.683084    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"
Sep 14 05:08:36.748057 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:36.747991    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2nm56\" (UniqueName: \"kubernetes.io/projected/f58465fe-e115-4ba7-be72-4d2b17957d90-kube-api-access-2nm56\") pod \"f58465fe-e115-4ba7-be72-4d2b17957d90\" (UID: \"f58465fe-e115-4ba7-be72-4d2b17957d90\") "
Sep 14 05:08:36.748057 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:36.748022    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f58465fe-e115-4ba7-be72-4d2b17957d90-istiod-ca-cert\") pod \"f58465fe-e115-4ba7-be72-4d2b17957d90\" (UID: \"f58465fe-e115-4ba7-be72-4d2b17957d90\") "
Sep 14 05:08:36.748057 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:36.748052    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f58465fe-e115-4ba7-be72-4d2b17957d90-workload-certs\") pod \"f58465fe-e115-4ba7-be72-4d2b17957d90\" (UID: \"f58465fe-e115-4ba7-be72-4d2b17957d90\") "
Sep 14 05:08:36.748271 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:36.748079    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f58465fe-e115-4ba7-be72-4d2b17957d90-istio-data\") pod \"f58465fe-e115-4ba7-be72-4d2b17957d90\" (UID: \"f58465fe-e115-4ba7-be72-4d2b17957d90\") "
Sep 14 05:08:36.748271 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:36.748108    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58465fe-e115-4ba7-be72-4d2b17957d90-credential-socket\") pod \"f58465fe-e115-4ba7-be72-4d2b17957d90\" (UID: \"f58465fe-e115-4ba7-be72-4d2b17957d90\") "
Sep 14 05:08:36.748271 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:36.748144    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f58465fe-e115-4ba7-be72-4d2b17957d90-istio-podinfo\") pod \"f58465fe-e115-4ba7-be72-4d2b17957d90\" (UID: \"f58465fe-e115-4ba7-be72-4d2b17957d90\") "
Sep 14 05:08:36.748271 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:36.748196    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f58465fe-e115-4ba7-be72-4d2b17957d90-istio-envoy\") pod \"f58465fe-e115-4ba7-be72-4d2b17957d90\" (UID: \"f58465fe-e115-4ba7-be72-4d2b17957d90\") "
Sep 14 05:08:36.748271 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:36.748219    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f58465fe-e115-4ba7-be72-4d2b17957d90-istio-token\") pod \"f58465fe-e115-4ba7-be72-4d2b17957d90\" (UID: \"f58465fe-e115-4ba7-be72-4d2b17957d90\") "
Sep 14 05:08:36.748271 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:36.748256    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58465fe-e115-4ba7-be72-4d2b17957d90-workload-socket\") pod \"f58465fe-e115-4ba7-be72-4d2b17957d90\" (UID: \"f58465fe-e115-4ba7-be72-4d2b17957d90\") "
Sep 14 05:08:36.748568 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:36.748344    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58465fe-e115-4ba7-be72-4d2b17957d90-istio-data" (OuterVolumeSpecName: "istio-data") pod "f58465fe-e115-4ba7-be72-4d2b17957d90" (UID: "f58465fe-e115-4ba7-be72-4d2b17957d90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:08:36.748568 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:36.748357    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58465fe-e115-4ba7-be72-4d2b17957d90-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f58465fe-e115-4ba7-be72-4d2b17957d90" (UID: "f58465fe-e115-4ba7-be72-4d2b17957d90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:08:36.748568 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:36.748435    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f58465fe-e115-4ba7-be72-4d2b17957d90-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f58465fe-e115-4ba7-be72-4d2b17957d90" (UID: "f58465fe-e115-4ba7-be72-4d2b17957d90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 05:08:36.748817 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:36.748793    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58465fe-e115-4ba7-be72-4d2b17957d90-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f58465fe-e115-4ba7-be72-4d2b17957d90" (UID: "f58465fe-e115-4ba7-be72-4d2b17957d90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:08:36.748904 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:36.748839    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f58465fe-e115-4ba7-be72-4d2b17957d90-istiod-ca-cert\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:08:36.748904 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:36.748858    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f58465fe-e115-4ba7-be72-4d2b17957d90-workload-certs\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:08:36.748904 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:36.748871    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f58465fe-e115-4ba7-be72-4d2b17957d90-istio-data\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:08:36.749159 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:36.749136    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58465fe-e115-4ba7-be72-4d2b17957d90-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f58465fe-e115-4ba7-be72-4d2b17957d90" (UID: "f58465fe-e115-4ba7-be72-4d2b17957d90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:08:36.750808 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:36.750782    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58465fe-e115-4ba7-be72-4d2b17957d90-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f58465fe-e115-4ba7-be72-4d2b17957d90" (UID: "f58465fe-e115-4ba7-be72-4d2b17957d90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:08:36.750898 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:36.750792    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f58465fe-e115-4ba7-be72-4d2b17957d90-istio-token" (OuterVolumeSpecName: "istio-token") pod "f58465fe-e115-4ba7-be72-4d2b17957d90" (UID: "f58465fe-e115-4ba7-be72-4d2b17957d90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:08:36.750982 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:36.750961    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f58465fe-e115-4ba7-be72-4d2b17957d90-kube-api-access-2nm56" (OuterVolumeSpecName: "kube-api-access-2nm56") pod "f58465fe-e115-4ba7-be72-4d2b17957d90" (UID: "f58465fe-e115-4ba7-be72-4d2b17957d90"). InnerVolumeSpecName "kube-api-access-2nm56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:08:36.751052 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:36.751021    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f58465fe-e115-4ba7-be72-4d2b17957d90-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f58465fe-e115-4ba7-be72-4d2b17957d90" (UID: "f58465fe-e115-4ba7-be72-4d2b17957d90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 05:08:36.850378 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:36.850294    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f58465fe-e115-4ba7-be72-4d2b17957d90-istio-envoy\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:08:36.850378 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:36.850328    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f58465fe-e115-4ba7-be72-4d2b17957d90-istio-token\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:08:36.850378 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:36.850342    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58465fe-e115-4ba7-be72-4d2b17957d90-workload-socket\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:08:36.850378 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:36.850356    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nm56\" (UniqueName: \"kubernetes.io/projected/f58465fe-e115-4ba7-be72-4d2b17957d90-kube-api-access-2nm56\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:08:36.850378 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:36.850370    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58465fe-e115-4ba7-be72-4d2b17957d90-credential-socket\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:08:36.850378 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:36.850384    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f58465fe-e115-4ba7-be72-4d2b17957d90-istio-podinfo\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:08:37.542288 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:37.542254    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c7c6a48c-7f9cf4474b-qrmzh"]
Sep 14 05:08:37.542612 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:37.542599    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f58465fe-e115-4ba7-be72-4d2b17957d90" containerName="istio-proxy"
Sep 14 05:08:37.542664 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:37.542615    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f58465fe-e115-4ba7-be72-4d2b17957d90" containerName="istio-proxy"
Sep 14 05:08:37.542699 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:37.542695    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f58465fe-e115-4ba7-be72-4d2b17957d90" containerName="istio-proxy"
Sep 14 05:08:37.546973 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:37.546958    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7c6a48c-7f9cf4474b-qrmzh"
Sep 14 05:08:37.549293 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:37.549267    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-c7c6a48c-dockercfg-2lgk2\""
Sep 14 05:08:37.549421 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:37.549274    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c7c6a48c\""
Sep 14 05:08:37.553254 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:37.553221    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c7c6a48c-7f9cf4474b-qrmzh"]
Sep 14 05:08:37.658207 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:37.658177    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v55ck\" (UniqueName: \"kubernetes.io/projected/7818bda4-c5be-42e4-a4cf-ca3ef45cf295-kube-api-access-v55ck\") pod \"payload-pre-processing-e2e-mt-c7c6a48c-7f9cf4474b-qrmzh\" (UID: \"7818bda4-c5be-42e4-a4cf-ca3ef45cf295\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c7c6a48c-7f9cf4474b-qrmzh"
Sep 14 05:08:37.658378 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:37.658284    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7818bda4-c5be-42e4-a4cf-ca3ef45cf295-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c7c6a48c-7f9cf4474b-qrmzh\" (UID: \"7818bda4-c5be-42e4-a4cf-ca3ef45cf295\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c7c6a48c-7f9cf4474b-qrmzh"
Sep 14 05:08:37.668035 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:37.668010    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr" event={"ID":"f58465fe-e115-4ba7-be72-4d2b17957d90","Type":"ContainerDied","Data":"3fd6759af74bfb68fe3cea0747088facf1293630f1e542f5fc5241f97bb0f4da"}
Sep 14 05:08:37.668035 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:37.668033    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"
Sep 14 05:08:37.668518 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:37.668056    2565 scope.go:117] "RemoveContainer" containerID="9e0b265fb8ddcc8a1a5f70dd83374f941ffad9e7a7d460a5c980d8d72bc26d1c"
Sep 14 05:08:37.697983 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:37.697954    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"]
Sep 14 05:08:37.701602 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:37.701582    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-9529a1-openshift-default-5fb7fd4688-px7qr"]
Sep 14 05:08:37.758887 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:37.758866    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7818bda4-c5be-42e4-a4cf-ca3ef45cf295-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c7c6a48c-7f9cf4474b-qrmzh\" (UID: \"7818bda4-c5be-42e4-a4cf-ca3ef45cf295\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c7c6a48c-7f9cf4474b-qrmzh"
Sep 14 05:08:37.759000 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:37.758970    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v55ck\" (UniqueName: \"kubernetes.io/projected/7818bda4-c5be-42e4-a4cf-ca3ef45cf295-kube-api-access-v55ck\") pod \"payload-pre-processing-e2e-mt-c7c6a48c-7f9cf4474b-qrmzh\" (UID: \"7818bda4-c5be-42e4-a4cf-ca3ef45cf295\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c7c6a48c-7f9cf4474b-qrmzh"
Sep 14 05:08:37.759561 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:37.759542    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7818bda4-c5be-42e4-a4cf-ca3ef45cf295-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c7c6a48c-7f9cf4474b-qrmzh\" (UID: \"7818bda4-c5be-42e4-a4cf-ca3ef45cf295\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c7c6a48c-7f9cf4474b-qrmzh"
Sep 14 05:08:37.766285 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:37.766266    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v55ck\" (UniqueName: \"kubernetes.io/projected/7818bda4-c5be-42e4-a4cf-ca3ef45cf295-kube-api-access-v55ck\") pod \"payload-pre-processing-e2e-mt-c7c6a48c-7f9cf4474b-qrmzh\" (UID: \"7818bda4-c5be-42e4-a4cf-ca3ef45cf295\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c7c6a48c-7f9cf4474b-qrmzh"
Sep 14 05:08:37.858837 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:37.858808    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7c6a48c-7f9cf4474b-qrmzh"
Sep 14 05:08:38.130411 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:38.130382    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c7c6a48c-7f9cf4474b-qrmzh"]
Sep 14 05:08:38.131776 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:08:38.131736    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7818bda4_c5be_42e4_a4cf_ca3ef45cf295.slice/crio-3929d0be97a40161dab8f6d8736a906cc8c61f357cd04a6056d268e77f394e9d WatchSource:0}: Error finding container 3929d0be97a40161dab8f6d8736a906cc8c61f357cd04a6056d268e77f394e9d: Status 404 returned error can't find the container with id 3929d0be97a40161dab8f6d8736a906cc8c61f357cd04a6056d268e77f394e9d
Sep 14 05:08:38.335985 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:38.335955    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f58465fe-e115-4ba7-be72-4d2b17957d90" path="/var/lib/kubelet/pods/f58465fe-e115-4ba7-be72-4d2b17957d90/volumes"
Sep 14 05:08:38.675699 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:38.675663    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7c6a48c-7f9cf4474b-qrmzh" event={"ID":"7818bda4-c5be-42e4-a4cf-ca3ef45cf295","Type":"ContainerStarted","Data":"c3b0c7b3267fb8197b2864747539cbba179db28a29821dec87e7cc4d9a7fdb64"}
Sep 14 05:08:38.676076 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:38.675709    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7c6a48c-7f9cf4474b-qrmzh" event={"ID":"7818bda4-c5be-42e4-a4cf-ca3ef45cf295","Type":"ContainerStarted","Data":"3929d0be97a40161dab8f6d8736a906cc8c61f357cd04a6056d268e77f394e9d"}
Sep 14 05:08:38.676076 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:38.675788    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7c6a48c-7f9cf4474b-qrmzh"
Sep 14 05:08:38.693140 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:38.693097    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7c6a48c-7f9cf4474b-qrmzh" podStartSLOduration=1.693082671 podStartE2EDuration="1.693082671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:08:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:08:38.690569185 +0000 UTC m=+1316.866340755" watchObservedRunningTime="2026-09-14 05:08:38.693082671 +0000 UTC m=+1316.868854230"
Sep 14 05:08:42.446491 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:42.446461    2565 scope.go:117] "RemoveContainer" containerID="5b17bb081a98e6baee6565187be7daaf2f39fdd56a141ca0b071ebb78f134e07"
Sep 14 05:08:42.455012 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:42.454983    2565 scope.go:117] "RemoveContainer" containerID="888de0ac61ea0e0b08467f3ba2357497e666265d769abed3cd1b53c143a7e0b4"
Sep 14 05:08:49.682719 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:49.682685    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7c6a48c-7f9cf4474b-qrmzh"
Sep 14 05:08:50.625903 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:50.625865    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:08:56.727669 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:56.727637    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-e08402/e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk"]
Sep 14 05:08:56.728052 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:56.728007    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f58465fe-e115-4ba7-be72-4d2b17957d90" containerName="istio-proxy"
Sep 14 05:08:56.728052 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:56.728020    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f58465fe-e115-4ba7-be72-4d2b17957d90" containerName="istio-proxy"
Sep 14 05:08:56.728125 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:56.728084    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f58465fe-e115-4ba7-be72-4d2b17957d90" containerName="istio-proxy"
Sep 14 05:08:56.732645 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:56.732629    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e08402/e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk"
Sep 14 05:08:56.734906 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:56.734887    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e08402\"/\"e2e-conflict-b-c6882fdf-kserve-self-signed-certs\""
Sep 14 05:08:56.739130 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:56.739104    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e08402/e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk"]
Sep 14 05:08:56.837418 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:56.837388    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/089ab012-ecd2-4d9d-9ce1-25f1cb8554fc-model-cache\") pod \"e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk\" (UID: \"089ab012-ecd2-4d9d-9ce1-25f1cb8554fc\") " pod="e2e-models-e2e-worker-w4-e08402/e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk"
Sep 14 05:08:56.837599 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:56.837462    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfg65\" (UniqueName: \"kubernetes.io/projected/089ab012-ecd2-4d9d-9ce1-25f1cb8554fc-kube-api-access-sfg65\") pod \"e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk\" (UID: \"089ab012-ecd2-4d9d-9ce1-25f1cb8554fc\") " pod="e2e-models-e2e-worker-w4-e08402/e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk"
Sep 14 05:08:56.837599 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:56.837517    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/089ab012-ecd2-4d9d-9ce1-25f1cb8554fc-tmp-dir\") pod \"e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk\" (UID: \"089ab012-ecd2-4d9d-9ce1-25f1cb8554fc\") " pod="e2e-models-e2e-worker-w4-e08402/e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk"
Sep 14 05:08:56.837599 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:56.837537    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/089ab012-ecd2-4d9d-9ce1-25f1cb8554fc-dshm\") pod \"e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk\" (UID: \"089ab012-ecd2-4d9d-9ce1-25f1cb8554fc\") " pod="e2e-models-e2e-worker-w4-e08402/e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk"
Sep 14 05:08:56.837786 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:56.837594    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/089ab012-ecd2-4d9d-9ce1-25f1cb8554fc-tls-certs\") pod \"e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk\" (UID: \"089ab012-ecd2-4d9d-9ce1-25f1cb8554fc\") " pod="e2e-models-e2e-worker-w4-e08402/e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk"
Sep 14 05:08:56.837786 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:56.837666    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/089ab012-ecd2-4d9d-9ce1-25f1cb8554fc-home\") pod \"e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk\" (UID: \"089ab012-ecd2-4d9d-9ce1-25f1cb8554fc\") " pod="e2e-models-e2e-worker-w4-e08402/e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk"
Sep 14 05:08:56.938575 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:56.938531    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfg65\" (UniqueName: \"kubernetes.io/projected/089ab012-ecd2-4d9d-9ce1-25f1cb8554fc-kube-api-access-sfg65\") pod \"e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk\" (UID: \"089ab012-ecd2-4d9d-9ce1-25f1cb8554fc\") " pod="e2e-models-e2e-worker-w4-e08402/e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk"
Sep 14 05:08:56.938799 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:56.938590    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/089ab012-ecd2-4d9d-9ce1-25f1cb8554fc-tmp-dir\") pod \"e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk\" (UID: \"089ab012-ecd2-4d9d-9ce1-25f1cb8554fc\") " pod="e2e-models-e2e-worker-w4-e08402/e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk"
Sep 14 05:08:56.938799 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:56.938622    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/089ab012-ecd2-4d9d-9ce1-25f1cb8554fc-dshm\") pod \"e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk\" (UID: \"089ab012-ecd2-4d9d-9ce1-25f1cb8554fc\") " pod="e2e-models-e2e-worker-w4-e08402/e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk"
Sep 14 05:08:56.938799 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:56.938654    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/089ab012-ecd2-4d9d-9ce1-25f1cb8554fc-tls-certs\") pod \"e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk\" (UID: \"089ab012-ecd2-4d9d-9ce1-25f1cb8554fc\") " pod="e2e-models-e2e-worker-w4-e08402/e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk"
Sep 14 05:08:56.938799 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:56.938713    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/089ab012-ecd2-4d9d-9ce1-25f1cb8554fc-home\") pod \"e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk\" (UID: \"089ab012-ecd2-4d9d-9ce1-25f1cb8554fc\") " pod="e2e-models-e2e-worker-w4-e08402/e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk"
Sep 14 05:08:56.938799 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:56.938772    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/089ab012-ecd2-4d9d-9ce1-25f1cb8554fc-model-cache\") pod \"e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk\" (UID: \"089ab012-ecd2-4d9d-9ce1-25f1cb8554fc\") " pod="e2e-models-e2e-worker-w4-e08402/e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk"
Sep 14 05:08:56.939225 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:56.939200    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/089ab012-ecd2-4d9d-9ce1-25f1cb8554fc-model-cache\") pod \"e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk\" (UID: \"089ab012-ecd2-4d9d-9ce1-25f1cb8554fc\") " pod="e2e-models-e2e-worker-w4-e08402/e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk"
Sep 14 05:08:56.939980 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:56.939958    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/089ab012-ecd2-4d9d-9ce1-25f1cb8554fc-tmp-dir\") pod \"e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk\" (UID: \"089ab012-ecd2-4d9d-9ce1-25f1cb8554fc\") " pod="e2e-models-e2e-worker-w4-e08402/e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk"
Sep 14 05:08:56.941029 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:56.941008    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/089ab012-ecd2-4d9d-9ce1-25f1cb8554fc-home\") pod \"e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk\" (UID: \"089ab012-ecd2-4d9d-9ce1-25f1cb8554fc\") " pod="e2e-models-e2e-worker-w4-e08402/e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk"
Sep 14 05:08:56.943747 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:56.943726    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/089ab012-ecd2-4d9d-9ce1-25f1cb8554fc-dshm\") pod \"e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk\" (UID: \"089ab012-ecd2-4d9d-9ce1-25f1cb8554fc\") " pod="e2e-models-e2e-worker-w4-e08402/e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk"
Sep 14 05:08:56.954003 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:56.953978    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfg65\" (UniqueName: \"kubernetes.io/projected/089ab012-ecd2-4d9d-9ce1-25f1cb8554fc-kube-api-access-sfg65\") pod \"e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk\" (UID: \"089ab012-ecd2-4d9d-9ce1-25f1cb8554fc\") " pod="e2e-models-e2e-worker-w4-e08402/e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk"
Sep 14 05:08:56.957312 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:56.957289    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/089ab012-ecd2-4d9d-9ce1-25f1cb8554fc-tls-certs\") pod \"e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk\" (UID: \"089ab012-ecd2-4d9d-9ce1-25f1cb8554fc\") " pod="e2e-models-e2e-worker-w4-e08402/e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk"
Sep 14 05:08:57.044007 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:57.043927    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e08402/e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk"
Sep 14 05:08:57.324111 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:57.324081    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e08402/e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk"]
Sep 14 05:08:57.325299 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:08:57.325274    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod089ab012_ecd2_4d9d_9ce1_25f1cb8554fc.slice/crio-9e802f8a5b07f7c53794095a24442c4528913f297a9458e8b8e71c7d92315c5c WatchSource:0}: Error finding container 9e802f8a5b07f7c53794095a24442c4528913f297a9458e8b8e71c7d92315c5c: Status 404 returned error can't find the container with id 9e802f8a5b07f7c53794095a24442c4528913f297a9458e8b8e71c7d92315c5c
Sep 14 05:08:57.800353 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:57.800319    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e08402/e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk" event={"ID":"089ab012-ecd2-4d9d-9ce1-25f1cb8554fc","Type":"ContainerStarted","Data":"a17e05025b86c6d928ac740db5988adaa24aaf23b388a24b600bec9430cbf66a"}
Sep 14 05:08:57.800778 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:57.800366    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e08402/e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk" event={"ID":"089ab012-ecd2-4d9d-9ce1-25f1cb8554fc","Type":"ContainerStarted","Data":"9e802f8a5b07f7c53794095a24442c4528913f297a9458e8b8e71c7d92315c5c"}
Sep 14 05:08:57.823779 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:08:57.819002    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-e08402/e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk" podStartSLOduration=1.818979475 podStartE2EDuration="1.818979475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:08:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:08:57.814646323 +0000 UTC m=+1335.990417881" watchObservedRunningTime="2026-09-14 05:08:57.818979475 +0000 UTC m=+1335.994751035"
Sep 14 05:09:01.824629 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:01.824595    2565 generic.go:356] "Generic (PLEG): container finished" podID="8753cc9c-d346-441f-ba35-725feb7818d3" containerID="82c716a1ab19896fc333b5d9e7ed94e74d6b1dca45aeb3919a5a67286586c2e8" exitCode=0
Sep 14 05:09:01.825081 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:01.824664    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd" event={"ID":"8753cc9c-d346-441f-ba35-725feb7818d3","Type":"ContainerDied","Data":"82c716a1ab19896fc333b5d9e7ed94e74d6b1dca45aeb3919a5a67286586c2e8"}
Sep 14 05:09:01.825140 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:01.825086    2565 scope.go:117] "RemoveContainer" containerID="82c716a1ab19896fc333b5d9e7ed94e74d6b1dca45aeb3919a5a67286586c2e8"
Sep 14 05:09:01.826222 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:01.826187    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:09:01.826285 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:01.826261    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:09:01.826320 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:01.826293    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:09:02.629968 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:02.629940    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"
Sep 14 05:09:02.831998 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:02.831962    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd" event={"ID":"8753cc9c-d346-441f-ba35-725feb7818d3","Type":"ContainerStarted","Data":"5cdbd7b9f400693d4f8b41fadbbd925c5d70dd6e839536b3ee5cf8624e4ae38a"}
Sep 14 05:09:03.655883 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:03.655838    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"
Sep 14 05:09:03.656100 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:03.655977    2565 patch_prober.go:28] interesting pod/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Sep 14 05:09:03.656100 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:03.656024    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd" podUID="8753cc9c-d346-441f-ba35-725feb7818d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Sep 14 05:09:04.654894 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:04.654848    2565 patch_prober.go:28] interesting pod/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Sep 14 05:09:04.655393 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:04.654918    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd" podUID="8753cc9c-d346-441f-ba35-725feb7818d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Sep 14 05:09:05.654648 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:05.654607    2565 patch_prober.go:28] interesting pod/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Sep 14 05:09:05.654986 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:05.654954    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd" podUID="8753cc9c-d346-441f-ba35-725feb7818d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Sep 14 05:09:06.654901 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:06.654861    2565 patch_prober.go:28] interesting pod/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Sep 14 05:09:06.655191 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:06.655160    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd" podUID="8753cc9c-d346-441f-ba35-725feb7818d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Sep 14 05:09:07.044622 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:07.044523    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e08402/e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk"
Sep 14 05:09:07.045895 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:07.045873    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-e08402/e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk"
Sep 14 05:09:07.074309 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:07.074280    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-e08402/e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk"
Sep 14 05:09:07.655281 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:07.655240    2565 patch_prober.go:28] interesting pod/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Sep 14 05:09:07.655783 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:07.655315    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd" podUID="8753cc9c-d346-441f-ba35-725feb7818d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Sep 14 05:09:07.878557 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:07.878531    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e08402/e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk"
Sep 14 05:09:08.660240 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:08.660205    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"
Sep 14 05:09:08.660670 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:08.660614    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"
Sep 14 05:09:08.661938 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:08.661919    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"
Sep 14 05:09:10.536093 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:10.536053    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:09:11.027445 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:11.027411    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-255ce3/noten-model-255ce3-kserve-7594d477f9-fwz8c"]
Sep 14 05:09:11.030807 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:11.030539    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-255ce3/noten-model-255ce3-kserve-7594d477f9-fwz8c"
Sep 14 05:09:11.034029 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:11.034002    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-255ce3\"/\"kube-root-ca.crt\""
Sep 14 05:09:11.034165 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:11.034027    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-255ce3\"/\"noten-model-255ce3-kserve-self-signed-certs\""
Sep 14 05:09:11.034165 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:11.034060    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-255ce3\"/\"openshift-service-ca.crt\""
Sep 14 05:09:11.034165 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:11.034134    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-255ce3\"/\"default-dockercfg-jlltw\""
Sep 14 05:09:11.040100 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:11.040080    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-255ce3/noten-model-255ce3-kserve-7594d477f9-fwz8c"]
Sep 14 05:09:11.091623 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:11.091592    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/728f28c2-15b8-4d33-9ed1-a99c2b02cd59-tls-certs\") pod \"noten-model-255ce3-kserve-7594d477f9-fwz8c\" (UID: \"728f28c2-15b8-4d33-9ed1-a99c2b02cd59\") " pod="ai-tenant-e2e-noten-255ce3/noten-model-255ce3-kserve-7594d477f9-fwz8c"
Sep 14 05:09:11.091817 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:11.091706    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/728f28c2-15b8-4d33-9ed1-a99c2b02cd59-dshm\") pod \"noten-model-255ce3-kserve-7594d477f9-fwz8c\" (UID: \"728f28c2-15b8-4d33-9ed1-a99c2b02cd59\") " pod="ai-tenant-e2e-noten-255ce3/noten-model-255ce3-kserve-7594d477f9-fwz8c"
Sep 14 05:09:11.091817 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:11.091731    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/728f28c2-15b8-4d33-9ed1-a99c2b02cd59-model-cache\") pod \"noten-model-255ce3-kserve-7594d477f9-fwz8c\" (UID: \"728f28c2-15b8-4d33-9ed1-a99c2b02cd59\") " pod="ai-tenant-e2e-noten-255ce3/noten-model-255ce3-kserve-7594d477f9-fwz8c"
Sep 14 05:09:11.091817 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:11.091780    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/728f28c2-15b8-4d33-9ed1-a99c2b02cd59-home\") pod \"noten-model-255ce3-kserve-7594d477f9-fwz8c\" (UID: \"728f28c2-15b8-4d33-9ed1-a99c2b02cd59\") " pod="ai-tenant-e2e-noten-255ce3/noten-model-255ce3-kserve-7594d477f9-fwz8c"
Sep 14 05:09:11.091817 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:11.091802    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/728f28c2-15b8-4d33-9ed1-a99c2b02cd59-tmp-dir\") pod \"noten-model-255ce3-kserve-7594d477f9-fwz8c\" (UID: \"728f28c2-15b8-4d33-9ed1-a99c2b02cd59\") " pod="ai-tenant-e2e-noten-255ce3/noten-model-255ce3-kserve-7594d477f9-fwz8c"
Sep 14 05:09:11.091979 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:11.091819    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fpjj\" (UniqueName: \"kubernetes.io/projected/728f28c2-15b8-4d33-9ed1-a99c2b02cd59-kube-api-access-2fpjj\") pod \"noten-model-255ce3-kserve-7594d477f9-fwz8c\" (UID: \"728f28c2-15b8-4d33-9ed1-a99c2b02cd59\") " pod="ai-tenant-e2e-noten-255ce3/noten-model-255ce3-kserve-7594d477f9-fwz8c"
Sep 14 05:09:11.192708 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:11.192665    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/728f28c2-15b8-4d33-9ed1-a99c2b02cd59-dshm\") pod \"noten-model-255ce3-kserve-7594d477f9-fwz8c\" (UID: \"728f28c2-15b8-4d33-9ed1-a99c2b02cd59\") " pod="ai-tenant-e2e-noten-255ce3/noten-model-255ce3-kserve-7594d477f9-fwz8c"
Sep 14 05:09:11.192919 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:11.192720    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/728f28c2-15b8-4d33-9ed1-a99c2b02cd59-model-cache\") pod \"noten-model-255ce3-kserve-7594d477f9-fwz8c\" (UID: \"728f28c2-15b8-4d33-9ed1-a99c2b02cd59\") " pod="ai-tenant-e2e-noten-255ce3/noten-model-255ce3-kserve-7594d477f9-fwz8c"
Sep 14 05:09:11.192919 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:11.192783    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/728f28c2-15b8-4d33-9ed1-a99c2b02cd59-home\") pod \"noten-model-255ce3-kserve-7594d477f9-fwz8c\" (UID: \"728f28c2-15b8-4d33-9ed1-a99c2b02cd59\") " pod="ai-tenant-e2e-noten-255ce3/noten-model-255ce3-kserve-7594d477f9-fwz8c"
Sep 14 05:09:11.192919 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:11.192808    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/728f28c2-15b8-4d33-9ed1-a99c2b02cd59-tmp-dir\") pod \"noten-model-255ce3-kserve-7594d477f9-fwz8c\" (UID: \"728f28c2-15b8-4d33-9ed1-a99c2b02cd59\") " pod="ai-tenant-e2e-noten-255ce3/noten-model-255ce3-kserve-7594d477f9-fwz8c"
Sep 14 05:09:11.192919 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:11.192833    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fpjj\" (UniqueName: \"kubernetes.io/projected/728f28c2-15b8-4d33-9ed1-a99c2b02cd59-kube-api-access-2fpjj\") pod \"noten-model-255ce3-kserve-7594d477f9-fwz8c\" (UID: \"728f28c2-15b8-4d33-9ed1-a99c2b02cd59\") " pod="ai-tenant-e2e-noten-255ce3/noten-model-255ce3-kserve-7594d477f9-fwz8c"
Sep 14 05:09:11.192919 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:11.192915    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/728f28c2-15b8-4d33-9ed1-a99c2b02cd59-tls-certs\") pod \"noten-model-255ce3-kserve-7594d477f9-fwz8c\" (UID: \"728f28c2-15b8-4d33-9ed1-a99c2b02cd59\") " pod="ai-tenant-e2e-noten-255ce3/noten-model-255ce3-kserve-7594d477f9-fwz8c"
Sep 14 05:09:11.193216 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:11.193189    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/728f28c2-15b8-4d33-9ed1-a99c2b02cd59-home\") pod \"noten-model-255ce3-kserve-7594d477f9-fwz8c\" (UID: \"728f28c2-15b8-4d33-9ed1-a99c2b02cd59\") " pod="ai-tenant-e2e-noten-255ce3/noten-model-255ce3-kserve-7594d477f9-fwz8c"
Sep 14 05:09:11.193287 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:11.193222    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/728f28c2-15b8-4d33-9ed1-a99c2b02cd59-model-cache\") pod \"noten-model-255ce3-kserve-7594d477f9-fwz8c\" (UID: \"728f28c2-15b8-4d33-9ed1-a99c2b02cd59\") " pod="ai-tenant-e2e-noten-255ce3/noten-model-255ce3-kserve-7594d477f9-fwz8c"
Sep 14 05:09:11.193287 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:11.193270    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/728f28c2-15b8-4d33-9ed1-a99c2b02cd59-tmp-dir\") pod \"noten-model-255ce3-kserve-7594d477f9-fwz8c\" (UID: \"728f28c2-15b8-4d33-9ed1-a99c2b02cd59\") " pod="ai-tenant-e2e-noten-255ce3/noten-model-255ce3-kserve-7594d477f9-fwz8c"
Sep 14 05:09:11.195003 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:11.194976    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/728f28c2-15b8-4d33-9ed1-a99c2b02cd59-dshm\") pod \"noten-model-255ce3-kserve-7594d477f9-fwz8c\" (UID: \"728f28c2-15b8-4d33-9ed1-a99c2b02cd59\") " pod="ai-tenant-e2e-noten-255ce3/noten-model-255ce3-kserve-7594d477f9-fwz8c"
Sep 14 05:09:11.195515 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:11.195494    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/728f28c2-15b8-4d33-9ed1-a99c2b02cd59-tls-certs\") pod \"noten-model-255ce3-kserve-7594d477f9-fwz8c\" (UID: \"728f28c2-15b8-4d33-9ed1-a99c2b02cd59\") " pod="ai-tenant-e2e-noten-255ce3/noten-model-255ce3-kserve-7594d477f9-fwz8c"
Sep 14 05:09:11.200116 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:11.200093    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fpjj\" (UniqueName: \"kubernetes.io/projected/728f28c2-15b8-4d33-9ed1-a99c2b02cd59-kube-api-access-2fpjj\") pod \"noten-model-255ce3-kserve-7594d477f9-fwz8c\" (UID: \"728f28c2-15b8-4d33-9ed1-a99c2b02cd59\") " pod="ai-tenant-e2e-noten-255ce3/noten-model-255ce3-kserve-7594d477f9-fwz8c"
Sep 14 05:09:11.342196 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:11.342156    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-255ce3/noten-model-255ce3-kserve-7594d477f9-fwz8c"
Sep 14 05:09:11.614845 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:11.614817    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-255ce3/noten-model-255ce3-kserve-7594d477f9-fwz8c"]
Sep 14 05:09:11.616080 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:09:11.616054    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod728f28c2_15b8_4d33_9ed1_a99c2b02cd59.slice/crio-55d4e95bb67bbfbc955f8f2530d6817ab640b5b9fc4415c952a05ba8a9449c79 WatchSource:0}: Error finding container 55d4e95bb67bbfbc955f8f2530d6817ab640b5b9fc4415c952a05ba8a9449c79: Status 404 returned error can't find the container with id 55d4e95bb67bbfbc955f8f2530d6817ab640b5b9fc4415c952a05ba8a9449c79
Sep 14 05:09:11.891078 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:11.890986    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-255ce3/noten-model-255ce3-kserve-7594d477f9-fwz8c" event={"ID":"728f28c2-15b8-4d33-9ed1-a99c2b02cd59","Type":"ContainerStarted","Data":"79ef385bcaa942644094327a0c507ad5de351dad4a34eb2b1e257430319a19da"}
Sep 14 05:09:11.891078 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:11.891030    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-255ce3/noten-model-255ce3-kserve-7594d477f9-fwz8c" event={"ID":"728f28c2-15b8-4d33-9ed1-a99c2b02cd59","Type":"ContainerStarted","Data":"55d4e95bb67bbfbc955f8f2530d6817ab640b5b9fc4415c952a05ba8a9449c79"}
Sep 14 05:09:11.906189 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:11.906100    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-255ce3/noten-model-255ce3-kserve-7594d477f9-fwz8c" podStartSLOduration=0.906083388 podStartE2EDuration="906.083388ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:09:11.905735197 +0000 UTC m=+1350.081506756" watchObservedRunningTime="2026-09-14 05:09:11.906083388 +0000 UTC m=+1350.081854947"
Sep 14 05:09:14.311972 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:14.311936    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-667664/test-model-667664-kserve-96bfc7c9f-ktdrk"]
Sep 14 05:09:21.342671 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:21.342631    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-255ce3/noten-model-255ce3-kserve-7594d477f9-fwz8c"
Sep 14 05:09:21.342671 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:21.342679    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-255ce3/noten-model-255ce3-kserve-7594d477f9-fwz8c"
Sep 14 05:09:21.354637 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:21.354615    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-255ce3/noten-model-255ce3-kserve-7594d477f9-fwz8c"
Sep 14 05:09:21.954654 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:21.954620    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-255ce3/noten-model-255ce3-kserve-7594d477f9-fwz8c"
Sep 14 05:09:29.332227 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:29.332170    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-667664/test-model-667664-kserve-96bfc7c9f-ktdrk" podUID="956bf287-009d-4eb1-be24-78c2cc343a76" containerName="main" containerID="cri-o://b4a500de9ff7a8b4e750207abdad0b470d475d5bd5d5dcde4b82fc5cf33ec993" gracePeriod=45
Sep 14 05:09:29.574737 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:29.574715    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-667664/test-model-667664-kserve-96bfc7c9f-ktdrk"
Sep 14 05:09:29.663344 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:29.663318    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/956bf287-009d-4eb1-be24-78c2cc343a76-dshm\") pod \"956bf287-009d-4eb1-be24-78c2cc343a76\" (UID: \"956bf287-009d-4eb1-be24-78c2cc343a76\") "
Sep 14 05:09:29.663512 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:29.663391    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/956bf287-009d-4eb1-be24-78c2cc343a76-home\") pod \"956bf287-009d-4eb1-be24-78c2cc343a76\" (UID: \"956bf287-009d-4eb1-be24-78c2cc343a76\") "
Sep 14 05:09:29.663512 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:29.663422    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/956bf287-009d-4eb1-be24-78c2cc343a76-tls-certs\") pod \"956bf287-009d-4eb1-be24-78c2cc343a76\" (UID: \"956bf287-009d-4eb1-be24-78c2cc343a76\") "
Sep 14 05:09:29.663512 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:29.663461    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9qdk\" (UniqueName: \"kubernetes.io/projected/956bf287-009d-4eb1-be24-78c2cc343a76-kube-api-access-p9qdk\") pod \"956bf287-009d-4eb1-be24-78c2cc343a76\" (UID: \"956bf287-009d-4eb1-be24-78c2cc343a76\") "
Sep 14 05:09:29.663645 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:29.663535    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/956bf287-009d-4eb1-be24-78c2cc343a76-model-cache\") pod \"956bf287-009d-4eb1-be24-78c2cc343a76\" (UID: \"956bf287-009d-4eb1-be24-78c2cc343a76\") "
Sep 14 05:09:29.663645 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:29.663583    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/956bf287-009d-4eb1-be24-78c2cc343a76-tmp-dir\") pod \"956bf287-009d-4eb1-be24-78c2cc343a76\" (UID: \"956bf287-009d-4eb1-be24-78c2cc343a76\") "
Sep 14 05:09:29.663772 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:29.663668    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/956bf287-009d-4eb1-be24-78c2cc343a76-home" (OuterVolumeSpecName: "home") pod "956bf287-009d-4eb1-be24-78c2cc343a76" (UID: "956bf287-009d-4eb1-be24-78c2cc343a76"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:09:29.663922 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:29.663892    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/956bf287-009d-4eb1-be24-78c2cc343a76-model-cache" (OuterVolumeSpecName: "model-cache") pod "956bf287-009d-4eb1-be24-78c2cc343a76" (UID: "956bf287-009d-4eb1-be24-78c2cc343a76"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:09:29.663922 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:29.663908    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/956bf287-009d-4eb1-be24-78c2cc343a76-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "956bf287-009d-4eb1-be24-78c2cc343a76" (UID: "956bf287-009d-4eb1-be24-78c2cc343a76"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:09:29.664037 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:29.663966    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/956bf287-009d-4eb1-be24-78c2cc343a76-home\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:09:29.665566 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:29.665544    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/956bf287-009d-4eb1-be24-78c2cc343a76-dshm" (OuterVolumeSpecName: "dshm") pod "956bf287-009d-4eb1-be24-78c2cc343a76" (UID: "956bf287-009d-4eb1-be24-78c2cc343a76"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:09:29.665639 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:29.665571    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/956bf287-009d-4eb1-be24-78c2cc343a76-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "956bf287-009d-4eb1-be24-78c2cc343a76" (UID: "956bf287-009d-4eb1-be24-78c2cc343a76"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 05:09:29.665639 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:29.665627    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/956bf287-009d-4eb1-be24-78c2cc343a76-kube-api-access-p9qdk" (OuterVolumeSpecName: "kube-api-access-p9qdk") pod "956bf287-009d-4eb1-be24-78c2cc343a76" (UID: "956bf287-009d-4eb1-be24-78c2cc343a76"). InnerVolumeSpecName "kube-api-access-p9qdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:09:29.765106 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:29.765073    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/956bf287-009d-4eb1-be24-78c2cc343a76-tls-certs\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:09:29.765106 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:29.765101    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9qdk\" (UniqueName: \"kubernetes.io/projected/956bf287-009d-4eb1-be24-78c2cc343a76-kube-api-access-p9qdk\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:09:29.765106 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:29.765110    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/956bf287-009d-4eb1-be24-78c2cc343a76-model-cache\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:09:29.765330 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:29.765134    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/956bf287-009d-4eb1-be24-78c2cc343a76-tmp-dir\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:09:29.765330 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:29.765143    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/956bf287-009d-4eb1-be24-78c2cc343a76-dshm\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:09:29.988875 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:29.988776    2565 generic.go:356] "Generic (PLEG): container finished" podID="956bf287-009d-4eb1-be24-78c2cc343a76" containerID="b4a500de9ff7a8b4e750207abdad0b470d475d5bd5d5dcde4b82fc5cf33ec993" exitCode=0
Sep 14 05:09:29.988875 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:29.988822    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-667664/test-model-667664-kserve-96bfc7c9f-ktdrk"
Sep 14 05:09:29.989110 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:29.988867    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-667664/test-model-667664-kserve-96bfc7c9f-ktdrk" event={"ID":"956bf287-009d-4eb1-be24-78c2cc343a76","Type":"ContainerDied","Data":"b4a500de9ff7a8b4e750207abdad0b470d475d5bd5d5dcde4b82fc5cf33ec993"}
Sep 14 05:09:29.989110 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:29.988921    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-667664/test-model-667664-kserve-96bfc7c9f-ktdrk" event={"ID":"956bf287-009d-4eb1-be24-78c2cc343a76","Type":"ContainerDied","Data":"2d80337243f7581244780534056584e45ecdfba8a5b655431bf1fee78a201033"}
Sep 14 05:09:29.989110 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:29.988943    2565 scope.go:117] "RemoveContainer" containerID="b4a500de9ff7a8b4e750207abdad0b470d475d5bd5d5dcde4b82fc5cf33ec993"
Sep 14 05:09:29.998434 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:29.998417    2565 scope.go:117] "RemoveContainer" containerID="b4a500de9ff7a8b4e750207abdad0b470d475d5bd5d5dcde4b82fc5cf33ec993"
Sep 14 05:09:29.998691 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:09:29.998673    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4a500de9ff7a8b4e750207abdad0b470d475d5bd5d5dcde4b82fc5cf33ec993\": container with ID starting with b4a500de9ff7a8b4e750207abdad0b470d475d5bd5d5dcde4b82fc5cf33ec993 not found: ID does not exist" containerID="b4a500de9ff7a8b4e750207abdad0b470d475d5bd5d5dcde4b82fc5cf33ec993"
Sep 14 05:09:29.998740 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:29.998700    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4a500de9ff7a8b4e750207abdad0b470d475d5bd5d5dcde4b82fc5cf33ec993"} err="failed to get container status \"b4a500de9ff7a8b4e750207abdad0b470d475d5bd5d5dcde4b82fc5cf33ec993\": rpc error: code = NotFound desc = could not find container \"b4a500de9ff7a8b4e750207abdad0b470d475d5bd5d5dcde4b82fc5cf33ec993\": container with ID starting with b4a500de9ff7a8b4e750207abdad0b470d475d5bd5d5dcde4b82fc5cf33ec993 not found: ID does not exist"
Sep 14 05:09:30.021231 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:30.021183    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-667664/test-model-667664-kserve-96bfc7c9f-ktdrk"]
Sep 14 05:09:30.023489 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:30.023463    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-667664/test-model-667664-kserve-96bfc7c9f-ktdrk"]
Sep 14 05:09:30.335260 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:30.335230    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="956bf287-009d-4eb1-be24-78c2cc343a76" path="/var/lib/kubelet/pods/956bf287-009d-4eb1-be24-78c2cc343a76/volumes"
Sep 14 05:09:35.631399 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:35.631365    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:09:44.133004 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:44.132970    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-c7c6a48c/e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm"]
Sep 14 05:09:44.133367 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:44.133357    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="956bf287-009d-4eb1-be24-78c2cc343a76" containerName="main"
Sep 14 05:09:44.133408 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:44.133368    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="956bf287-009d-4eb1-be24-78c2cc343a76" containerName="main"
Sep 14 05:09:44.133443 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:44.133438    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="956bf287-009d-4eb1-be24-78c2cc343a76" containerName="main"
Sep 14 05:09:44.137330 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:44.137312    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c7c6a48c/e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm"
Sep 14 05:09:44.140451 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:44.140426    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-c7c6a48c\"/\"e2e-lifecycle-model-c7c6a48c-kserve-self-signed-certs\""
Sep 14 05:09:44.140451 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:44.140445    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-c7c6a48c\"/\"kube-root-ca.crt\""
Sep 14 05:09:44.140617 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:44.140425    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-c7c6a48c\"/\"default-dockercfg-rfxsj\""
Sep 14 05:09:44.140617 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:44.140425    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-c7c6a48c\"/\"openshift-service-ca.crt\""
Sep 14 05:09:44.146043 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:44.146021    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-c7c6a48c/e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm"]
Sep 14 05:09:44.199479 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:44.199440    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dff61ad-71c3-4c07-b6fc-34d4fff8b68a-tls-certs\") pod \"e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm\" (UID: \"0dff61ad-71c3-4c07-b6fc-34d4fff8b68a\") " pod="ai-tenant-e2e-mt-c7c6a48c/e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm"
Sep 14 05:09:44.199646 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:44.199492    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkflg\" (UniqueName: \"kubernetes.io/projected/0dff61ad-71c3-4c07-b6fc-34d4fff8b68a-kube-api-access-bkflg\") pod \"e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm\" (UID: \"0dff61ad-71c3-4c07-b6fc-34d4fff8b68a\") " pod="ai-tenant-e2e-mt-c7c6a48c/e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm"
Sep 14 05:09:44.199646 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:44.199566    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dff61ad-71c3-4c07-b6fc-34d4fff8b68a-home\") pod \"e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm\" (UID: \"0dff61ad-71c3-4c07-b6fc-34d4fff8b68a\") " pod="ai-tenant-e2e-mt-c7c6a48c/e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm"
Sep 14 05:09:44.199646 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:44.199613    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dff61ad-71c3-4c07-b6fc-34d4fff8b68a-tmp-dir\") pod \"e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm\" (UID: \"0dff61ad-71c3-4c07-b6fc-34d4fff8b68a\") " pod="ai-tenant-e2e-mt-c7c6a48c/e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm"
Sep 14 05:09:44.199785 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:44.199668    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dff61ad-71c3-4c07-b6fc-34d4fff8b68a-dshm\") pod \"e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm\" (UID: \"0dff61ad-71c3-4c07-b6fc-34d4fff8b68a\") " pod="ai-tenant-e2e-mt-c7c6a48c/e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm"
Sep 14 05:09:44.199785 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:44.199693    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dff61ad-71c3-4c07-b6fc-34d4fff8b68a-model-cache\") pod \"e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm\" (UID: \"0dff61ad-71c3-4c07-b6fc-34d4fff8b68a\") " pod="ai-tenant-e2e-mt-c7c6a48c/e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm"
Sep 14 05:09:44.300729 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:44.300688    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dff61ad-71c3-4c07-b6fc-34d4fff8b68a-dshm\") pod \"e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm\" (UID: \"0dff61ad-71c3-4c07-b6fc-34d4fff8b68a\") " pod="ai-tenant-e2e-mt-c7c6a48c/e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm"
Sep 14 05:09:44.300958 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:44.300746    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dff61ad-71c3-4c07-b6fc-34d4fff8b68a-model-cache\") pod \"e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm\" (UID: \"0dff61ad-71c3-4c07-b6fc-34d4fff8b68a\") " pod="ai-tenant-e2e-mt-c7c6a48c/e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm"
Sep 14 05:09:44.300958 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:44.300826    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dff61ad-71c3-4c07-b6fc-34d4fff8b68a-tls-certs\") pod \"e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm\" (UID: \"0dff61ad-71c3-4c07-b6fc-34d4fff8b68a\") " pod="ai-tenant-e2e-mt-c7c6a48c/e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm"
Sep 14 05:09:44.300958 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:44.300873    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkflg\" (UniqueName: \"kubernetes.io/projected/0dff61ad-71c3-4c07-b6fc-34d4fff8b68a-kube-api-access-bkflg\") pod \"e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm\" (UID: \"0dff61ad-71c3-4c07-b6fc-34d4fff8b68a\") " pod="ai-tenant-e2e-mt-c7c6a48c/e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm"
Sep 14 05:09:44.300958 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:44.300915    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dff61ad-71c3-4c07-b6fc-34d4fff8b68a-home\") pod \"e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm\" (UID: \"0dff61ad-71c3-4c07-b6fc-34d4fff8b68a\") " pod="ai-tenant-e2e-mt-c7c6a48c/e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm"
Sep 14 05:09:44.300958 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:44.300944    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dff61ad-71c3-4c07-b6fc-34d4fff8b68a-tmp-dir\") pod \"e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm\" (UID: \"0dff61ad-71c3-4c07-b6fc-34d4fff8b68a\") " pod="ai-tenant-e2e-mt-c7c6a48c/e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm"
Sep 14 05:09:44.301281 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:44.301255    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dff61ad-71c3-4c07-b6fc-34d4fff8b68a-model-cache\") pod \"e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm\" (UID: \"0dff61ad-71c3-4c07-b6fc-34d4fff8b68a\") " pod="ai-tenant-e2e-mt-c7c6a48c/e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm"
Sep 14 05:09:44.301354 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:44.301266    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dff61ad-71c3-4c07-b6fc-34d4fff8b68a-home\") pod \"e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm\" (UID: \"0dff61ad-71c3-4c07-b6fc-34d4fff8b68a\") " pod="ai-tenant-e2e-mt-c7c6a48c/e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm"
Sep 14 05:09:44.301354 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:44.301328    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dff61ad-71c3-4c07-b6fc-34d4fff8b68a-tmp-dir\") pod \"e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm\" (UID: \"0dff61ad-71c3-4c07-b6fc-34d4fff8b68a\") " pod="ai-tenant-e2e-mt-c7c6a48c/e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm"
Sep 14 05:09:44.303075 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:44.303044    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dff61ad-71c3-4c07-b6fc-34d4fff8b68a-dshm\") pod \"e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm\" (UID: \"0dff61ad-71c3-4c07-b6fc-34d4fff8b68a\") " pod="ai-tenant-e2e-mt-c7c6a48c/e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm"
Sep 14 05:09:44.303419 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:44.303401    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dff61ad-71c3-4c07-b6fc-34d4fff8b68a-tls-certs\") pod \"e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm\" (UID: \"0dff61ad-71c3-4c07-b6fc-34d4fff8b68a\") " pod="ai-tenant-e2e-mt-c7c6a48c/e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm"
Sep 14 05:09:44.308186 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:44.308166    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkflg\" (UniqueName: \"kubernetes.io/projected/0dff61ad-71c3-4c07-b6fc-34d4fff8b68a-kube-api-access-bkflg\") pod \"e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm\" (UID: \"0dff61ad-71c3-4c07-b6fc-34d4fff8b68a\") " pod="ai-tenant-e2e-mt-c7c6a48c/e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm"
Sep 14 05:09:44.447567 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:44.447477    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c7c6a48c/e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm"
Sep 14 05:09:44.722243 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:44.722213    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-c7c6a48c/e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm"]
Sep 14 05:09:44.723727 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:09:44.723701    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0dff61ad_71c3_4c07_b6fc_34d4fff8b68a.slice/crio-aa46b66df609bb9b0ec969886f318f32eb8f1e68ad90362b3a37d0a17cb72237 WatchSource:0}: Error finding container aa46b66df609bb9b0ec969886f318f32eb8f1e68ad90362b3a37d0a17cb72237: Status 404 returned error can't find the container with id aa46b66df609bb9b0ec969886f318f32eb8f1e68ad90362b3a37d0a17cb72237
Sep 14 05:09:45.073216 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:45.073122    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c7c6a48c/e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm" event={"ID":"0dff61ad-71c3-4c07-b6fc-34d4fff8b68a","Type":"ContainerStarted","Data":"34e1221159072d9ccdc1c49c5381d47ae5630a92cf0d0624a24a0936ce16d241"}
Sep 14 05:09:45.073216 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:45.073168    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c7c6a48c/e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm" event={"ID":"0dff61ad-71c3-4c07-b6fc-34d4fff8b68a","Type":"ContainerStarted","Data":"aa46b66df609bb9b0ec969886f318f32eb8f1e68ad90362b3a37d0a17cb72237"}
Sep 14 05:09:45.088767 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:45.088710    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-c7c6a48c/e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm" podStartSLOduration=1.088694547 podStartE2EDuration="1.088694547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:09:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:09:45.087848292 +0000 UTC m=+1383.263619863" watchObservedRunningTime="2026-09-14 05:09:45.088694547 +0000 UTC m=+1383.264466105"
Sep 14 05:09:46.863846 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:46.863811    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh"]
Sep 14 05:09:46.864253 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:46.864071    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh" podUID="595352d9-0e7d-41a9-a318-7ad8bc720e26" containerName="istio-proxy" containerID="cri-o://204df6b0d42a11a2a29018601a725f28cd2cedeb14a3998d16b5b539db6f389a" gracePeriod=30
Sep 14 05:09:47.804891 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:47.804855    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-255ce3/noten-model-255ce3-kserve-7594d477f9-fwz8c"]
Sep 14 05:09:48.610622 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:48.610589    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e7fdd6/test-model-e7fdd6-kserve-78dc566997-7ssq4"]
Sep 14 05:09:49.645550 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:49.645515    2565 patch_prober.go:28] interesting pod/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 14 05:09:49.646006 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:49.645576    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh" podUID="595352d9-0e7d-41a9-a318-7ad8bc720e26" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 14 05:09:49.805950 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:49.805917    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e08402/e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk"]
Sep 14 05:09:52.112299 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:52.112271    2565 generic.go:356] "Generic (PLEG): container finished" podID="595352d9-0e7d-41a9-a318-7ad8bc720e26" containerID="204df6b0d42a11a2a29018601a725f28cd2cedeb14a3998d16b5b539db6f389a" exitCode=0
Sep 14 05:09:52.112720 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:52.112333    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh" event={"ID":"595352d9-0e7d-41a9-a318-7ad8bc720e26","Type":"ContainerDied","Data":"204df6b0d42a11a2a29018601a725f28cd2cedeb14a3998d16b5b539db6f389a"}
Sep 14 05:09:52.112720 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:52.112385    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh" event={"ID":"595352d9-0e7d-41a9-a318-7ad8bc720e26","Type":"ContainerDied","Data":"2706bc9f67f9b35f85b0b94cf8aeb7c1a55f80da67bbd7492fe7727aa767b861"}
Sep 14 05:09:52.112720 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:52.112403    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2706bc9f67f9b35f85b0b94cf8aeb7c1a55f80da67bbd7492fe7727aa767b861"
Sep 14 05:09:52.118386 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:52.118368    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh"
Sep 14 05:09:52.176507 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:52.176434    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/595352d9-0e7d-41a9-a318-7ad8bc720e26-workload-certs\") pod \"595352d9-0e7d-41a9-a318-7ad8bc720e26\" (UID: \"595352d9-0e7d-41a9-a318-7ad8bc720e26\") "
Sep 14 05:09:52.176507 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:52.176471    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpml4\" (UniqueName: \"kubernetes.io/projected/595352d9-0e7d-41a9-a318-7ad8bc720e26-kube-api-access-lpml4\") pod \"595352d9-0e7d-41a9-a318-7ad8bc720e26\" (UID: \"595352d9-0e7d-41a9-a318-7ad8bc720e26\") "
Sep 14 05:09:52.176507 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:52.176507    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/595352d9-0e7d-41a9-a318-7ad8bc720e26-istio-data\") pod \"595352d9-0e7d-41a9-a318-7ad8bc720e26\" (UID: \"595352d9-0e7d-41a9-a318-7ad8bc720e26\") "
Sep 14 05:09:52.176740 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:52.176538    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/595352d9-0e7d-41a9-a318-7ad8bc720e26-istio-envoy\") pod \"595352d9-0e7d-41a9-a318-7ad8bc720e26\" (UID: \"595352d9-0e7d-41a9-a318-7ad8bc720e26\") "
Sep 14 05:09:52.176740 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:52.176560    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/595352d9-0e7d-41a9-a318-7ad8bc720e26-workload-socket\") pod \"595352d9-0e7d-41a9-a318-7ad8bc720e26\" (UID: \"595352d9-0e7d-41a9-a318-7ad8bc720e26\") "
Sep 14 05:09:52.176740 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:52.176617    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/595352d9-0e7d-41a9-a318-7ad8bc720e26-istiod-ca-cert\") pod \"595352d9-0e7d-41a9-a318-7ad8bc720e26\" (UID: \"595352d9-0e7d-41a9-a318-7ad8bc720e26\") "
Sep 14 05:09:52.176740 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:52.176658    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/595352d9-0e7d-41a9-a318-7ad8bc720e26-istio-token\") pod \"595352d9-0e7d-41a9-a318-7ad8bc720e26\" (UID: \"595352d9-0e7d-41a9-a318-7ad8bc720e26\") "
Sep 14 05:09:52.176740 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:52.176714    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/595352d9-0e7d-41a9-a318-7ad8bc720e26-istio-podinfo\") pod \"595352d9-0e7d-41a9-a318-7ad8bc720e26\" (UID: \"595352d9-0e7d-41a9-a318-7ad8bc720e26\") "
Sep 14 05:09:52.177042 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:52.176741    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/595352d9-0e7d-41a9-a318-7ad8bc720e26-credential-socket\") pod \"595352d9-0e7d-41a9-a318-7ad8bc720e26\" (UID: \"595352d9-0e7d-41a9-a318-7ad8bc720e26\") "
Sep 14 05:09:52.177042 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:52.176740    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595352d9-0e7d-41a9-a318-7ad8bc720e26-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "595352d9-0e7d-41a9-a318-7ad8bc720e26" (UID: "595352d9-0e7d-41a9-a318-7ad8bc720e26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:09:52.177042 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:52.176816    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595352d9-0e7d-41a9-a318-7ad8bc720e26-istio-data" (OuterVolumeSpecName: "istio-data") pod "595352d9-0e7d-41a9-a318-7ad8bc720e26" (UID: "595352d9-0e7d-41a9-a318-7ad8bc720e26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:09:52.177042 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:52.176987    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/595352d9-0e7d-41a9-a318-7ad8bc720e26-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "595352d9-0e7d-41a9-a318-7ad8bc720e26" (UID: "595352d9-0e7d-41a9-a318-7ad8bc720e26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 05:09:52.177249 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:52.177085    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595352d9-0e7d-41a9-a318-7ad8bc720e26-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "595352d9-0e7d-41a9-a318-7ad8bc720e26" (UID: "595352d9-0e7d-41a9-a318-7ad8bc720e26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:09:52.177249 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:52.177188    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595352d9-0e7d-41a9-a318-7ad8bc720e26-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "595352d9-0e7d-41a9-a318-7ad8bc720e26" (UID: "595352d9-0e7d-41a9-a318-7ad8bc720e26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:09:52.177249 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:52.177202    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/595352d9-0e7d-41a9-a318-7ad8bc720e26-istio-data\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:09:52.177249 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:52.177223    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/595352d9-0e7d-41a9-a318-7ad8bc720e26-workload-socket\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:09:52.177249 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:52.177237    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/595352d9-0e7d-41a9-a318-7ad8bc720e26-istiod-ca-cert\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:09:52.177249 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:52.177250    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/595352d9-0e7d-41a9-a318-7ad8bc720e26-workload-certs\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:09:52.179118 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:52.179098    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595352d9-0e7d-41a9-a318-7ad8bc720e26-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "595352d9-0e7d-41a9-a318-7ad8bc720e26" (UID: "595352d9-0e7d-41a9-a318-7ad8bc720e26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:09:52.179215 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:52.179124    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/595352d9-0e7d-41a9-a318-7ad8bc720e26-kube-api-access-lpml4" (OuterVolumeSpecName: "kube-api-access-lpml4") pod "595352d9-0e7d-41a9-a318-7ad8bc720e26" (UID: "595352d9-0e7d-41a9-a318-7ad8bc720e26"). InnerVolumeSpecName "kube-api-access-lpml4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:09:52.179279 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:52.179231    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/595352d9-0e7d-41a9-a318-7ad8bc720e26-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "595352d9-0e7d-41a9-a318-7ad8bc720e26" (UID: "595352d9-0e7d-41a9-a318-7ad8bc720e26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 05:09:52.179279 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:52.179231    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/595352d9-0e7d-41a9-a318-7ad8bc720e26-istio-token" (OuterVolumeSpecName: "istio-token") pod "595352d9-0e7d-41a9-a318-7ad8bc720e26" (UID: "595352d9-0e7d-41a9-a318-7ad8bc720e26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:09:52.278288 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:52.278255    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpml4\" (UniqueName: \"kubernetes.io/projected/595352d9-0e7d-41a9-a318-7ad8bc720e26-kube-api-access-lpml4\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:09:52.278288 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:52.278283    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/595352d9-0e7d-41a9-a318-7ad8bc720e26-istio-envoy\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:09:52.278288 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:52.278292    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/595352d9-0e7d-41a9-a318-7ad8bc720e26-istio-token\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:09:52.278492 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:52.278300    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/595352d9-0e7d-41a9-a318-7ad8bc720e26-istio-podinfo\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:09:52.278492 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:52.278308    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/595352d9-0e7d-41a9-a318-7ad8bc720e26-credential-socket\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:09:53.118163 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:53.118128    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh"
Sep 14 05:09:53.135673 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:53.135639    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh"]
Sep 14 05:09:53.141741 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:53.141712    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-255ce3-openshift-default-dd5459cf8-4mjlh"]
Sep 14 05:09:53.432905 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:53.432824    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:09:54.335295 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:54.335260    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="595352d9-0e7d-41a9-a318-7ad8bc720e26" path="/var/lib/kubelet/pods/595352d9-0e7d-41a9-a318-7ad8bc720e26/volumes"
Sep 14 05:09:54.447914 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:54.447878    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-c7c6a48c/e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm"
Sep 14 05:09:54.448083 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:54.447937    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-c7c6a48c/e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm"
Sep 14 05:09:54.459798 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:54.459773    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-c7c6a48c/e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm"
Sep 14 05:09:55.142028 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:55.141995    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-c7c6a48c/e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm"
Sep 14 05:09:55.820436 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:55.820405    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-667664-5bd57bd994-52n8z"]
Sep 14 05:09:55.820904 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:55.820610    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-667664-5bd57bd994-52n8z" podUID="eec5515b-3b3f-4fdd-9cf0-a8f2cf77a226" containerName="payload-pre-processing" containerID="cri-o://bd753d2797174af7a089b061f9a5221f9b5601272b4d5ce0e92bd39708639016" gracePeriod=30
Sep 14 05:09:56.074968 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:56.074907    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-667664-5bd57bd994-52n8z"
Sep 14 05:09:56.114334 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:56.114311    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jphlh\" (UniqueName: \"kubernetes.io/projected/eec5515b-3b3f-4fdd-9cf0-a8f2cf77a226-kube-api-access-jphlh\") pod \"eec5515b-3b3f-4fdd-9cf0-a8f2cf77a226\" (UID: \"eec5515b-3b3f-4fdd-9cf0-a8f2cf77a226\") "
Sep 14 05:09:56.114494 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:56.114477    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eec5515b-3b3f-4fdd-9cf0-a8f2cf77a226-plugins-config-volume\") pod \"eec5515b-3b3f-4fdd-9cf0-a8f2cf77a226\" (UID: \"eec5515b-3b3f-4fdd-9cf0-a8f2cf77a226\") "
Sep 14 05:09:56.114832 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:56.114809    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eec5515b-3b3f-4fdd-9cf0-a8f2cf77a226-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eec5515b-3b3f-4fdd-9cf0-a8f2cf77a226" (UID: "eec5515b-3b3f-4fdd-9cf0-a8f2cf77a226"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 05:09:56.116494 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:56.116469    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eec5515b-3b3f-4fdd-9cf0-a8f2cf77a226-kube-api-access-jphlh" (OuterVolumeSpecName: "kube-api-access-jphlh") pod "eec5515b-3b3f-4fdd-9cf0-a8f2cf77a226" (UID: "eec5515b-3b3f-4fdd-9cf0-a8f2cf77a226"). InnerVolumeSpecName "kube-api-access-jphlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:09:56.135791 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:56.135747    2565 generic.go:356] "Generic (PLEG): container finished" podID="eec5515b-3b3f-4fdd-9cf0-a8f2cf77a226" containerID="bd753d2797174af7a089b061f9a5221f9b5601272b4d5ce0e92bd39708639016" exitCode=0
Sep 14 05:09:56.135898 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:56.135806    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-667664-5bd57bd994-52n8z"
Sep 14 05:09:56.135898 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:56.135844    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-667664-5bd57bd994-52n8z" event={"ID":"eec5515b-3b3f-4fdd-9cf0-a8f2cf77a226","Type":"ContainerDied","Data":"bd753d2797174af7a089b061f9a5221f9b5601272b4d5ce0e92bd39708639016"}
Sep 14 05:09:56.135898 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:56.135896    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-667664-5bd57bd994-52n8z" event={"ID":"eec5515b-3b3f-4fdd-9cf0-a8f2cf77a226","Type":"ContainerDied","Data":"6d95c4a1106b32bcf469a2e5accdd7462e7516d2f95d64256e1d8ef5c1389fab"}
Sep 14 05:09:56.136049 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:56.135919    2565 scope.go:117] "RemoveContainer" containerID="bd753d2797174af7a089b061f9a5221f9b5601272b4d5ce0e92bd39708639016"
Sep 14 05:09:56.145666 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:56.145648    2565 scope.go:117] "RemoveContainer" containerID="bd753d2797174af7a089b061f9a5221f9b5601272b4d5ce0e92bd39708639016"
Sep 14 05:09:56.145977 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:09:56.145958    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd753d2797174af7a089b061f9a5221f9b5601272b4d5ce0e92bd39708639016\": container with ID starting with bd753d2797174af7a089b061f9a5221f9b5601272b4d5ce0e92bd39708639016 not found: ID does not exist" containerID="bd753d2797174af7a089b061f9a5221f9b5601272b4d5ce0e92bd39708639016"
Sep 14 05:09:56.146051 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:56.145989    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd753d2797174af7a089b061f9a5221f9b5601272b4d5ce0e92bd39708639016"} err="failed to get container status \"bd753d2797174af7a089b061f9a5221f9b5601272b4d5ce0e92bd39708639016\": rpc error: code = NotFound desc = could not find container \"bd753d2797174af7a089b061f9a5221f9b5601272b4d5ce0e92bd39708639016\": container with ID starting with bd753d2797174af7a089b061f9a5221f9b5601272b4d5ce0e92bd39708639016 not found: ID does not exist"
Sep 14 05:09:56.157520 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:56.157500    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-667664-5bd57bd994-52n8z"]
Sep 14 05:09:56.160768 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:56.160735    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-667664-5bd57bd994-52n8z"]
Sep 14 05:09:56.216212 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:56.216186    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eec5515b-3b3f-4fdd-9cf0-a8f2cf77a226-plugins-config-volume\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:09:56.216212 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:56.216209    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jphlh\" (UniqueName: \"kubernetes.io/projected/eec5515b-3b3f-4fdd-9cf0-a8f2cf77a226-kube-api-access-jphlh\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:09:56.335310 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:09:56.335281    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eec5515b-3b3f-4fdd-9cf0-a8f2cf77a226" path="/var/lib/kubelet/pods/eec5515b-3b3f-4fdd-9cf0-a8f2cf77a226/volumes"
Sep 14 05:10:02.823967 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:02.823921    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-255ce3/noten-model-255ce3-kserve-7594d477f9-fwz8c" podUID="728f28c2-15b8-4d33-9ed1-a99c2b02cd59" containerName="main" containerID="cri-o://79ef385bcaa942644094327a0c507ad5de351dad4a34eb2b1e257430319a19da" gracePeriod=45
Sep 14 05:10:03.065646 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.065617    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e7fdd6-lxqgm"]
Sep 14 05:10:03.066049 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.066033    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eec5515b-3b3f-4fdd-9cf0-a8f2cf77a226" containerName="payload-pre-processing"
Sep 14 05:10:03.066103 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.066050    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="eec5515b-3b3f-4fdd-9cf0-a8f2cf77a226" containerName="payload-pre-processing"
Sep 14 05:10:03.066103 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.066059    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="595352d9-0e7d-41a9-a318-7ad8bc720e26" containerName="istio-proxy"
Sep 14 05:10:03.066103 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.066065    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="595352d9-0e7d-41a9-a318-7ad8bc720e26" containerName="istio-proxy"
Sep 14 05:10:03.066216 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.066118    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="595352d9-0e7d-41a9-a318-7ad8bc720e26" containerName="istio-proxy"
Sep 14 05:10:03.066216 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.066127    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="eec5515b-3b3f-4fdd-9cf0-a8f2cf77a226" containerName="payload-pre-processing"
Sep 14 05:10:03.069316 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.069297    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e7fdd6-lxqgm"
Sep 14 05:10:03.071136 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.071118    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-255ce3/noten-model-255ce3-kserve-7594d477f9-fwz8c"
Sep 14 05:10:03.071474 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.071458    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 05:10:03.071711 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.071696    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k6hpm\""
Sep 14 05:10:03.076530 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.076304    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e7fdd6-lxqgm"]
Sep 14 05:10:03.176345 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.176307    2565 generic.go:356] "Generic (PLEG): container finished" podID="728f28c2-15b8-4d33-9ed1-a99c2b02cd59" containerID="79ef385bcaa942644094327a0c507ad5de351dad4a34eb2b1e257430319a19da" exitCode=0
Sep 14 05:10:03.176524 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.176353    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-255ce3/noten-model-255ce3-kserve-7594d477f9-fwz8c"
Sep 14 05:10:03.176524 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.176388    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-255ce3/noten-model-255ce3-kserve-7594d477f9-fwz8c" event={"ID":"728f28c2-15b8-4d33-9ed1-a99c2b02cd59","Type":"ContainerDied","Data":"79ef385bcaa942644094327a0c507ad5de351dad4a34eb2b1e257430319a19da"}
Sep 14 05:10:03.176524 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.176423    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-255ce3/noten-model-255ce3-kserve-7594d477f9-fwz8c" event={"ID":"728f28c2-15b8-4d33-9ed1-a99c2b02cd59","Type":"ContainerDied","Data":"55d4e95bb67bbfbc955f8f2530d6817ab640b5b9fc4415c952a05ba8a9449c79"}
Sep 14 05:10:03.176524 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.176441    2565 scope.go:117] "RemoveContainer" containerID="79ef385bcaa942644094327a0c507ad5de351dad4a34eb2b1e257430319a19da"
Sep 14 05:10:03.180157 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.180135    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/728f28c2-15b8-4d33-9ed1-a99c2b02cd59-tls-certs\") pod \"728f28c2-15b8-4d33-9ed1-a99c2b02cd59\" (UID: \"728f28c2-15b8-4d33-9ed1-a99c2b02cd59\") "
Sep 14 05:10:03.180297 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.180167    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/728f28c2-15b8-4d33-9ed1-a99c2b02cd59-dshm\") pod \"728f28c2-15b8-4d33-9ed1-a99c2b02cd59\" (UID: \"728f28c2-15b8-4d33-9ed1-a99c2b02cd59\") "
Sep 14 05:10:03.180297 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.180216    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fpjj\" (UniqueName: \"kubernetes.io/projected/728f28c2-15b8-4d33-9ed1-a99c2b02cd59-kube-api-access-2fpjj\") pod \"728f28c2-15b8-4d33-9ed1-a99c2b02cd59\" (UID: \"728f28c2-15b8-4d33-9ed1-a99c2b02cd59\") "
Sep 14 05:10:03.180297 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.180249    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/728f28c2-15b8-4d33-9ed1-a99c2b02cd59-model-cache\") pod \"728f28c2-15b8-4d33-9ed1-a99c2b02cd59\" (UID: \"728f28c2-15b8-4d33-9ed1-a99c2b02cd59\") "
Sep 14 05:10:03.180297 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.180292    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/728f28c2-15b8-4d33-9ed1-a99c2b02cd59-tmp-dir\") pod \"728f28c2-15b8-4d33-9ed1-a99c2b02cd59\" (UID: \"728f28c2-15b8-4d33-9ed1-a99c2b02cd59\") "
Sep 14 05:10:03.180531 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.180403    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/728f28c2-15b8-4d33-9ed1-a99c2b02cd59-home\") pod \"728f28c2-15b8-4d33-9ed1-a99c2b02cd59\" (UID: \"728f28c2-15b8-4d33-9ed1-a99c2b02cd59\") "
Sep 14 05:10:03.180689 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.180668    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44df6175-a921-422a-99dd-52d62a2205be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e7fdd6-lxqgm\" (UID: \"44df6175-a921-422a-99dd-52d62a2205be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e7fdd6-lxqgm"
Sep 14 05:10:03.180799 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.180668    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728f28c2-15b8-4d33-9ed1-a99c2b02cd59-model-cache" (OuterVolumeSpecName: "model-cache") pod "728f28c2-15b8-4d33-9ed1-a99c2b02cd59" (UID: "728f28c2-15b8-4d33-9ed1-a99c2b02cd59"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:10:03.180799 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.180680    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728f28c2-15b8-4d33-9ed1-a99c2b02cd59-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "728f28c2-15b8-4d33-9ed1-a99c2b02cd59" (UID: "728f28c2-15b8-4d33-9ed1-a99c2b02cd59"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:10:03.180928 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.180788    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728f28c2-15b8-4d33-9ed1-a99c2b02cd59-home" (OuterVolumeSpecName: "home") pod "728f28c2-15b8-4d33-9ed1-a99c2b02cd59" (UID: "728f28c2-15b8-4d33-9ed1-a99c2b02cd59"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:10:03.180928 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.180893    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/728f28c2-15b8-4d33-9ed1-a99c2b02cd59-tmp-dir\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:10:03.180928 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.180911    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/728f28c2-15b8-4d33-9ed1-a99c2b02cd59-home\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:10:03.180928 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.180925    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/728f28c2-15b8-4d33-9ed1-a99c2b02cd59-model-cache\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:10:03.182652 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.182629    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728f28c2-15b8-4d33-9ed1-a99c2b02cd59-dshm" (OuterVolumeSpecName: "dshm") pod "728f28c2-15b8-4d33-9ed1-a99c2b02cd59" (UID: "728f28c2-15b8-4d33-9ed1-a99c2b02cd59"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:10:03.182777 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.182689    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/728f28c2-15b8-4d33-9ed1-a99c2b02cd59-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "728f28c2-15b8-4d33-9ed1-a99c2b02cd59" (UID: "728f28c2-15b8-4d33-9ed1-a99c2b02cd59"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 05:10:03.182844 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.182791    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/728f28c2-15b8-4d33-9ed1-a99c2b02cd59-kube-api-access-2fpjj" (OuterVolumeSpecName: "kube-api-access-2fpjj") pod "728f28c2-15b8-4d33-9ed1-a99c2b02cd59" (UID: "728f28c2-15b8-4d33-9ed1-a99c2b02cd59"). InnerVolumeSpecName "kube-api-access-2fpjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:10:03.186397 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.186379    2565 scope.go:117] "RemoveContainer" containerID="79ef385bcaa942644094327a0c507ad5de351dad4a34eb2b1e257430319a19da"
Sep 14 05:10:03.186735 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:10:03.186715    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79ef385bcaa942644094327a0c507ad5de351dad4a34eb2b1e257430319a19da\": container with ID starting with 79ef385bcaa942644094327a0c507ad5de351dad4a34eb2b1e257430319a19da not found: ID does not exist" containerID="79ef385bcaa942644094327a0c507ad5de351dad4a34eb2b1e257430319a19da"
Sep 14 05:10:03.186834 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.186743    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79ef385bcaa942644094327a0c507ad5de351dad4a34eb2b1e257430319a19da"} err="failed to get container status \"79ef385bcaa942644094327a0c507ad5de351dad4a34eb2b1e257430319a19da\": rpc error: code = NotFound desc = could not find container \"79ef385bcaa942644094327a0c507ad5de351dad4a34eb2b1e257430319a19da\": container with ID starting with 79ef385bcaa942644094327a0c507ad5de351dad4a34eb2b1e257430319a19da not found: ID does not exist"
Sep 14 05:10:03.281685 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.281644    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44df6175-a921-422a-99dd-52d62a2205be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e7fdd6-lxqgm\" (UID: \"44df6175-a921-422a-99dd-52d62a2205be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e7fdd6-lxqgm"
Sep 14 05:10:03.281931 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.281783    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/728f28c2-15b8-4d33-9ed1-a99c2b02cd59-tls-certs\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:10:03.281931 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.281794    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/728f28c2-15b8-4d33-9ed1-a99c2b02cd59-dshm\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:10:03.281931 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.281803    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fpjj\" (UniqueName: \"kubernetes.io/projected/728f28c2-15b8-4d33-9ed1-a99c2b02cd59-kube-api-access-2fpjj\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:10:03.282415 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.282386    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44df6175-a921-422a-99dd-52d62a2205be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e7fdd6-lxqgm\" (UID: \"44df6175-a921-422a-99dd-52d62a2205be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e7fdd6-lxqgm"
Sep 14 05:10:03.381089 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.381054    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e7fdd6-lxqgm"
Sep 14 05:10:03.497264 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.497233    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-255ce3/noten-model-255ce3-kserve-7594d477f9-fwz8c"]
Sep 14 05:10:03.500694 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.500669    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-255ce3/noten-model-255ce3-kserve-7594d477f9-fwz8c"]
Sep 14 05:10:03.629848 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.629811    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-e7fdd6/test-model-e7fdd6-kserve-78dc566997-7ssq4" podUID="81dac50a-cd9d-4762-907b-878715836a9c" containerName="main" containerID="cri-o://a9e37c2b8fc6aab77fe729031734c627b1ea6592aef49da3b5927ee48ab585f6" gracePeriod=45
Sep 14 05:10:03.655873 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.655847    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e7fdd6-lxqgm"]
Sep 14 05:10:03.678777 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:10:03.678724    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44df6175_a921_422a_99dd_52d62a2205be.slice/crio-26477d9b35f8328d5daadc829f2952c275fa0ba25591ab83c89f7deef9bb5468 WatchSource:0}: Error finding container 26477d9b35f8328d5daadc829f2952c275fa0ba25591ab83c89f7deef9bb5468: Status 404 returned error can't find the container with id 26477d9b35f8328d5daadc829f2952c275fa0ba25591ab83c89f7deef9bb5468
Sep 14 05:10:03.872120 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.872093    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e7fdd6/test-model-e7fdd6-kserve-78dc566997-7ssq4"
Sep 14 05:10:03.987693 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.987665    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81dac50a-cd9d-4762-907b-878715836a9c-dshm\") pod \"81dac50a-cd9d-4762-907b-878715836a9c\" (UID: \"81dac50a-cd9d-4762-907b-878715836a9c\") "
Sep 14 05:10:03.987909 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.987735    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81dac50a-cd9d-4762-907b-878715836a9c-tmp-dir\") pod \"81dac50a-cd9d-4762-907b-878715836a9c\" (UID: \"81dac50a-cd9d-4762-907b-878715836a9c\") "
Sep 14 05:10:03.987909 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.987790    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81dac50a-cd9d-4762-907b-878715836a9c-home\") pod \"81dac50a-cd9d-4762-907b-878715836a9c\" (UID: \"81dac50a-cd9d-4762-907b-878715836a9c\") "
Sep 14 05:10:03.987909 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.987841    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmtjb\" (UniqueName: \"kubernetes.io/projected/81dac50a-cd9d-4762-907b-878715836a9c-kube-api-access-vmtjb\") pod \"81dac50a-cd9d-4762-907b-878715836a9c\" (UID: \"81dac50a-cd9d-4762-907b-878715836a9c\") "
Sep 14 05:10:03.987909 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.987880    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81dac50a-cd9d-4762-907b-878715836a9c-model-cache\") pod \"81dac50a-cd9d-4762-907b-878715836a9c\" (UID: \"81dac50a-cd9d-4762-907b-878715836a9c\") "
Sep 14 05:10:03.988126 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.987944    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81dac50a-cd9d-4762-907b-878715836a9c-tls-certs\") pod \"81dac50a-cd9d-4762-907b-878715836a9c\" (UID: \"81dac50a-cd9d-4762-907b-878715836a9c\") "
Sep 14 05:10:03.988126 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.988006    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81dac50a-cd9d-4762-907b-878715836a9c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "81dac50a-cd9d-4762-907b-878715836a9c" (UID: "81dac50a-cd9d-4762-907b-878715836a9c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:10:03.988244 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.988111    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81dac50a-cd9d-4762-907b-878715836a9c-home" (OuterVolumeSpecName: "home") pod "81dac50a-cd9d-4762-907b-878715836a9c" (UID: "81dac50a-cd9d-4762-907b-878715836a9c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:10:03.988244 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.988174    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81dac50a-cd9d-4762-907b-878715836a9c-model-cache" (OuterVolumeSpecName: "model-cache") pod "81dac50a-cd9d-4762-907b-878715836a9c" (UID: "81dac50a-cd9d-4762-907b-878715836a9c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:10:03.988369 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.988273    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81dac50a-cd9d-4762-907b-878715836a9c-model-cache\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:10:03.988369 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.988297    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81dac50a-cd9d-4762-907b-878715836a9c-tmp-dir\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:10:03.988369 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:03.988306    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81dac50a-cd9d-4762-907b-878715836a9c-home\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:10:04.038306 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:04.038269    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81dac50a-cd9d-4762-907b-878715836a9c-dshm" (OuterVolumeSpecName: "dshm") pod "81dac50a-cd9d-4762-907b-878715836a9c" (UID: "81dac50a-cd9d-4762-907b-878715836a9c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:10:04.038460 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:04.038402    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81dac50a-cd9d-4762-907b-878715836a9c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "81dac50a-cd9d-4762-907b-878715836a9c" (UID: "81dac50a-cd9d-4762-907b-878715836a9c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 05:10:04.038554 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:04.038487    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81dac50a-cd9d-4762-907b-878715836a9c-kube-api-access-vmtjb" (OuterVolumeSpecName: "kube-api-access-vmtjb") pod "81dac50a-cd9d-4762-907b-878715836a9c" (UID: "81dac50a-cd9d-4762-907b-878715836a9c"). InnerVolumeSpecName "kube-api-access-vmtjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:10:04.089645 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:04.089618    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81dac50a-cd9d-4762-907b-878715836a9c-dshm\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:10:04.089645 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:04.089641    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmtjb\" (UniqueName: \"kubernetes.io/projected/81dac50a-cd9d-4762-907b-878715836a9c-kube-api-access-vmtjb\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:10:04.089818 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:04.089652    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81dac50a-cd9d-4762-907b-878715836a9c-tls-certs\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:10:04.182949 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:04.182911    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e7fdd6-lxqgm" event={"ID":"44df6175-a921-422a-99dd-52d62a2205be","Type":"ContainerStarted","Data":"954240132f4e6c4e71c6313e2ac878c137fccaec0ee3b46be5349a26d4dbc9e8"}
Sep 14 05:10:04.183137 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:04.182976    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e7fdd6-lxqgm" event={"ID":"44df6175-a921-422a-99dd-52d62a2205be","Type":"ContainerStarted","Data":"26477d9b35f8328d5daadc829f2952c275fa0ba25591ab83c89f7deef9bb5468"}
Sep 14 05:10:04.184278 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:04.184250    2565 generic.go:356] "Generic (PLEG): container finished" podID="81dac50a-cd9d-4762-907b-878715836a9c" containerID="a9e37c2b8fc6aab77fe729031734c627b1ea6592aef49da3b5927ee48ab585f6" exitCode=0
Sep 14 05:10:04.184413 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:04.184290    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e7fdd6/test-model-e7fdd6-kserve-78dc566997-7ssq4" event={"ID":"81dac50a-cd9d-4762-907b-878715836a9c","Type":"ContainerDied","Data":"a9e37c2b8fc6aab77fe729031734c627b1ea6592aef49da3b5927ee48ab585f6"}
Sep 14 05:10:04.184413 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:04.184327    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e7fdd6/test-model-e7fdd6-kserve-78dc566997-7ssq4" event={"ID":"81dac50a-cd9d-4762-907b-878715836a9c","Type":"ContainerDied","Data":"c7e19f11f1bd7949866c0afe943e8205374e86dae5b5b040721b2b5a6ffb3249"}
Sep 14 05:10:04.184413 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:04.184346    2565 scope.go:117] "RemoveContainer" containerID="a9e37c2b8fc6aab77fe729031734c627b1ea6592aef49da3b5927ee48ab585f6"
Sep 14 05:10:04.184413 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:04.184298    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e7fdd6/test-model-e7fdd6-kserve-78dc566997-7ssq4"
Sep 14 05:10:04.195889 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:04.195871    2565 scope.go:117] "RemoveContainer" containerID="a9e37c2b8fc6aab77fe729031734c627b1ea6592aef49da3b5927ee48ab585f6"
Sep 14 05:10:04.196188 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:10:04.196171    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9e37c2b8fc6aab77fe729031734c627b1ea6592aef49da3b5927ee48ab585f6\": container with ID starting with a9e37c2b8fc6aab77fe729031734c627b1ea6592aef49da3b5927ee48ab585f6 not found: ID does not exist" containerID="a9e37c2b8fc6aab77fe729031734c627b1ea6592aef49da3b5927ee48ab585f6"
Sep 14 05:10:04.196230 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:04.196198    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9e37c2b8fc6aab77fe729031734c627b1ea6592aef49da3b5927ee48ab585f6"} err="failed to get container status \"a9e37c2b8fc6aab77fe729031734c627b1ea6592aef49da3b5927ee48ab585f6\": rpc error: code = NotFound desc = could not find container \"a9e37c2b8fc6aab77fe729031734c627b1ea6592aef49da3b5927ee48ab585f6\": container with ID starting with a9e37c2b8fc6aab77fe729031734c627b1ea6592aef49da3b5927ee48ab585f6 not found: ID does not exist"
Sep 14 05:10:04.200618 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:04.200550    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e7fdd6-lxqgm" podStartSLOduration=1.200537967 podStartE2EDuration="1.200537967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:10:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:10:04.198375405 +0000 UTC m=+1402.374146986" watchObservedRunningTime="2026-09-14 05:10:04.200537967 +0000 UTC m=+1402.376309568"
Sep 14 05:10:04.209971 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:04.209911    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e7fdd6/test-model-e7fdd6-kserve-78dc566997-7ssq4"]
Sep 14 05:10:04.215334 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:04.215314    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-e7fdd6/test-model-e7fdd6-kserve-78dc566997-7ssq4"]
Sep 14 05:10:04.334219 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:04.334186    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="728f28c2-15b8-4d33-9ed1-a99c2b02cd59" path="/var/lib/kubelet/pods/728f28c2-15b8-4d33-9ed1-a99c2b02cd59/volumes"
Sep 14 05:10:04.334587 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:04.334575    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81dac50a-cd9d-4762-907b-878715836a9c" path="/var/lib/kubelet/pods/81dac50a-cd9d-4762-907b-878715836a9c/volumes"
Sep 14 05:10:04.825063 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:04.824973    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-e08402/e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk" podUID="089ab012-ecd2-4d9d-9ce1-25f1cb8554fc" containerName="main" containerID="cri-o://a17e05025b86c6d928ac740db5988adaa24aaf23b388a24b600bec9430cbf66a" gracePeriod=45
Sep 14 05:10:05.081183 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:05.081116    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e08402/e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk"
Sep 14 05:10:05.192654 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:05.192616    2565 generic.go:356] "Generic (PLEG): container finished" podID="44df6175-a921-422a-99dd-52d62a2205be" containerID="954240132f4e6c4e71c6313e2ac878c137fccaec0ee3b46be5349a26d4dbc9e8" exitCode=0
Sep 14 05:10:05.192854 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:05.192707    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e7fdd6-lxqgm" event={"ID":"44df6175-a921-422a-99dd-52d62a2205be","Type":"ContainerDied","Data":"954240132f4e6c4e71c6313e2ac878c137fccaec0ee3b46be5349a26d4dbc9e8"}
Sep 14 05:10:05.194079 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:05.194057    2565 generic.go:356] "Generic (PLEG): container finished" podID="089ab012-ecd2-4d9d-9ce1-25f1cb8554fc" containerID="a17e05025b86c6d928ac740db5988adaa24aaf23b388a24b600bec9430cbf66a" exitCode=0
Sep 14 05:10:05.194195 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:05.194106    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e08402/e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk"
Sep 14 05:10:05.194195 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:05.194142    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e08402/e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk" event={"ID":"089ab012-ecd2-4d9d-9ce1-25f1cb8554fc","Type":"ContainerDied","Data":"a17e05025b86c6d928ac740db5988adaa24aaf23b388a24b600bec9430cbf66a"}
Sep 14 05:10:05.194195 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:05.194188    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e08402/e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk" event={"ID":"089ab012-ecd2-4d9d-9ce1-25f1cb8554fc","Type":"ContainerDied","Data":"9e802f8a5b07f7c53794095a24442c4528913f297a9458e8b8e71c7d92315c5c"}
Sep 14 05:10:05.194327 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:05.194214    2565 scope.go:117] "RemoveContainer" containerID="a17e05025b86c6d928ac740db5988adaa24aaf23b388a24b600bec9430cbf66a"
Sep 14 05:10:05.199354 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:05.199335    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/089ab012-ecd2-4d9d-9ce1-25f1cb8554fc-dshm\") pod \"089ab012-ecd2-4d9d-9ce1-25f1cb8554fc\" (UID: \"089ab012-ecd2-4d9d-9ce1-25f1cb8554fc\") "
Sep 14 05:10:05.199465 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:05.199370    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfg65\" (UniqueName: \"kubernetes.io/projected/089ab012-ecd2-4d9d-9ce1-25f1cb8554fc-kube-api-access-sfg65\") pod \"089ab012-ecd2-4d9d-9ce1-25f1cb8554fc\" (UID: \"089ab012-ecd2-4d9d-9ce1-25f1cb8554fc\") "
Sep 14 05:10:05.199465 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:05.199409    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/089ab012-ecd2-4d9d-9ce1-25f1cb8554fc-model-cache\") pod \"089ab012-ecd2-4d9d-9ce1-25f1cb8554fc\" (UID: \"089ab012-ecd2-4d9d-9ce1-25f1cb8554fc\") "
Sep 14 05:10:05.199577 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:05.199492    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/089ab012-ecd2-4d9d-9ce1-25f1cb8554fc-tls-certs\") pod \"089ab012-ecd2-4d9d-9ce1-25f1cb8554fc\" (UID: \"089ab012-ecd2-4d9d-9ce1-25f1cb8554fc\") "
Sep 14 05:10:05.199577 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:05.199520    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/089ab012-ecd2-4d9d-9ce1-25f1cb8554fc-home\") pod \"089ab012-ecd2-4d9d-9ce1-25f1cb8554fc\" (UID: \"089ab012-ecd2-4d9d-9ce1-25f1cb8554fc\") "
Sep 14 05:10:05.199577 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:05.199548    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/089ab012-ecd2-4d9d-9ce1-25f1cb8554fc-tmp-dir\") pod \"089ab012-ecd2-4d9d-9ce1-25f1cb8554fc\" (UID: \"089ab012-ecd2-4d9d-9ce1-25f1cb8554fc\") "
Sep 14 05:10:05.199720 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:05.199701    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/089ab012-ecd2-4d9d-9ce1-25f1cb8554fc-model-cache" (OuterVolumeSpecName: "model-cache") pod "089ab012-ecd2-4d9d-9ce1-25f1cb8554fc" (UID: "089ab012-ecd2-4d9d-9ce1-25f1cb8554fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:10:05.199869 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:05.199824    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/089ab012-ecd2-4d9d-9ce1-25f1cb8554fc-home" (OuterVolumeSpecName: "home") pod "089ab012-ecd2-4d9d-9ce1-25f1cb8554fc" (UID: "089ab012-ecd2-4d9d-9ce1-25f1cb8554fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:10:05.199962 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:05.199941    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/089ab012-ecd2-4d9d-9ce1-25f1cb8554fc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "089ab012-ecd2-4d9d-9ce1-25f1cb8554fc" (UID: "089ab012-ecd2-4d9d-9ce1-25f1cb8554fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:10:05.200023 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:05.200006    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/089ab012-ecd2-4d9d-9ce1-25f1cb8554fc-model-cache\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:10:05.200083 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:05.200029    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/089ab012-ecd2-4d9d-9ce1-25f1cb8554fc-home\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:10:05.201636 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:05.201613    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/089ab012-ecd2-4d9d-9ce1-25f1cb8554fc-dshm" (OuterVolumeSpecName: "dshm") pod "089ab012-ecd2-4d9d-9ce1-25f1cb8554fc" (UID: "089ab012-ecd2-4d9d-9ce1-25f1cb8554fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:10:05.201954 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:05.201936    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/089ab012-ecd2-4d9d-9ce1-25f1cb8554fc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "089ab012-ecd2-4d9d-9ce1-25f1cb8554fc" (UID: "089ab012-ecd2-4d9d-9ce1-25f1cb8554fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 05:10:05.202030 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:05.201939    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/089ab012-ecd2-4d9d-9ce1-25f1cb8554fc-kube-api-access-sfg65" (OuterVolumeSpecName: "kube-api-access-sfg65") pod "089ab012-ecd2-4d9d-9ce1-25f1cb8554fc" (UID: "089ab012-ecd2-4d9d-9ce1-25f1cb8554fc"). InnerVolumeSpecName "kube-api-access-sfg65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:10:05.204173 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:05.204152    2565 scope.go:117] "RemoveContainer" containerID="a17e05025b86c6d928ac740db5988adaa24aaf23b388a24b600bec9430cbf66a"
Sep 14 05:10:05.204496 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:10:05.204478    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a17e05025b86c6d928ac740db5988adaa24aaf23b388a24b600bec9430cbf66a\": container with ID starting with a17e05025b86c6d928ac740db5988adaa24aaf23b388a24b600bec9430cbf66a not found: ID does not exist" containerID="a17e05025b86c6d928ac740db5988adaa24aaf23b388a24b600bec9430cbf66a"
Sep 14 05:10:05.204563 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:05.204502    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a17e05025b86c6d928ac740db5988adaa24aaf23b388a24b600bec9430cbf66a"} err="failed to get container status \"a17e05025b86c6d928ac740db5988adaa24aaf23b388a24b600bec9430cbf66a\": rpc error: code = NotFound desc = could not find container \"a17e05025b86c6d928ac740db5988adaa24aaf23b388a24b600bec9430cbf66a\": container with ID starting with a17e05025b86c6d928ac740db5988adaa24aaf23b388a24b600bec9430cbf66a not found: ID does not exist"
Sep 14 05:10:05.301159 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:05.301121    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/089ab012-ecd2-4d9d-9ce1-25f1cb8554fc-dshm\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:10:05.301159 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:05.301155    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfg65\" (UniqueName: \"kubernetes.io/projected/089ab012-ecd2-4d9d-9ce1-25f1cb8554fc-kube-api-access-sfg65\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:10:05.301159 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:05.301166    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/089ab012-ecd2-4d9d-9ce1-25f1cb8554fc-tls-certs\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:10:05.301400 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:05.301175    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/089ab012-ecd2-4d9d-9ce1-25f1cb8554fc-tmp-dir\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:10:05.515980 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:05.515941    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e08402/e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk"]
Sep 14 05:10:05.521008 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:05.520983    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-e08402/e2e-conflict-b-c6882fdf-kserve-6b68685758-dn4vk"]
Sep 14 05:10:06.334619 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:06.334588    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="089ab012-ecd2-4d9d-9ce1-25f1cb8554fc" path="/var/lib/kubelet/pods/089ab012-ecd2-4d9d-9ce1-25f1cb8554fc/volumes"
Sep 14 05:10:06.335903 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:06.335882    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e7fdd6-lxqgm"
Sep 14 05:10:06.414488 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:06.414453    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44df6175-a921-422a-99dd-52d62a2205be-maas-api-service-ca\") pod \"44df6175-a921-422a-99dd-52d62a2205be\" (UID: \"44df6175-a921-422a-99dd-52d62a2205be\") "
Sep 14 05:10:06.414835 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:06.414813    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44df6175-a921-422a-99dd-52d62a2205be-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "44df6175-a921-422a-99dd-52d62a2205be" (UID: "44df6175-a921-422a-99dd-52d62a2205be"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 05:10:06.516087 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:06.516000    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44df6175-a921-422a-99dd-52d62a2205be-maas-api-service-ca\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:10:07.208211 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:07.208176    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e7fdd6-lxqgm"
Sep 14 05:10:07.208211 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:07.208192    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e7fdd6-lxqgm" event={"ID":"44df6175-a921-422a-99dd-52d62a2205be","Type":"ContainerDied","Data":"26477d9b35f8328d5daadc829f2952c275fa0ba25591ab83c89f7deef9bb5468"}
Sep 14 05:10:07.208429 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:07.208243    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26477d9b35f8328d5daadc829f2952c275fa0ba25591ab83c89f7deef9bb5468"
Sep 14 05:10:13.629935 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:13.629895    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e7fdd6-56c7655575-4ltrp"]
Sep 14 05:10:13.630311 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:13.630207    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-e7fdd6-56c7655575-4ltrp" podUID="170ea597-3f51-42e8-8590-497cd6f7761b" containerName="payload-processing" containerID="cri-o://749d63e628ef63857628fc56081150138140d3f03dca308b3b7b64cf95f72c70" gracePeriod=30
Sep 14 05:10:13.788944 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:13.788911    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e7fdd6-lxqgm"]
Sep 14 05:10:13.792350 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:13.792328    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e7fdd6-lxqgm"]
Sep 14 05:10:13.887022 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:13.886964    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e7fdd6-56c7655575-4ltrp"
Sep 14 05:10:13.988006 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:13.987973    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/170ea597-3f51-42e8-8590-497cd6f7761b-plugins-config-volume\") pod \"170ea597-3f51-42e8-8590-497cd6f7761b\" (UID: \"170ea597-3f51-42e8-8590-497cd6f7761b\") "
Sep 14 05:10:13.988185 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:13.988032    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nv2t\" (UniqueName: \"kubernetes.io/projected/170ea597-3f51-42e8-8590-497cd6f7761b-kube-api-access-7nv2t\") pod \"170ea597-3f51-42e8-8590-497cd6f7761b\" (UID: \"170ea597-3f51-42e8-8590-497cd6f7761b\") "
Sep 14 05:10:13.988343 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:13.988318    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/170ea597-3f51-42e8-8590-497cd6f7761b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "170ea597-3f51-42e8-8590-497cd6f7761b" (UID: "170ea597-3f51-42e8-8590-497cd6f7761b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 05:10:13.990087 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:13.990061    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/170ea597-3f51-42e8-8590-497cd6f7761b-kube-api-access-7nv2t" (OuterVolumeSpecName: "kube-api-access-7nv2t") pod "170ea597-3f51-42e8-8590-497cd6f7761b" (UID: "170ea597-3f51-42e8-8590-497cd6f7761b"). InnerVolumeSpecName "kube-api-access-7nv2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:10:14.089392 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:14.089361    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/170ea597-3f51-42e8-8590-497cd6f7761b-plugins-config-volume\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:10:14.089392 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:14.089387    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nv2t\" (UniqueName: \"kubernetes.io/projected/170ea597-3f51-42e8-8590-497cd6f7761b-kube-api-access-7nv2t\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:10:14.248356 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:14.248262    2565 generic.go:356] "Generic (PLEG): container finished" podID="170ea597-3f51-42e8-8590-497cd6f7761b" containerID="749d63e628ef63857628fc56081150138140d3f03dca308b3b7b64cf95f72c70" exitCode=0
Sep 14 05:10:14.248356 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:14.248318    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e7fdd6-56c7655575-4ltrp"
Sep 14 05:10:14.248516 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:14.248350    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e7fdd6-56c7655575-4ltrp" event={"ID":"170ea597-3f51-42e8-8590-497cd6f7761b","Type":"ContainerDied","Data":"749d63e628ef63857628fc56081150138140d3f03dca308b3b7b64cf95f72c70"}
Sep 14 05:10:14.248516 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:14.248405    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e7fdd6-56c7655575-4ltrp" event={"ID":"170ea597-3f51-42e8-8590-497cd6f7761b","Type":"ContainerDied","Data":"e59a93c4de827d5f7b71134b9720509d6018764542f12925f3284b1257b54e5a"}
Sep 14 05:10:14.248516 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:14.248430    2565 scope.go:117] "RemoveContainer" containerID="749d63e628ef63857628fc56081150138140d3f03dca308b3b7b64cf95f72c70"
Sep 14 05:10:14.257995 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:14.257979    2565 scope.go:117] "RemoveContainer" containerID="749d63e628ef63857628fc56081150138140d3f03dca308b3b7b64cf95f72c70"
Sep 14 05:10:14.258243 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:10:14.258227    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"749d63e628ef63857628fc56081150138140d3f03dca308b3b7b64cf95f72c70\": container with ID starting with 749d63e628ef63857628fc56081150138140d3f03dca308b3b7b64cf95f72c70 not found: ID does not exist" containerID="749d63e628ef63857628fc56081150138140d3f03dca308b3b7b64cf95f72c70"
Sep 14 05:10:14.258317 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:14.258248    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"749d63e628ef63857628fc56081150138140d3f03dca308b3b7b64cf95f72c70"} err="failed to get container status \"749d63e628ef63857628fc56081150138140d3f03dca308b3b7b64cf95f72c70\": rpc error: code = NotFound desc = could not find container \"749d63e628ef63857628fc56081150138140d3f03dca308b3b7b64cf95f72c70\": container with ID starting with 749d63e628ef63857628fc56081150138140d3f03dca308b3b7b64cf95f72c70 not found: ID does not exist"
Sep 14 05:10:14.268945 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:14.268919    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e7fdd6-56c7655575-4ltrp"]
Sep 14 05:10:14.271036 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:14.271013    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e7fdd6-56c7655575-4ltrp"]
Sep 14 05:10:14.335589 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:14.335547    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="170ea597-3f51-42e8-8590-497cd6f7761b" path="/var/lib/kubelet/pods/170ea597-3f51-42e8-8590-497cd6f7761b/volumes"
Sep 14 05:10:14.336016 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:14.335996    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44df6175-a921-422a-99dd-52d62a2205be" path="/var/lib/kubelet/pods/44df6175-a921-422a-99dd-52d62a2205be/volumes"
Sep 14 05:10:17.627208 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:17.627175    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-e08402/distinct-w4-e08402-kserve-68d4c58cb4-wqjff"]
Sep 14 05:10:17.627556 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:17.627489    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="728f28c2-15b8-4d33-9ed1-a99c2b02cd59" containerName="main"
Sep 14 05:10:17.627556 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:17.627499    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="728f28c2-15b8-4d33-9ed1-a99c2b02cd59" containerName="main"
Sep 14 05:10:17.627556 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:17.627513    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44df6175-a921-422a-99dd-52d62a2205be" containerName="revoke-keys"
Sep 14 05:10:17.627556 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:17.627519    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="44df6175-a921-422a-99dd-52d62a2205be" containerName="revoke-keys"
Sep 14 05:10:17.627556 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:17.627527    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="089ab012-ecd2-4d9d-9ce1-25f1cb8554fc" containerName="main"
Sep 14 05:10:17.627556 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:17.627532    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="089ab012-ecd2-4d9d-9ce1-25f1cb8554fc" containerName="main"
Sep 14 05:10:17.627556 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:17.627547    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81dac50a-cd9d-4762-907b-878715836a9c" containerName="main"
Sep 14 05:10:17.627556 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:17.627552    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="81dac50a-cd9d-4762-907b-878715836a9c" containerName="main"
Sep 14 05:10:17.627556 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:17.627559    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="170ea597-3f51-42e8-8590-497cd6f7761b" containerName="payload-processing"
Sep 14 05:10:17.627888 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:17.627565    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="170ea597-3f51-42e8-8590-497cd6f7761b" containerName="payload-processing"
Sep 14 05:10:17.627888 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:17.627612    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="170ea597-3f51-42e8-8590-497cd6f7761b" containerName="payload-processing"
Sep 14 05:10:17.627888 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:17.627620    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="089ab012-ecd2-4d9d-9ce1-25f1cb8554fc" containerName="main"
Sep 14 05:10:17.627888 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:17.627626    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="728f28c2-15b8-4d33-9ed1-a99c2b02cd59" containerName="main"
Sep 14 05:10:17.627888 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:17.627631    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="44df6175-a921-422a-99dd-52d62a2205be" containerName="revoke-keys"
Sep 14 05:10:17.627888 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:17.627640    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="81dac50a-cd9d-4762-907b-878715836a9c" containerName="main"
Sep 14 05:10:17.630850 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:17.630834    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e08402/distinct-w4-e08402-kserve-68d4c58cb4-wqjff"
Sep 14 05:10:17.633008 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:17.632989    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e08402\"/\"distinct-w4-e08402-kserve-self-signed-certs\""
Sep 14 05:10:17.638715 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:17.638693    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e08402/distinct-w4-e08402-kserve-68d4c58cb4-wqjff"]
Sep 14 05:10:17.823954 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:17.823915    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbbkh\" (UniqueName: \"kubernetes.io/projected/c3d361b8-037d-4d5d-951c-f954c7d6a31c-kube-api-access-vbbkh\") pod \"distinct-w4-e08402-kserve-68d4c58cb4-wqjff\" (UID: \"c3d361b8-037d-4d5d-951c-f954c7d6a31c\") " pod="e2e-models-e2e-worker-w4-e08402/distinct-w4-e08402-kserve-68d4c58cb4-wqjff"
Sep 14 05:10:17.823954 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:17.823965    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3d361b8-037d-4d5d-951c-f954c7d6a31c-tmp-dir\") pod \"distinct-w4-e08402-kserve-68d4c58cb4-wqjff\" (UID: \"c3d361b8-037d-4d5d-951c-f954c7d6a31c\") " pod="e2e-models-e2e-worker-w4-e08402/distinct-w4-e08402-kserve-68d4c58cb4-wqjff"
Sep 14 05:10:17.824173 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:17.824103    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3d361b8-037d-4d5d-951c-f954c7d6a31c-model-cache\") pod \"distinct-w4-e08402-kserve-68d4c58cb4-wqjff\" (UID: \"c3d361b8-037d-4d5d-951c-f954c7d6a31c\") " pod="e2e-models-e2e-worker-w4-e08402/distinct-w4-e08402-kserve-68d4c58cb4-wqjff"
Sep 14 05:10:17.824173 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:17.824130    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3d361b8-037d-4d5d-951c-f954c7d6a31c-tls-certs\") pod \"distinct-w4-e08402-kserve-68d4c58cb4-wqjff\" (UID: \"c3d361b8-037d-4d5d-951c-f954c7d6a31c\") " pod="e2e-models-e2e-worker-w4-e08402/distinct-w4-e08402-kserve-68d4c58cb4-wqjff"
Sep 14 05:10:17.824173 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:17.824159    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3d361b8-037d-4d5d-951c-f954c7d6a31c-home\") pod \"distinct-w4-e08402-kserve-68d4c58cb4-wqjff\" (UID: \"c3d361b8-037d-4d5d-951c-f954c7d6a31c\") " pod="e2e-models-e2e-worker-w4-e08402/distinct-w4-e08402-kserve-68d4c58cb4-wqjff"
Sep 14 05:10:17.824270 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:17.824220    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3d361b8-037d-4d5d-951c-f954c7d6a31c-dshm\") pod \"distinct-w4-e08402-kserve-68d4c58cb4-wqjff\" (UID: \"c3d361b8-037d-4d5d-951c-f954c7d6a31c\") " pod="e2e-models-e2e-worker-w4-e08402/distinct-w4-e08402-kserve-68d4c58cb4-wqjff"
Sep 14 05:10:17.925485 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:17.925398    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbbkh\" (UniqueName: \"kubernetes.io/projected/c3d361b8-037d-4d5d-951c-f954c7d6a31c-kube-api-access-vbbkh\") pod \"distinct-w4-e08402-kserve-68d4c58cb4-wqjff\" (UID: \"c3d361b8-037d-4d5d-951c-f954c7d6a31c\") " pod="e2e-models-e2e-worker-w4-e08402/distinct-w4-e08402-kserve-68d4c58cb4-wqjff"
Sep 14 05:10:17.925485 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:17.925431    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3d361b8-037d-4d5d-951c-f954c7d6a31c-tmp-dir\") pod \"distinct-w4-e08402-kserve-68d4c58cb4-wqjff\" (UID: \"c3d361b8-037d-4d5d-951c-f954c7d6a31c\") " pod="e2e-models-e2e-worker-w4-e08402/distinct-w4-e08402-kserve-68d4c58cb4-wqjff"
Sep 14 05:10:17.925485 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:17.925468    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3d361b8-037d-4d5d-951c-f954c7d6a31c-model-cache\") pod \"distinct-w4-e08402-kserve-68d4c58cb4-wqjff\" (UID: \"c3d361b8-037d-4d5d-951c-f954c7d6a31c\") " pod="e2e-models-e2e-worker-w4-e08402/distinct-w4-e08402-kserve-68d4c58cb4-wqjff"
Sep 14 05:10:17.925727 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:17.925487    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3d361b8-037d-4d5d-951c-f954c7d6a31c-tls-certs\") pod \"distinct-w4-e08402-kserve-68d4c58cb4-wqjff\" (UID: \"c3d361b8-037d-4d5d-951c-f954c7d6a31c\") " pod="e2e-models-e2e-worker-w4-e08402/distinct-w4-e08402-kserve-68d4c58cb4-wqjff"
Sep 14 05:10:17.925727 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:17.925522    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3d361b8-037d-4d5d-951c-f954c7d6a31c-home\") pod \"distinct-w4-e08402-kserve-68d4c58cb4-wqjff\" (UID: \"c3d361b8-037d-4d5d-951c-f954c7d6a31c\") " pod="e2e-models-e2e-worker-w4-e08402/distinct-w4-e08402-kserve-68d4c58cb4-wqjff"
Sep 14 05:10:17.925727 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:17.925556    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3d361b8-037d-4d5d-951c-f954c7d6a31c-dshm\") pod \"distinct-w4-e08402-kserve-68d4c58cb4-wqjff\" (UID: \"c3d361b8-037d-4d5d-951c-f954c7d6a31c\") " pod="e2e-models-e2e-worker-w4-e08402/distinct-w4-e08402-kserve-68d4c58cb4-wqjff"
Sep 14 05:10:17.925916 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:17.925890    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3d361b8-037d-4d5d-951c-f954c7d6a31c-tmp-dir\") pod \"distinct-w4-e08402-kserve-68d4c58cb4-wqjff\" (UID: \"c3d361b8-037d-4d5d-951c-f954c7d6a31c\") " pod="e2e-models-e2e-worker-w4-e08402/distinct-w4-e08402-kserve-68d4c58cb4-wqjff"
Sep 14 05:10:17.925978 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:17.925910    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3d361b8-037d-4d5d-951c-f954c7d6a31c-model-cache\") pod \"distinct-w4-e08402-kserve-68d4c58cb4-wqjff\" (UID: \"c3d361b8-037d-4d5d-951c-f954c7d6a31c\") " pod="e2e-models-e2e-worker-w4-e08402/distinct-w4-e08402-kserve-68d4c58cb4-wqjff"
Sep 14 05:10:17.925978 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:17.925958    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3d361b8-037d-4d5d-951c-f954c7d6a31c-home\") pod \"distinct-w4-e08402-kserve-68d4c58cb4-wqjff\" (UID: \"c3d361b8-037d-4d5d-951c-f954c7d6a31c\") " pod="e2e-models-e2e-worker-w4-e08402/distinct-w4-e08402-kserve-68d4c58cb4-wqjff"
Sep 14 05:10:17.927952 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:17.927925    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3d361b8-037d-4d5d-951c-f954c7d6a31c-dshm\") pod \"distinct-w4-e08402-kserve-68d4c58cb4-wqjff\" (UID: \"c3d361b8-037d-4d5d-951c-f954c7d6a31c\") " pod="e2e-models-e2e-worker-w4-e08402/distinct-w4-e08402-kserve-68d4c58cb4-wqjff"
Sep 14 05:10:17.928089 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:17.928070    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3d361b8-037d-4d5d-951c-f954c7d6a31c-tls-certs\") pod \"distinct-w4-e08402-kserve-68d4c58cb4-wqjff\" (UID: \"c3d361b8-037d-4d5d-951c-f954c7d6a31c\") " pod="e2e-models-e2e-worker-w4-e08402/distinct-w4-e08402-kserve-68d4c58cb4-wqjff"
Sep 14 05:10:17.932418 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:17.932399    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbbkh\" (UniqueName: \"kubernetes.io/projected/c3d361b8-037d-4d5d-951c-f954c7d6a31c-kube-api-access-vbbkh\") pod \"distinct-w4-e08402-kserve-68d4c58cb4-wqjff\" (UID: \"c3d361b8-037d-4d5d-951c-f954c7d6a31c\") " pod="e2e-models-e2e-worker-w4-e08402/distinct-w4-e08402-kserve-68d4c58cb4-wqjff"
Sep 14 05:10:17.940299 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:17.940272    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e08402/distinct-w4-e08402-kserve-68d4c58cb4-wqjff"
Sep 14 05:10:18.212772 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:18.212728    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e08402/distinct-w4-e08402-kserve-68d4c58cb4-wqjff"]
Sep 14 05:10:18.214429 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:10:18.214400    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3d361b8_037d_4d5d_951c_f954c7d6a31c.slice/crio-95c0b25ca59b954d668b8b0be820fb4e60e11c8e7de52ada0bb6bfdd7f71d251 WatchSource:0}: Error finding container 95c0b25ca59b954d668b8b0be820fb4e60e11c8e7de52ada0bb6bfdd7f71d251: Status 404 returned error can't find the container with id 95c0b25ca59b954d668b8b0be820fb4e60e11c8e7de52ada0bb6bfdd7f71d251
Sep 14 05:10:18.278703 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:18.278674    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e08402/distinct-w4-e08402-kserve-68d4c58cb4-wqjff" event={"ID":"c3d361b8-037d-4d5d-951c-f954c7d6a31c","Type":"ContainerStarted","Data":"95c0b25ca59b954d668b8b0be820fb4e60e11c8e7de52ada0bb6bfdd7f71d251"}
Sep 14 05:10:19.285331 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:19.285293    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e08402/distinct-w4-e08402-kserve-68d4c58cb4-wqjff" event={"ID":"c3d361b8-037d-4d5d-951c-f954c7d6a31c","Type":"ContainerStarted","Data":"4418ba2cb28b9aa07603aede7e0f2a658f9c68af3712575bcc74ed422ae6cd33"}
Sep 14 05:10:19.301892 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:19.301843    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-e08402/distinct-w4-e08402-kserve-68d4c58cb4-wqjff" podStartSLOduration=2.301827673 podStartE2EDuration="2.301827673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:10:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:10:19.300871935 +0000 UTC m=+1417.476643496" watchObservedRunningTime="2026-09-14 05:10:19.301827673 +0000 UTC m=+1417.477599230"
Sep 14 05:10:21.601585 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:21.601550    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"]
Sep 14 05:10:21.605090 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:21.605073    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"
Sep 14 05:10:21.607438 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:21.607418    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-2ff4a6-openshift-default-dockercfg-4ffcd\""
Sep 14 05:10:21.617966 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:21.617942    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"]
Sep 14 05:10:21.653190 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:21.653161    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-istio-podinfo\") pod \"e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5\" (UID: \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\") " pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"
Sep 14 05:10:21.653301 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:21.653212    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-credential-socket\") pod \"e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5\" (UID: \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\") " pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"
Sep 14 05:10:21.653301 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:21.653265    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-workload-socket\") pod \"e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5\" (UID: \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\") " pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"
Sep 14 05:10:21.653301 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:21.653290    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-istiod-ca-cert\") pod \"e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5\" (UID: \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\") " pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"
Sep 14 05:10:21.653413 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:21.653309    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-istio-envoy\") pod \"e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5\" (UID: \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\") " pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"
Sep 14 05:10:21.653413 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:21.653349    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwkk7\" (UniqueName: \"kubernetes.io/projected/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-kube-api-access-qwkk7\") pod \"e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5\" (UID: \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\") " pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"
Sep 14 05:10:21.653413 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:21.653387    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-workload-certs\") pod \"e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5\" (UID: \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\") " pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"
Sep 14 05:10:21.653537 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:21.653416    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-istio-data\") pod \"e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5\" (UID: \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\") " pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"
Sep 14 05:10:21.653537 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:21.653431    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-istio-token\") pod \"e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5\" (UID: \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\") " pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"
Sep 14 05:10:21.754197 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:21.754164    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-workload-certs\") pod \"e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5\" (UID: \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\") " pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"
Sep 14 05:10:21.754376 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:21.754204    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-istio-data\") pod \"e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5\" (UID: \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\") " pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"
Sep 14 05:10:21.754376 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:21.754226    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-istio-token\") pod \"e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5\" (UID: \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\") " pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"
Sep 14 05:10:21.754376 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:21.754302    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-istio-podinfo\") pod \"e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5\" (UID: \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\") " pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"
Sep 14 05:10:21.754376 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:21.754352    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-credential-socket\") pod \"e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5\" (UID: \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\") " pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"
Sep 14 05:10:21.754590 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:21.754390    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-workload-socket\") pod \"e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5\" (UID: \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\") " pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"
Sep 14 05:10:21.754590 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:21.754413    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-istiod-ca-cert\") pod \"e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5\" (UID: \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\") " pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"
Sep 14 05:10:21.754590 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:21.754441    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-istio-envoy\") pod \"e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5\" (UID: \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\") " pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"
Sep 14 05:10:21.754590 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:21.754485    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwkk7\" (UniqueName: \"kubernetes.io/projected/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-kube-api-access-qwkk7\") pod \"e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5\" (UID: \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\") " pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"
Sep 14 05:10:21.754813 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:21.754588    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-workload-certs\") pod \"e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5\" (UID: \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\") " pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"
Sep 14 05:10:21.754983 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:21.754960    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-workload-socket\") pod \"e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5\" (UID: \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\") " pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"
Sep 14 05:10:21.755047 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:21.754971    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-istio-data\") pod \"e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5\" (UID: \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\") " pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"
Sep 14 05:10:21.755047 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:21.755020    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-credential-socket\") pod \"e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5\" (UID: \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\") " pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"
Sep 14 05:10:21.755261 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:21.755241    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-istiod-ca-cert\") pod \"e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5\" (UID: \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\") " pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"
Sep 14 05:10:21.756769 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:21.756734    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-istio-envoy\") pod \"e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5\" (UID: \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\") " pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"
Sep 14 05:10:21.756876 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:21.756859    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-istio-podinfo\") pod \"e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5\" (UID: \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\") " pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"
Sep 14 05:10:21.761573 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:21.761543    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-istio-token\") pod \"e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5\" (UID: \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\") " pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"
Sep 14 05:10:21.761647 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:21.761635    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwkk7\" (UniqueName: \"kubernetes.io/projected/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-kube-api-access-qwkk7\") pod \"e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5\" (UID: \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\") " pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"
Sep 14 05:10:21.915844 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:21.915739    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"
Sep 14 05:10:22.194478 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:22.194386    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"]
Sep 14 05:10:22.196404 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:10:22.196363    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a5a2e92_8a45_4f45_a89a_afd3d81f0699.slice/crio-e9acbefbf5130751fffd73e254777ac364b12f65facb329597cdbfd276bd6a74 WatchSource:0}: Error finding container e9acbefbf5130751fffd73e254777ac364b12f65facb329597cdbfd276bd6a74: Status 404 returned error can't find the container with id e9acbefbf5130751fffd73e254777ac364b12f65facb329597cdbfd276bd6a74
Sep 14 05:10:22.198791 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:22.198742    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:10:22.198870 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:22.198827    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:10:22.198870 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:22.198854    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:10:22.305023 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:22.304991    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5" event={"ID":"6a5a2e92-8a45-4f45-a89a-afd3d81f0699","Type":"ContainerStarted","Data":"69fa9bfb9ec720356935fcaae89bf89fbd08dff90b0ba33ab7aa670e25fe898d"}
Sep 14 05:10:22.305223 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:22.305040    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5" event={"ID":"6a5a2e92-8a45-4f45-a89a-afd3d81f0699","Type":"ContainerStarted","Data":"e9acbefbf5130751fffd73e254777ac364b12f65facb329597cdbfd276bd6a74"}
Sep 14 05:10:22.326188 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:22.326132    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5" podStartSLOduration=1.326111399 podStartE2EDuration="1.326111399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:10:22.323728514 +0000 UTC m=+1420.499500083" watchObservedRunningTime="2026-09-14 05:10:22.326111399 +0000 UTC m=+1420.501882961"
Sep 14 05:10:22.632544 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:22.632509    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:10:22.916242 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:22.916160    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"
Sep 14 05:10:23.921944 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:23.921914    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"
Sep 14 05:10:24.316313 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:24.316219    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"
Sep 14 05:10:24.317130 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:24.317112    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"
Sep 14 05:10:27.940412 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:27.940377    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e08402/distinct-w4-e08402-kserve-68d4c58cb4-wqjff"
Sep 14 05:10:27.940841 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:27.940425    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-e08402/distinct-w4-e08402-kserve-68d4c58cb4-wqjff"
Sep 14 05:10:27.954047 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:27.954017    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-e08402/distinct-w4-e08402-kserve-68d4c58cb4-wqjff"
Sep 14 05:10:28.349515 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:28.349488    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e08402/distinct-w4-e08402-kserve-68d4c58cb4-wqjff"
Sep 14 05:10:29.537084 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:29.537047    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7c6a48c-qspn4"]
Sep 14 05:10:29.540792 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:29.540775    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7c6a48c-qspn4"
Sep 14 05:10:29.543038 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:29.543019    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 05:10:29.543104 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:29.543033    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k6hpm\""
Sep 14 05:10:29.547781 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:29.547744    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7c6a48c-qspn4"]
Sep 14 05:10:29.627276 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:29.627224    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/513dafb2-e0c9-4ae9-93c3-35e3c0793043-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c7c6a48c-qspn4\" (UID: \"513dafb2-e0c9-4ae9-93c3-35e3c0793043\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7c6a48c-qspn4"
Sep 14 05:10:29.727906 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:29.727875    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/513dafb2-e0c9-4ae9-93c3-35e3c0793043-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c7c6a48c-qspn4\" (UID: \"513dafb2-e0c9-4ae9-93c3-35e3c0793043\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7c6a48c-qspn4"
Sep 14 05:10:29.728486 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:29.728469    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/513dafb2-e0c9-4ae9-93c3-35e3c0793043-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c7c6a48c-qspn4\" (UID: \"513dafb2-e0c9-4ae9-93c3-35e3c0793043\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7c6a48c-qspn4"
Sep 14 05:10:29.852187 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:29.852154    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7c6a48c-qspn4"
Sep 14 05:10:30.122228 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:30.122203    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7c6a48c-qspn4"]
Sep 14 05:10:30.124049 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:10:30.124023    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod513dafb2_e0c9_4ae9_93c3_35e3c0793043.slice/crio-cd0b6e652d4ac76052ecdaaed63125a8c49254074c2077a0a7c2f49e72038985 WatchSource:0}: Error finding container cd0b6e652d4ac76052ecdaaed63125a8c49254074c2077a0a7c2f49e72038985: Status 404 returned error can't find the container with id cd0b6e652d4ac76052ecdaaed63125a8c49254074c2077a0a7c2f49e72038985
Sep 14 05:10:30.351181 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:30.351148    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7c6a48c-qspn4" event={"ID":"513dafb2-e0c9-4ae9-93c3-35e3c0793043","Type":"ContainerStarted","Data":"cd0b6e652d4ac76052ecdaaed63125a8c49254074c2077a0a7c2f49e72038985"}
Sep 14 05:10:31.360986 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:31.360948    2565 generic.go:356] "Generic (PLEG): container finished" podID="513dafb2-e0c9-4ae9-93c3-35e3c0793043" containerID="86f5423f15a599665da405455b5467b700d9975ba8235b174f0f1dc896f04e0d" exitCode=0
Sep 14 05:10:31.361402 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:31.360998    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7c6a48c-qspn4" event={"ID":"513dafb2-e0c9-4ae9-93c3-35e3c0793043","Type":"ContainerDied","Data":"86f5423f15a599665da405455b5467b700d9975ba8235b174f0f1dc896f04e0d"}
Sep 14 05:10:32.498264 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:32.498238    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7c6a48c-qspn4"
Sep 14 05:10:32.554213 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:32.554182    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/513dafb2-e0c9-4ae9-93c3-35e3c0793043-maas-api-service-ca\") pod \"513dafb2-e0c9-4ae9-93c3-35e3c0793043\" (UID: \"513dafb2-e0c9-4ae9-93c3-35e3c0793043\") "
Sep 14 05:10:32.554530 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:32.554508    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/513dafb2-e0c9-4ae9-93c3-35e3c0793043-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "513dafb2-e0c9-4ae9-93c3-35e3c0793043" (UID: "513dafb2-e0c9-4ae9-93c3-35e3c0793043"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 05:10:32.654980 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:32.654891    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/513dafb2-e0c9-4ae9-93c3-35e3c0793043-maas-api-service-ca\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:10:33.372664 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:33.372633    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7c6a48c-qspn4"
Sep 14 05:10:33.372664 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:33.372642    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7c6a48c-qspn4" event={"ID":"513dafb2-e0c9-4ae9-93c3-35e3c0793043","Type":"ContainerDied","Data":"cd0b6e652d4ac76052ecdaaed63125a8c49254074c2077a0a7c2f49e72038985"}
Sep 14 05:10:33.372910 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:33.372682    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd0b6e652d4ac76052ecdaaed63125a8c49254074c2077a0a7c2f49e72038985"
Sep 14 05:10:45.267191 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:45.267158    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c7c6a48c-7f9cf4474b-qrmzh"]
Sep 14 05:10:45.267667 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:45.267430    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7c6a48c-7f9cf4474b-qrmzh" podUID="7818bda4-c5be-42e4-a4cf-ca3ef45cf295" containerName="payload-pre-processing" containerID="cri-o://c3b0c7b3267fb8197b2864747539cbba179db28a29821dec87e7cc4d9a7fdb64" gracePeriod=30
Sep 14 05:10:45.414876 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:45.414833    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-e08402/distinct-2-w4-e08402-kserve-78545b6fcf-8w74w"]
Sep 14 05:10:45.415270 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:45.415252    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="513dafb2-e0c9-4ae9-93c3-35e3c0793043" containerName="revoke-keys"
Sep 14 05:10:45.415355 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:45.415273    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="513dafb2-e0c9-4ae9-93c3-35e3c0793043" containerName="revoke-keys"
Sep 14 05:10:45.415413 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:45.415360    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="513dafb2-e0c9-4ae9-93c3-35e3c0793043" containerName="revoke-keys"
Sep 14 05:10:45.418518 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:45.418498    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e08402/distinct-2-w4-e08402-kserve-78545b6fcf-8w74w"
Sep 14 05:10:45.420820 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:45.420798    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e08402\"/\"distinct-2-w4-e08402-kserve-self-signed-certs\""
Sep 14 05:10:45.429600 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:45.429578    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e08402/distinct-2-w4-e08402-kserve-78545b6fcf-8w74w"]
Sep 14 05:10:45.440139 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:45.440108    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7c6a48c-qspn4"]
Sep 14 05:10:45.448524 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:45.448497    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7c6a48c-qspn4"]
Sep 14 05:10:45.448781 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:45.448735    2565 generic.go:356] "Generic (PLEG): container finished" podID="7818bda4-c5be-42e4-a4cf-ca3ef45cf295" containerID="c3b0c7b3267fb8197b2864747539cbba179db28a29821dec87e7cc4d9a7fdb64" exitCode=0
Sep 14 05:10:45.448871 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:45.448809    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7c6a48c-7f9cf4474b-qrmzh" event={"ID":"7818bda4-c5be-42e4-a4cf-ca3ef45cf295","Type":"ContainerDied","Data":"c3b0c7b3267fb8197b2864747539cbba179db28a29821dec87e7cc4d9a7fdb64"}
Sep 14 05:10:45.481253 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:45.481224    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ea11a2f-42c5-4e7f-988e-b980d3803291-dshm\") pod \"distinct-2-w4-e08402-kserve-78545b6fcf-8w74w\" (UID: \"5ea11a2f-42c5-4e7f-988e-b980d3803291\") " pod="e2e-models-e2e-worker-w4-e08402/distinct-2-w4-e08402-kserve-78545b6fcf-8w74w"
Sep 14 05:10:45.481383 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:45.481273    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ea11a2f-42c5-4e7f-988e-b980d3803291-model-cache\") pod \"distinct-2-w4-e08402-kserve-78545b6fcf-8w74w\" (UID: \"5ea11a2f-42c5-4e7f-988e-b980d3803291\") " pod="e2e-models-e2e-worker-w4-e08402/distinct-2-w4-e08402-kserve-78545b6fcf-8w74w"
Sep 14 05:10:45.481383 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:45.481357    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ea11a2f-42c5-4e7f-988e-b980d3803291-tmp-dir\") pod \"distinct-2-w4-e08402-kserve-78545b6fcf-8w74w\" (UID: \"5ea11a2f-42c5-4e7f-988e-b980d3803291\") " pod="e2e-models-e2e-worker-w4-e08402/distinct-2-w4-e08402-kserve-78545b6fcf-8w74w"
Sep 14 05:10:45.481466 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:45.481405    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ea11a2f-42c5-4e7f-988e-b980d3803291-tls-certs\") pod \"distinct-2-w4-e08402-kserve-78545b6fcf-8w74w\" (UID: \"5ea11a2f-42c5-4e7f-988e-b980d3803291\") " pod="e2e-models-e2e-worker-w4-e08402/distinct-2-w4-e08402-kserve-78545b6fcf-8w74w"
Sep 14 05:10:45.481466 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:45.481445    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ea11a2f-42c5-4e7f-988e-b980d3803291-home\") pod \"distinct-2-w4-e08402-kserve-78545b6fcf-8w74w\" (UID: \"5ea11a2f-42c5-4e7f-988e-b980d3803291\") " pod="e2e-models-e2e-worker-w4-e08402/distinct-2-w4-e08402-kserve-78545b6fcf-8w74w"
Sep 14 05:10:45.481544 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:45.481495    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5h2g\" (UniqueName: \"kubernetes.io/projected/5ea11a2f-42c5-4e7f-988e-b980d3803291-kube-api-access-z5h2g\") pod \"distinct-2-w4-e08402-kserve-78545b6fcf-8w74w\" (UID: \"5ea11a2f-42c5-4e7f-988e-b980d3803291\") " pod="e2e-models-e2e-worker-w4-e08402/distinct-2-w4-e08402-kserve-78545b6fcf-8w74w"
Sep 14 05:10:45.536061 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:45.536022    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7c6a48c-7f9cf4474b-qrmzh"
Sep 14 05:10:45.582117 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:45.582085    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ea11a2f-42c5-4e7f-988e-b980d3803291-home\") pod \"distinct-2-w4-e08402-kserve-78545b6fcf-8w74w\" (UID: \"5ea11a2f-42c5-4e7f-988e-b980d3803291\") " pod="e2e-models-e2e-worker-w4-e08402/distinct-2-w4-e08402-kserve-78545b6fcf-8w74w"
Sep 14 05:10:45.582416 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:45.582126    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5h2g\" (UniqueName: \"kubernetes.io/projected/5ea11a2f-42c5-4e7f-988e-b980d3803291-kube-api-access-z5h2g\") pod \"distinct-2-w4-e08402-kserve-78545b6fcf-8w74w\" (UID: \"5ea11a2f-42c5-4e7f-988e-b980d3803291\") " pod="e2e-models-e2e-worker-w4-e08402/distinct-2-w4-e08402-kserve-78545b6fcf-8w74w"
Sep 14 05:10:45.582416 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:45.582175    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ea11a2f-42c5-4e7f-988e-b980d3803291-dshm\") pod \"distinct-2-w4-e08402-kserve-78545b6fcf-8w74w\" (UID: \"5ea11a2f-42c5-4e7f-988e-b980d3803291\") " pod="e2e-models-e2e-worker-w4-e08402/distinct-2-w4-e08402-kserve-78545b6fcf-8w74w"
Sep 14 05:10:45.582416 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:45.582202    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ea11a2f-42c5-4e7f-988e-b980d3803291-model-cache\") pod \"distinct-2-w4-e08402-kserve-78545b6fcf-8w74w\" (UID: \"5ea11a2f-42c5-4e7f-988e-b980d3803291\") " pod="e2e-models-e2e-worker-w4-e08402/distinct-2-w4-e08402-kserve-78545b6fcf-8w74w"
Sep 14 05:10:45.582416 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:45.582230    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ea11a2f-42c5-4e7f-988e-b980d3803291-tmp-dir\") pod \"distinct-2-w4-e08402-kserve-78545b6fcf-8w74w\" (UID: \"5ea11a2f-42c5-4e7f-988e-b980d3803291\") " pod="e2e-models-e2e-worker-w4-e08402/distinct-2-w4-e08402-kserve-78545b6fcf-8w74w"
Sep 14 05:10:45.582416 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:45.582272    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ea11a2f-42c5-4e7f-988e-b980d3803291-tls-certs\") pod \"distinct-2-w4-e08402-kserve-78545b6fcf-8w74w\" (UID: \"5ea11a2f-42c5-4e7f-988e-b980d3803291\") " pod="e2e-models-e2e-worker-w4-e08402/distinct-2-w4-e08402-kserve-78545b6fcf-8w74w"
Sep 14 05:10:45.582768 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:45.582489    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ea11a2f-42c5-4e7f-988e-b980d3803291-home\") pod \"distinct-2-w4-e08402-kserve-78545b6fcf-8w74w\" (UID: \"5ea11a2f-42c5-4e7f-988e-b980d3803291\") " pod="e2e-models-e2e-worker-w4-e08402/distinct-2-w4-e08402-kserve-78545b6fcf-8w74w"
Sep 14 05:10:45.582768 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:45.582594    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ea11a2f-42c5-4e7f-988e-b980d3803291-model-cache\") pod \"distinct-2-w4-e08402-kserve-78545b6fcf-8w74w\" (UID: \"5ea11a2f-42c5-4e7f-988e-b980d3803291\") " pod="e2e-models-e2e-worker-w4-e08402/distinct-2-w4-e08402-kserve-78545b6fcf-8w74w"
Sep 14 05:10:45.582768 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:45.582676    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ea11a2f-42c5-4e7f-988e-b980d3803291-tmp-dir\") pod \"distinct-2-w4-e08402-kserve-78545b6fcf-8w74w\" (UID: \"5ea11a2f-42c5-4e7f-988e-b980d3803291\") " pod="e2e-models-e2e-worker-w4-e08402/distinct-2-w4-e08402-kserve-78545b6fcf-8w74w"
Sep 14 05:10:45.584551 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:45.584522    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ea11a2f-42c5-4e7f-988e-b980d3803291-dshm\") pod \"distinct-2-w4-e08402-kserve-78545b6fcf-8w74w\" (UID: \"5ea11a2f-42c5-4e7f-988e-b980d3803291\") " pod="e2e-models-e2e-worker-w4-e08402/distinct-2-w4-e08402-kserve-78545b6fcf-8w74w"
Sep 14 05:10:45.584852 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:45.584831    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ea11a2f-42c5-4e7f-988e-b980d3803291-tls-certs\") pod \"distinct-2-w4-e08402-kserve-78545b6fcf-8w74w\" (UID: \"5ea11a2f-42c5-4e7f-988e-b980d3803291\") " pod="e2e-models-e2e-worker-w4-e08402/distinct-2-w4-e08402-kserve-78545b6fcf-8w74w"
Sep 14 05:10:45.589536 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:45.589511    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5h2g\" (UniqueName: \"kubernetes.io/projected/5ea11a2f-42c5-4e7f-988e-b980d3803291-kube-api-access-z5h2g\") pod \"distinct-2-w4-e08402-kserve-78545b6fcf-8w74w\" (UID: \"5ea11a2f-42c5-4e7f-988e-b980d3803291\") " pod="e2e-models-e2e-worker-w4-e08402/distinct-2-w4-e08402-kserve-78545b6fcf-8w74w"
Sep 14 05:10:45.683046 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:45.683013    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7818bda4-c5be-42e4-a4cf-ca3ef45cf295-plugins-config-volume\") pod \"7818bda4-c5be-42e4-a4cf-ca3ef45cf295\" (UID: \"7818bda4-c5be-42e4-a4cf-ca3ef45cf295\") "
Sep 14 05:10:45.683226 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:45.683130    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v55ck\" (UniqueName: \"kubernetes.io/projected/7818bda4-c5be-42e4-a4cf-ca3ef45cf295-kube-api-access-v55ck\") pod \"7818bda4-c5be-42e4-a4cf-ca3ef45cf295\" (UID: \"7818bda4-c5be-42e4-a4cf-ca3ef45cf295\") "
Sep 14 05:10:45.683462 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:45.683433    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7818bda4-c5be-42e4-a4cf-ca3ef45cf295-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7818bda4-c5be-42e4-a4cf-ca3ef45cf295" (UID: "7818bda4-c5be-42e4-a4cf-ca3ef45cf295"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 05:10:45.685220 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:45.685199    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7818bda4-c5be-42e4-a4cf-ca3ef45cf295-kube-api-access-v55ck" (OuterVolumeSpecName: "kube-api-access-v55ck") pod "7818bda4-c5be-42e4-a4cf-ca3ef45cf295" (UID: "7818bda4-c5be-42e4-a4cf-ca3ef45cf295"). InnerVolumeSpecName "kube-api-access-v55ck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:10:45.729496 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:45.729463    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e08402/distinct-2-w4-e08402-kserve-78545b6fcf-8w74w"
Sep 14 05:10:45.784788 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:45.784735    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v55ck\" (UniqueName: \"kubernetes.io/projected/7818bda4-c5be-42e4-a4cf-ca3ef45cf295-kube-api-access-v55ck\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:10:45.784934 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:45.784791    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7818bda4-c5be-42e4-a4cf-ca3ef45cf295-plugins-config-volume\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:10:46.005562 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:46.005537    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e08402/distinct-2-w4-e08402-kserve-78545b6fcf-8w74w"]
Sep 14 05:10:46.007309 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:10:46.007284    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ea11a2f_42c5_4e7f_988e_b980d3803291.slice/crio-147ff2e0350944300534dba6b1a2b8be6481afb8c4d99dff56b200b97497371e WatchSource:0}: Error finding container 147ff2e0350944300534dba6b1a2b8be6481afb8c4d99dff56b200b97497371e: Status 404 returned error can't find the container with id 147ff2e0350944300534dba6b1a2b8be6481afb8c4d99dff56b200b97497371e
Sep 14 05:10:46.335199 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:46.335170    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="513dafb2-e0c9-4ae9-93c3-35e3c0793043" path="/var/lib/kubelet/pods/513dafb2-e0c9-4ae9-93c3-35e3c0793043/volumes"
Sep 14 05:10:46.349903 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:46.349869    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:10:46.457654 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:46.457610    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e08402/distinct-2-w4-e08402-kserve-78545b6fcf-8w74w" event={"ID":"5ea11a2f-42c5-4e7f-988e-b980d3803291","Type":"ContainerStarted","Data":"10037950990b968a94604d5279011762813fc0df7ac1312ed55446365a46d804"}
Sep 14 05:10:46.457857 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:46.457666    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e08402/distinct-2-w4-e08402-kserve-78545b6fcf-8w74w" event={"ID":"5ea11a2f-42c5-4e7f-988e-b980d3803291","Type":"ContainerStarted","Data":"147ff2e0350944300534dba6b1a2b8be6481afb8c4d99dff56b200b97497371e"}
Sep 14 05:10:46.459045 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:46.459020    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7c6a48c-7f9cf4474b-qrmzh"
Sep 14 05:10:46.459045 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:46.459018    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7c6a48c-7f9cf4474b-qrmzh" event={"ID":"7818bda4-c5be-42e4-a4cf-ca3ef45cf295","Type":"ContainerDied","Data":"3929d0be97a40161dab8f6d8736a906cc8c61f357cd04a6056d268e77f394e9d"}
Sep 14 05:10:46.459181 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:46.459097    2565 scope.go:117] "RemoveContainer" containerID="c3b0c7b3267fb8197b2864747539cbba179db28a29821dec87e7cc4d9a7fdb64"
Sep 14 05:10:46.475587 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:46.475544    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-e08402/distinct-2-w4-e08402-kserve-78545b6fcf-8w74w" podStartSLOduration=1.47552845 podStartE2EDuration="1.47552845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:10:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:10:46.473276123 +0000 UTC m=+1444.649047682" watchObservedRunningTime="2026-09-14 05:10:46.47552845 +0000 UTC m=+1444.651300009"
Sep 14 05:10:46.485440 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:46.485408    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c7c6a48c-7f9cf4474b-qrmzh"]
Sep 14 05:10:46.492453 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:46.492429    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c7c6a48c-7f9cf4474b-qrmzh"]
Sep 14 05:10:47.467943 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:47.467909    2565 generic.go:356] "Generic (PLEG): container finished" podID="6a5a2e92-8a45-4f45-a89a-afd3d81f0699" containerID="69fa9bfb9ec720356935fcaae89bf89fbd08dff90b0ba33ab7aa670e25fe898d" exitCode=0
Sep 14 05:10:47.468359 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:47.467981    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5" event={"ID":"6a5a2e92-8a45-4f45-a89a-afd3d81f0699","Type":"ContainerDied","Data":"69fa9bfb9ec720356935fcaae89bf89fbd08dff90b0ba33ab7aa670e25fe898d"}
Sep 14 05:10:47.468404 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:47.468396    2565 scope.go:117] "RemoveContainer" containerID="69fa9bfb9ec720356935fcaae89bf89fbd08dff90b0ba33ab7aa670e25fe898d"
Sep 14 05:10:47.469458 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:47.469433    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:10:47.469515 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:47.469500    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:10:47.469550 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:47.469533    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:10:48.335830 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:48.335796    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7818bda4-c5be-42e4-a4cf-ca3ef45cf295" path="/var/lib/kubelet/pods/7818bda4-c5be-42e4-a4cf-ca3ef45cf295/volumes"
Sep 14 05:10:48.475785 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:48.475723    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5" event={"ID":"6a5a2e92-8a45-4f45-a89a-afd3d81f0699","Type":"ContainerStarted","Data":"732d8ac1fb86b3447c8eeb98092f735d22bb94e0a684f731b5d9df085ad103a3"}
Sep 14 05:10:48.916228 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:48.916185    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"
Sep 14 05:10:48.916500 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:48.916472    2565 patch_prober.go:28] interesting pod/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Sep 14 05:10:48.916583 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:48.916548    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5" podUID="6a5a2e92-8a45-4f45-a89a-afd3d81f0699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Sep 14 05:10:49.916455 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:49.916413    2565 patch_prober.go:28] interesting pod/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Sep 14 05:10:49.916917 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:49.916471    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5" podUID="6a5a2e92-8a45-4f45-a89a-afd3d81f0699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Sep 14 05:10:50.112377 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:50.112333    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"]
Sep 14 05:10:50.112792 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:50.112768    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7818bda4-c5be-42e4-a4cf-ca3ef45cf295" containerName="payload-pre-processing"
Sep 14 05:10:50.112792 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:50.112790    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7818bda4-c5be-42e4-a4cf-ca3ef45cf295" containerName="payload-pre-processing"
Sep 14 05:10:50.112994 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:50.112873    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7818bda4-c5be-42e4-a4cf-ca3ef45cf295" containerName="payload-pre-processing"
Sep 14 05:10:50.116096 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:50.116077    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"
Sep 14 05:10:50.118277 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:50.118256    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-27be50-openshift-default-dockercfg-z7g5l\""
Sep 14 05:10:50.129673 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:50.129646    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"]
Sep 14 05:10:50.226570 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:50.226496    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09dff87c-b6f1-45da-8d31-c6a906b19a6a-istiod-ca-cert\") pod \"e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4\" (UID: \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\") " pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"
Sep 14 05:10:50.226570 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:50.226543    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09dff87c-b6f1-45da-8d31-c6a906b19a6a-workload-socket\") pod \"e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4\" (UID: \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\") " pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"
Sep 14 05:10:50.226771 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:50.226625    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09dff87c-b6f1-45da-8d31-c6a906b19a6a-istio-envoy\") pod \"e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4\" (UID: \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\") " pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"
Sep 14 05:10:50.226771 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:50.226655    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09dff87c-b6f1-45da-8d31-c6a906b19a6a-credential-socket\") pod \"e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4\" (UID: \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\") " pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"
Sep 14 05:10:50.226771 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:50.226716    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09dff87c-b6f1-45da-8d31-c6a906b19a6a-workload-certs\") pod \"e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4\" (UID: \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\") " pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"
Sep 14 05:10:50.226771 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:50.226738    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09dff87c-b6f1-45da-8d31-c6a906b19a6a-istio-token\") pod \"e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4\" (UID: \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\") " pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"
Sep 14 05:10:50.226910 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:50.226811    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09dff87c-b6f1-45da-8d31-c6a906b19a6a-istio-podinfo\") pod \"e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4\" (UID: \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\") " pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"
Sep 14 05:10:50.226910 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:50.226883    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xmpj\" (UniqueName: \"kubernetes.io/projected/09dff87c-b6f1-45da-8d31-c6a906b19a6a-kube-api-access-6xmpj\") pod \"e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4\" (UID: \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\") " pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"
Sep 14 05:10:50.226978 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:50.226922    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09dff87c-b6f1-45da-8d31-c6a906b19a6a-istio-data\") pod \"e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4\" (UID: \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\") " pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"
Sep 14 05:10:50.328265 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:50.328225    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xmpj\" (UniqueName: \"kubernetes.io/projected/09dff87c-b6f1-45da-8d31-c6a906b19a6a-kube-api-access-6xmpj\") pod \"e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4\" (UID: \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\") " pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"
Sep 14 05:10:50.328477 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:50.328286    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09dff87c-b6f1-45da-8d31-c6a906b19a6a-istio-data\") pod \"e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4\" (UID: \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\") " pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"
Sep 14 05:10:50.328477 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:50.328316    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09dff87c-b6f1-45da-8d31-c6a906b19a6a-istiod-ca-cert\") pod \"e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4\" (UID: \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\") " pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"
Sep 14 05:10:50.328477 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:50.328359    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09dff87c-b6f1-45da-8d31-c6a906b19a6a-workload-socket\") pod \"e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4\" (UID: \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\") " pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"
Sep 14 05:10:50.328477 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:50.328412    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09dff87c-b6f1-45da-8d31-c6a906b19a6a-istio-envoy\") pod \"e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4\" (UID: \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\") " pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"
Sep 14 05:10:50.328477 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:50.328437    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09dff87c-b6f1-45da-8d31-c6a906b19a6a-credential-socket\") pod \"e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4\" (UID: \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\") " pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"
Sep 14 05:10:50.328831 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:50.328561    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09dff87c-b6f1-45da-8d31-c6a906b19a6a-workload-certs\") pod \"e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4\" (UID: \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\") " pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"
Sep 14 05:10:50.328831 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:50.328614    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09dff87c-b6f1-45da-8d31-c6a906b19a6a-istio-token\") pod \"e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4\" (UID: \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\") " pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"
Sep 14 05:10:50.328831 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:50.328652    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09dff87c-b6f1-45da-8d31-c6a906b19a6a-istio-podinfo\") pod \"e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4\" (UID: \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\") " pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"
Sep 14 05:10:50.328831 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:50.328784    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09dff87c-b6f1-45da-8d31-c6a906b19a6a-istio-data\") pod \"e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4\" (UID: \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\") " pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"
Sep 14 05:10:50.329250 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:50.329228    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09dff87c-b6f1-45da-8d31-c6a906b19a6a-credential-socket\") pod \"e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4\" (UID: \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\") " pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"
Sep 14 05:10:50.329385 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:50.329287    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09dff87c-b6f1-45da-8d31-c6a906b19a6a-workload-certs\") pod \"e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4\" (UID: \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\") " pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"
Sep 14 05:10:50.329465 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:50.329419    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09dff87c-b6f1-45da-8d31-c6a906b19a6a-workload-socket\") pod \"e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4\" (UID: \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\") " pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"
Sep 14 05:10:50.329698 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:50.329673    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09dff87c-b6f1-45da-8d31-c6a906b19a6a-istiod-ca-cert\") pod \"e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4\" (UID: \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\") " pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"
Sep 14 05:10:50.331056 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:50.331032    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09dff87c-b6f1-45da-8d31-c6a906b19a6a-istio-envoy\") pod \"e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4\" (UID: \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\") " pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"
Sep 14 05:10:50.331506 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:50.331477    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09dff87c-b6f1-45da-8d31-c6a906b19a6a-istio-podinfo\") pod \"e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4\" (UID: \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\") " pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"
Sep 14 05:10:50.349776 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:50.349724    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09dff87c-b6f1-45da-8d31-c6a906b19a6a-istio-token\") pod \"e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4\" (UID: \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\") " pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"
Sep 14 05:10:50.369777 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:50.369412    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xmpj\" (UniqueName: \"kubernetes.io/projected/09dff87c-b6f1-45da-8d31-c6a906b19a6a-kube-api-access-6xmpj\") pod \"e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4\" (UID: \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\") " pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"
Sep 14 05:10:50.429375 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:50.429336    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"
Sep 14 05:10:50.726669 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:50.726639    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"]
Sep 14 05:10:50.727624 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:10:50.727596    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09dff87c_b6f1_45da_8d31_c6a906b19a6a.slice/crio-322d64e5efcdd6f23b5c7579d63841d98911c5949ce023730bee1bf46c27237a WatchSource:0}: Error finding container 322d64e5efcdd6f23b5c7579d63841d98911c5949ce023730bee1bf46c27237a: Status 404 returned error can't find the container with id 322d64e5efcdd6f23b5c7579d63841d98911c5949ce023730bee1bf46c27237a
Sep 14 05:10:50.730030 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:50.729992    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:10:50.730135 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:50.730075    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:10:50.730135 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:50.730121    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:10:50.916461 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:50.916430    2565 patch_prober.go:28] interesting pod/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Sep 14 05:10:50.916826 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:50.916481    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5" podUID="6a5a2e92-8a45-4f45-a89a-afd3d81f0699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Sep 14 05:10:51.495175 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:51.495131    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4" event={"ID":"09dff87c-b6f1-45da-8d31-c6a906b19a6a","Type":"ContainerStarted","Data":"f31d9520cd40097950f3a6a0278b85a54a05fb845fa9679cf8aed98b4b6380c1"}
Sep 14 05:10:51.495333 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:51.495197    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4" event={"ID":"09dff87c-b6f1-45da-8d31-c6a906b19a6a","Type":"ContainerStarted","Data":"322d64e5efcdd6f23b5c7579d63841d98911c5949ce023730bee1bf46c27237a"}
Sep 14 05:10:51.515112 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:51.515063    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4" podStartSLOduration=1.515046458 podStartE2EDuration="1.515046458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:10:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:10:51.513272015 +0000 UTC m=+1449.689043576" watchObservedRunningTime="2026-09-14 05:10:51.515046458 +0000 UTC m=+1449.690818017"
Sep 14 05:10:51.917003 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:51.916965    2565 patch_prober.go:28] interesting pod/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Sep 14 05:10:51.917421 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:51.917024    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5" podUID="6a5a2e92-8a45-4f45-a89a-afd3d81f0699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Sep 14 05:10:52.430281 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:52.430250    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"
Sep 14 05:10:52.435445 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:52.435417    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"
Sep 14 05:10:52.500749 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:52.500715    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"
Sep 14 05:10:52.501683 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:52.501662    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"
Sep 14 05:10:52.916258 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:52.916226    2565 patch_prober.go:28] interesting pod/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Sep 14 05:10:52.916440 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:52.916280    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5" podUID="6a5a2e92-8a45-4f45-a89a-afd3d81f0699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Sep 14 05:10:53.920542 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:53.920509    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"
Sep 14 05:10:53.921086 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:53.920959    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"
Sep 14 05:10:53.921858 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:53.921830    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"
Sep 14 05:10:53.974642 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:53.974607    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-c7c6a48c/e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm"]
Sep 14 05:10:55.730021 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:55.729986    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-e08402/distinct-2-w4-e08402-kserve-78545b6fcf-8w74w"
Sep 14 05:10:55.730412 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:55.730070    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e08402/distinct-2-w4-e08402-kserve-78545b6fcf-8w74w"
Sep 14 05:10:55.742665 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:55.742635    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-e08402/distinct-2-w4-e08402-kserve-78545b6fcf-8w74w"
Sep 14 05:10:56.540372 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:10:56.540335    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e08402/distinct-2-w4-e08402-kserve-78545b6fcf-8w74w"
Sep 14 05:11:07.831908 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:07.831872    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:11:08.994514 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:08.994451    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-c7c6a48c/e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm" podUID="0dff61ad-71c3-4c07-b6fc-34d4fff8b68a" containerName="main" containerID="cri-o://34e1221159072d9ccdc1c49c5381d47ae5630a92cf0d0624a24a0936ce16d241" gracePeriod=45
Sep 14 05:11:09.282770 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:09.282738    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c7c6a48c/e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm"
Sep 14 05:11:09.325559 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:09.325523    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dff61ad-71c3-4c07-b6fc-34d4fff8b68a-tls-certs\") pod \"0dff61ad-71c3-4c07-b6fc-34d4fff8b68a\" (UID: \"0dff61ad-71c3-4c07-b6fc-34d4fff8b68a\") "
Sep 14 05:11:09.325744 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:09.325580    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dff61ad-71c3-4c07-b6fc-34d4fff8b68a-dshm\") pod \"0dff61ad-71c3-4c07-b6fc-34d4fff8b68a\" (UID: \"0dff61ad-71c3-4c07-b6fc-34d4fff8b68a\") "
Sep 14 05:11:09.325744 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:09.325630    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dff61ad-71c3-4c07-b6fc-34d4fff8b68a-home\") pod \"0dff61ad-71c3-4c07-b6fc-34d4fff8b68a\" (UID: \"0dff61ad-71c3-4c07-b6fc-34d4fff8b68a\") "
Sep 14 05:11:09.325744 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:09.325673    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkflg\" (UniqueName: \"kubernetes.io/projected/0dff61ad-71c3-4c07-b6fc-34d4fff8b68a-kube-api-access-bkflg\") pod \"0dff61ad-71c3-4c07-b6fc-34d4fff8b68a\" (UID: \"0dff61ad-71c3-4c07-b6fc-34d4fff8b68a\") "
Sep 14 05:11:09.325920 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:09.325746    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dff61ad-71c3-4c07-b6fc-34d4fff8b68a-tmp-dir\") pod \"0dff61ad-71c3-4c07-b6fc-34d4fff8b68a\" (UID: \"0dff61ad-71c3-4c07-b6fc-34d4fff8b68a\") "
Sep 14 05:11:09.325920 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:09.325831    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dff61ad-71c3-4c07-b6fc-34d4fff8b68a-model-cache\") pod \"0dff61ad-71c3-4c07-b6fc-34d4fff8b68a\" (UID: \"0dff61ad-71c3-4c07-b6fc-34d4fff8b68a\") "
Sep 14 05:11:09.326022 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:09.325947    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dff61ad-71c3-4c07-b6fc-34d4fff8b68a-home" (OuterVolumeSpecName: "home") pod "0dff61ad-71c3-4c07-b6fc-34d4fff8b68a" (UID: "0dff61ad-71c3-4c07-b6fc-34d4fff8b68a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:11:09.326083 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:09.326047    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dff61ad-71c3-4c07-b6fc-34d4fff8b68a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0dff61ad-71c3-4c07-b6fc-34d4fff8b68a" (UID: "0dff61ad-71c3-4c07-b6fc-34d4fff8b68a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:11:09.326167 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:09.326141    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dff61ad-71c3-4c07-b6fc-34d4fff8b68a-model-cache" (OuterVolumeSpecName: "model-cache") pod "0dff61ad-71c3-4c07-b6fc-34d4fff8b68a" (UID: "0dff61ad-71c3-4c07-b6fc-34d4fff8b68a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:11:09.326272 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:09.326248    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dff61ad-71c3-4c07-b6fc-34d4fff8b68a-tmp-dir\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:11:09.326272 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:09.326273    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dff61ad-71c3-4c07-b6fc-34d4fff8b68a-model-cache\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:11:09.326457 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:09.326288    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dff61ad-71c3-4c07-b6fc-34d4fff8b68a-home\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:11:09.328024 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:09.327995    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dff61ad-71c3-4c07-b6fc-34d4fff8b68a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0dff61ad-71c3-4c07-b6fc-34d4fff8b68a" (UID: "0dff61ad-71c3-4c07-b6fc-34d4fff8b68a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 05:11:09.328142 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:09.328030    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dff61ad-71c3-4c07-b6fc-34d4fff8b68a-dshm" (OuterVolumeSpecName: "dshm") pod "0dff61ad-71c3-4c07-b6fc-34d4fff8b68a" (UID: "0dff61ad-71c3-4c07-b6fc-34d4fff8b68a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:11:09.328700 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:09.328667    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dff61ad-71c3-4c07-b6fc-34d4fff8b68a-kube-api-access-bkflg" (OuterVolumeSpecName: "kube-api-access-bkflg") pod "0dff61ad-71c3-4c07-b6fc-34d4fff8b68a" (UID: "0dff61ad-71c3-4c07-b6fc-34d4fff8b68a"). InnerVolumeSpecName "kube-api-access-bkflg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:11:09.427711 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:09.427681    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkflg\" (UniqueName: \"kubernetes.io/projected/0dff61ad-71c3-4c07-b6fc-34d4fff8b68a-kube-api-access-bkflg\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:11:09.427711 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:09.427708    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dff61ad-71c3-4c07-b6fc-34d4fff8b68a-tls-certs\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:11:09.427711 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:09.427718    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dff61ad-71c3-4c07-b6fc-34d4fff8b68a-dshm\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:11:09.604050 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:09.604013    2565 generic.go:356] "Generic (PLEG): container finished" podID="0dff61ad-71c3-4c07-b6fc-34d4fff8b68a" containerID="34e1221159072d9ccdc1c49c5381d47ae5630a92cf0d0624a24a0936ce16d241" exitCode=0
Sep 14 05:11:09.604223 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:09.604056    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c7c6a48c/e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm"
Sep 14 05:11:09.604223 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:09.604099    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c7c6a48c/e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm" event={"ID":"0dff61ad-71c3-4c07-b6fc-34d4fff8b68a","Type":"ContainerDied","Data":"34e1221159072d9ccdc1c49c5381d47ae5630a92cf0d0624a24a0936ce16d241"}
Sep 14 05:11:09.604223 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:09.604135    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c7c6a48c/e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm" event={"ID":"0dff61ad-71c3-4c07-b6fc-34d4fff8b68a","Type":"ContainerDied","Data":"aa46b66df609bb9b0ec969886f318f32eb8f1e68ad90362b3a37d0a17cb72237"}
Sep 14 05:11:09.604223 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:09.604152    2565 scope.go:117] "RemoveContainer" containerID="34e1221159072d9ccdc1c49c5381d47ae5630a92cf0d0624a24a0936ce16d241"
Sep 14 05:11:09.613467 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:09.613449    2565 scope.go:117] "RemoveContainer" containerID="34e1221159072d9ccdc1c49c5381d47ae5630a92cf0d0624a24a0936ce16d241"
Sep 14 05:11:09.613785 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:11:09.613738    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34e1221159072d9ccdc1c49c5381d47ae5630a92cf0d0624a24a0936ce16d241\": container with ID starting with 34e1221159072d9ccdc1c49c5381d47ae5630a92cf0d0624a24a0936ce16d241 not found: ID does not exist" containerID="34e1221159072d9ccdc1c49c5381d47ae5630a92cf0d0624a24a0936ce16d241"
Sep 14 05:11:09.613859 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:09.613789    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34e1221159072d9ccdc1c49c5381d47ae5630a92cf0d0624a24a0936ce16d241"} err="failed to get container status \"34e1221159072d9ccdc1c49c5381d47ae5630a92cf0d0624a24a0936ce16d241\": rpc error: code = NotFound desc = could not find container \"34e1221159072d9ccdc1c49c5381d47ae5630a92cf0d0624a24a0936ce16d241\": container with ID starting with 34e1221159072d9ccdc1c49c5381d47ae5630a92cf0d0624a24a0936ce16d241 not found: ID does not exist"
Sep 14 05:11:09.625502 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:09.625475    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-c7c6a48c/e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm"]
Sep 14 05:11:09.630658 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:09.630634    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-c7c6a48c/e2e-lifecycle-model-c7c6a48c-kserve-78bb95fd7-dm4fm"]
Sep 14 05:11:10.334467 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:10.334433    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0dff61ad-71c3-4c07-b6fc-34d4fff8b68a" path="/var/lib/kubelet/pods/0dff61ad-71c3-4c07-b6fc-34d4fff8b68a/volumes"
Sep 14 05:11:18.655956 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:18.655879    2565 generic.go:356] "Generic (PLEG): container finished" podID="09dff87c-b6f1-45da-8d31-c6a906b19a6a" containerID="f31d9520cd40097950f3a6a0278b85a54a05fb845fa9679cf8aed98b4b6380c1" exitCode=0
Sep 14 05:11:18.655956 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:18.655923    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4" event={"ID":"09dff87c-b6f1-45da-8d31-c6a906b19a6a","Type":"ContainerDied","Data":"f31d9520cd40097950f3a6a0278b85a54a05fb845fa9679cf8aed98b4b6380c1"}
Sep 14 05:11:18.656418 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:18.656328    2565 scope.go:117] "RemoveContainer" containerID="f31d9520cd40097950f3a6a0278b85a54a05fb845fa9679cf8aed98b4b6380c1"
Sep 14 05:11:18.657406 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:18.657375    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:11:18.657481 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:18.657448    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:11:18.657518 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:18.657481    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:11:19.669891 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:19.669792    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4" event={"ID":"09dff87c-b6f1-45da-8d31-c6a906b19a6a","Type":"ContainerStarted","Data":"2872ff980962ce39aa52d51b989f7b8f36ee80fce9ea4ec05d77c2dd085eba66"}
Sep 14 05:11:20.430067 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:20.430032    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"
Sep 14 05:11:20.431355 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:20.431331    2565 patch_prober.go:28] interesting pod/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Sep 14 05:11:20.431466 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:20.431381    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4" podUID="09dff87c-b6f1-45da-8d31-c6a906b19a6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Sep 14 05:11:21.430088 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:21.430049    2565 patch_prober.go:28] interesting pod/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Sep 14 05:11:21.430496 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:21.430106    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4" podUID="09dff87c-b6f1-45da-8d31-c6a906b19a6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Sep 14 05:11:22.430254 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:22.430220    2565 patch_prober.go:28] interesting pod/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Sep 14 05:11:22.430658 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:22.430275    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4" podUID="09dff87c-b6f1-45da-8d31-c6a906b19a6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Sep 14 05:11:22.501138 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:22.501106    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"
Sep 14 05:11:23.430178 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:23.430145    2565 patch_prober.go:28] interesting pod/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Sep 14 05:11:23.430353 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:23.430208    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4" podUID="09dff87c-b6f1-45da-8d31-c6a906b19a6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Sep 14 05:11:24.429720 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:24.429680    2565 patch_prober.go:28] interesting pod/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Sep 14 05:11:24.429991 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:24.429745    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4" podUID="09dff87c-b6f1-45da-8d31-c6a906b19a6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Sep 14 05:11:25.434439 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:25.434404    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"
Sep 14 05:11:25.435393 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:25.435376    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"
Sep 14 05:11:26.518082 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:26.518050    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-2ff4a6/auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl"]
Sep 14 05:11:26.518544 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:26.518412    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dff61ad-71c3-4c07-b6fc-34d4fff8b68a" containerName="main"
Sep 14 05:11:26.518544 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:26.518424    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dff61ad-71c3-4c07-b6fc-34d4fff8b68a" containerName="main"
Sep 14 05:11:26.518544 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:26.518480    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dff61ad-71c3-4c07-b6fc-34d4fff8b68a" containerName="main"
Sep 14 05:11:26.522856 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:26.522833    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2ff4a6/auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl"
Sep 14 05:11:26.525232 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:26.525213    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-2ff4a6\"/\"kube-root-ca.crt\""
Sep 14 05:11:26.526217 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:26.526200    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-2ff4a6\"/\"openshift-service-ca.crt\""
Sep 14 05:11:26.526333 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:26.526229    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2ff4a6\"/\"auth-test-model-2ff4a6-kserve-self-signed-certs\""
Sep 14 05:11:26.526333 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:26.526248    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2ff4a6\"/\"default-dockercfg-qnm7q\""
Sep 14 05:11:26.532288 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:26.532265    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2ff4a6/auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl"]
Sep 14 05:11:26.596771 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:26.596736    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df1311ed-d356-49b5-b319-f247b65a4abd-dshm\") pod \"auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl\" (UID: \"df1311ed-d356-49b5-b319-f247b65a4abd\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl"
Sep 14 05:11:26.596921 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:26.596804    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df1311ed-d356-49b5-b319-f247b65a4abd-home\") pod \"auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl\" (UID: \"df1311ed-d356-49b5-b319-f247b65a4abd\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl"
Sep 14 05:11:26.596921 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:26.596844    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df1311ed-d356-49b5-b319-f247b65a4abd-model-cache\") pod \"auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl\" (UID: \"df1311ed-d356-49b5-b319-f247b65a4abd\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl"
Sep 14 05:11:26.596921 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:26.596893    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df1311ed-d356-49b5-b319-f247b65a4abd-tls-certs\") pod \"auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl\" (UID: \"df1311ed-d356-49b5-b319-f247b65a4abd\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl"
Sep 14 05:11:26.597031 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:26.596931    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzb5n\" (UniqueName: \"kubernetes.io/projected/df1311ed-d356-49b5-b319-f247b65a4abd-kube-api-access-pzb5n\") pod \"auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl\" (UID: \"df1311ed-d356-49b5-b319-f247b65a4abd\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl"
Sep 14 05:11:26.597031 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:26.596965    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df1311ed-d356-49b5-b319-f247b65a4abd-tmp-dir\") pod \"auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl\" (UID: \"df1311ed-d356-49b5-b319-f247b65a4abd\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl"
Sep 14 05:11:26.698393 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:26.698352    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df1311ed-d356-49b5-b319-f247b65a4abd-model-cache\") pod \"auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl\" (UID: \"df1311ed-d356-49b5-b319-f247b65a4abd\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl"
Sep 14 05:11:26.698591 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:26.698467    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df1311ed-d356-49b5-b319-f247b65a4abd-tls-certs\") pod \"auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl\" (UID: \"df1311ed-d356-49b5-b319-f247b65a4abd\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl"
Sep 14 05:11:26.698591 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:26.698524    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzb5n\" (UniqueName: \"kubernetes.io/projected/df1311ed-d356-49b5-b319-f247b65a4abd-kube-api-access-pzb5n\") pod \"auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl\" (UID: \"df1311ed-d356-49b5-b319-f247b65a4abd\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl"
Sep 14 05:11:26.698591 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:26.698544    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df1311ed-d356-49b5-b319-f247b65a4abd-tmp-dir\") pod \"auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl\" (UID: \"df1311ed-d356-49b5-b319-f247b65a4abd\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl"
Sep 14 05:11:26.698591 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:26.698584    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df1311ed-d356-49b5-b319-f247b65a4abd-dshm\") pod \"auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl\" (UID: \"df1311ed-d356-49b5-b319-f247b65a4abd\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl"
Sep 14 05:11:26.698854 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:26.698611    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df1311ed-d356-49b5-b319-f247b65a4abd-home\") pod \"auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl\" (UID: \"df1311ed-d356-49b5-b319-f247b65a4abd\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl"
Sep 14 05:11:26.698917 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:26.698863    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df1311ed-d356-49b5-b319-f247b65a4abd-model-cache\") pod \"auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl\" (UID: \"df1311ed-d356-49b5-b319-f247b65a4abd\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl"
Sep 14 05:11:26.698967 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:26.698945    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df1311ed-d356-49b5-b319-f247b65a4abd-tmp-dir\") pod \"auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl\" (UID: \"df1311ed-d356-49b5-b319-f247b65a4abd\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl"
Sep 14 05:11:26.699007 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:26.698987    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df1311ed-d356-49b5-b319-f247b65a4abd-home\") pod \"auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl\" (UID: \"df1311ed-d356-49b5-b319-f247b65a4abd\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl"
Sep 14 05:11:26.700810 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:26.700788    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df1311ed-d356-49b5-b319-f247b65a4abd-dshm\") pod \"auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl\" (UID: \"df1311ed-d356-49b5-b319-f247b65a4abd\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl"
Sep 14 05:11:26.701022 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:26.701004    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df1311ed-d356-49b5-b319-f247b65a4abd-tls-certs\") pod \"auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl\" (UID: \"df1311ed-d356-49b5-b319-f247b65a4abd\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl"
Sep 14 05:11:26.706778 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:26.706741    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzb5n\" (UniqueName: \"kubernetes.io/projected/df1311ed-d356-49b5-b319-f247b65a4abd-kube-api-access-pzb5n\") pod \"auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl\" (UID: \"df1311ed-d356-49b5-b319-f247b65a4abd\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl"
Sep 14 05:11:26.834151 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:26.834125    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2ff4a6/auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl"
Sep 14 05:11:27.105517 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:27.105492    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2ff4a6/auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl"]
Sep 14 05:11:27.109519 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:11:27.109461    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf1311ed_d356_49b5_b319_f247b65a4abd.slice/crio-282ee520bb485cbcba08c8a70692293d8dab9426e10901d736f99a297adf420e WatchSource:0}: Error finding container 282ee520bb485cbcba08c8a70692293d8dab9426e10901d736f99a297adf420e: Status 404 returned error can't find the container with id 282ee520bb485cbcba08c8a70692293d8dab9426e10901d736f99a297adf420e
Sep 14 05:11:27.713099 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:27.713057    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2ff4a6/auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl" event={"ID":"df1311ed-d356-49b5-b319-f247b65a4abd","Type":"ContainerStarted","Data":"a02b5fbe4b565c0973210023572f4c3facb35d1e916691f676c1d4bbe7b662f9"}
Sep 14 05:11:27.713453 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:27.713107    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2ff4a6/auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl" event={"ID":"df1311ed-d356-49b5-b319-f247b65a4abd","Type":"ContainerStarted","Data":"282ee520bb485cbcba08c8a70692293d8dab9426e10901d736f99a297adf420e"}
Sep 14 05:11:27.730040 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:27.729995    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-2ff4a6/auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl" podStartSLOduration=1.729980754 podStartE2EDuration="1.729980754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:11:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:11:27.727860709 +0000 UTC m=+1485.903632268" watchObservedRunningTime="2026-09-14 05:11:27.729980754 +0000 UTC m=+1485.905752358"
Sep 14 05:11:32.334131 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:32.334102    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:11:36.834492 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:36.834443    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-2ff4a6/auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl"
Sep 14 05:11:36.834492 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:36.834499    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-2ff4a6/auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl"
Sep 14 05:11:36.847217 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:36.847193    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-2ff4a6/auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl"
Sep 14 05:11:37.779291 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:37.779260    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-2ff4a6/auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl"
Sep 14 05:11:42.395382 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:42.395355    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kndxq_e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae/ovn-acl-logging/0.log"
Sep 14 05:11:42.398843 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:42.398821    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kndxq_e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae/ovn-acl-logging/0.log"
Sep 14 05:11:56.926796 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:11:56.926750    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:12:10.011804 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:10.011713    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"]
Sep 14 05:12:10.012215 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:10.011971    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd" podUID="8753cc9c-d346-441f-ba35-725feb7818d3" containerName="istio-proxy" containerID="cri-o://5cdbd7b9f400693d4f8b41fadbbd925c5d70dd6e839536b3ee5cf8624e4ae38a" gracePeriod=30
Sep 14 05:12:15.248368 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:15.248341    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"
Sep 14 05:12:15.348647 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:15.348609    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8753cc9c-d346-441f-ba35-725feb7818d3-istio-data\") pod \"8753cc9c-d346-441f-ba35-725feb7818d3\" (UID: \"8753cc9c-d346-441f-ba35-725feb7818d3\") "
Sep 14 05:12:15.348843 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:15.348670    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8753cc9c-d346-441f-ba35-725feb7818d3-istiod-ca-cert\") pod \"8753cc9c-d346-441f-ba35-725feb7818d3\" (UID: \"8753cc9c-d346-441f-ba35-725feb7818d3\") "
Sep 14 05:12:15.348843 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:15.348709    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8753cc9c-d346-441f-ba35-725feb7818d3-workload-certs\") pod \"8753cc9c-d346-441f-ba35-725feb7818d3\" (UID: \"8753cc9c-d346-441f-ba35-725feb7818d3\") "
Sep 14 05:12:15.348843 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:15.348733    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8753cc9c-d346-441f-ba35-725feb7818d3-istio-podinfo\") pod \"8753cc9c-d346-441f-ba35-725feb7818d3\" (UID: \"8753cc9c-d346-441f-ba35-725feb7818d3\") "
Sep 14 05:12:15.348843 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:15.348748    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8753cc9c-d346-441f-ba35-725feb7818d3-credential-socket\") pod \"8753cc9c-d346-441f-ba35-725feb7818d3\" (UID: \"8753cc9c-d346-441f-ba35-725feb7818d3\") "
Sep 14 05:12:15.348843 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:15.348811    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8753cc9c-d346-441f-ba35-725feb7818d3-istio-envoy\") pod \"8753cc9c-d346-441f-ba35-725feb7818d3\" (UID: \"8753cc9c-d346-441f-ba35-725feb7818d3\") "
Sep 14 05:12:15.349116 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:15.348857    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cp824\" (UniqueName: \"kubernetes.io/projected/8753cc9c-d346-441f-ba35-725feb7818d3-kube-api-access-cp824\") pod \"8753cc9c-d346-441f-ba35-725feb7818d3\" (UID: \"8753cc9c-d346-441f-ba35-725feb7818d3\") "
Sep 14 05:12:15.349116 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:15.348902    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8753cc9c-d346-441f-ba35-725feb7818d3-workload-socket\") pod \"8753cc9c-d346-441f-ba35-725feb7818d3\" (UID: \"8753cc9c-d346-441f-ba35-725feb7818d3\") "
Sep 14 05:12:15.349116 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:15.348928    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8753cc9c-d346-441f-ba35-725feb7818d3-istio-data" (OuterVolumeSpecName: "istio-data") pod "8753cc9c-d346-441f-ba35-725feb7818d3" (UID: "8753cc9c-d346-441f-ba35-725feb7818d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:12:15.349116 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:15.348990    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8753cc9c-d346-441f-ba35-725feb7818d3-istio-token\") pod \"8753cc9c-d346-441f-ba35-725feb7818d3\" (UID: \"8753cc9c-d346-441f-ba35-725feb7818d3\") "
Sep 14 05:12:15.349116 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:15.349046    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8753cc9c-d346-441f-ba35-725feb7818d3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8753cc9c-d346-441f-ba35-725feb7818d3" (UID: "8753cc9c-d346-441f-ba35-725feb7818d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:12:15.349343 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:15.349115    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8753cc9c-d346-441f-ba35-725feb7818d3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8753cc9c-d346-441f-ba35-725feb7818d3" (UID: "8753cc9c-d346-441f-ba35-725feb7818d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:12:15.349343 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:15.349178    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8753cc9c-d346-441f-ba35-725feb7818d3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8753cc9c-d346-441f-ba35-725feb7818d3" (UID: "8753cc9c-d346-441f-ba35-725feb7818d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 05:12:15.349343 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:15.349266    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8753cc9c-d346-441f-ba35-725feb7818d3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8753cc9c-d346-441f-ba35-725feb7818d3" (UID: "8753cc9c-d346-441f-ba35-725feb7818d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:12:15.349471 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:15.349453    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8753cc9c-d346-441f-ba35-725feb7818d3-istiod-ca-cert\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:12:15.349508 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:15.349477    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8753cc9c-d346-441f-ba35-725feb7818d3-workload-certs\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:12:15.349508 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:15.349490    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8753cc9c-d346-441f-ba35-725feb7818d3-credential-socket\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:12:15.349508 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:15.349501    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8753cc9c-d346-441f-ba35-725feb7818d3-workload-socket\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:12:15.349649 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:15.349514    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8753cc9c-d346-441f-ba35-725feb7818d3-istio-data\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:12:15.351026 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:15.350999    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8753cc9c-d346-441f-ba35-725feb7818d3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8753cc9c-d346-441f-ba35-725feb7818d3" (UID: "8753cc9c-d346-441f-ba35-725feb7818d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 05:12:15.351134 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:15.351056    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8753cc9c-d346-441f-ba35-725feb7818d3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8753cc9c-d346-441f-ba35-725feb7818d3" (UID: "8753cc9c-d346-441f-ba35-725feb7818d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:12:15.351449 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:15.351423    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8753cc9c-d346-441f-ba35-725feb7818d3-istio-token" (OuterVolumeSpecName: "istio-token") pod "8753cc9c-d346-441f-ba35-725feb7818d3" (UID: "8753cc9c-d346-441f-ba35-725feb7818d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:12:15.351449 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:15.351431    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8753cc9c-d346-441f-ba35-725feb7818d3-kube-api-access-cp824" (OuterVolumeSpecName: "kube-api-access-cp824") pod "8753cc9c-d346-441f-ba35-725feb7818d3" (UID: "8753cc9c-d346-441f-ba35-725feb7818d3"). InnerVolumeSpecName "kube-api-access-cp824". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:12:15.450206 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:15.450169    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8753cc9c-d346-441f-ba35-725feb7818d3-istio-podinfo\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:12:15.450206 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:15.450201    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8753cc9c-d346-441f-ba35-725feb7818d3-istio-envoy\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:12:15.450206 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:15.450210    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp824\" (UniqueName: \"kubernetes.io/projected/8753cc9c-d346-441f-ba35-725feb7818d3-kube-api-access-cp824\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:12:15.450452 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:15.450220    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8753cc9c-d346-441f-ba35-725feb7818d3-istio-token\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:12:15.983460 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:15.983425    2565 generic.go:356] "Generic (PLEG): container finished" podID="8753cc9c-d346-441f-ba35-725feb7818d3" containerID="5cdbd7b9f400693d4f8b41fadbbd925c5d70dd6e839536b3ee5cf8624e4ae38a" exitCode=0
Sep 14 05:12:15.983636 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:15.983471    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"
Sep 14 05:12:15.983636 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:15.983514    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd" event={"ID":"8753cc9c-d346-441f-ba35-725feb7818d3","Type":"ContainerDied","Data":"5cdbd7b9f400693d4f8b41fadbbd925c5d70dd6e839536b3ee5cf8624e4ae38a"}
Sep 14 05:12:15.983636 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:15.983570    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd" event={"ID":"8753cc9c-d346-441f-ba35-725feb7818d3","Type":"ContainerDied","Data":"845e00c63970f55bfb2b94f81e95f2c372c193100444392ad0cb23d9f030935b"}
Sep 14 05:12:15.983636 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:15.983598    2565 scope.go:117] "RemoveContainer" containerID="5cdbd7b9f400693d4f8b41fadbbd925c5d70dd6e839536b3ee5cf8624e4ae38a"
Sep 14 05:12:15.993129 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:15.993112    2565 scope.go:117] "RemoveContainer" containerID="82c716a1ab19896fc333b5d9e7ed94e74d6b1dca45aeb3919a5a67286586c2e8"
Sep 14 05:12:16.001155 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:16.001132    2565 scope.go:117] "RemoveContainer" containerID="5cdbd7b9f400693d4f8b41fadbbd925c5d70dd6e839536b3ee5cf8624e4ae38a"
Sep 14 05:12:16.001424 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:12:16.001402    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cdbd7b9f400693d4f8b41fadbbd925c5d70dd6e839536b3ee5cf8624e4ae38a\": container with ID starting with 5cdbd7b9f400693d4f8b41fadbbd925c5d70dd6e839536b3ee5cf8624e4ae38a not found: ID does not exist" containerID="5cdbd7b9f400693d4f8b41fadbbd925c5d70dd6e839536b3ee5cf8624e4ae38a"
Sep 14 05:12:16.001474 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:16.001433    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cdbd7b9f400693d4f8b41fadbbd925c5d70dd6e839536b3ee5cf8624e4ae38a"} err="failed to get container status \"5cdbd7b9f400693d4f8b41fadbbd925c5d70dd6e839536b3ee5cf8624e4ae38a\": rpc error: code = NotFound desc = could not find container \"5cdbd7b9f400693d4f8b41fadbbd925c5d70dd6e839536b3ee5cf8624e4ae38a\": container with ID starting with 5cdbd7b9f400693d4f8b41fadbbd925c5d70dd6e839536b3ee5cf8624e4ae38a not found: ID does not exist"
Sep 14 05:12:16.001474 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:16.001449    2565 scope.go:117] "RemoveContainer" containerID="82c716a1ab19896fc333b5d9e7ed94e74d6b1dca45aeb3919a5a67286586c2e8"
Sep 14 05:12:16.001698 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:12:16.001679    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82c716a1ab19896fc333b5d9e7ed94e74d6b1dca45aeb3919a5a67286586c2e8\": container with ID starting with 82c716a1ab19896fc333b5d9e7ed94e74d6b1dca45aeb3919a5a67286586c2e8 not found: ID does not exist" containerID="82c716a1ab19896fc333b5d9e7ed94e74d6b1dca45aeb3919a5a67286586c2e8"
Sep 14 05:12:16.001865 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:16.001704    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82c716a1ab19896fc333b5d9e7ed94e74d6b1dca45aeb3919a5a67286586c2e8"} err="failed to get container status \"82c716a1ab19896fc333b5d9e7ed94e74d6b1dca45aeb3919a5a67286586c2e8\": rpc error: code = NotFound desc = could not find container \"82c716a1ab19896fc333b5d9e7ed94e74d6b1dca45aeb3919a5a67286586c2e8\": container with ID starting with 82c716a1ab19896fc333b5d9e7ed94e74d6b1dca45aeb3919a5a67286586c2e8 not found: ID does not exist"
Sep 14 05:12:16.004790 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:16.004764    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"]
Sep 14 05:12:16.008617 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:16.008595    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c7c6a48c-openshift-default-785bf6b7f6-8w7qd"]
Sep 14 05:12:16.335458 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:16.335425    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8753cc9c-d346-441f-ba35-725feb7818d3" path="/var/lib/kubelet/pods/8753cc9c-d346-441f-ba35-725feb7818d3/volumes"
Sep 14 05:12:20.230630 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:20.230593    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:12:45.324449 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:12:45.324413    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:13:05.485099 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:05.485066    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"]
Sep 14 05:13:05.485605 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:05.485585    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8753cc9c-d346-441f-ba35-725feb7818d3" containerName="istio-proxy"
Sep 14 05:13:05.485682 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:05.485609    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8753cc9c-d346-441f-ba35-725feb7818d3" containerName="istio-proxy"
Sep 14 05:13:05.485682 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:05.485623    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8753cc9c-d346-441f-ba35-725feb7818d3" containerName="istio-proxy"
Sep 14 05:13:05.485682 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:05.485631    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8753cc9c-d346-441f-ba35-725feb7818d3" containerName="istio-proxy"
Sep 14 05:13:05.485914 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:05.485729    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8753cc9c-d346-441f-ba35-725feb7818d3" containerName="istio-proxy"
Sep 14 05:13:05.488986 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:05.488963    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"
Sep 14 05:13:05.491281 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:05.491262    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-6cb472-openshift-default-dockercfg-zng7v\""
Sep 14 05:13:05.500503 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:05.500478    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"]
Sep 14 05:13:05.588887 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:05.588842    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dada4eda-9779-48ff-a77f-ec7a4cc6852c-istio-podinfo\") pod \"e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr\" (UID: \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\") " pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"
Sep 14 05:13:05.589094 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:05.588915    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dada4eda-9779-48ff-a77f-ec7a4cc6852c-workload-socket\") pod \"e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr\" (UID: \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\") " pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"
Sep 14 05:13:05.589094 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:05.588978    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dada4eda-9779-48ff-a77f-ec7a4cc6852c-istio-data\") pod \"e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr\" (UID: \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\") " pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"
Sep 14 05:13:05.589094 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:05.589065    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dada4eda-9779-48ff-a77f-ec7a4cc6852c-istio-envoy\") pod \"e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr\" (UID: \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\") " pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"
Sep 14 05:13:05.589271 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:05.589109    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dada4eda-9779-48ff-a77f-ec7a4cc6852c-workload-certs\") pod \"e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr\" (UID: \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\") " pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"
Sep 14 05:13:05.589271 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:05.589190    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dada4eda-9779-48ff-a77f-ec7a4cc6852c-istiod-ca-cert\") pod \"e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr\" (UID: \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\") " pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"
Sep 14 05:13:05.589271 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:05.589236    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9s7c\" (UniqueName: \"kubernetes.io/projected/dada4eda-9779-48ff-a77f-ec7a4cc6852c-kube-api-access-l9s7c\") pod \"e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr\" (UID: \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\") " pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"
Sep 14 05:13:05.589402 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:05.589270    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dada4eda-9779-48ff-a77f-ec7a4cc6852c-credential-socket\") pod \"e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr\" (UID: \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\") " pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"
Sep 14 05:13:05.589402 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:05.589293    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dada4eda-9779-48ff-a77f-ec7a4cc6852c-istio-token\") pod \"e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr\" (UID: \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\") " pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"
Sep 14 05:13:05.690558 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:05.690523    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dada4eda-9779-48ff-a77f-ec7a4cc6852c-istio-podinfo\") pod \"e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr\" (UID: \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\") " pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"
Sep 14 05:13:05.690737 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:05.690577    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dada4eda-9779-48ff-a77f-ec7a4cc6852c-workload-socket\") pod \"e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr\" (UID: \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\") " pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"
Sep 14 05:13:05.690737 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:05.690630    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dada4eda-9779-48ff-a77f-ec7a4cc6852c-istio-data\") pod \"e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr\" (UID: \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\") " pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"
Sep 14 05:13:05.690737 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:05.690668    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dada4eda-9779-48ff-a77f-ec7a4cc6852c-istio-envoy\") pod \"e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr\" (UID: \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\") " pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"
Sep 14 05:13:05.690737 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:05.690699    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dada4eda-9779-48ff-a77f-ec7a4cc6852c-workload-certs\") pod \"e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr\" (UID: \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\") " pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"
Sep 14 05:13:05.690737 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:05.690730    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dada4eda-9779-48ff-a77f-ec7a4cc6852c-istiod-ca-cert\") pod \"e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr\" (UID: \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\") " pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"
Sep 14 05:13:05.691050 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:05.690784    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9s7c\" (UniqueName: \"kubernetes.io/projected/dada4eda-9779-48ff-a77f-ec7a4cc6852c-kube-api-access-l9s7c\") pod \"e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr\" (UID: \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\") " pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"
Sep 14 05:13:05.691050 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:05.690819    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dada4eda-9779-48ff-a77f-ec7a4cc6852c-credential-socket\") pod \"e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr\" (UID: \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\") " pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"
Sep 14 05:13:05.691050 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:05.690843    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dada4eda-9779-48ff-a77f-ec7a4cc6852c-istio-token\") pod \"e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr\" (UID: \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\") " pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"
Sep 14 05:13:05.691220 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:05.691148    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dada4eda-9779-48ff-a77f-ec7a4cc6852c-istio-data\") pod \"e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr\" (UID: \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\") " pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"
Sep 14 05:13:05.691220 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:05.691159    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dada4eda-9779-48ff-a77f-ec7a4cc6852c-workload-certs\") pod \"e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr\" (UID: \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\") " pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"
Sep 14 05:13:05.691328 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:05.691277    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dada4eda-9779-48ff-a77f-ec7a4cc6852c-credential-socket\") pod \"e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr\" (UID: \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\") " pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"
Sep 14 05:13:05.691400 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:05.691379    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dada4eda-9779-48ff-a77f-ec7a4cc6852c-workload-socket\") pod \"e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr\" (UID: \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\") " pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"
Sep 14 05:13:05.691609 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:05.691589    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dada4eda-9779-48ff-a77f-ec7a4cc6852c-istiod-ca-cert\") pod \"e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr\" (UID: \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\") " pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"
Sep 14 05:13:05.693467 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:05.693441    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dada4eda-9779-48ff-a77f-ec7a4cc6852c-istio-envoy\") pod \"e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr\" (UID: \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\") " pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"
Sep 14 05:13:05.693670 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:05.693655    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dada4eda-9779-48ff-a77f-ec7a4cc6852c-istio-podinfo\") pod \"e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr\" (UID: \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\") " pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"
Sep 14 05:13:05.699539 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:05.699511    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dada4eda-9779-48ff-a77f-ec7a4cc6852c-istio-token\") pod \"e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr\" (UID: \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\") " pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"
Sep 14 05:13:05.699958 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:05.699932    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9s7c\" (UniqueName: \"kubernetes.io/projected/dada4eda-9779-48ff-a77f-ec7a4cc6852c-kube-api-access-l9s7c\") pod \"e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr\" (UID: \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\") " pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"
Sep 14 05:13:05.800253 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:05.800162    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"
Sep 14 05:13:06.075610 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:06.075578    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"]
Sep 14 05:13:06.077139 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:13:06.077110    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddada4eda_9779_48ff_a77f_ec7a4cc6852c.slice/crio-f917527edf3b9054b4e926c8885c819852dcba1349d467824d942312621094ea WatchSource:0}: Error finding container f917527edf3b9054b4e926c8885c819852dcba1349d467824d942312621094ea: Status 404 returned error can't find the container with id f917527edf3b9054b4e926c8885c819852dcba1349d467824d942312621094ea
Sep 14 05:13:06.079960 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:06.079930    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:13:06.080033 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:06.079990    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:13:06.080033 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:06.080019    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:13:06.261266 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:06.261101    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr" event={"ID":"dada4eda-9779-48ff-a77f-ec7a4cc6852c","Type":"ContainerStarted","Data":"5732d669cb81c2d523f22c47c9d67cf728608862b519ec97c0bd43471b6ce0d4"}
Sep 14 05:13:06.261266 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:06.261210    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr" event={"ID":"dada4eda-9779-48ff-a77f-ec7a4cc6852c","Type":"ContainerStarted","Data":"f917527edf3b9054b4e926c8885c819852dcba1349d467824d942312621094ea"}
Sep 14 05:13:06.289938 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:06.289886    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr" podStartSLOduration=1.289870388 podStartE2EDuration="1.289870388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:13:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:13:06.286091802 +0000 UTC m=+1584.461863358" watchObservedRunningTime="2026-09-14 05:13:06.289870388 +0000 UTC m=+1584.465641947"
Sep 14 05:13:06.800498 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:06.800462    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"
Sep 14 05:13:07.805948 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:07.805921    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"
Sep 14 05:13:08.272847 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:08.272811    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"
Sep 14 05:13:08.273782 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:08.273739    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"
Sep 14 05:13:10.625844 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:10.625810    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:13:11.297910 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:11.297878    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e08402/distinct-2-w4-e08402-kserve-78545b6fcf-8w74w"]
Sep 14 05:13:11.302102 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:11.302073    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e08402/distinct-w4-e08402-kserve-68d4c58cb4-wqjff"]
Sep 14 05:13:11.306806 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:11.306782    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e08402/premium-w4-e08402-kserve-d7dff54c7-fddfc"]
Sep 14 05:13:11.321102 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:11.321078    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e08402/simulator-w4-e08402-kserve-5f5788cc6-cxtfd"]
Sep 14 05:13:12.435012 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:12.434977    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6cb472-74ccccb955-7jkt4"]
Sep 14 05:13:12.435377 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:12.435364    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8753cc9c-d346-441f-ba35-725feb7818d3" containerName="istio-proxy"
Sep 14 05:13:12.438668 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:12.438638    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6cb472-74ccccb955-7jkt4"
Sep 14 05:13:12.440892 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:12.440867    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-6cb472-dockercfg-zmkr5\""
Sep 14 05:13:12.440892 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:12.440887    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-6cb472\""
Sep 14 05:13:12.447345 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:12.447326    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6cb472-74ccccb955-7jkt4"]
Sep 14 05:13:12.559982 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:12.559947    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcb50e9d-dec3-4c96-b8b4-ed5bb9aee247-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6cb472-74ccccb955-7jkt4\" (UID: \"fcb50e9d-dec3-4c96-b8b4-ed5bb9aee247\") " pod="openshift-ingress/payload-processing-e2e-api-a-6cb472-74ccccb955-7jkt4"
Sep 14 05:13:12.560148 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:12.560038    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfdpv\" (UniqueName: \"kubernetes.io/projected/fcb50e9d-dec3-4c96-b8b4-ed5bb9aee247-kube-api-access-bfdpv\") pod \"payload-processing-e2e-api-a-6cb472-74ccccb955-7jkt4\" (UID: \"fcb50e9d-dec3-4c96-b8b4-ed5bb9aee247\") " pod="openshift-ingress/payload-processing-e2e-api-a-6cb472-74ccccb955-7jkt4"
Sep 14 05:13:12.661213 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:12.661177    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfdpv\" (UniqueName: \"kubernetes.io/projected/fcb50e9d-dec3-4c96-b8b4-ed5bb9aee247-kube-api-access-bfdpv\") pod \"payload-processing-e2e-api-a-6cb472-74ccccb955-7jkt4\" (UID: \"fcb50e9d-dec3-4c96-b8b4-ed5bb9aee247\") " pod="openshift-ingress/payload-processing-e2e-api-a-6cb472-74ccccb955-7jkt4"
Sep 14 05:13:12.661391 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:12.661250    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcb50e9d-dec3-4c96-b8b4-ed5bb9aee247-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6cb472-74ccccb955-7jkt4\" (UID: \"fcb50e9d-dec3-4c96-b8b4-ed5bb9aee247\") " pod="openshift-ingress/payload-processing-e2e-api-a-6cb472-74ccccb955-7jkt4"
Sep 14 05:13:12.662000 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:12.661982    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcb50e9d-dec3-4c96-b8b4-ed5bb9aee247-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6cb472-74ccccb955-7jkt4\" (UID: \"fcb50e9d-dec3-4c96-b8b4-ed5bb9aee247\") " pod="openshift-ingress/payload-processing-e2e-api-a-6cb472-74ccccb955-7jkt4"
Sep 14 05:13:12.669215 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:12.669189    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfdpv\" (UniqueName: \"kubernetes.io/projected/fcb50e9d-dec3-4c96-b8b4-ed5bb9aee247-kube-api-access-bfdpv\") pod \"payload-processing-e2e-api-a-6cb472-74ccccb955-7jkt4\" (UID: \"fcb50e9d-dec3-4c96-b8b4-ed5bb9aee247\") " pod="openshift-ingress/payload-processing-e2e-api-a-6cb472-74ccccb955-7jkt4"
Sep 14 05:13:12.750291 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:12.750223    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6cb472-74ccccb955-7jkt4"
Sep 14 05:13:13.027465 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:13.027437    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6cb472-74ccccb955-7jkt4"]
Sep 14 05:13:13.029274 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:13:13.029249    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcb50e9d_dec3_4c96_b8b4_ed5bb9aee247.slice/crio-ef3487618bbcd6b5d78f579bf174c5aee8d979d6a4aaa7aba7305780d3f8d58a WatchSource:0}: Error finding container ef3487618bbcd6b5d78f579bf174c5aee8d979d6a4aaa7aba7305780d3f8d58a: Status 404 returned error can't find the container with id ef3487618bbcd6b5d78f579bf174c5aee8d979d6a4aaa7aba7305780d3f8d58a
Sep 14 05:13:13.305288 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:13.305191    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6cb472-74ccccb955-7jkt4" event={"ID":"fcb50e9d-dec3-4c96-b8b4-ed5bb9aee247","Type":"ContainerStarted","Data":"590d7a9c35325ff485225af806d140b6e12f9278ca2c1a9c6fd1c1555111d52a"}
Sep 14 05:13:13.305288 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:13.305252    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6cb472-74ccccb955-7jkt4" event={"ID":"fcb50e9d-dec3-4c96-b8b4-ed5bb9aee247","Type":"ContainerStarted","Data":"ef3487618bbcd6b5d78f579bf174c5aee8d979d6a4aaa7aba7305780d3f8d58a"}
Sep 14 05:13:13.305288 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:13.305283    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-6cb472-74ccccb955-7jkt4"
Sep 14 05:13:13.323026 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:13.322972    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-6cb472-74ccccb955-7jkt4" podStartSLOduration=1.322958446 podStartE2EDuration="1.322958446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:13:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:13:13.32114629 +0000 UTC m=+1591.496917840" watchObservedRunningTime="2026-09-14 05:13:13.322958446 +0000 UTC m=+1591.498730056"
Sep 14 05:13:24.314921 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:24.314886    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-6cb472-74ccccb955-7jkt4"
Sep 14 05:13:26.318669 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.318609    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-e08402/distinct-2-w4-e08402-kserve-78545b6fcf-8w74w" podUID="5ea11a2f-42c5-4e7f-988e-b980d3803291" containerName="main" containerID="cri-o://10037950990b968a94604d5279011762813fc0df7ac1312ed55446365a46d804" gracePeriod=45
Sep 14 05:13:26.328243 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.328181    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-e08402/distinct-w4-e08402-kserve-68d4c58cb4-wqjff" podUID="c3d361b8-037d-4d5d-951c-f954c7d6a31c" containerName="main" containerID="cri-o://4418ba2cb28b9aa07603aede7e0f2a658f9c68af3712575bcc74ed422ae6cd33" gracePeriod=45
Sep 14 05:13:26.333154 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.333119    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-e08402/premium-w4-e08402-kserve-d7dff54c7-fddfc" podUID="10e39a9c-3c5c-4e22-a788-a09735193dcd" containerName="main" containerID="cri-o://8a62d9d2e0bc976d488a655872314e3614404a8d9ba2f927b0df5da6b0ca6ed6" gracePeriod=45
Sep 14 05:13:26.338725 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.338692    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-e08402/simulator-w4-e08402-kserve-5f5788cc6-cxtfd" podUID="d925517d-719f-41bb-ae99-09e538af6104" containerName="main" containerID="cri-o://168b33a1b881ffafadb62f50332bbf0975d67d698732ff74b81e43ddc8ed7091" gracePeriod=45
Sep 14 05:13:26.529570 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.529527    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e08402/distinct-2-w4-e08402-kserve-78545b6fcf-8w74w" podUID="5ea11a2f-42c5-4e7f-988e-b980d3803291" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": dial tcp 10.133.0.78:8000: connect: connection refused"
Sep 14 05:13:26.615528 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.615502    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e08402/simulator-w4-e08402-kserve-5f5788cc6-cxtfd"
Sep 14 05:13:26.644664 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.644605    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e08402/distinct-w4-e08402-kserve-68d4c58cb4-wqjff"
Sep 14 05:13:26.658010 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.657988    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e08402/premium-w4-e08402-kserve-d7dff54c7-fddfc"
Sep 14 05:13:26.670223 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.670196    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e08402/distinct-2-w4-e08402-kserve-78545b6fcf-8w74w"
Sep 14 05:13:26.691626 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.691600    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d925517d-719f-41bb-ae99-09e538af6104-tmp-dir\") pod \"d925517d-719f-41bb-ae99-09e538af6104\" (UID: \"d925517d-719f-41bb-ae99-09e538af6104\") "
Sep 14 05:13:26.691781 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.691639    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbbkh\" (UniqueName: \"kubernetes.io/projected/c3d361b8-037d-4d5d-951c-f954c7d6a31c-kube-api-access-vbbkh\") pod \"c3d361b8-037d-4d5d-951c-f954c7d6a31c\" (UID: \"c3d361b8-037d-4d5d-951c-f954c7d6a31c\") "
Sep 14 05:13:26.691781 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.691662    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3d361b8-037d-4d5d-951c-f954c7d6a31c-model-cache\") pod \"c3d361b8-037d-4d5d-951c-f954c7d6a31c\" (UID: \"c3d361b8-037d-4d5d-951c-f954c7d6a31c\") "
Sep 14 05:13:26.691781 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.691737    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3d361b8-037d-4d5d-951c-f954c7d6a31c-tmp-dir\") pod \"c3d361b8-037d-4d5d-951c-f954c7d6a31c\" (UID: \"c3d361b8-037d-4d5d-951c-f954c7d6a31c\") "
Sep 14 05:13:26.691973 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.691811    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10e39a9c-3c5c-4e22-a788-a09735193dcd-model-cache\") pod \"10e39a9c-3c5c-4e22-a788-a09735193dcd\" (UID: \"10e39a9c-3c5c-4e22-a788-a09735193dcd\") "
Sep 14 05:13:26.691973 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.691855    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d925517d-719f-41bb-ae99-09e538af6104-home\") pod \"d925517d-719f-41bb-ae99-09e538af6104\" (UID: \"d925517d-719f-41bb-ae99-09e538af6104\") "
Sep 14 05:13:26.691973 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.691908    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3d361b8-037d-4d5d-951c-f954c7d6a31c-tls-certs\") pod \"c3d361b8-037d-4d5d-951c-f954c7d6a31c\" (UID: \"c3d361b8-037d-4d5d-951c-f954c7d6a31c\") "
Sep 14 05:13:26.691973 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.691902    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d925517d-719f-41bb-ae99-09e538af6104-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d925517d-719f-41bb-ae99-09e538af6104" (UID: "d925517d-719f-41bb-ae99-09e538af6104"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:13:26.691973 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.691932    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3d361b8-037d-4d5d-951c-f954c7d6a31c-home\") pod \"c3d361b8-037d-4d5d-951c-f954c7d6a31c\" (UID: \"c3d361b8-037d-4d5d-951c-f954c7d6a31c\") "
Sep 14 05:13:26.691973 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.691957    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3d361b8-037d-4d5d-951c-f954c7d6a31c-dshm\") pod \"c3d361b8-037d-4d5d-951c-f954c7d6a31c\" (UID: \"c3d361b8-037d-4d5d-951c-f954c7d6a31c\") "
Sep 14 05:13:26.692292 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.692000    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d925517d-719f-41bb-ae99-09e538af6104-dshm\") pod \"d925517d-719f-41bb-ae99-09e538af6104\" (UID: \"d925517d-719f-41bb-ae99-09e538af6104\") "
Sep 14 05:13:26.692292 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.692042    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d925517d-719f-41bb-ae99-09e538af6104-model-cache\") pod \"d925517d-719f-41bb-ae99-09e538af6104\" (UID: \"d925517d-719f-41bb-ae99-09e538af6104\") "
Sep 14 05:13:26.692292 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.692079    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10e39a9c-3c5c-4e22-a788-a09735193dcd-home\") pod \"10e39a9c-3c5c-4e22-a788-a09735193dcd\" (UID: \"10e39a9c-3c5c-4e22-a788-a09735193dcd\") "
Sep 14 05:13:26.692292 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.692137    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqrhl\" (UniqueName: \"kubernetes.io/projected/d925517d-719f-41bb-ae99-09e538af6104-kube-api-access-nqrhl\") pod \"d925517d-719f-41bb-ae99-09e538af6104\" (UID: \"d925517d-719f-41bb-ae99-09e538af6104\") "
Sep 14 05:13:26.692292 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.692154    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d925517d-719f-41bb-ae99-09e538af6104-home" (OuterVolumeSpecName: "home") pod "d925517d-719f-41bb-ae99-09e538af6104" (UID: "d925517d-719f-41bb-ae99-09e538af6104"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:13:26.692292 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.692193    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10e39a9c-3c5c-4e22-a788-a09735193dcd-dshm\") pod \"10e39a9c-3c5c-4e22-a788-a09735193dcd\" (UID: \"10e39a9c-3c5c-4e22-a788-a09735193dcd\") "
Sep 14 05:13:26.692292 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.692224    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmxkj\" (UniqueName: \"kubernetes.io/projected/10e39a9c-3c5c-4e22-a788-a09735193dcd-kube-api-access-gmxkj\") pod \"10e39a9c-3c5c-4e22-a788-a09735193dcd\" (UID: \"10e39a9c-3c5c-4e22-a788-a09735193dcd\") "
Sep 14 05:13:26.692292 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.692287    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10e39a9c-3c5c-4e22-a788-a09735193dcd-tmp-dir\") pod \"10e39a9c-3c5c-4e22-a788-a09735193dcd\" (UID: \"10e39a9c-3c5c-4e22-a788-a09735193dcd\") "
Sep 14 05:13:26.692703 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.692386    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10e39a9c-3c5c-4e22-a788-a09735193dcd-tls-certs\") pod \"10e39a9c-3c5c-4e22-a788-a09735193dcd\" (UID: \"10e39a9c-3c5c-4e22-a788-a09735193dcd\") "
Sep 14 05:13:26.692703 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.692437    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d925517d-719f-41bb-ae99-09e538af6104-tls-certs\") pod \"d925517d-719f-41bb-ae99-09e538af6104\" (UID: \"d925517d-719f-41bb-ae99-09e538af6104\") "
Sep 14 05:13:26.692703 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.692521    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10e39a9c-3c5c-4e22-a788-a09735193dcd-model-cache" (OuterVolumeSpecName: "model-cache") pod "10e39a9c-3c5c-4e22-a788-a09735193dcd" (UID: "10e39a9c-3c5c-4e22-a788-a09735193dcd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:13:26.693004 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.692982    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10e39a9c-3c5c-4e22-a788-a09735193dcd-model-cache\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:13:26.693382 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.693006    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d925517d-719f-41bb-ae99-09e538af6104-home\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:13:26.693382 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.693019    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d361b8-037d-4d5d-951c-f954c7d6a31c-model-cache" (OuterVolumeSpecName: "model-cache") pod "c3d361b8-037d-4d5d-951c-f954c7d6a31c" (UID: "c3d361b8-037d-4d5d-951c-f954c7d6a31c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:13:26.695031 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.693395    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d925517d-719f-41bb-ae99-09e538af6104-model-cache" (OuterVolumeSpecName: "model-cache") pod "d925517d-719f-41bb-ae99-09e538af6104" (UID: "d925517d-719f-41bb-ae99-09e538af6104"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:13:26.695031 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.693571    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d361b8-037d-4d5d-951c-f954c7d6a31c-home" (OuterVolumeSpecName: "home") pod "c3d361b8-037d-4d5d-951c-f954c7d6a31c" (UID: "c3d361b8-037d-4d5d-951c-f954c7d6a31c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:13:26.695031 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.693697    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d361b8-037d-4d5d-951c-f954c7d6a31c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c3d361b8-037d-4d5d-951c-f954c7d6a31c" (UID: "c3d361b8-037d-4d5d-951c-f954c7d6a31c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:13:26.695031 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.694560    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10e39a9c-3c5c-4e22-a788-a09735193dcd-home" (OuterVolumeSpecName: "home") pod "10e39a9c-3c5c-4e22-a788-a09735193dcd" (UID: "10e39a9c-3c5c-4e22-a788-a09735193dcd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:13:26.695031 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.694703    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10e39a9c-3c5c-4e22-a788-a09735193dcd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "10e39a9c-3c5c-4e22-a788-a09735193dcd" (UID: "10e39a9c-3c5c-4e22-a788-a09735193dcd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:13:26.695031 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.694807    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3d361b8-037d-4d5d-951c-f954c7d6a31c-kube-api-access-vbbkh" (OuterVolumeSpecName: "kube-api-access-vbbkh") pod "c3d361b8-037d-4d5d-951c-f954c7d6a31c" (UID: "c3d361b8-037d-4d5d-951c-f954c7d6a31c"). InnerVolumeSpecName "kube-api-access-vbbkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:13:26.696495 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.693861    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d925517d-719f-41bb-ae99-09e538af6104-tmp-dir\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:13:26.697903 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.697872    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3d361b8-037d-4d5d-951c-f954c7d6a31c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c3d361b8-037d-4d5d-951c-f954c7d6a31c" (UID: "c3d361b8-037d-4d5d-951c-f954c7d6a31c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 05:13:26.698370 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.698333    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d361b8-037d-4d5d-951c-f954c7d6a31c-dshm" (OuterVolumeSpecName: "dshm") pod "c3d361b8-037d-4d5d-951c-f954c7d6a31c" (UID: "c3d361b8-037d-4d5d-951c-f954c7d6a31c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:13:26.698570 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.698536    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10e39a9c-3c5c-4e22-a788-a09735193dcd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "10e39a9c-3c5c-4e22-a788-a09735193dcd" (UID: "10e39a9c-3c5c-4e22-a788-a09735193dcd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 05:13:26.698723 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.698697    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d925517d-719f-41bb-ae99-09e538af6104-kube-api-access-nqrhl" (OuterVolumeSpecName: "kube-api-access-nqrhl") pod "d925517d-719f-41bb-ae99-09e538af6104" (UID: "d925517d-719f-41bb-ae99-09e538af6104"). InnerVolumeSpecName "kube-api-access-nqrhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:13:26.699201 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.699167    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10e39a9c-3c5c-4e22-a788-a09735193dcd-dshm" (OuterVolumeSpecName: "dshm") pod "10e39a9c-3c5c-4e22-a788-a09735193dcd" (UID: "10e39a9c-3c5c-4e22-a788-a09735193dcd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:13:26.699313 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.699294    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d925517d-719f-41bb-ae99-09e538af6104-dshm" (OuterVolumeSpecName: "dshm") pod "d925517d-719f-41bb-ae99-09e538af6104" (UID: "d925517d-719f-41bb-ae99-09e538af6104"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:13:26.699470 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.699443    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10e39a9c-3c5c-4e22-a788-a09735193dcd-kube-api-access-gmxkj" (OuterVolumeSpecName: "kube-api-access-gmxkj") pod "10e39a9c-3c5c-4e22-a788-a09735193dcd" (UID: "10e39a9c-3c5c-4e22-a788-a09735193dcd"). InnerVolumeSpecName "kube-api-access-gmxkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:13:26.700418 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.700397    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d925517d-719f-41bb-ae99-09e538af6104-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d925517d-719f-41bb-ae99-09e538af6104" (UID: "d925517d-719f-41bb-ae99-09e538af6104"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 05:13:26.795874 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.795839    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ea11a2f-42c5-4e7f-988e-b980d3803291-home\") pod \"5ea11a2f-42c5-4e7f-988e-b980d3803291\" (UID: \"5ea11a2f-42c5-4e7f-988e-b980d3803291\") "
Sep 14 05:13:26.796058 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.795895    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ea11a2f-42c5-4e7f-988e-b980d3803291-tmp-dir\") pod \"5ea11a2f-42c5-4e7f-988e-b980d3803291\" (UID: \"5ea11a2f-42c5-4e7f-988e-b980d3803291\") "
Sep 14 05:13:26.796058 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.795917    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ea11a2f-42c5-4e7f-988e-b980d3803291-dshm\") pod \"5ea11a2f-42c5-4e7f-988e-b980d3803291\" (UID: \"5ea11a2f-42c5-4e7f-988e-b980d3803291\") "
Sep 14 05:13:26.796058 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.795953    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5h2g\" (UniqueName: \"kubernetes.io/projected/5ea11a2f-42c5-4e7f-988e-b980d3803291-kube-api-access-z5h2g\") pod \"5ea11a2f-42c5-4e7f-988e-b980d3803291\" (UID: \"5ea11a2f-42c5-4e7f-988e-b980d3803291\") "
Sep 14 05:13:26.796058 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.795992    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ea11a2f-42c5-4e7f-988e-b980d3803291-tls-certs\") pod \"5ea11a2f-42c5-4e7f-988e-b980d3803291\" (UID: \"5ea11a2f-42c5-4e7f-988e-b980d3803291\") "
Sep 14 05:13:26.796275 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.796110    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ea11a2f-42c5-4e7f-988e-b980d3803291-model-cache\") pod \"5ea11a2f-42c5-4e7f-988e-b980d3803291\" (UID: \"5ea11a2f-42c5-4e7f-988e-b980d3803291\") "
Sep 14 05:13:26.796275 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.796157    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ea11a2f-42c5-4e7f-988e-b980d3803291-home" (OuterVolumeSpecName: "home") pod "5ea11a2f-42c5-4e7f-988e-b980d3803291" (UID: "5ea11a2f-42c5-4e7f-988e-b980d3803291"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:13:26.796275 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.796172    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ea11a2f-42c5-4e7f-988e-b980d3803291-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5ea11a2f-42c5-4e7f-988e-b980d3803291" (UID: "5ea11a2f-42c5-4e7f-988e-b980d3803291"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:13:26.796439 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.796420    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ea11a2f-42c5-4e7f-988e-b980d3803291-model-cache" (OuterVolumeSpecName: "model-cache") pod "5ea11a2f-42c5-4e7f-988e-b980d3803291" (UID: "5ea11a2f-42c5-4e7f-988e-b980d3803291"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:13:26.796439 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.796430    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqrhl\" (UniqueName: \"kubernetes.io/projected/d925517d-719f-41bb-ae99-09e538af6104-kube-api-access-nqrhl\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:13:26.796549 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.796451    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10e39a9c-3c5c-4e22-a788-a09735193dcd-dshm\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:13:26.796549 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.796467    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmxkj\" (UniqueName: \"kubernetes.io/projected/10e39a9c-3c5c-4e22-a788-a09735193dcd-kube-api-access-gmxkj\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:13:26.796549 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.796482    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10e39a9c-3c5c-4e22-a788-a09735193dcd-tmp-dir\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:13:26.796549 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.796496    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10e39a9c-3c5c-4e22-a788-a09735193dcd-tls-certs\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:13:26.796549 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.796507    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d925517d-719f-41bb-ae99-09e538af6104-tls-certs\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:13:26.796549 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.796519    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ea11a2f-42c5-4e7f-988e-b980d3803291-home\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:13:26.796549 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.796531    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbbkh\" (UniqueName: \"kubernetes.io/projected/c3d361b8-037d-4d5d-951c-f954c7d6a31c-kube-api-access-vbbkh\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:13:26.796549 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.796546    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3d361b8-037d-4d5d-951c-f954c7d6a31c-model-cache\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:13:26.796967 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.796558    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3d361b8-037d-4d5d-951c-f954c7d6a31c-tmp-dir\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:13:26.796967 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.796566    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ea11a2f-42c5-4e7f-988e-b980d3803291-tmp-dir\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:13:26.796967 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.796574    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3d361b8-037d-4d5d-951c-f954c7d6a31c-tls-certs\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:13:26.796967 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.796582    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3d361b8-037d-4d5d-951c-f954c7d6a31c-home\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:13:26.796967 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.796589    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3d361b8-037d-4d5d-951c-f954c7d6a31c-dshm\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:13:26.796967 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.796597    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d925517d-719f-41bb-ae99-09e538af6104-dshm\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:13:26.796967 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.796605    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d925517d-719f-41bb-ae99-09e538af6104-model-cache\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:13:26.796967 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.796614    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10e39a9c-3c5c-4e22-a788-a09735193dcd-home\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:13:26.798419 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.798386    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ea11a2f-42c5-4e7f-988e-b980d3803291-dshm" (OuterVolumeSpecName: "dshm") pod "5ea11a2f-42c5-4e7f-988e-b980d3803291" (UID: "5ea11a2f-42c5-4e7f-988e-b980d3803291"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:13:26.798530 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.798441    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ea11a2f-42c5-4e7f-988e-b980d3803291-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5ea11a2f-42c5-4e7f-988e-b980d3803291" (UID: "5ea11a2f-42c5-4e7f-988e-b980d3803291"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 05:13:26.798530 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.798478    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ea11a2f-42c5-4e7f-988e-b980d3803291-kube-api-access-z5h2g" (OuterVolumeSpecName: "kube-api-access-z5h2g") pod "5ea11a2f-42c5-4e7f-988e-b980d3803291" (UID: "5ea11a2f-42c5-4e7f-988e-b980d3803291"). InnerVolumeSpecName "kube-api-access-z5h2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:13:26.897356 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.897272    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5h2g\" (UniqueName: \"kubernetes.io/projected/5ea11a2f-42c5-4e7f-988e-b980d3803291-kube-api-access-z5h2g\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:13:26.897356 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.897300    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ea11a2f-42c5-4e7f-988e-b980d3803291-tls-certs\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:13:26.897356 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.897312    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ea11a2f-42c5-4e7f-988e-b980d3803291-model-cache\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:13:26.897356 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:26.897320    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ea11a2f-42c5-4e7f-988e-b980d3803291-dshm\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:13:27.384277 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:27.384229    2565 generic.go:356] "Generic (PLEG): container finished" podID="c3d361b8-037d-4d5d-951c-f954c7d6a31c" containerID="4418ba2cb28b9aa07603aede7e0f2a658f9c68af3712575bcc74ed422ae6cd33" exitCode=0
Sep 14 05:13:27.384277 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:27.384281    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e08402/distinct-w4-e08402-kserve-68d4c58cb4-wqjff"
Sep 14 05:13:27.384793 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:27.384322    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e08402/distinct-w4-e08402-kserve-68d4c58cb4-wqjff" event={"ID":"c3d361b8-037d-4d5d-951c-f954c7d6a31c","Type":"ContainerDied","Data":"4418ba2cb28b9aa07603aede7e0f2a658f9c68af3712575bcc74ed422ae6cd33"}
Sep 14 05:13:27.384793 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:27.384374    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e08402/distinct-w4-e08402-kserve-68d4c58cb4-wqjff" event={"ID":"c3d361b8-037d-4d5d-951c-f954c7d6a31c","Type":"ContainerDied","Data":"95c0b25ca59b954d668b8b0be820fb4e60e11c8e7de52ada0bb6bfdd7f71d251"}
Sep 14 05:13:27.384793 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:27.384399    2565 scope.go:117] "RemoveContainer" containerID="4418ba2cb28b9aa07603aede7e0f2a658f9c68af3712575bcc74ed422ae6cd33"
Sep 14 05:13:27.385749 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:27.385726    2565 generic.go:356] "Generic (PLEG): container finished" podID="10e39a9c-3c5c-4e22-a788-a09735193dcd" containerID="8a62d9d2e0bc976d488a655872314e3614404a8d9ba2f927b0df5da6b0ca6ed6" exitCode=0
Sep 14 05:13:27.385838 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:27.385797    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e08402/premium-w4-e08402-kserve-d7dff54c7-fddfc"
Sep 14 05:13:27.385838 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:27.385816    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e08402/premium-w4-e08402-kserve-d7dff54c7-fddfc" event={"ID":"10e39a9c-3c5c-4e22-a788-a09735193dcd","Type":"ContainerDied","Data":"8a62d9d2e0bc976d488a655872314e3614404a8d9ba2f927b0df5da6b0ca6ed6"}
Sep 14 05:13:27.385944 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:27.385857    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e08402/premium-w4-e08402-kserve-d7dff54c7-fddfc" event={"ID":"10e39a9c-3c5c-4e22-a788-a09735193dcd","Type":"ContainerDied","Data":"fcb19d063a2bd6fe8ffc7d0b706b4763563158509f006038496a1bcfb1c7a004"}
Sep 14 05:13:27.391076 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:27.391048    2565 generic.go:356] "Generic (PLEG): container finished" podID="d925517d-719f-41bb-ae99-09e538af6104" containerID="168b33a1b881ffafadb62f50332bbf0975d67d698732ff74b81e43ddc8ed7091" exitCode=0
Sep 14 05:13:27.391211 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:27.391193    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e08402/simulator-w4-e08402-kserve-5f5788cc6-cxtfd"
Sep 14 05:13:27.391270 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:27.391251    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e08402/simulator-w4-e08402-kserve-5f5788cc6-cxtfd" event={"ID":"d925517d-719f-41bb-ae99-09e538af6104","Type":"ContainerDied","Data":"168b33a1b881ffafadb62f50332bbf0975d67d698732ff74b81e43ddc8ed7091"}
Sep 14 05:13:27.391325 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:27.391281    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e08402/simulator-w4-e08402-kserve-5f5788cc6-cxtfd" event={"ID":"d925517d-719f-41bb-ae99-09e538af6104","Type":"ContainerDied","Data":"edce1b8c01c0d656afb2761e0e8fc62d77c90e611a7f904c0269ff45027cc20e"}
Sep 14 05:13:27.396353 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:27.396330    2565 scope.go:117] "RemoveContainer" containerID="4418ba2cb28b9aa07603aede7e0f2a658f9c68af3712575bcc74ed422ae6cd33"
Sep 14 05:13:27.396682 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:13:27.396655    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4418ba2cb28b9aa07603aede7e0f2a658f9c68af3712575bcc74ed422ae6cd33\": container with ID starting with 4418ba2cb28b9aa07603aede7e0f2a658f9c68af3712575bcc74ed422ae6cd33 not found: ID does not exist" containerID="4418ba2cb28b9aa07603aede7e0f2a658f9c68af3712575bcc74ed422ae6cd33"
Sep 14 05:13:27.396799 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:27.396695    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4418ba2cb28b9aa07603aede7e0f2a658f9c68af3712575bcc74ed422ae6cd33"} err="failed to get container status \"4418ba2cb28b9aa07603aede7e0f2a658f9c68af3712575bcc74ed422ae6cd33\": rpc error: code = NotFound desc = could not find container \"4418ba2cb28b9aa07603aede7e0f2a658f9c68af3712575bcc74ed422ae6cd33\": container with ID starting with 4418ba2cb28b9aa07603aede7e0f2a658f9c68af3712575bcc74ed422ae6cd33 not found: ID does not exist"
Sep 14 05:13:27.396799 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:27.396717    2565 scope.go:117] "RemoveContainer" containerID="8a62d9d2e0bc976d488a655872314e3614404a8d9ba2f927b0df5da6b0ca6ed6"
Sep 14 05:13:27.397156 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:27.397137    2565 generic.go:356] "Generic (PLEG): container finished" podID="5ea11a2f-42c5-4e7f-988e-b980d3803291" containerID="10037950990b968a94604d5279011762813fc0df7ac1312ed55446365a46d804" exitCode=0
Sep 14 05:13:27.397263 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:27.397172    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e08402/distinct-2-w4-e08402-kserve-78545b6fcf-8w74w" event={"ID":"5ea11a2f-42c5-4e7f-988e-b980d3803291","Type":"ContainerDied","Data":"10037950990b968a94604d5279011762813fc0df7ac1312ed55446365a46d804"}
Sep 14 05:13:27.397263 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:27.397194    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e08402/distinct-2-w4-e08402-kserve-78545b6fcf-8w74w"
Sep 14 05:13:27.397263 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:27.397201    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e08402/distinct-2-w4-e08402-kserve-78545b6fcf-8w74w" event={"ID":"5ea11a2f-42c5-4e7f-988e-b980d3803291","Type":"ContainerDied","Data":"147ff2e0350944300534dba6b1a2b8be6481afb8c4d99dff56b200b97497371e"}
Sep 14 05:13:27.406913 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:27.406877    2565 scope.go:117] "RemoveContainer" containerID="8a62d9d2e0bc976d488a655872314e3614404a8d9ba2f927b0df5da6b0ca6ed6"
Sep 14 05:13:27.407254 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:13:27.407227    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a62d9d2e0bc976d488a655872314e3614404a8d9ba2f927b0df5da6b0ca6ed6\": container with ID starting with 8a62d9d2e0bc976d488a655872314e3614404a8d9ba2f927b0df5da6b0ca6ed6 not found: ID does not exist" containerID="8a62d9d2e0bc976d488a655872314e3614404a8d9ba2f927b0df5da6b0ca6ed6"
Sep 14 05:13:27.407326 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:27.407263    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a62d9d2e0bc976d488a655872314e3614404a8d9ba2f927b0df5da6b0ca6ed6"} err="failed to get container status \"8a62d9d2e0bc976d488a655872314e3614404a8d9ba2f927b0df5da6b0ca6ed6\": rpc error: code = NotFound desc = could not find container \"8a62d9d2e0bc976d488a655872314e3614404a8d9ba2f927b0df5da6b0ca6ed6\": container with ID starting with 8a62d9d2e0bc976d488a655872314e3614404a8d9ba2f927b0df5da6b0ca6ed6 not found: ID does not exist"
Sep 14 05:13:27.407326 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:27.407278    2565 scope.go:117] "RemoveContainer" containerID="168b33a1b881ffafadb62f50332bbf0975d67d698732ff74b81e43ddc8ed7091"
Sep 14 05:13:27.415424 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:27.415409    2565 scope.go:117] "RemoveContainer" containerID="168b33a1b881ffafadb62f50332bbf0975d67d698732ff74b81e43ddc8ed7091"
Sep 14 05:13:27.415688 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:13:27.415660    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"168b33a1b881ffafadb62f50332bbf0975d67d698732ff74b81e43ddc8ed7091\": container with ID starting with 168b33a1b881ffafadb62f50332bbf0975d67d698732ff74b81e43ddc8ed7091 not found: ID does not exist" containerID="168b33a1b881ffafadb62f50332bbf0975d67d698732ff74b81e43ddc8ed7091"
Sep 14 05:13:27.415791 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:27.415691    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"168b33a1b881ffafadb62f50332bbf0975d67d698732ff74b81e43ddc8ed7091"} err="failed to get container status \"168b33a1b881ffafadb62f50332bbf0975d67d698732ff74b81e43ddc8ed7091\": rpc error: code = NotFound desc = could not find container \"168b33a1b881ffafadb62f50332bbf0975d67d698732ff74b81e43ddc8ed7091\": container with ID starting with 168b33a1b881ffafadb62f50332bbf0975d67d698732ff74b81e43ddc8ed7091 not found: ID does not exist"
Sep 14 05:13:27.415791 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:27.415707    2565 scope.go:117] "RemoveContainer" containerID="10037950990b968a94604d5279011762813fc0df7ac1312ed55446365a46d804"
Sep 14 05:13:27.423494 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:27.423478    2565 scope.go:117] "RemoveContainer" containerID="10037950990b968a94604d5279011762813fc0df7ac1312ed55446365a46d804"
Sep 14 05:13:27.423788 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:13:27.423770    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10037950990b968a94604d5279011762813fc0df7ac1312ed55446365a46d804\": container with ID starting with 10037950990b968a94604d5279011762813fc0df7ac1312ed55446365a46d804 not found: ID does not exist" containerID="10037950990b968a94604d5279011762813fc0df7ac1312ed55446365a46d804"
Sep 14 05:13:27.423869 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:27.423795    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10037950990b968a94604d5279011762813fc0df7ac1312ed55446365a46d804"} err="failed to get container status \"10037950990b968a94604d5279011762813fc0df7ac1312ed55446365a46d804\": rpc error: code = NotFound desc = could not find container \"10037950990b968a94604d5279011762813fc0df7ac1312ed55446365a46d804\": container with ID starting with 10037950990b968a94604d5279011762813fc0df7ac1312ed55446365a46d804 not found: ID does not exist"
Sep 14 05:13:27.428606 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:27.428584    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e08402/distinct-w4-e08402-kserve-68d4c58cb4-wqjff"]
Sep 14 05:13:27.434483 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:27.434461    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-e08402/distinct-w4-e08402-kserve-68d4c58cb4-wqjff"]
Sep 14 05:13:27.443976 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:27.443952    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e08402/simulator-w4-e08402-kserve-5f5788cc6-cxtfd"]
Sep 14 05:13:27.446345 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:27.446325    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-e08402/simulator-w4-e08402-kserve-5f5788cc6-cxtfd"]
Sep 14 05:13:27.453712 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:27.453690    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e08402/premium-w4-e08402-kserve-d7dff54c7-fddfc"]
Sep 14 05:13:27.457331 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:27.457312    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-e08402/premium-w4-e08402-kserve-d7dff54c7-fddfc"]
Sep 14 05:13:27.467037 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:27.467004    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e08402/distinct-2-w4-e08402-kserve-78545b6fcf-8w74w"]
Sep 14 05:13:27.472130 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:27.472105    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-e08402/distinct-2-w4-e08402-kserve-78545b6fcf-8w74w"]
Sep 14 05:13:28.334020 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:28.333980    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10e39a9c-3c5c-4e22-a788-a09735193dcd" path="/var/lib/kubelet/pods/10e39a9c-3c5c-4e22-a788-a09735193dcd/volumes"
Sep 14 05:13:28.334479 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:28.334456    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ea11a2f-42c5-4e7f-988e-b980d3803291" path="/var/lib/kubelet/pods/5ea11a2f-42c5-4e7f-988e-b980d3803291/volumes"
Sep 14 05:13:28.334938 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:28.334918    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3d361b8-037d-4d5d-951c-f954c7d6a31c" path="/var/lib/kubelet/pods/c3d361b8-037d-4d5d-951c-f954c7d6a31c/volumes"
Sep 14 05:13:28.335635 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:28.335613    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d925517d-719f-41bb-ae99-09e538af6104" path="/var/lib/kubelet/pods/d925517d-719f-41bb-ae99-09e538af6104/volumes"
Sep 14 05:13:28.889720 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:28.889686    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"]
Sep 14 05:13:28.890781 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:28.890725    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ea11a2f-42c5-4e7f-988e-b980d3803291" containerName="main"
Sep 14 05:13:28.890781 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:28.890767    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ea11a2f-42c5-4e7f-988e-b980d3803291" containerName="main"
Sep 14 05:13:28.890967 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:28.890790    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3d361b8-037d-4d5d-951c-f954c7d6a31c" containerName="main"
Sep 14 05:13:28.890967 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:28.890799    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3d361b8-037d-4d5d-951c-f954c7d6a31c" containerName="main"
Sep 14 05:13:28.890967 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:28.890823    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d925517d-719f-41bb-ae99-09e538af6104" containerName="main"
Sep 14 05:13:28.890967 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:28.890831    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d925517d-719f-41bb-ae99-09e538af6104" containerName="main"
Sep 14 05:13:28.890967 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:28.890847    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10e39a9c-3c5c-4e22-a788-a09735193dcd" containerName="main"
Sep 14 05:13:28.890967 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:28.890855    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="10e39a9c-3c5c-4e22-a788-a09735193dcd" containerName="main"
Sep 14 05:13:28.890967 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:28.890944    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="10e39a9c-3c5c-4e22-a788-a09735193dcd" containerName="main"
Sep 14 05:13:28.890967 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:28.890957    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3d361b8-037d-4d5d-951c-f954c7d6a31c" containerName="main"
Sep 14 05:13:28.890967 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:28.890967    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d925517d-719f-41bb-ae99-09e538af6104" containerName="main"
Sep 14 05:13:28.891426 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:28.890977    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ea11a2f-42c5-4e7f-988e-b980d3803291" containerName="main"
Sep 14 05:13:28.895987 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:28.895967    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"
Sep 14 05:13:28.898225 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:28.898206    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-09203c-openshift-default-dockercfg-hqxl9\""
Sep 14 05:13:28.903310 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:28.903058    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"]
Sep 14 05:13:29.015867 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:29.015836    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdb4e455-7001-4113-bd1d-16894513c9da-istio-token\") pod \"e2e-api-b-09203c-openshift-default-6d94744596-vzrnv\" (UID: \"cdb4e455-7001-4113-bd1d-16894513c9da\") " pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"
Sep 14 05:13:29.016027 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:29.015871    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdb4e455-7001-4113-bd1d-16894513c9da-workload-socket\") pod \"e2e-api-b-09203c-openshift-default-6d94744596-vzrnv\" (UID: \"cdb4e455-7001-4113-bd1d-16894513c9da\") " pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"
Sep 14 05:13:29.016027 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:29.015897    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdb4e455-7001-4113-bd1d-16894513c9da-istio-podinfo\") pod \"e2e-api-b-09203c-openshift-default-6d94744596-vzrnv\" (UID: \"cdb4e455-7001-4113-bd1d-16894513c9da\") " pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"
Sep 14 05:13:29.016027 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:29.015925    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdb4e455-7001-4113-bd1d-16894513c9da-istiod-ca-cert\") pod \"e2e-api-b-09203c-openshift-default-6d94744596-vzrnv\" (UID: \"cdb4e455-7001-4113-bd1d-16894513c9da\") " pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"
Sep 14 05:13:29.016027 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:29.015988    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdb4e455-7001-4113-bd1d-16894513c9da-credential-socket\") pod \"e2e-api-b-09203c-openshift-default-6d94744596-vzrnv\" (UID: \"cdb4e455-7001-4113-bd1d-16894513c9da\") " pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"
Sep 14 05:13:29.016027 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:29.016011    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdb4e455-7001-4113-bd1d-16894513c9da-istio-data\") pod \"e2e-api-b-09203c-openshift-default-6d94744596-vzrnv\" (UID: \"cdb4e455-7001-4113-bd1d-16894513c9da\") " pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"
Sep 14 05:13:29.016226 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:29.016040    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdb4e455-7001-4113-bd1d-16894513c9da-istio-envoy\") pod \"e2e-api-b-09203c-openshift-default-6d94744596-vzrnv\" (UID: \"cdb4e455-7001-4113-bd1d-16894513c9da\") " pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"
Sep 14 05:13:29.016226 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:29.016070    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdb4e455-7001-4113-bd1d-16894513c9da-workload-certs\") pod \"e2e-api-b-09203c-openshift-default-6d94744596-vzrnv\" (UID: \"cdb4e455-7001-4113-bd1d-16894513c9da\") " pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"
Sep 14 05:13:29.016226 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:29.016161    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clnzs\" (UniqueName: \"kubernetes.io/projected/cdb4e455-7001-4113-bd1d-16894513c9da-kube-api-access-clnzs\") pod \"e2e-api-b-09203c-openshift-default-6d94744596-vzrnv\" (UID: \"cdb4e455-7001-4113-bd1d-16894513c9da\") " pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"
Sep 14 05:13:29.116619 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:29.116587    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clnzs\" (UniqueName: \"kubernetes.io/projected/cdb4e455-7001-4113-bd1d-16894513c9da-kube-api-access-clnzs\") pod \"e2e-api-b-09203c-openshift-default-6d94744596-vzrnv\" (UID: \"cdb4e455-7001-4113-bd1d-16894513c9da\") " pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"
Sep 14 05:13:29.116808 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:29.116631    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdb4e455-7001-4113-bd1d-16894513c9da-istio-token\") pod \"e2e-api-b-09203c-openshift-default-6d94744596-vzrnv\" (UID: \"cdb4e455-7001-4113-bd1d-16894513c9da\") " pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"
Sep 14 05:13:29.116808 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:29.116649    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdb4e455-7001-4113-bd1d-16894513c9da-workload-socket\") pod \"e2e-api-b-09203c-openshift-default-6d94744596-vzrnv\" (UID: \"cdb4e455-7001-4113-bd1d-16894513c9da\") " pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"
Sep 14 05:13:29.116808 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:29.116675    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdb4e455-7001-4113-bd1d-16894513c9da-istio-podinfo\") pod \"e2e-api-b-09203c-openshift-default-6d94744596-vzrnv\" (UID: \"cdb4e455-7001-4113-bd1d-16894513c9da\") " pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"
Sep 14 05:13:29.116808 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:29.116702    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdb4e455-7001-4113-bd1d-16894513c9da-istiod-ca-cert\") pod \"e2e-api-b-09203c-openshift-default-6d94744596-vzrnv\" (UID: \"cdb4e455-7001-4113-bd1d-16894513c9da\") " pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"
Sep 14 05:13:29.116808 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:29.116767    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdb4e455-7001-4113-bd1d-16894513c9da-credential-socket\") pod \"e2e-api-b-09203c-openshift-default-6d94744596-vzrnv\" (UID: \"cdb4e455-7001-4113-bd1d-16894513c9da\") " pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"
Sep 14 05:13:29.116808 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:29.116790    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdb4e455-7001-4113-bd1d-16894513c9da-istio-data\") pod \"e2e-api-b-09203c-openshift-default-6d94744596-vzrnv\" (UID: \"cdb4e455-7001-4113-bd1d-16894513c9da\") " pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"
Sep 14 05:13:29.117153 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:29.116827    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdb4e455-7001-4113-bd1d-16894513c9da-istio-envoy\") pod \"e2e-api-b-09203c-openshift-default-6d94744596-vzrnv\" (UID: \"cdb4e455-7001-4113-bd1d-16894513c9da\") " pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"
Sep 14 05:13:29.117153 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:29.116872    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdb4e455-7001-4113-bd1d-16894513c9da-workload-certs\") pod \"e2e-api-b-09203c-openshift-default-6d94744596-vzrnv\" (UID: \"cdb4e455-7001-4113-bd1d-16894513c9da\") " pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"
Sep 14 05:13:29.117153 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:29.117096    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdb4e455-7001-4113-bd1d-16894513c9da-workload-socket\") pod \"e2e-api-b-09203c-openshift-default-6d94744596-vzrnv\" (UID: \"cdb4e455-7001-4113-bd1d-16894513c9da\") " pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"
Sep 14 05:13:29.117311 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:29.117189    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdb4e455-7001-4113-bd1d-16894513c9da-workload-certs\") pod \"e2e-api-b-09203c-openshift-default-6d94744596-vzrnv\" (UID: \"cdb4e455-7001-4113-bd1d-16894513c9da\") " pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"
Sep 14 05:13:29.117311 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:29.117288    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdb4e455-7001-4113-bd1d-16894513c9da-istio-data\") pod \"e2e-api-b-09203c-openshift-default-6d94744596-vzrnv\" (UID: \"cdb4e455-7001-4113-bd1d-16894513c9da\") " pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"
Sep 14 05:13:29.117551 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:29.117527    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdb4e455-7001-4113-bd1d-16894513c9da-credential-socket\") pod \"e2e-api-b-09203c-openshift-default-6d94744596-vzrnv\" (UID: \"cdb4e455-7001-4113-bd1d-16894513c9da\") " pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"
Sep 14 05:13:29.117711 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:29.117567    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdb4e455-7001-4113-bd1d-16894513c9da-istiod-ca-cert\") pod \"e2e-api-b-09203c-openshift-default-6d94744596-vzrnv\" (UID: \"cdb4e455-7001-4113-bd1d-16894513c9da\") " pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"
Sep 14 05:13:29.119635 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:29.119612    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdb4e455-7001-4113-bd1d-16894513c9da-istio-envoy\") pod \"e2e-api-b-09203c-openshift-default-6d94744596-vzrnv\" (UID: \"cdb4e455-7001-4113-bd1d-16894513c9da\") " pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"
Sep 14 05:13:29.119941 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:29.119920    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdb4e455-7001-4113-bd1d-16894513c9da-istio-podinfo\") pod \"e2e-api-b-09203c-openshift-default-6d94744596-vzrnv\" (UID: \"cdb4e455-7001-4113-bd1d-16894513c9da\") " pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"
Sep 14 05:13:29.125652 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:29.125629    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdb4e455-7001-4113-bd1d-16894513c9da-istio-token\") pod \"e2e-api-b-09203c-openshift-default-6d94744596-vzrnv\" (UID: \"cdb4e455-7001-4113-bd1d-16894513c9da\") " pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"
Sep 14 05:13:29.125809 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:29.125792    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clnzs\" (UniqueName: \"kubernetes.io/projected/cdb4e455-7001-4113-bd1d-16894513c9da-kube-api-access-clnzs\") pod \"e2e-api-b-09203c-openshift-default-6d94744596-vzrnv\" (UID: \"cdb4e455-7001-4113-bd1d-16894513c9da\") " pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"
Sep 14 05:13:29.207208 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:29.207115    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"
Sep 14 05:13:29.514578 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:29.514543    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"]
Sep 14 05:13:29.514926 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:13:29.514905    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdb4e455_7001_4113_bd1d_16894513c9da.slice/crio-0c2aeae96d91fbc8ef08e54303eb826ab25db8872262d2fffa541fbadc510065 WatchSource:0}: Error finding container 0c2aeae96d91fbc8ef08e54303eb826ab25db8872262d2fffa541fbadc510065: Status 404 returned error can't find the container with id 0c2aeae96d91fbc8ef08e54303eb826ab25db8872262d2fffa541fbadc510065
Sep 14 05:13:29.517273 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:29.517239    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:13:29.517375 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:29.517301    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:13:29.517375 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:29.517330    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:13:30.419174 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:30.419134    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv" event={"ID":"cdb4e455-7001-4113-bd1d-16894513c9da","Type":"ContainerStarted","Data":"9d61608f2238599d1baa90d3aa6e17484500a42650e4884ff06507a5564059b0"}
Sep 14 05:13:30.419544 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:30.419198    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv" event={"ID":"cdb4e455-7001-4113-bd1d-16894513c9da","Type":"ContainerStarted","Data":"0c2aeae96d91fbc8ef08e54303eb826ab25db8872262d2fffa541fbadc510065"}
Sep 14 05:13:30.442068 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:30.442007    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv" podStartSLOduration=2.441987432 podStartE2EDuration="2.441987432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:13:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:13:30.439151963 +0000 UTC m=+1608.614923524" watchObservedRunningTime="2026-09-14 05:13:30.441987432 +0000 UTC m=+1608.617758990"
Sep 14 05:13:31.207908 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:31.207874    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"
Sep 14 05:13:31.213155 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:31.213132    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"
Sep 14 05:13:31.424192 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:31.424163    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"
Sep 14 05:13:31.425079 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:31.425062    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"
Sep 14 05:13:37.458725 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:37.458691    2565 generic.go:356] "Generic (PLEG): container finished" podID="dada4eda-9779-48ff-a77f-ec7a4cc6852c" containerID="5732d669cb81c2d523f22c47c9d67cf728608862b519ec97c0bd43471b6ce0d4" exitCode=0
Sep 14 05:13:37.459176 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:37.458782    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr" event={"ID":"dada4eda-9779-48ff-a77f-ec7a4cc6852c","Type":"ContainerDied","Data":"5732d669cb81c2d523f22c47c9d67cf728608862b519ec97c0bd43471b6ce0d4"}
Sep 14 05:13:37.459220 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:37.459199    2565 scope.go:117] "RemoveContainer" containerID="5732d669cb81c2d523f22c47c9d67cf728608862b519ec97c0bd43471b6ce0d4"
Sep 14 05:13:37.460279 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:37.460245    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:13:37.460328 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:37.460317    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:13:37.460375 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:37.460363    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:13:38.273555 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:38.273522    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"
Sep 14 05:13:38.467828 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:38.467786    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr" event={"ID":"dada4eda-9779-48ff-a77f-ec7a4cc6852c","Type":"ContainerStarted","Data":"2840181df55222e511b7ce094f83aa6edfc87a5d6f368efc482d3279a0c48fd8"}
Sep 14 05:13:38.526536 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:38.526463    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:13:38.800734 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:38.800654    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"
Sep 14 05:13:38.801946 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:38.801920    2565 patch_prober.go:28] interesting pod/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Sep 14 05:13:38.802070 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:38.801970    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr" podUID="dada4eda-9779-48ff-a77f-ec7a4cc6852c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Sep 14 05:13:39.801381 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:39.801341    2565 patch_prober.go:28] interesting pod/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Sep 14 05:13:39.801811 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:39.801432    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr" podUID="dada4eda-9779-48ff-a77f-ec7a4cc6852c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Sep 14 05:13:40.801189 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:40.801147    2565 patch_prober.go:28] interesting pod/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Sep 14 05:13:40.801395 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:40.801207    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr" podUID="dada4eda-9779-48ff-a77f-ec7a4cc6852c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Sep 14 05:13:41.801005 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:41.800965    2565 patch_prober.go:28] interesting pod/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Sep 14 05:13:41.801459 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:41.801027    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr" podUID="dada4eda-9779-48ff-a77f-ec7a4cc6852c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Sep 14 05:13:42.800808 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:42.800743    2565 patch_prober.go:28] interesting pod/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Sep 14 05:13:42.801001 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:42.800830    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr" podUID="dada4eda-9779-48ff-a77f-ec7a4cc6852c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Sep 14 05:13:43.801547 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:43.801511    2565 patch_prober.go:28] interesting pod/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Sep 14 05:13:43.801970 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:43.801567    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr" podUID="dada4eda-9779-48ff-a77f-ec7a4cc6852c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Sep 14 05:13:44.804748 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:44.804720    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"
Sep 14 05:13:44.805175 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:44.805008    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"
Sep 14 05:13:44.805578 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:13:44.805559    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"
Sep 14 05:14:00.029718 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:00.029681    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:14:01.425744 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:01.425709    2565 patch_prober.go:28] interesting pod/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Sep 14 05:14:01.426140 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:01.425783    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv" podUID="cdb4e455-7001-4113-bd1d-16894513c9da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Sep 14 05:14:01.606044 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:01.606008    2565 generic.go:356] "Generic (PLEG): container finished" podID="cdb4e455-7001-4113-bd1d-16894513c9da" containerID="9d61608f2238599d1baa90d3aa6e17484500a42650e4884ff06507a5564059b0" exitCode=0
Sep 14 05:14:01.606216 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:01.606077    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv" event={"ID":"cdb4e455-7001-4113-bd1d-16894513c9da","Type":"ContainerDied","Data":"9d61608f2238599d1baa90d3aa6e17484500a42650e4884ff06507a5564059b0"}
Sep 14 05:14:01.606489 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:01.606475    2565 scope.go:117] "RemoveContainer" containerID="9d61608f2238599d1baa90d3aa6e17484500a42650e4884ff06507a5564059b0"
Sep 14 05:14:01.607575 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:01.607547    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:14:01.607670 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:01.607611    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:14:01.607670 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:01.607646    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:14:02.613201 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:02.613170    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv" event={"ID":"cdb4e455-7001-4113-bd1d-16894513c9da","Type":"ContainerStarted","Data":"288b1c3524d771905748e2cf69d27055bc75bea0ef42ddbda07eb3906f8e1331"}
Sep 14 05:14:03.207792 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:03.207718    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"
Sep 14 05:14:03.208043 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:03.208011    2565 patch_prober.go:28] interesting pod/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Sep 14 05:14:03.208222 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:03.208068    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv" podUID="cdb4e455-7001-4113-bd1d-16894513c9da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Sep 14 05:14:04.207936 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:04.207899    2565 patch_prober.go:28] interesting pod/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Sep 14 05:14:04.208312 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:04.207951    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv" podUID="cdb4e455-7001-4113-bd1d-16894513c9da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Sep 14 05:14:05.208441 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:05.208402    2565 patch_prober.go:28] interesting pod/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Sep 14 05:14:05.208870 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:05.208457    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv" podUID="cdb4e455-7001-4113-bd1d-16894513c9da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Sep 14 05:14:05.622934 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:05.622898    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cb472-l6gnv"]
Sep 14 05:14:05.626404 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:05.626387    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cb472-l6gnv"
Sep 14 05:14:05.628532 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:05.628509    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k6hpm\""
Sep 14 05:14:05.628644 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:05.628578    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 05:14:05.633809 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:05.633742    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cb472-l6gnv"]
Sep 14 05:14:05.666259 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:05.666217    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b85a2690-347c-482e-b013-e5a1ed7244fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6cb472-l6gnv\" (UID: \"b85a2690-347c-482e-b013-e5a1ed7244fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cb472-l6gnv"
Sep 14 05:14:05.767456 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:05.767421    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b85a2690-347c-482e-b013-e5a1ed7244fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6cb472-l6gnv\" (UID: \"b85a2690-347c-482e-b013-e5a1ed7244fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cb472-l6gnv"
Sep 14 05:14:05.768102 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:05.768081    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b85a2690-347c-482e-b013-e5a1ed7244fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6cb472-l6gnv\" (UID: \"b85a2690-347c-482e-b013-e5a1ed7244fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cb472-l6gnv"
Sep 14 05:14:05.938319 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:05.938216    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cb472-l6gnv"
Sep 14 05:14:06.208244 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:06.208171    2565 patch_prober.go:28] interesting pod/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Sep 14 05:14:06.208389 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:06.208236    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv" podUID="cdb4e455-7001-4113-bd1d-16894513c9da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Sep 14 05:14:06.221385 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:06.221357    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cb472-l6gnv"]
Sep 14 05:14:06.222560 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:14:06.222528    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb85a2690_347c_482e_b013_e5a1ed7244fe.slice/crio-4158facf31ae2918e0d176adaed2d881574aa088e6ff22d3e241ecfa4e2172af WatchSource:0}: Error finding container 4158facf31ae2918e0d176adaed2d881574aa088e6ff22d3e241ecfa4e2172af: Status 404 returned error can't find the container with id 4158facf31ae2918e0d176adaed2d881574aa088e6ff22d3e241ecfa4e2172af
Sep 14 05:14:06.638466 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:06.638426    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cb472-l6gnv" event={"ID":"b85a2690-347c-482e-b013-e5a1ed7244fe","Type":"ContainerStarted","Data":"4158facf31ae2918e0d176adaed2d881574aa088e6ff22d3e241ecfa4e2172af"}
Sep 14 05:14:07.207506 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:07.207477    2565 patch_prober.go:28] interesting pod/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Sep 14 05:14:07.207674 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:07.207530    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv" podUID="cdb4e455-7001-4113-bd1d-16894513c9da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Sep 14 05:14:07.649583 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:07.649542    2565 generic.go:356] "Generic (PLEG): container finished" podID="b85a2690-347c-482e-b013-e5a1ed7244fe" containerID="41b4bb50bbde6e62b2d29093db8f2f2b618a3efa6e9573b10b06333a6ec3886e" exitCode=0
Sep 14 05:14:07.650089 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:07.649643    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cb472-l6gnv" event={"ID":"b85a2690-347c-482e-b013-e5a1ed7244fe","Type":"ContainerDied","Data":"41b4bb50bbde6e62b2d29093db8f2f2b618a3efa6e9573b10b06333a6ec3886e"}
Sep 14 05:14:08.208402 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:08.208356    2565 patch_prober.go:28] interesting pod/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Sep 14 05:14:08.208578 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:08.208420    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv" podUID="cdb4e455-7001-4113-bd1d-16894513c9da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Sep 14 05:14:08.816377 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:08.816353    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cb472-l6gnv"
Sep 14 05:14:08.896032 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:08.895995    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b85a2690-347c-482e-b013-e5a1ed7244fe-maas-api-service-ca\") pod \"b85a2690-347c-482e-b013-e5a1ed7244fe\" (UID: \"b85a2690-347c-482e-b013-e5a1ed7244fe\") "
Sep 14 05:14:08.896495 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:08.896463    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b85a2690-347c-482e-b013-e5a1ed7244fe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b85a2690-347c-482e-b013-e5a1ed7244fe" (UID: "b85a2690-347c-482e-b013-e5a1ed7244fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 05:14:08.997213 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:08.997115    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b85a2690-347c-482e-b013-e5a1ed7244fe-maas-api-service-ca\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:14:09.212107 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:09.212080    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"
Sep 14 05:14:09.212345 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:09.212324    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"
Sep 14 05:14:09.212991 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:09.212973    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"
Sep 14 05:14:09.665337 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:09.665299    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cb472-l6gnv" event={"ID":"b85a2690-347c-482e-b013-e5a1ed7244fe","Type":"ContainerDied","Data":"4158facf31ae2918e0d176adaed2d881574aa088e6ff22d3e241ecfa4e2172af"}
Sep 14 05:14:09.665515 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:09.665367    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4158facf31ae2918e0d176adaed2d881574aa088e6ff22d3e241ecfa4e2172af"
Sep 14 05:14:09.665515 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:09.665319    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cb472-l6gnv"
Sep 14 05:14:19.745902 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:19.745822    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:14:21.552370 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:21.552338    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6cb472-74ccccb955-7jkt4"]
Sep 14 05:14:21.552983 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:21.552559    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-6cb472-74ccccb955-7jkt4" podUID="fcb50e9d-dec3-4c96-b8b4-ed5bb9aee247" containerName="payload-processing" containerID="cri-o://590d7a9c35325ff485225af806d140b6e12f9278ca2c1a9c6fd1c1555111d52a" gracePeriod=30
Sep 14 05:14:21.717961 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:21.717911    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cb472-l6gnv"]
Sep 14 05:14:21.720637 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:21.720607    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cb472-l6gnv"]
Sep 14 05:14:21.739483 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:21.739450    2565 generic.go:356] "Generic (PLEG): container finished" podID="fcb50e9d-dec3-4c96-b8b4-ed5bb9aee247" containerID="590d7a9c35325ff485225af806d140b6e12f9278ca2c1a9c6fd1c1555111d52a" exitCode=0
Sep 14 05:14:21.739657 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:21.739520    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6cb472-74ccccb955-7jkt4" event={"ID":"fcb50e9d-dec3-4c96-b8b4-ed5bb9aee247","Type":"ContainerDied","Data":"590d7a9c35325ff485225af806d140b6e12f9278ca2c1a9c6fd1c1555111d52a"}
Sep 14 05:14:21.808424 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:21.808365    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6cb472-74ccccb955-7jkt4"
Sep 14 05:14:21.909537 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:21.909504    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcb50e9d-dec3-4c96-b8b4-ed5bb9aee247-plugins-config-volume\") pod \"fcb50e9d-dec3-4c96-b8b4-ed5bb9aee247\" (UID: \"fcb50e9d-dec3-4c96-b8b4-ed5bb9aee247\") "
Sep 14 05:14:21.909783 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:21.909568    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bfdpv\" (UniqueName: \"kubernetes.io/projected/fcb50e9d-dec3-4c96-b8b4-ed5bb9aee247-kube-api-access-bfdpv\") pod \"fcb50e9d-dec3-4c96-b8b4-ed5bb9aee247\" (UID: \"fcb50e9d-dec3-4c96-b8b4-ed5bb9aee247\") "
Sep 14 05:14:21.910071 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:21.910040    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcb50e9d-dec3-4c96-b8b4-ed5bb9aee247-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fcb50e9d-dec3-4c96-b8b4-ed5bb9aee247" (UID: "fcb50e9d-dec3-4c96-b8b4-ed5bb9aee247"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 05:14:21.911791 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:21.911748    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcb50e9d-dec3-4c96-b8b4-ed5bb9aee247-kube-api-access-bfdpv" (OuterVolumeSpecName: "kube-api-access-bfdpv") pod "fcb50e9d-dec3-4c96-b8b4-ed5bb9aee247" (UID: "fcb50e9d-dec3-4c96-b8b4-ed5bb9aee247"). InnerVolumeSpecName "kube-api-access-bfdpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:14:22.010665 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:22.010622    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcb50e9d-dec3-4c96-b8b4-ed5bb9aee247-plugins-config-volume\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:14:22.010665 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:22.010664    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfdpv\" (UniqueName: \"kubernetes.io/projected/fcb50e9d-dec3-4c96-b8b4-ed5bb9aee247-kube-api-access-bfdpv\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:14:22.334958 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:22.334921    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b85a2690-347c-482e-b013-e5a1ed7244fe" path="/var/lib/kubelet/pods/b85a2690-347c-482e-b013-e5a1ed7244fe/volumes"
Sep 14 05:14:22.746249 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:22.746174    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6cb472-74ccccb955-7jkt4"
Sep 14 05:14:22.746617 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:22.746167    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6cb472-74ccccb955-7jkt4" event={"ID":"fcb50e9d-dec3-4c96-b8b4-ed5bb9aee247","Type":"ContainerDied","Data":"ef3487618bbcd6b5d78f579bf174c5aee8d979d6a4aaa7aba7305780d3f8d58a"}
Sep 14 05:14:22.746617 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:22.746304    2565 scope.go:117] "RemoveContainer" containerID="590d7a9c35325ff485225af806d140b6e12f9278ca2c1a9c6fd1c1555111d52a"
Sep 14 05:14:22.761894 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:22.761871    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6cb472-74ccccb955-7jkt4"]
Sep 14 05:14:22.768395 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:22.768372    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6cb472-74ccccb955-7jkt4"]
Sep 14 05:14:24.334118 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:24.334084    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcb50e9d-dec3-4c96-b8b4-ed5bb9aee247" path="/var/lib/kubelet/pods/fcb50e9d-dec3-4c96-b8b4-ed5bb9aee247/volumes"
Sep 14 05:14:28.457364 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:28.457321    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"]
Sep 14 05:14:28.457811 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:28.457612    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr" podUID="dada4eda-9779-48ff-a77f-ec7a4cc6852c" containerName="istio-proxy" containerID="cri-o://2840181df55222e511b7ce094f83aa6edfc87a5d6f368efc482d3279a0c48fd8" gracePeriod=30
Sep 14 05:14:28.828100 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:28.828016    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-09203c-tsdp7"]
Sep 14 05:14:28.828400 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:28.828386    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b85a2690-347c-482e-b013-e5a1ed7244fe" containerName="revoke-keys"
Sep 14 05:14:28.828442 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:28.828401    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b85a2690-347c-482e-b013-e5a1ed7244fe" containerName="revoke-keys"
Sep 14 05:14:28.828442 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:28.828413    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcb50e9d-dec3-4c96-b8b4-ed5bb9aee247" containerName="payload-processing"
Sep 14 05:14:28.828442 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:28.828422    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcb50e9d-dec3-4c96-b8b4-ed5bb9aee247" containerName="payload-processing"
Sep 14 05:14:28.828558 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:28.828487    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b85a2690-347c-482e-b013-e5a1ed7244fe" containerName="revoke-keys"
Sep 14 05:14:28.828558 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:28.828498    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcb50e9d-dec3-4c96-b8b4-ed5bb9aee247" containerName="payload-processing"
Sep 14 05:14:28.832710 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:28.832693    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-09203c-tsdp7"
Sep 14 05:14:28.835266 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:28.835240    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 05:14:28.835638 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:28.835613    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k6hpm\""
Sep 14 05:14:28.838676 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:28.838640    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-09203c-tsdp7"]
Sep 14 05:14:28.971746 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:28.971710    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59a9c2c5-8958-4807-a0ab-62df4f1bafb5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-09203c-tsdp7\" (UID: \"59a9c2c5-8958-4807-a0ab-62df4f1bafb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-09203c-tsdp7"
Sep 14 05:14:29.072199 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:29.072165    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59a9c2c5-8958-4807-a0ab-62df4f1bafb5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-09203c-tsdp7\" (UID: \"59a9c2c5-8958-4807-a0ab-62df4f1bafb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-09203c-tsdp7"
Sep 14 05:14:29.072762 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:29.072734    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59a9c2c5-8958-4807-a0ab-62df4f1bafb5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-09203c-tsdp7\" (UID: \"59a9c2c5-8958-4807-a0ab-62df4f1bafb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-09203c-tsdp7"
Sep 14 05:14:29.144486 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:29.144448    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-09203c-tsdp7"
Sep 14 05:14:29.417470 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:29.417428    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-09203c-tsdp7"]
Sep 14 05:14:29.419014 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:14:29.418988    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59a9c2c5_8958_4807_a0ab_62df4f1bafb5.slice/crio-6664f8fc496c4a6c7e7a09f46d637f234306f6bf9f113e1005158afdde3316fa WatchSource:0}: Error finding container 6664f8fc496c4a6c7e7a09f46d637f234306f6bf9f113e1005158afdde3316fa: Status 404 returned error can't find the container with id 6664f8fc496c4a6c7e7a09f46d637f234306f6bf9f113e1005158afdde3316fa
Sep 14 05:14:29.786275 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:29.786244    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-09203c-tsdp7" event={"ID":"59a9c2c5-8958-4807-a0ab-62df4f1bafb5","Type":"ContainerStarted","Data":"281af8a0142143ff5c2d99a6ea23cf72aafcd8014a22f782ff63fd88722bca87"}
Sep 14 05:14:29.786626 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:29.786300    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-09203c-tsdp7" event={"ID":"59a9c2c5-8958-4807-a0ab-62df4f1bafb5","Type":"ContainerStarted","Data":"6664f8fc496c4a6c7e7a09f46d637f234306f6bf9f113e1005158afdde3316fa"}
Sep 14 05:14:29.801664 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:29.801618    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-09203c-tsdp7" podStartSLOduration=1.8016022 podStartE2EDuration="1.8016022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:14:29.799975262 +0000 UTC m=+1667.975746823" watchObservedRunningTime="2026-09-14 05:14:29.8016022 +0000 UTC m=+1667.977373759"
Sep 14 05:14:29.808036 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:29.808006    2565 patch_prober.go:28] interesting pod/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 14 05:14:29.808132 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:29.808060    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr" podUID="dada4eda-9779-48ff-a77f-ec7a4cc6852c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 14 05:14:30.793656 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:30.793616    2565 generic.go:356] "Generic (PLEG): container finished" podID="59a9c2c5-8958-4807-a0ab-62df4f1bafb5" containerID="281af8a0142143ff5c2d99a6ea23cf72aafcd8014a22f782ff63fd88722bca87" exitCode=0
Sep 14 05:14:30.794083 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:30.793664    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-09203c-tsdp7" event={"ID":"59a9c2c5-8958-4807-a0ab-62df4f1bafb5","Type":"ContainerDied","Data":"281af8a0142143ff5c2d99a6ea23cf72aafcd8014a22f782ff63fd88722bca87"}
Sep 14 05:14:31.936993 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:31.936966    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-09203c-tsdp7"
Sep 14 05:14:32.101349 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:32.101320    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59a9c2c5-8958-4807-a0ab-62df4f1bafb5-maas-api-service-ca\") pod \"59a9c2c5-8958-4807-a0ab-62df4f1bafb5\" (UID: \"59a9c2c5-8958-4807-a0ab-62df4f1bafb5\") "
Sep 14 05:14:32.101690 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:32.101664    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59a9c2c5-8958-4807-a0ab-62df4f1bafb5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "59a9c2c5-8958-4807-a0ab-62df4f1bafb5" (UID: "59a9c2c5-8958-4807-a0ab-62df4f1bafb5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 05:14:32.202961 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:32.202925    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59a9c2c5-8958-4807-a0ab-62df4f1bafb5-maas-api-service-ca\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:14:32.805366 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:32.805339    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-09203c-tsdp7"
Sep 14 05:14:32.805573 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:32.805379    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-09203c-tsdp7" event={"ID":"59a9c2c5-8958-4807-a0ab-62df4f1bafb5","Type":"ContainerDied","Data":"6664f8fc496c4a6c7e7a09f46d637f234306f6bf9f113e1005158afdde3316fa"}
Sep 14 05:14:32.805573 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:32.805427    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6664f8fc496c4a6c7e7a09f46d637f234306f6bf9f113e1005158afdde3316fa"
Sep 14 05:14:33.709629 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:33.709604    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"
Sep 14 05:14:33.812576 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:33.812490    2565 generic.go:356] "Generic (PLEG): container finished" podID="dada4eda-9779-48ff-a77f-ec7a4cc6852c" containerID="2840181df55222e511b7ce094f83aa6edfc87a5d6f368efc482d3279a0c48fd8" exitCode=0
Sep 14 05:14:33.812576 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:33.812539    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"
Sep 14 05:14:33.812734 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:33.812571    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr" event={"ID":"dada4eda-9779-48ff-a77f-ec7a4cc6852c","Type":"ContainerDied","Data":"2840181df55222e511b7ce094f83aa6edfc87a5d6f368efc482d3279a0c48fd8"}
Sep 14 05:14:33.812734 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:33.812627    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr" event={"ID":"dada4eda-9779-48ff-a77f-ec7a4cc6852c","Type":"ContainerDied","Data":"f917527edf3b9054b4e926c8885c819852dcba1349d467824d942312621094ea"}
Sep 14 05:14:33.812734 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:33.812651    2565 scope.go:117] "RemoveContainer" containerID="2840181df55222e511b7ce094f83aa6edfc87a5d6f368efc482d3279a0c48fd8"
Sep 14 05:14:33.814820 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:33.814800    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9s7c\" (UniqueName: \"kubernetes.io/projected/dada4eda-9779-48ff-a77f-ec7a4cc6852c-kube-api-access-l9s7c\") pod \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\" (UID: \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\") "
Sep 14 05:14:33.814947 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:33.814842    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dada4eda-9779-48ff-a77f-ec7a4cc6852c-istio-envoy\") pod \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\" (UID: \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\") "
Sep 14 05:14:33.814947 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:33.814865    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dada4eda-9779-48ff-a77f-ec7a4cc6852c-istio-podinfo\") pod \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\" (UID: \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\") "
Sep 14 05:14:33.814947 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:33.814882    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dada4eda-9779-48ff-a77f-ec7a4cc6852c-istiod-ca-cert\") pod \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\" (UID: \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\") "
Sep 14 05:14:33.815114 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:33.814978    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dada4eda-9779-48ff-a77f-ec7a4cc6852c-istio-data\") pod \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\" (UID: \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\") "
Sep 14 05:14:33.815114 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:33.815045    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dada4eda-9779-48ff-a77f-ec7a4cc6852c-istio-token\") pod \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\" (UID: \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\") "
Sep 14 05:14:33.815114 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:33.815109    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dada4eda-9779-48ff-a77f-ec7a4cc6852c-workload-certs\") pod \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\" (UID: \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\") "
Sep 14 05:14:33.815270 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:33.815148    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dada4eda-9779-48ff-a77f-ec7a4cc6852c-workload-socket\") pod \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\" (UID: \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\") "
Sep 14 05:14:33.815270 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:33.815173    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dada4eda-9779-48ff-a77f-ec7a4cc6852c-credential-socket\") pod \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\" (UID: \"dada4eda-9779-48ff-a77f-ec7a4cc6852c\") "
Sep 14 05:14:33.815376 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:33.815286    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dada4eda-9779-48ff-a77f-ec7a4cc6852c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dada4eda-9779-48ff-a77f-ec7a4cc6852c" (UID: "dada4eda-9779-48ff-a77f-ec7a4cc6852c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 05:14:33.815791 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:33.815530    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dada4eda-9779-48ff-a77f-ec7a4cc6852c-istiod-ca-cert\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:14:33.815791 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:33.815599    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dada4eda-9779-48ff-a77f-ec7a4cc6852c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dada4eda-9779-48ff-a77f-ec7a4cc6852c" (UID: "dada4eda-9779-48ff-a77f-ec7a4cc6852c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:14:33.815791 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:33.815611    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dada4eda-9779-48ff-a77f-ec7a4cc6852c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dada4eda-9779-48ff-a77f-ec7a4cc6852c" (UID: "dada4eda-9779-48ff-a77f-ec7a4cc6852c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:14:33.815791 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:33.815667    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dada4eda-9779-48ff-a77f-ec7a4cc6852c-istio-data" (OuterVolumeSpecName: "istio-data") pod "dada4eda-9779-48ff-a77f-ec7a4cc6852c" (UID: "dada4eda-9779-48ff-a77f-ec7a4cc6852c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:14:33.815791 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:33.815745    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dada4eda-9779-48ff-a77f-ec7a4cc6852c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dada4eda-9779-48ff-a77f-ec7a4cc6852c" (UID: "dada4eda-9779-48ff-a77f-ec7a4cc6852c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:14:33.817487 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:33.817465    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dada4eda-9779-48ff-a77f-ec7a4cc6852c-kube-api-access-l9s7c" (OuterVolumeSpecName: "kube-api-access-l9s7c") pod "dada4eda-9779-48ff-a77f-ec7a4cc6852c" (UID: "dada4eda-9779-48ff-a77f-ec7a4cc6852c"). InnerVolumeSpecName "kube-api-access-l9s7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:14:33.817605 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:33.817488    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dada4eda-9779-48ff-a77f-ec7a4cc6852c-istio-token" (OuterVolumeSpecName: "istio-token") pod "dada4eda-9779-48ff-a77f-ec7a4cc6852c" (UID: "dada4eda-9779-48ff-a77f-ec7a4cc6852c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:14:33.817674 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:33.817610    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dada4eda-9779-48ff-a77f-ec7a4cc6852c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dada4eda-9779-48ff-a77f-ec7a4cc6852c" (UID: "dada4eda-9779-48ff-a77f-ec7a4cc6852c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 05:14:33.817674 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:33.817623    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dada4eda-9779-48ff-a77f-ec7a4cc6852c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dada4eda-9779-48ff-a77f-ec7a4cc6852c" (UID: "dada4eda-9779-48ff-a77f-ec7a4cc6852c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:14:33.822119 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:33.822093    2565 scope.go:117] "RemoveContainer" containerID="5732d669cb81c2d523f22c47c9d67cf728608862b519ec97c0bd43471b6ce0d4"
Sep 14 05:14:33.830527 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:33.830505    2565 scope.go:117] "RemoveContainer" containerID="2840181df55222e511b7ce094f83aa6edfc87a5d6f368efc482d3279a0c48fd8"
Sep 14 05:14:33.830851 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:14:33.830827    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2840181df55222e511b7ce094f83aa6edfc87a5d6f368efc482d3279a0c48fd8\": container with ID starting with 2840181df55222e511b7ce094f83aa6edfc87a5d6f368efc482d3279a0c48fd8 not found: ID does not exist" containerID="2840181df55222e511b7ce094f83aa6edfc87a5d6f368efc482d3279a0c48fd8"
Sep 14 05:14:33.830920 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:33.830856    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2840181df55222e511b7ce094f83aa6edfc87a5d6f368efc482d3279a0c48fd8"} err="failed to get container status \"2840181df55222e511b7ce094f83aa6edfc87a5d6f368efc482d3279a0c48fd8\": rpc error: code = NotFound desc = could not find container \"2840181df55222e511b7ce094f83aa6edfc87a5d6f368efc482d3279a0c48fd8\": container with ID starting with 2840181df55222e511b7ce094f83aa6edfc87a5d6f368efc482d3279a0c48fd8 not found: ID does not exist"
Sep 14 05:14:33.830920 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:33.830873    2565 scope.go:117] "RemoveContainer" containerID="5732d669cb81c2d523f22c47c9d67cf728608862b519ec97c0bd43471b6ce0d4"
Sep 14 05:14:33.831121 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:14:33.831105    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5732d669cb81c2d523f22c47c9d67cf728608862b519ec97c0bd43471b6ce0d4\": container with ID starting with 5732d669cb81c2d523f22c47c9d67cf728608862b519ec97c0bd43471b6ce0d4 not found: ID does not exist" containerID="5732d669cb81c2d523f22c47c9d67cf728608862b519ec97c0bd43471b6ce0d4"
Sep 14 05:14:33.831158 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:33.831126    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5732d669cb81c2d523f22c47c9d67cf728608862b519ec97c0bd43471b6ce0d4"} err="failed to get container status \"5732d669cb81c2d523f22c47c9d67cf728608862b519ec97c0bd43471b6ce0d4\": rpc error: code = NotFound desc = could not find container \"5732d669cb81c2d523f22c47c9d67cf728608862b519ec97c0bd43471b6ce0d4\": container with ID starting with 5732d669cb81c2d523f22c47c9d67cf728608862b519ec97c0bd43471b6ce0d4 not found: ID does not exist"
Sep 14 05:14:33.916504 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:33.916473    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dada4eda-9779-48ff-a77f-ec7a4cc6852c-istio-token\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:14:33.916504 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:33.916498    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dada4eda-9779-48ff-a77f-ec7a4cc6852c-workload-certs\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:14:33.916504 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:33.916508    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dada4eda-9779-48ff-a77f-ec7a4cc6852c-workload-socket\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:14:33.916731 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:33.916517    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dada4eda-9779-48ff-a77f-ec7a4cc6852c-credential-socket\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:14:33.916731 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:33.916528    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9s7c\" (UniqueName: \"kubernetes.io/projected/dada4eda-9779-48ff-a77f-ec7a4cc6852c-kube-api-access-l9s7c\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:14:33.916731 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:33.916537    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dada4eda-9779-48ff-a77f-ec7a4cc6852c-istio-envoy\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:14:33.916731 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:33.916546    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dada4eda-9779-48ff-a77f-ec7a4cc6852c-istio-podinfo\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:14:33.916731 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:33.916553    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dada4eda-9779-48ff-a77f-ec7a4cc6852c-istio-data\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:14:34.133545 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:34.133518    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"]
Sep 14 05:14:34.136995 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:34.136971    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-6cb472-openshift-default-56b886b47b-c78dr"]
Sep 14 05:14:34.334215 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:34.334183    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dada4eda-9779-48ff-a77f-ec7a4cc6852c" path="/var/lib/kubelet/pods/dada4eda-9779-48ff-a77f-ec7a4cc6852c/volumes"
Sep 14 05:14:41.902693 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:41.902659    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:14:42.597262 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:42.597230    2565 scope.go:117] "RemoveContainer" containerID="204df6b0d42a11a2a29018601a725f28cd2cedeb14a3998d16b5b539db6f389a"
Sep 14 05:14:44.503684 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:44.503645    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-09203c-tsdp7"]
Sep 14 05:14:44.505733 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:44.505707    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-09203c-tsdp7"]
Sep 14 05:14:46.337827 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:46.337798    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59a9c2c5-8958-4807-a0ab-62df4f1bafb5" path="/var/lib/kubelet/pods/59a9c2c5-8958-4807-a0ab-62df4f1bafb5/volumes"
Sep 14 05:14:48.840574 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:48.840535    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7536-7f032d"]
Sep 14 05:14:48.840979 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:48.840913    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59a9c2c5-8958-4807-a0ab-62df4f1bafb5" containerName="revoke-keys"
Sep 14 05:14:48.840979 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:48.840926    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="59a9c2c5-8958-4807-a0ab-62df4f1bafb5" containerName="revoke-keys"
Sep 14 05:14:48.840979 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:48.840943    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dada4eda-9779-48ff-a77f-ec7a4cc6852c" containerName="istio-proxy"
Sep 14 05:14:48.840979 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:48.840948    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dada4eda-9779-48ff-a77f-ec7a4cc6852c" containerName="istio-proxy"
Sep 14 05:14:48.841115 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:48.841002    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="dada4eda-9779-48ff-a77f-ec7a4cc6852c" containerName="istio-proxy"
Sep 14 05:14:48.841115 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:48.841011    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="dada4eda-9779-48ff-a77f-ec7a4cc6852c" containerName="istio-proxy"
Sep 14 05:14:48.841115 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:48.841016    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="59a9c2c5-8958-4807-a0ab-62df4f1bafb5" containerName="revoke-keys"
Sep 14 05:14:48.844168 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:48.844152    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7536-7f032d"
Sep 14 05:14:48.846150 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:48.846119    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8p5k\" (UniqueName: \"kubernetes.io/projected/8d4f09c4-010b-4676-988b-fb1679ef3af1-kube-api-access-p8p5k\") pod \"test-curl-7536-7f032d\" (UID: \"8d4f09c4-010b-4676-988b-fb1679ef3af1\") " pod="opendatahub/test-curl-7536-7f032d"
Sep 14 05:14:48.846382 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:48.846363    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dcz7p\""
Sep 14 05:14:48.850086 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:48.850063    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7536-7f032d"]
Sep 14 05:14:48.947460 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:48.947425    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8p5k\" (UniqueName: \"kubernetes.io/projected/8d4f09c4-010b-4676-988b-fb1679ef3af1-kube-api-access-p8p5k\") pod \"test-curl-7536-7f032d\" (UID: \"8d4f09c4-010b-4676-988b-fb1679ef3af1\") " pod="opendatahub/test-curl-7536-7f032d"
Sep 14 05:14:48.955170 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:48.955136    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8p5k\" (UniqueName: \"kubernetes.io/projected/8d4f09c4-010b-4676-988b-fb1679ef3af1-kube-api-access-p8p5k\") pod \"test-curl-7536-7f032d\" (UID: \"8d4f09c4-010b-4676-988b-fb1679ef3af1\") " pod="opendatahub/test-curl-7536-7f032d"
Sep 14 05:14:49.155249 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:49.155161    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7536-7f032d"
Sep 14 05:14:49.424494 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:49.424470    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7536-7f032d"]
Sep 14 05:14:49.426709 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:14:49.426681    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d4f09c4_010b_4676_988b_fb1679ef3af1.slice/crio-bbca998b3bdeebf9ba6340225393152e0e8ba48ce18b12589e0c17485bcc0029 WatchSource:0}: Error finding container bbca998b3bdeebf9ba6340225393152e0e8ba48ce18b12589e0c17485bcc0029: Status 404 returned error can't find the container with id bbca998b3bdeebf9ba6340225393152e0e8ba48ce18b12589e0c17485bcc0029
Sep 14 05:14:49.906800 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:49.906743    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7536-7f032d" event={"ID":"8d4f09c4-010b-4676-988b-fb1679ef3af1","Type":"ContainerStarted","Data":"0c214a59de8e26cd81552f1ebea769edf0448a7821166c8c7680c1af71943bdc"}
Sep 14 05:14:49.907156 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:49.906810    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7536-7f032d" event={"ID":"8d4f09c4-010b-4676-988b-fb1679ef3af1","Type":"ContainerStarted","Data":"bbca998b3bdeebf9ba6340225393152e0e8ba48ce18b12589e0c17485bcc0029"}
Sep 14 05:14:49.922837 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:49.922787    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7536-7f032d" podStartSLOduration=1.922773082 podStartE2EDuration="1.922773082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:14:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:14:49.922733952 +0000 UTC m=+1688.098505511" watchObservedRunningTime="2026-09-14 05:14:49.922773082 +0000 UTC m=+1688.098544632"
Sep 14 05:14:50.529789 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:50.529733    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7536-7f032d"]
Sep 14 05:14:50.536818 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:50.536790    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7536-7f032d"]
Sep 14 05:14:50.912631 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:14:50.912602    2565 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7536-7f032d" container="test-curl-7536-7f032d"
Sep 14 05:14:50.989604 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:50.989570    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"]
Sep 14 05:14:50.989814 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:50.989787    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv" podUID="cdb4e455-7001-4113-bd1d-16894513c9da" containerName="istio-proxy" containerID="cri-o://288b1c3524d771905748e2cf69d27055bc75bea0ef42ddbda07eb3906f8e1331" gracePeriod=30
Sep 14 05:14:51.917590 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:51.917547    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7536-7f032d" podUID="8d4f09c4-010b-4676-988b-fb1679ef3af1" containerName="test-curl-7536-7f032d" containerID="cri-o://0c214a59de8e26cd81552f1ebea769edf0448a7821166c8c7680c1af71943bdc" gracePeriod=2
Sep 14 05:14:51.919642 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:14:51.919614    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7536-7f032d\" is forbidden: User \"system:node:ip-10-0-133-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-99.ec2.internal' and this object" podUID="8d4f09c4-010b-4676-988b-fb1679ef3af1" pod="opendatahub/test-curl-7536-7f032d"
Sep 14 05:14:52.016569 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:52.016540    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2ff4a6/auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl"]
Sep 14 05:14:52.033478 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:14:52.033442    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7536-7f032d\" is forbidden: User \"system:node:ip-10-0-133-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-99.ec2.internal' and this object" podUID="8d4f09c4-010b-4676-988b-fb1679ef3af1" pod="opendatahub/test-curl-7536-7f032d"
Sep 14 05:14:52.335473 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:14:52.335431    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7536-7f032d\" is forbidden: User \"system:node:ip-10-0-133-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-99.ec2.internal' and this object" podUID="8d4f09c4-010b-4676-988b-fb1679ef3af1" pod="opendatahub/test-curl-7536-7f032d"
Sep 14 05:14:53.326762 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:53.326717    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:14:54.094299 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:54.094276    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7536-7f032d"
Sep 14 05:14:54.194460 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:54.194376    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8p5k\" (UniqueName: \"kubernetes.io/projected/8d4f09c4-010b-4676-988b-fb1679ef3af1-kube-api-access-p8p5k\") pod \"8d4f09c4-010b-4676-988b-fb1679ef3af1\" (UID: \"8d4f09c4-010b-4676-988b-fb1679ef3af1\") "
Sep 14 05:14:54.196404 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:54.196374    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d4f09c4-010b-4676-988b-fb1679ef3af1-kube-api-access-p8p5k" (OuterVolumeSpecName: "kube-api-access-p8p5k") pod "8d4f09c4-010b-4676-988b-fb1679ef3af1" (UID: "8d4f09c4-010b-4676-988b-fb1679ef3af1"). InnerVolumeSpecName "kube-api-access-p8p5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:14:54.215348 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:54.215320    2565 patch_prober.go:28] interesting pod/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 14 05:14:54.215462 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:54.215372    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv" podUID="cdb4e455-7001-4113-bd1d-16894513c9da" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 14 05:14:54.295586 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:54.295551    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8p5k\" (UniqueName: \"kubernetes.io/projected/8d4f09c4-010b-4676-988b-fb1679ef3af1-kube-api-access-p8p5k\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:14:54.334413 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:54.334384    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d4f09c4-010b-4676-988b-fb1679ef3af1" path="/var/lib/kubelet/pods/8d4f09c4-010b-4676-988b-fb1679ef3af1/volumes"
Sep 14 05:14:54.935902 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:54.935869    2565 generic.go:356] "Generic (PLEG): container finished" podID="8d4f09c4-010b-4676-988b-fb1679ef3af1" containerID="0c214a59de8e26cd81552f1ebea769edf0448a7821166c8c7680c1af71943bdc" exitCode=137
Sep 14 05:14:54.936097 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:54.935906    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7536-7f032d"
Sep 14 05:14:54.936097 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:54.935962    2565 scope.go:117] "RemoveContainer" containerID="0c214a59de8e26cd81552f1ebea769edf0448a7821166c8c7680c1af71943bdc"
Sep 14 05:14:54.944527 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:54.944510    2565 scope.go:117] "RemoveContainer" containerID="0c214a59de8e26cd81552f1ebea769edf0448a7821166c8c7680c1af71943bdc"
Sep 14 05:14:54.944811 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:14:54.944793    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c214a59de8e26cd81552f1ebea769edf0448a7821166c8c7680c1af71943bdc\": container with ID starting with 0c214a59de8e26cd81552f1ebea769edf0448a7821166c8c7680c1af71943bdc not found: ID does not exist" containerID="0c214a59de8e26cd81552f1ebea769edf0448a7821166c8c7680c1af71943bdc"
Sep 14 05:14:54.944865 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:54.944819    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c214a59de8e26cd81552f1ebea769edf0448a7821166c8c7680c1af71943bdc"} err="failed to get container status \"0c214a59de8e26cd81552f1ebea769edf0448a7821166c8c7680c1af71943bdc\": rpc error: code = NotFound desc = could not find container \"0c214a59de8e26cd81552f1ebea769edf0448a7821166c8c7680c1af71943bdc\": container with ID starting with 0c214a59de8e26cd81552f1ebea769edf0448a7821166c8c7680c1af71943bdc not found: ID does not exist"
Sep 14 05:14:56.239033 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:56.239012    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"
Sep 14 05:14:56.311984 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:56.311911    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdb4e455-7001-4113-bd1d-16894513c9da-istio-data\") pod \"cdb4e455-7001-4113-bd1d-16894513c9da\" (UID: \"cdb4e455-7001-4113-bd1d-16894513c9da\") "
Sep 14 05:14:56.311984 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:56.311957    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdb4e455-7001-4113-bd1d-16894513c9da-istio-podinfo\") pod \"cdb4e455-7001-4113-bd1d-16894513c9da\" (UID: \"cdb4e455-7001-4113-bd1d-16894513c9da\") "
Sep 14 05:14:56.311984 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:56.311983    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdb4e455-7001-4113-bd1d-16894513c9da-workload-socket\") pod \"cdb4e455-7001-4113-bd1d-16894513c9da\" (UID: \"cdb4e455-7001-4113-bd1d-16894513c9da\") "
Sep 14 05:14:56.312222 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:56.312000    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdb4e455-7001-4113-bd1d-16894513c9da-istio-token\") pod \"cdb4e455-7001-4113-bd1d-16894513c9da\" (UID: \"cdb4e455-7001-4113-bd1d-16894513c9da\") "
Sep 14 05:14:56.312222 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:56.312037    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdb4e455-7001-4113-bd1d-16894513c9da-istio-envoy\") pod \"cdb4e455-7001-4113-bd1d-16894513c9da\" (UID: \"cdb4e455-7001-4113-bd1d-16894513c9da\") "
Sep 14 05:14:56.312222 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:56.312086    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdb4e455-7001-4113-bd1d-16894513c9da-credential-socket\") pod \"cdb4e455-7001-4113-bd1d-16894513c9da\" (UID: \"cdb4e455-7001-4113-bd1d-16894513c9da\") "
Sep 14 05:14:56.312222 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:56.312109    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clnzs\" (UniqueName: \"kubernetes.io/projected/cdb4e455-7001-4113-bd1d-16894513c9da-kube-api-access-clnzs\") pod \"cdb4e455-7001-4113-bd1d-16894513c9da\" (UID: \"cdb4e455-7001-4113-bd1d-16894513c9da\") "
Sep 14 05:14:56.312222 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:56.312144    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdb4e455-7001-4113-bd1d-16894513c9da-istiod-ca-cert\") pod \"cdb4e455-7001-4113-bd1d-16894513c9da\" (UID: \"cdb4e455-7001-4113-bd1d-16894513c9da\") "
Sep 14 05:14:56.312222 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:56.312181    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdb4e455-7001-4113-bd1d-16894513c9da-workload-certs\") pod \"cdb4e455-7001-4113-bd1d-16894513c9da\" (UID: \"cdb4e455-7001-4113-bd1d-16894513c9da\") "
Sep 14 05:14:56.312592 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:56.312236    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdb4e455-7001-4113-bd1d-16894513c9da-istio-data" (OuterVolumeSpecName: "istio-data") pod "cdb4e455-7001-4113-bd1d-16894513c9da" (UID: "cdb4e455-7001-4113-bd1d-16894513c9da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:14:56.312592 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:56.312246    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdb4e455-7001-4113-bd1d-16894513c9da-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cdb4e455-7001-4113-bd1d-16894513c9da" (UID: "cdb4e455-7001-4113-bd1d-16894513c9da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:14:56.312592 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:56.312470    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdb4e455-7001-4113-bd1d-16894513c9da-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cdb4e455-7001-4113-bd1d-16894513c9da" (UID: "cdb4e455-7001-4113-bd1d-16894513c9da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:14:56.312592 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:56.312529    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdb4e455-7001-4113-bd1d-16894513c9da-istio-data\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:14:56.312592 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:56.312549    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdb4e455-7001-4113-bd1d-16894513c9da-workload-socket\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:14:56.312783 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:56.312564    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdb4e455-7001-4113-bd1d-16894513c9da-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cdb4e455-7001-4113-bd1d-16894513c9da" (UID: "cdb4e455-7001-4113-bd1d-16894513c9da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 05:14:56.312783 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:56.312639    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdb4e455-7001-4113-bd1d-16894513c9da-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cdb4e455-7001-4113-bd1d-16894513c9da" (UID: "cdb4e455-7001-4113-bd1d-16894513c9da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:14:56.314263 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:56.314229    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cdb4e455-7001-4113-bd1d-16894513c9da-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cdb4e455-7001-4113-bd1d-16894513c9da" (UID: "cdb4e455-7001-4113-bd1d-16894513c9da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 05:14:56.314428 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:56.314403    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdb4e455-7001-4113-bd1d-16894513c9da-istio-token" (OuterVolumeSpecName: "istio-token") pod "cdb4e455-7001-4113-bd1d-16894513c9da" (UID: "cdb4e455-7001-4113-bd1d-16894513c9da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:14:56.314685 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:56.314666    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdb4e455-7001-4113-bd1d-16894513c9da-kube-api-access-clnzs" (OuterVolumeSpecName: "kube-api-access-clnzs") pod "cdb4e455-7001-4113-bd1d-16894513c9da" (UID: "cdb4e455-7001-4113-bd1d-16894513c9da"). InnerVolumeSpecName "kube-api-access-clnzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:14:56.314685 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:56.314668    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdb4e455-7001-4113-bd1d-16894513c9da-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cdb4e455-7001-4113-bd1d-16894513c9da" (UID: "cdb4e455-7001-4113-bd1d-16894513c9da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:14:56.413581 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:56.413549    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdb4e455-7001-4113-bd1d-16894513c9da-credential-socket\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:14:56.413581 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:56.413576    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clnzs\" (UniqueName: \"kubernetes.io/projected/cdb4e455-7001-4113-bd1d-16894513c9da-kube-api-access-clnzs\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:14:56.413742 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:56.413590    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdb4e455-7001-4113-bd1d-16894513c9da-istiod-ca-cert\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:14:56.413742 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:56.413602    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdb4e455-7001-4113-bd1d-16894513c9da-workload-certs\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:14:56.413742 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:56.413614    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdb4e455-7001-4113-bd1d-16894513c9da-istio-podinfo\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:14:56.413742 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:56.413625    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdb4e455-7001-4113-bd1d-16894513c9da-istio-token\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:14:56.413742 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:56.413635    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdb4e455-7001-4113-bd1d-16894513c9da-istio-envoy\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:14:56.949214 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:56.949177    2565 generic.go:356] "Generic (PLEG): container finished" podID="cdb4e455-7001-4113-bd1d-16894513c9da" containerID="288b1c3524d771905748e2cf69d27055bc75bea0ef42ddbda07eb3906f8e1331" exitCode=0
Sep 14 05:14:56.949364 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:56.949230    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"
Sep 14 05:14:56.949364 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:56.949260    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv" event={"ID":"cdb4e455-7001-4113-bd1d-16894513c9da","Type":"ContainerDied","Data":"288b1c3524d771905748e2cf69d27055bc75bea0ef42ddbda07eb3906f8e1331"}
Sep 14 05:14:56.949364 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:56.949308    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv" event={"ID":"cdb4e455-7001-4113-bd1d-16894513c9da","Type":"ContainerDied","Data":"0c2aeae96d91fbc8ef08e54303eb826ab25db8872262d2fffa541fbadc510065"}
Sep 14 05:14:56.949364 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:56.949330    2565 scope.go:117] "RemoveContainer" containerID="288b1c3524d771905748e2cf69d27055bc75bea0ef42ddbda07eb3906f8e1331"
Sep 14 05:14:56.960537 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:56.960521    2565 scope.go:117] "RemoveContainer" containerID="9d61608f2238599d1baa90d3aa6e17484500a42650e4884ff06507a5564059b0"
Sep 14 05:14:56.970148 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:56.970128    2565 scope.go:117] "RemoveContainer" containerID="288b1c3524d771905748e2cf69d27055bc75bea0ef42ddbda07eb3906f8e1331"
Sep 14 05:14:56.970499 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:14:56.970474    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"288b1c3524d771905748e2cf69d27055bc75bea0ef42ddbda07eb3906f8e1331\": container with ID starting with 288b1c3524d771905748e2cf69d27055bc75bea0ef42ddbda07eb3906f8e1331 not found: ID does not exist" containerID="288b1c3524d771905748e2cf69d27055bc75bea0ef42ddbda07eb3906f8e1331"
Sep 14 05:14:56.970629 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:56.970508    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"288b1c3524d771905748e2cf69d27055bc75bea0ef42ddbda07eb3906f8e1331"} err="failed to get container status \"288b1c3524d771905748e2cf69d27055bc75bea0ef42ddbda07eb3906f8e1331\": rpc error: code = NotFound desc = could not find container \"288b1c3524d771905748e2cf69d27055bc75bea0ef42ddbda07eb3906f8e1331\": container with ID starting with 288b1c3524d771905748e2cf69d27055bc75bea0ef42ddbda07eb3906f8e1331 not found: ID does not exist"
Sep 14 05:14:56.970629 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:56.970531    2565 scope.go:117] "RemoveContainer" containerID="9d61608f2238599d1baa90d3aa6e17484500a42650e4884ff06507a5564059b0"
Sep 14 05:14:56.970931 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:14:56.970911    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d61608f2238599d1baa90d3aa6e17484500a42650e4884ff06507a5564059b0\": container with ID starting with 9d61608f2238599d1baa90d3aa6e17484500a42650e4884ff06507a5564059b0 not found: ID does not exist" containerID="9d61608f2238599d1baa90d3aa6e17484500a42650e4884ff06507a5564059b0"
Sep 14 05:14:56.971034 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:56.970936    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d61608f2238599d1baa90d3aa6e17484500a42650e4884ff06507a5564059b0"} err="failed to get container status \"9d61608f2238599d1baa90d3aa6e17484500a42650e4884ff06507a5564059b0\": rpc error: code = NotFound desc = could not find container \"9d61608f2238599d1baa90d3aa6e17484500a42650e4884ff06507a5564059b0\": container with ID starting with 9d61608f2238599d1baa90d3aa6e17484500a42650e4884ff06507a5564059b0 not found: ID does not exist"
Sep 14 05:14:56.971034 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:56.970990    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"]
Sep 14 05:14:56.973339 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:56.973315    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-09203c-openshift-default-6d94744596-vzrnv"]
Sep 14 05:14:58.334391 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:58.334356    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdb4e455-7001-4113-bd1d-16894513c9da" path="/var/lib/kubelet/pods/cdb4e455-7001-4113-bd1d-16894513c9da/volumes"
Sep 14 05:14:58.814691 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:58.814617    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-2ff4a6/sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg"]
Sep 14 05:14:58.814969 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:58.814955    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdb4e455-7001-4113-bd1d-16894513c9da" containerName="istio-proxy"
Sep 14 05:14:58.815020 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:58.814971    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdb4e455-7001-4113-bd1d-16894513c9da" containerName="istio-proxy"
Sep 14 05:14:58.815020 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:58.814978    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d4f09c4-010b-4676-988b-fb1679ef3af1" containerName="test-curl-7536-7f032d"
Sep 14 05:14:58.815020 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:58.814984    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d4f09c4-010b-4676-988b-fb1679ef3af1" containerName="test-curl-7536-7f032d"
Sep 14 05:14:58.815020 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:58.814997    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdb4e455-7001-4113-bd1d-16894513c9da" containerName="istio-proxy"
Sep 14 05:14:58.815020 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:58.815004    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdb4e455-7001-4113-bd1d-16894513c9da" containerName="istio-proxy"
Sep 14 05:14:58.815020 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:58.815016    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dada4eda-9779-48ff-a77f-ec7a4cc6852c" containerName="istio-proxy"
Sep 14 05:14:58.815020 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:58.815021    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dada4eda-9779-48ff-a77f-ec7a4cc6852c" containerName="istio-proxy"
Sep 14 05:14:58.815258 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:58.815069    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdb4e455-7001-4113-bd1d-16894513c9da" containerName="istio-proxy"
Sep 14 05:14:58.815258 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:58.815075    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdb4e455-7001-4113-bd1d-16894513c9da" containerName="istio-proxy"
Sep 14 05:14:58.815258 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:58.815085    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d4f09c4-010b-4676-988b-fb1679ef3af1" containerName="test-curl-7536-7f032d"
Sep 14 05:14:58.819699 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:58.819681    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2ff4a6/sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg"
Sep 14 05:14:58.822122 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:58.822104    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2ff4a6\"/\"sub-test-model-2ff4a6-kserve-self-signed-certs\""
Sep 14 05:14:58.826660 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:58.826642    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2ff4a6/sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg"]
Sep 14 05:14:58.935373 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:58.935339    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3929840-c8be-43d0-ad00-d669e75f8cdc-dshm\") pod \"sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg\" (UID: \"f3929840-c8be-43d0-ad00-d669e75f8cdc\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg"
Sep 14 05:14:58.935373 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:58.935375    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3929840-c8be-43d0-ad00-d669e75f8cdc-tmp-dir\") pod \"sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg\" (UID: \"f3929840-c8be-43d0-ad00-d669e75f8cdc\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg"
Sep 14 05:14:58.935555 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:58.935396    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3929840-c8be-43d0-ad00-d669e75f8cdc-home\") pod \"sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg\" (UID: \"f3929840-c8be-43d0-ad00-d669e75f8cdc\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg"
Sep 14 05:14:58.935555 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:58.935479    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3929840-c8be-43d0-ad00-d669e75f8cdc-tls-certs\") pod \"sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg\" (UID: \"f3929840-c8be-43d0-ad00-d669e75f8cdc\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg"
Sep 14 05:14:58.935555 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:58.935518    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6kxt\" (UniqueName: \"kubernetes.io/projected/f3929840-c8be-43d0-ad00-d669e75f8cdc-kube-api-access-k6kxt\") pod \"sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg\" (UID: \"f3929840-c8be-43d0-ad00-d669e75f8cdc\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg"
Sep 14 05:14:58.935555 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:58.935539    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3929840-c8be-43d0-ad00-d669e75f8cdc-model-cache\") pod \"sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg\" (UID: \"f3929840-c8be-43d0-ad00-d669e75f8cdc\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg"
Sep 14 05:14:59.036857 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:59.036825    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3929840-c8be-43d0-ad00-d669e75f8cdc-dshm\") pod \"sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg\" (UID: \"f3929840-c8be-43d0-ad00-d669e75f8cdc\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg"
Sep 14 05:14:59.036857 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:59.036859    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3929840-c8be-43d0-ad00-d669e75f8cdc-tmp-dir\") pod \"sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg\" (UID: \"f3929840-c8be-43d0-ad00-d669e75f8cdc\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg"
Sep 14 05:14:59.037100 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:59.036887    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3929840-c8be-43d0-ad00-d669e75f8cdc-home\") pod \"sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg\" (UID: \"f3929840-c8be-43d0-ad00-d669e75f8cdc\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg"
Sep 14 05:14:59.037100 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:59.036952    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3929840-c8be-43d0-ad00-d669e75f8cdc-tls-certs\") pod \"sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg\" (UID: \"f3929840-c8be-43d0-ad00-d669e75f8cdc\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg"
Sep 14 05:14:59.037100 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:59.036986    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6kxt\" (UniqueName: \"kubernetes.io/projected/f3929840-c8be-43d0-ad00-d669e75f8cdc-kube-api-access-k6kxt\") pod \"sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg\" (UID: \"f3929840-c8be-43d0-ad00-d669e75f8cdc\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg"
Sep 14 05:14:59.037100 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:59.037007    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3929840-c8be-43d0-ad00-d669e75f8cdc-model-cache\") pod \"sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg\" (UID: \"f3929840-c8be-43d0-ad00-d669e75f8cdc\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg"
Sep 14 05:14:59.037374 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:59.037348    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3929840-c8be-43d0-ad00-d669e75f8cdc-home\") pod \"sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg\" (UID: \"f3929840-c8be-43d0-ad00-d669e75f8cdc\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg"
Sep 14 05:14:59.037374 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:59.037364    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3929840-c8be-43d0-ad00-d669e75f8cdc-tmp-dir\") pod \"sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg\" (UID: \"f3929840-c8be-43d0-ad00-d669e75f8cdc\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg"
Sep 14 05:14:59.037468 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:59.037416    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3929840-c8be-43d0-ad00-d669e75f8cdc-model-cache\") pod \"sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg\" (UID: \"f3929840-c8be-43d0-ad00-d669e75f8cdc\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg"
Sep 14 05:14:59.039269 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:59.039245    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3929840-c8be-43d0-ad00-d669e75f8cdc-dshm\") pod \"sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg\" (UID: \"f3929840-c8be-43d0-ad00-d669e75f8cdc\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg"
Sep 14 05:14:59.039459 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:59.039439    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3929840-c8be-43d0-ad00-d669e75f8cdc-tls-certs\") pod \"sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg\" (UID: \"f3929840-c8be-43d0-ad00-d669e75f8cdc\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg"
Sep 14 05:14:59.045000 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:59.044982    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6kxt\" (UniqueName: \"kubernetes.io/projected/f3929840-c8be-43d0-ad00-d669e75f8cdc-kube-api-access-k6kxt\") pod \"sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg\" (UID: \"f3929840-c8be-43d0-ad00-d669e75f8cdc\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg"
Sep 14 05:14:59.130977 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:59.130941    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2ff4a6/sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg"
Sep 14 05:14:59.404158 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:59.404133    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2ff4a6/sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg"]
Sep 14 05:14:59.406252 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:14:59.406226    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3929840_c8be_43d0_ad00_d669e75f8cdc.slice/crio-8ff7024e31d671f7318efa7bf5059cdd00d688a54314113de5dfdc9d927bdf3f WatchSource:0}: Error finding container 8ff7024e31d671f7318efa7bf5059cdd00d688a54314113de5dfdc9d927bdf3f: Status 404 returned error can't find the container with id 8ff7024e31d671f7318efa7bf5059cdd00d688a54314113de5dfdc9d927bdf3f
Sep 14 05:14:59.968257 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:59.968223    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2ff4a6/sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg" event={"ID":"f3929840-c8be-43d0-ad00-d669e75f8cdc","Type":"ContainerStarted","Data":"8503981043f042244e8116cb8a6fd4f7e6a42ea58db774b1518e8a13d0693243"}
Sep 14 05:14:59.968410 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:59.968263    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2ff4a6/sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg" event={"ID":"f3929840-c8be-43d0-ad00-d669e75f8cdc","Type":"ContainerStarted","Data":"8ff7024e31d671f7318efa7bf5059cdd00d688a54314113de5dfdc9d927bdf3f"}
Sep 14 05:14:59.986069 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:14:59.986016    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-2ff4a6/sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg" podStartSLOduration=1.986001984 podStartE2EDuration="1.986001984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:14:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:14:59.983485954 +0000 UTC m=+1698.159257515" watchObservedRunningTime="2026-09-14 05:14:59.986001984 +0000 UTC m=+1698.161773543"
Sep 14 05:15:00.137888 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:00.137849    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a8c46f-29822715-dx6rz"]
Sep 14 05:15:00.141248 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:00.141226    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29822715-rxqzw"]
Sep 14 05:15:00.141394 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:00.141341    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a8c46f-29822715-dx6rz"
Sep 14 05:15:00.143865 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:00.143843    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k6hpm\""
Sep 14 05:15:00.144407 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:00.144380    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-27be50-29822715-2hm9r"]
Sep 14 05:15:00.144497 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:00.144478    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29822715-rxqzw"
Sep 14 05:15:00.151263 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:00.151239    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a8c46f-29822715-dx6rz"]
Sep 14 05:15:00.151380 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:00.151341    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-27be50-29822715-2hm9r"
Sep 14 05:15:00.153918 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:00.153890    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-27be50-29822715-2hm9r"]
Sep 14 05:15:00.157232 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:00.157210    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29822715-rxqzw"]
Sep 14 05:15:00.159629 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:00.159453    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a8c46f-29822715-dx6rz"
Sep 14 05:15:00.168749 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:00.168726    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29822715-rxqzw"
Sep 14 05:15:00.200098 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:00.200073    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-27be50-29822715-2hm9r"
Sep 14 05:15:00.442254 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:00.442179    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a8c46f-29822715-dx6rz"]
Sep 14 05:15:00.444047 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:15:00.444023    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e94be06_9c94_44f2_89e6_02f9098e7aae.slice/crio-e1d8b8306204d468d1a93695c49a6870df6f3ca7314aed283a54865a62cb2282 WatchSource:0}: Error finding container e1d8b8306204d468d1a93695c49a6870df6f3ca7314aed283a54865a62cb2282: Status 404 returned error can't find the container with id e1d8b8306204d468d1a93695c49a6870df6f3ca7314aed283a54865a62cb2282
Sep 14 05:15:00.445386 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:00.445362    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29822715-rxqzw"]
Sep 14 05:15:00.445483 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:15:00.445370    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d362a61_8b9d_4ea8_95b6_cd6ba9537e8d.slice/crio-f86c4590289b0567580ba556968441a458faf9bc431715551ba8a9c639fd1759 WatchSource:0}: Error finding container f86c4590289b0567580ba556968441a458faf9bc431715551ba8a9c639fd1759: Status 404 returned error can't find the container with id f86c4590289b0567580ba556968441a458faf9bc431715551ba8a9c639fd1759
Sep 14 05:15:00.496071 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:00.496045    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-27be50-29822715-2hm9r"]
Sep 14 05:15:00.497475 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:15:00.497452    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2244a16f_ad71_4fd5_a783_3bf80befdde4.slice/crio-cba5814ba6b61064ffa62980df29c1fafb1dac14e731a51964871cd44fb6a31e WatchSource:0}: Error finding container cba5814ba6b61064ffa62980df29c1fafb1dac14e731a51964871cd44fb6a31e: Status 404 returned error can't find the container with id cba5814ba6b61064ffa62980df29c1fafb1dac14e731a51964871cd44fb6a31e
Sep 14 05:15:00.975495 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:00.975457    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29822715-rxqzw" event={"ID":"7d362a61-8b9d-4ea8-95b6-cd6ba9537e8d","Type":"ContainerStarted","Data":"3b0d5b6fc6878372daf648e547ecce35fe2ac69325ef5188fe4cab464447170f"}
Sep 14 05:15:00.975677 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:00.975515    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29822715-rxqzw" event={"ID":"7d362a61-8b9d-4ea8-95b6-cd6ba9537e8d","Type":"ContainerStarted","Data":"f86c4590289b0567580ba556968441a458faf9bc431715551ba8a9c639fd1759"}
Sep 14 05:15:00.977062 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:00.977035    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-27be50-29822715-2hm9r" event={"ID":"2244a16f-ad71-4fd5-a783-3bf80befdde4","Type":"ContainerStarted","Data":"8600ebb76fa1b7ccfc4e003e8f10a923a38c857bd1ac5f9d7323e501da449efa"}
Sep 14 05:15:00.977183 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:00.977084    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-27be50-29822715-2hm9r" event={"ID":"2244a16f-ad71-4fd5-a783-3bf80befdde4","Type":"ContainerStarted","Data":"cba5814ba6b61064ffa62980df29c1fafb1dac14e731a51964871cd44fb6a31e"}
Sep 14 05:15:00.978534 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:00.978514    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a8c46f-29822715-dx6rz" event={"ID":"0e94be06-9c94-44f2-89e6-02f9098e7aae","Type":"ContainerStarted","Data":"7dcedd3c00577d42c6f997a25bd60c4e00b21825b078736e399372c0ca7038f4"}
Sep 14 05:15:00.978661 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:00.978553    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a8c46f-29822715-dx6rz" event={"ID":"0e94be06-9c94-44f2-89e6-02f9098e7aae","Type":"ContainerStarted","Data":"e1d8b8306204d468d1a93695c49a6870df6f3ca7314aed283a54865a62cb2282"}
Sep 14 05:15:00.992529 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:00.992488    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29822715-rxqzw" podStartSLOduration=0.992473105 podStartE2EDuration="992.473105ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:15:00.991555384 +0000 UTC m=+1699.167326944" watchObservedRunningTime="2026-09-14 05:15:00.992473105 +0000 UTC m=+1699.168244730"
Sep 14 05:15:01.007855 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:01.007769    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-27be50-29822715-2hm9r" podStartSLOduration=1.007739989 podStartE2EDuration="1.007739989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:15:01.006016484 +0000 UTC m=+1699.181788045" watchObservedRunningTime="2026-09-14 05:15:01.007739989 +0000 UTC m=+1699.183511548"
Sep 14 05:15:01.024323 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:01.024279    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a8c46f-29822715-dx6rz" podStartSLOduration=1.024261847 podStartE2EDuration="1.024261847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:15:01.02158349 +0000 UTC m=+1699.197355050" watchObservedRunningTime="2026-09-14 05:15:01.024261847 +0000 UTC m=+1699.200033409"
Sep 14 05:15:01.985768 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:01.985722    2565 generic.go:356] "Generic (PLEG): container finished" podID="7d362a61-8b9d-4ea8-95b6-cd6ba9537e8d" containerID="3b0d5b6fc6878372daf648e547ecce35fe2ac69325ef5188fe4cab464447170f" exitCode=0
Sep 14 05:15:01.986170 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:01.985789    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29822715-rxqzw" event={"ID":"7d362a61-8b9d-4ea8-95b6-cd6ba9537e8d","Type":"ContainerDied","Data":"3b0d5b6fc6878372daf648e547ecce35fe2ac69325ef5188fe4cab464447170f"}
Sep 14 05:15:01.987312 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:01.987289    2565 generic.go:356] "Generic (PLEG): container finished" podID="2244a16f-ad71-4fd5-a783-3bf80befdde4" containerID="8600ebb76fa1b7ccfc4e003e8f10a923a38c857bd1ac5f9d7323e501da449efa" exitCode=0
Sep 14 05:15:01.987436 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:01.987316    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-27be50-29822715-2hm9r" event={"ID":"2244a16f-ad71-4fd5-a783-3bf80befdde4","Type":"ContainerDied","Data":"8600ebb76fa1b7ccfc4e003e8f10a923a38c857bd1ac5f9d7323e501da449efa"}
Sep 14 05:15:01.988908 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:01.988889    2565 generic.go:356] "Generic (PLEG): container finished" podID="0e94be06-9c94-44f2-89e6-02f9098e7aae" containerID="7dcedd3c00577d42c6f997a25bd60c4e00b21825b078736e399372c0ca7038f4" exitCode=0
Sep 14 05:15:01.988982 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:01.988942    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a8c46f-29822715-dx6rz" event={"ID":"0e94be06-9c94-44f2-89e6-02f9098e7aae","Type":"ContainerDied","Data":"7dcedd3c00577d42c6f997a25bd60c4e00b21825b078736e399372c0ca7038f4"}
Sep 14 05:15:02.115385 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:02.115353    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2eeb50-68db997bf4-s7kpn"]
Sep 14 05:15:02.115659 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:02.115610    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2eeb50-68db997bf4-s7kpn" podUID="97875616-7a3a-4ef7-963d-be811d6ddf46" containerName="payload-pre-processing" containerID="cri-o://667ca8ce211b27b279db436c66e458b32ba7ac182c92da64e83c35b2d86042e2" gracePeriod=30
Sep 14 05:15:02.363520 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:02.363493    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2eeb50-68db997bf4-s7kpn"
Sep 14 05:15:02.471962 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:02.471927    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97875616-7a3a-4ef7-963d-be811d6ddf46-plugins-config-volume\") pod \"97875616-7a3a-4ef7-963d-be811d6ddf46\" (UID: \"97875616-7a3a-4ef7-963d-be811d6ddf46\") "
Sep 14 05:15:02.472134 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:02.472008    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkvvf\" (UniqueName: \"kubernetes.io/projected/97875616-7a3a-4ef7-963d-be811d6ddf46-kube-api-access-gkvvf\") pod \"97875616-7a3a-4ef7-963d-be811d6ddf46\" (UID: \"97875616-7a3a-4ef7-963d-be811d6ddf46\") "
Sep 14 05:15:02.472396 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:02.472371    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97875616-7a3a-4ef7-963d-be811d6ddf46-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "97875616-7a3a-4ef7-963d-be811d6ddf46" (UID: "97875616-7a3a-4ef7-963d-be811d6ddf46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 05:15:02.474156 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:02.474121    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97875616-7a3a-4ef7-963d-be811d6ddf46-kube-api-access-gkvvf" (OuterVolumeSpecName: "kube-api-access-gkvvf") pod "97875616-7a3a-4ef7-963d-be811d6ddf46" (UID: "97875616-7a3a-4ef7-963d-be811d6ddf46"). InnerVolumeSpecName "kube-api-access-gkvvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:15:02.573221 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:02.573136    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkvvf\" (UniqueName: \"kubernetes.io/projected/97875616-7a3a-4ef7-963d-be811d6ddf46-kube-api-access-gkvvf\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:15:02.573221 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:02.573169    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97875616-7a3a-4ef7-963d-be811d6ddf46-plugins-config-volume\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:15:02.997236 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:02.997192    2565 generic.go:356] "Generic (PLEG): container finished" podID="97875616-7a3a-4ef7-963d-be811d6ddf46" containerID="667ca8ce211b27b279db436c66e458b32ba7ac182c92da64e83c35b2d86042e2" exitCode=0
Sep 14 05:15:02.997236 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:02.997241    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2eeb50-68db997bf4-s7kpn"
Sep 14 05:15:02.997917 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:02.997277    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2eeb50-68db997bf4-s7kpn" event={"ID":"97875616-7a3a-4ef7-963d-be811d6ddf46","Type":"ContainerDied","Data":"667ca8ce211b27b279db436c66e458b32ba7ac182c92da64e83c35b2d86042e2"}
Sep 14 05:15:02.997917 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:02.997334    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2eeb50-68db997bf4-s7kpn" event={"ID":"97875616-7a3a-4ef7-963d-be811d6ddf46","Type":"ContainerDied","Data":"2f75f625d760d7818efbc05ad864067f0e6c0775facf2bb87baff37ae6657622"}
Sep 14 05:15:02.997917 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:02.997358    2565 scope.go:117] "RemoveContainer" containerID="667ca8ce211b27b279db436c66e458b32ba7ac182c92da64e83c35b2d86042e2"
Sep 14 05:15:03.010785 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:03.010741    2565 scope.go:117] "RemoveContainer" containerID="667ca8ce211b27b279db436c66e458b32ba7ac182c92da64e83c35b2d86042e2"
Sep 14 05:15:03.011158 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:15:03.011126    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"667ca8ce211b27b279db436c66e458b32ba7ac182c92da64e83c35b2d86042e2\": container with ID starting with 667ca8ce211b27b279db436c66e458b32ba7ac182c92da64e83c35b2d86042e2 not found: ID does not exist" containerID="667ca8ce211b27b279db436c66e458b32ba7ac182c92da64e83c35b2d86042e2"
Sep 14 05:15:03.011265 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:03.011171    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"667ca8ce211b27b279db436c66e458b32ba7ac182c92da64e83c35b2d86042e2"} err="failed to get container status \"667ca8ce211b27b279db436c66e458b32ba7ac182c92da64e83c35b2d86042e2\": rpc error: code = NotFound desc = could not find container \"667ca8ce211b27b279db436c66e458b32ba7ac182c92da64e83c35b2d86042e2\": container with ID starting with 667ca8ce211b27b279db436c66e458b32ba7ac182c92da64e83c35b2d86042e2 not found: ID does not exist"
Sep 14 05:15:03.024551 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:03.024301    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2eeb50-68db997bf4-s7kpn"]
Sep 14 05:15:03.031205 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:03.031167    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2eeb50-68db997bf4-s7kpn"]
Sep 14 05:15:03.156522 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:03.156487    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a8c46f-29822715-dx6rz"
Sep 14 05:15:03.196997 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:03.196969    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-27be50-29822715-2hm9r"
Sep 14 05:15:03.200105 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:03.200084    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29822715-rxqzw"
Sep 14 05:15:04.004821 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:04.004793    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29822715-rxqzw"
Sep 14 05:15:04.005266 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:04.004819    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29822715-rxqzw" event={"ID":"7d362a61-8b9d-4ea8-95b6-cd6ba9537e8d","Type":"ContainerDied","Data":"f86c4590289b0567580ba556968441a458faf9bc431715551ba8a9c639fd1759"}
Sep 14 05:15:04.005266 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:04.004873    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f86c4590289b0567580ba556968441a458faf9bc431715551ba8a9c639fd1759"
Sep 14 05:15:04.007092 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:04.007072    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-27be50-29822715-2hm9r"
Sep 14 05:15:04.007240 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:04.007091    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-27be50-29822715-2hm9r" event={"ID":"2244a16f-ad71-4fd5-a783-3bf80befdde4","Type":"ContainerDied","Data":"cba5814ba6b61064ffa62980df29c1fafb1dac14e731a51964871cd44fb6a31e"}
Sep 14 05:15:04.007240 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:04.007135    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cba5814ba6b61064ffa62980df29c1fafb1dac14e731a51964871cd44fb6a31e"
Sep 14 05:15:04.008473 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:04.008449    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a8c46f-29822715-dx6rz" event={"ID":"0e94be06-9c94-44f2-89e6-02f9098e7aae","Type":"ContainerDied","Data":"e1d8b8306204d468d1a93695c49a6870df6f3ca7314aed283a54865a62cb2282"}
Sep 14 05:15:04.008586 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:04.008489    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1d8b8306204d468d1a93695c49a6870df6f3ca7314aed283a54865a62cb2282"
Sep 14 05:15:04.008586 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:04.008492    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a8c46f-29822715-dx6rz"
Sep 14 05:15:04.334869 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:04.334832    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97875616-7a3a-4ef7-963d-be811d6ddf46" path="/var/lib/kubelet/pods/97875616-7a3a-4ef7-963d-be811d6ddf46/volumes"
Sep 14 05:15:07.035854 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:07.035795    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-2ff4a6/auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl" podUID="df1311ed-d356-49b5-b319-f247b65a4abd" containerName="main" containerID="cri-o://a02b5fbe4b565c0973210023572f4c3facb35d1e916691f676c1d4bbe7b662f9" gracePeriod=45
Sep 14 05:15:07.288953 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:07.288885    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2ff4a6/auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl"
Sep 14 05:15:07.420019 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:07.419961    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df1311ed-d356-49b5-b319-f247b65a4abd-model-cache\") pod \"df1311ed-d356-49b5-b319-f247b65a4abd\" (UID: \"df1311ed-d356-49b5-b319-f247b65a4abd\") "
Sep 14 05:15:07.420019 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:07.420025    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df1311ed-d356-49b5-b319-f247b65a4abd-tmp-dir\") pod \"df1311ed-d356-49b5-b319-f247b65a4abd\" (UID: \"df1311ed-d356-49b5-b319-f247b65a4abd\") "
Sep 14 05:15:07.420292 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:07.420060    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df1311ed-d356-49b5-b319-f247b65a4abd-home\") pod \"df1311ed-d356-49b5-b319-f247b65a4abd\" (UID: \"df1311ed-d356-49b5-b319-f247b65a4abd\") "
Sep 14 05:15:07.420292 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:07.420102    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df1311ed-d356-49b5-b319-f247b65a4abd-dshm\") pod \"df1311ed-d356-49b5-b319-f247b65a4abd\" (UID: \"df1311ed-d356-49b5-b319-f247b65a4abd\") "
Sep 14 05:15:07.420292 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:07.420194    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df1311ed-d356-49b5-b319-f247b65a4abd-tls-certs\") pod \"df1311ed-d356-49b5-b319-f247b65a4abd\" (UID: \"df1311ed-d356-49b5-b319-f247b65a4abd\") "
Sep 14 05:15:07.420292 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:07.420255    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzb5n\" (UniqueName: \"kubernetes.io/projected/df1311ed-d356-49b5-b319-f247b65a4abd-kube-api-access-pzb5n\") pod \"df1311ed-d356-49b5-b319-f247b65a4abd\" (UID: \"df1311ed-d356-49b5-b319-f247b65a4abd\") "
Sep 14 05:15:07.420512 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:07.420304    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df1311ed-d356-49b5-b319-f247b65a4abd-model-cache" (OuterVolumeSpecName: "model-cache") pod "df1311ed-d356-49b5-b319-f247b65a4abd" (UID: "df1311ed-d356-49b5-b319-f247b65a4abd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:15:07.420512 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:07.420349    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df1311ed-d356-49b5-b319-f247b65a4abd-home" (OuterVolumeSpecName: "home") pod "df1311ed-d356-49b5-b319-f247b65a4abd" (UID: "df1311ed-d356-49b5-b319-f247b65a4abd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:15:07.420512 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:07.420331    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df1311ed-d356-49b5-b319-f247b65a4abd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "df1311ed-d356-49b5-b319-f247b65a4abd" (UID: "df1311ed-d356-49b5-b319-f247b65a4abd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:15:07.420661 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:07.420518    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df1311ed-d356-49b5-b319-f247b65a4abd-model-cache\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:15:07.420661 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:07.420530    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df1311ed-d356-49b5-b319-f247b65a4abd-tmp-dir\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:15:07.420661 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:07.420538    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df1311ed-d356-49b5-b319-f247b65a4abd-home\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:15:07.422387 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:07.422358    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df1311ed-d356-49b5-b319-f247b65a4abd-kube-api-access-pzb5n" (OuterVolumeSpecName: "kube-api-access-pzb5n") pod "df1311ed-d356-49b5-b319-f247b65a4abd" (UID: "df1311ed-d356-49b5-b319-f247b65a4abd"). InnerVolumeSpecName "kube-api-access-pzb5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:15:07.422387 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:07.422375    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df1311ed-d356-49b5-b319-f247b65a4abd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "df1311ed-d356-49b5-b319-f247b65a4abd" (UID: "df1311ed-d356-49b5-b319-f247b65a4abd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 05:15:07.422536 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:07.422385    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df1311ed-d356-49b5-b319-f247b65a4abd-dshm" (OuterVolumeSpecName: "dshm") pod "df1311ed-d356-49b5-b319-f247b65a4abd" (UID: "df1311ed-d356-49b5-b319-f247b65a4abd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:15:07.521502 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:07.521462    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df1311ed-d356-49b5-b319-f247b65a4abd-tls-certs\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:15:07.521502 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:07.521495    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzb5n\" (UniqueName: \"kubernetes.io/projected/df1311ed-d356-49b5-b319-f247b65a4abd-kube-api-access-pzb5n\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:15:07.521711 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:07.521506    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df1311ed-d356-49b5-b319-f247b65a4abd-dshm\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:15:08.033535 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:08.033497    2565 generic.go:356] "Generic (PLEG): container finished" podID="df1311ed-d356-49b5-b319-f247b65a4abd" containerID="a02b5fbe4b565c0973210023572f4c3facb35d1e916691f676c1d4bbe7b662f9" exitCode=0
Sep 14 05:15:08.033712 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:08.033547    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2ff4a6/auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl"
Sep 14 05:15:08.033712 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:08.033580    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2ff4a6/auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl" event={"ID":"df1311ed-d356-49b5-b319-f247b65a4abd","Type":"ContainerDied","Data":"a02b5fbe4b565c0973210023572f4c3facb35d1e916691f676c1d4bbe7b662f9"}
Sep 14 05:15:08.033712 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:08.033625    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2ff4a6/auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl" event={"ID":"df1311ed-d356-49b5-b319-f247b65a4abd","Type":"ContainerDied","Data":"282ee520bb485cbcba08c8a70692293d8dab9426e10901d736f99a297adf420e"}
Sep 14 05:15:08.033712 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:08.033642    2565 scope.go:117] "RemoveContainer" containerID="a02b5fbe4b565c0973210023572f4c3facb35d1e916691f676c1d4bbe7b662f9"
Sep 14 05:15:08.043106 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:08.042868    2565 scope.go:117] "RemoveContainer" containerID="a02b5fbe4b565c0973210023572f4c3facb35d1e916691f676c1d4bbe7b662f9"
Sep 14 05:15:08.043426 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:15:08.043188    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a02b5fbe4b565c0973210023572f4c3facb35d1e916691f676c1d4bbe7b662f9\": container with ID starting with a02b5fbe4b565c0973210023572f4c3facb35d1e916691f676c1d4bbe7b662f9 not found: ID does not exist" containerID="a02b5fbe4b565c0973210023572f4c3facb35d1e916691f676c1d4bbe7b662f9"
Sep 14 05:15:08.043426 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:08.043224    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a02b5fbe4b565c0973210023572f4c3facb35d1e916691f676c1d4bbe7b662f9"} err="failed to get container status \"a02b5fbe4b565c0973210023572f4c3facb35d1e916691f676c1d4bbe7b662f9\": rpc error: code = NotFound desc = could not find container \"a02b5fbe4b565c0973210023572f4c3facb35d1e916691f676c1d4bbe7b662f9\": container with ID starting with a02b5fbe4b565c0973210023572f4c3facb35d1e916691f676c1d4bbe7b662f9 not found: ID does not exist"
Sep 14 05:15:08.057236 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:08.057212    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2ff4a6/auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl"]
Sep 14 05:15:08.059367 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:08.059346    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2ff4a6/auth-test-model-2ff4a6-kserve-7bdfb8885f-rdzpl"]
Sep 14 05:15:08.334547 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:08.334521    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df1311ed-d356-49b5-b319-f247b65a4abd" path="/var/lib/kubelet/pods/df1311ed-d356-49b5-b319-f247b65a4abd/volumes"
Sep 14 05:15:09.131465 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:09.131435    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-2ff4a6/sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg"
Sep 14 05:15:09.131889 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:09.131495    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-2ff4a6/sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg"
Sep 14 05:15:09.143944 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:09.143921    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-2ff4a6/sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg"
Sep 14 05:15:09.575959 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:09.575870    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a8c46f-6qnrc"]
Sep 14 05:15:09.576280 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:09.576256    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97875616-7a3a-4ef7-963d-be811d6ddf46" containerName="payload-pre-processing"
Sep 14 05:15:09.576374 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:09.576283    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="97875616-7a3a-4ef7-963d-be811d6ddf46" containerName="payload-pre-processing"
Sep 14 05:15:09.576374 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:09.576299    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df1311ed-d356-49b5-b319-f247b65a4abd" containerName="main"
Sep 14 05:15:09.576374 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:09.576307    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="df1311ed-d356-49b5-b319-f247b65a4abd" containerName="main"
Sep 14 05:15:09.576374 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:09.576326    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d362a61-8b9d-4ea8-95b6-cd6ba9537e8d" containerName="cleanup"
Sep 14 05:15:09.576374 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:09.576336    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d362a61-8b9d-4ea8-95b6-cd6ba9537e8d" containerName="cleanup"
Sep 14 05:15:09.576374 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:09.576360    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e94be06-9c94-44f2-89e6-02f9098e7aae" containerName="cleanup"
Sep 14 05:15:09.576374 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:09.576368    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e94be06-9c94-44f2-89e6-02f9098e7aae" containerName="cleanup"
Sep 14 05:15:09.576705 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:09.576385    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2244a16f-ad71-4fd5-a783-3bf80befdde4" containerName="cleanup"
Sep 14 05:15:09.576705 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:09.576394    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2244a16f-ad71-4fd5-a783-3bf80befdde4" containerName="cleanup"
Sep 14 05:15:09.576705 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:09.576481    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d362a61-8b9d-4ea8-95b6-cd6ba9537e8d" containerName="cleanup"
Sep 14 05:15:09.576705 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:09.576493    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="97875616-7a3a-4ef7-963d-be811d6ddf46" containerName="payload-pre-processing"
Sep 14 05:15:09.576705 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:09.576507    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2244a16f-ad71-4fd5-a783-3bf80befdde4" containerName="cleanup"
Sep 14 05:15:09.576705 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:09.576516    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e94be06-9c94-44f2-89e6-02f9098e7aae" containerName="cleanup"
Sep 14 05:15:09.576705 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:09.576525    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="df1311ed-d356-49b5-b319-f247b65a4abd" containerName="main"
Sep 14 05:15:09.580863 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:09.580842    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a8c46f-6qnrc"
Sep 14 05:15:09.584147 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:09.584124    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 05:15:09.584338 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:09.584131    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k6hpm\""
Sep 14 05:15:09.587350 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:09.587328    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a8c46f-6qnrc"]
Sep 14 05:15:09.639930 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:09.639899    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a8e3c54-b7db-486b-83c5-568572c1901f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a8c46f-6qnrc\" (UID: \"4a8e3c54-b7db-486b-83c5-568572c1901f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a8c46f-6qnrc"
Sep 14 05:15:09.740473 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:09.740442    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a8e3c54-b7db-486b-83c5-568572c1901f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a8c46f-6qnrc\" (UID: \"4a8e3c54-b7db-486b-83c5-568572c1901f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a8c46f-6qnrc"
Sep 14 05:15:09.741048 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:09.741028    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a8e3c54-b7db-486b-83c5-568572c1901f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a8c46f-6qnrc\" (UID: \"4a8e3c54-b7db-486b-83c5-568572c1901f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a8c46f-6qnrc"
Sep 14 05:15:09.893661 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:09.893631    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a8c46f-6qnrc"
Sep 14 05:15:10.060211 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:10.060182    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-2ff4a6/sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg"
Sep 14 05:15:10.168961 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:10.168938    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a8c46f-6qnrc"]
Sep 14 05:15:10.170897 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:15:10.170867    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a8e3c54_b7db_486b_83c5_568572c1901f.slice/crio-03b466cb58b2b317d0be7292a5811b726c90517620669eb732947285b15cfc26 WatchSource:0}: Error finding container 03b466cb58b2b317d0be7292a5811b726c90517620669eb732947285b15cfc26: Status 404 returned error can't find the container with id 03b466cb58b2b317d0be7292a5811b726c90517620669eb732947285b15cfc26
Sep 14 05:15:11.057488 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:11.057401    2565 generic.go:356] "Generic (PLEG): container finished" podID="4a8e3c54-b7db-486b-83c5-568572c1901f" containerID="715da2ea1683fd50c7b655b0ddcd5763a995e1ddc65f2ad5022bd5cf07fc3e78" exitCode=0
Sep 14 05:15:11.057633 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:11.057488    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a8c46f-6qnrc" event={"ID":"4a8e3c54-b7db-486b-83c5-568572c1901f","Type":"ContainerDied","Data":"715da2ea1683fd50c7b655b0ddcd5763a995e1ddc65f2ad5022bd5cf07fc3e78"}
Sep 14 05:15:11.057633 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:11.057535    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a8c46f-6qnrc" event={"ID":"4a8e3c54-b7db-486b-83c5-568572c1901f","Type":"ContainerStarted","Data":"03b466cb58b2b317d0be7292a5811b726c90517620669eb732947285b15cfc26"}
Sep 14 05:15:12.192068 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:12.192042    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a8c46f-6qnrc"
Sep 14 05:15:12.261107 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:12.261075    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a8e3c54-b7db-486b-83c5-568572c1901f-maas-api-service-ca\") pod \"4a8e3c54-b7db-486b-83c5-568572c1901f\" (UID: \"4a8e3c54-b7db-486b-83c5-568572c1901f\") "
Sep 14 05:15:12.261427 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:12.261402    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a8e3c54-b7db-486b-83c5-568572c1901f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4a8e3c54-b7db-486b-83c5-568572c1901f" (UID: "4a8e3c54-b7db-486b-83c5-568572c1901f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 05:15:12.362293 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:12.362263    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a8e3c54-b7db-486b-83c5-568572c1901f-maas-api-service-ca\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:15:12.432368 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:12.432334    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:15:13.078160 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:13.078132    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a8c46f-6qnrc"
Sep 14 05:15:13.078160 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:13.078148    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a8c46f-6qnrc" event={"ID":"4a8e3c54-b7db-486b-83c5-568572c1901f","Type":"ContainerDied","Data":"03b466cb58b2b317d0be7292a5811b726c90517620669eb732947285b15cfc26"}
Sep 14 05:15:13.078382 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:13.078192    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03b466cb58b2b317d0be7292a5811b726c90517620669eb732947285b15cfc26"
Sep 14 05:15:20.105479 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:20.105443    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a8c46f-29822715-dx6rz"]
Sep 14 05:15:20.111122 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:20.111099    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a8c46f-29822715-dx6rz"]
Sep 14 05:15:20.138392 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:20.138355    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a8c46f-746b7b7d67-844g5"]
Sep 14 05:15:20.138582 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:20.138557    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a8c46f-746b7b7d67-844g5" podUID="993a22f2-b2c6-465b-b468-ce054475e45b" containerName="payload-processing" containerID="cri-o://85af6f68fff5ec45dc7c39029f0d011f73b1f04bbf43018424678dad8137eaf6" gracePeriod=30
Sep 14 05:15:20.147036 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:20.147010    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a8c46f-8585cb8f4-vjtwx"]
Sep 14 05:15:20.147289 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:20.147243    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a8c46f-8585cb8f4-vjtwx" podUID="fc1256d8-060b-45b4-a613-5b57ea866bc9" containerName="payload-pre-processing" containerID="cri-o://8f5085adadb13bcee960f3221962f4413f2e64cc38cb9a50e66a41736b3c0336" gracePeriod=30
Sep 14 05:15:20.316087 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:20.316052    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a8c46f-6qnrc"]
Sep 14 05:15:20.319905 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:20.319867    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a8c46f-6qnrc"]
Sep 14 05:15:20.334215 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:20.334185    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e94be06-9c94-44f2-89e6-02f9098e7aae" path="/var/lib/kubelet/pods/0e94be06-9c94-44f2-89e6-02f9098e7aae/volumes"
Sep 14 05:15:20.334484 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:20.334470    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a8e3c54-b7db-486b-83c5-568572c1901f" path="/var/lib/kubelet/pods/4a8e3c54-b7db-486b-83c5-568572c1901f/volumes"
Sep 14 05:15:20.406240 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:20.406218    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a8c46f-746b7b7d67-844g5"
Sep 14 05:15:20.435383 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:20.435363    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a8c46f-8585cb8f4-vjtwx"
Sep 14 05:15:20.530740 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:20.530705    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lc7fw\" (UniqueName: \"kubernetes.io/projected/993a22f2-b2c6-465b-b468-ce054475e45b-kube-api-access-lc7fw\") pod \"993a22f2-b2c6-465b-b468-ce054475e45b\" (UID: \"993a22f2-b2c6-465b-b468-ce054475e45b\") "
Sep 14 05:15:20.530905 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:20.530791    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qsnwp\" (UniqueName: \"kubernetes.io/projected/fc1256d8-060b-45b4-a613-5b57ea866bc9-kube-api-access-qsnwp\") pod \"fc1256d8-060b-45b4-a613-5b57ea866bc9\" (UID: \"fc1256d8-060b-45b4-a613-5b57ea866bc9\") "
Sep 14 05:15:20.530905 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:20.530886    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/993a22f2-b2c6-465b-b468-ce054475e45b-plugins-config-volume\") pod \"993a22f2-b2c6-465b-b468-ce054475e45b\" (UID: \"993a22f2-b2c6-465b-b468-ce054475e45b\") "
Sep 14 05:15:20.531041 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:20.530928    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc1256d8-060b-45b4-a613-5b57ea866bc9-plugins-config-volume\") pod \"fc1256d8-060b-45b4-a613-5b57ea866bc9\" (UID: \"fc1256d8-060b-45b4-a613-5b57ea866bc9\") "
Sep 14 05:15:20.531273 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:20.531243    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/993a22f2-b2c6-465b-b468-ce054475e45b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "993a22f2-b2c6-465b-b468-ce054475e45b" (UID: "993a22f2-b2c6-465b-b468-ce054475e45b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 05:15:20.531374 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:20.531278    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc1256d8-060b-45b4-a613-5b57ea866bc9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fc1256d8-060b-45b4-a613-5b57ea866bc9" (UID: "fc1256d8-060b-45b4-a613-5b57ea866bc9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 05:15:20.532917 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:20.532889    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc1256d8-060b-45b4-a613-5b57ea866bc9-kube-api-access-qsnwp" (OuterVolumeSpecName: "kube-api-access-qsnwp") pod "fc1256d8-060b-45b4-a613-5b57ea866bc9" (UID: "fc1256d8-060b-45b4-a613-5b57ea866bc9"). InnerVolumeSpecName "kube-api-access-qsnwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:15:20.533010 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:20.532924    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/993a22f2-b2c6-465b-b468-ce054475e45b-kube-api-access-lc7fw" (OuterVolumeSpecName: "kube-api-access-lc7fw") pod "993a22f2-b2c6-465b-b468-ce054475e45b" (UID: "993a22f2-b2c6-465b-b468-ce054475e45b"). InnerVolumeSpecName "kube-api-access-lc7fw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:15:20.632463 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:20.632420    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc7fw\" (UniqueName: \"kubernetes.io/projected/993a22f2-b2c6-465b-b468-ce054475e45b-kube-api-access-lc7fw\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:15:20.632463 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:20.632458    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsnwp\" (UniqueName: \"kubernetes.io/projected/fc1256d8-060b-45b4-a613-5b57ea866bc9-kube-api-access-qsnwp\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:15:20.632463 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:20.632469    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/993a22f2-b2c6-465b-b468-ce054475e45b-plugins-config-volume\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:15:20.632463 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:20.632478    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc1256d8-060b-45b4-a613-5b57ea866bc9-plugins-config-volume\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:15:21.122545 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:21.122509    2565 generic.go:356] "Generic (PLEG): container finished" podID="993a22f2-b2c6-465b-b468-ce054475e45b" containerID="85af6f68fff5ec45dc7c39029f0d011f73b1f04bbf43018424678dad8137eaf6" exitCode=0
Sep 14 05:15:21.122997 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:21.122557    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a8c46f-746b7b7d67-844g5"
Sep 14 05:15:21.122997 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:21.122597    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a8c46f-746b7b7d67-844g5" event={"ID":"993a22f2-b2c6-465b-b468-ce054475e45b","Type":"ContainerDied","Data":"85af6f68fff5ec45dc7c39029f0d011f73b1f04bbf43018424678dad8137eaf6"}
Sep 14 05:15:21.122997 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:21.122658    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a8c46f-746b7b7d67-844g5" event={"ID":"993a22f2-b2c6-465b-b468-ce054475e45b","Type":"ContainerDied","Data":"d056a4dfc78438f65ed71693f3f94150fac812bc1b88252ab0f58124208bde0a"}
Sep 14 05:15:21.122997 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:21.122683    2565 scope.go:117] "RemoveContainer" containerID="85af6f68fff5ec45dc7c39029f0d011f73b1f04bbf43018424678dad8137eaf6"
Sep 14 05:15:21.124061 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:21.124040    2565 generic.go:356] "Generic (PLEG): container finished" podID="fc1256d8-060b-45b4-a613-5b57ea866bc9" containerID="8f5085adadb13bcee960f3221962f4413f2e64cc38cb9a50e66a41736b3c0336" exitCode=0
Sep 14 05:15:21.124156 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:21.124085    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a8c46f-8585cb8f4-vjtwx"
Sep 14 05:15:21.124156 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:21.124121    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a8c46f-8585cb8f4-vjtwx" event={"ID":"fc1256d8-060b-45b4-a613-5b57ea866bc9","Type":"ContainerDied","Data":"8f5085adadb13bcee960f3221962f4413f2e64cc38cb9a50e66a41736b3c0336"}
Sep 14 05:15:21.124269 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:21.124178    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a8c46f-8585cb8f4-vjtwx" event={"ID":"fc1256d8-060b-45b4-a613-5b57ea866bc9","Type":"ContainerDied","Data":"d20cecff66725e7855a8b823543670225801cf72fbb4a074356da243e7d91280"}
Sep 14 05:15:21.133065 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:21.132920    2565 scope.go:117] "RemoveContainer" containerID="85af6f68fff5ec45dc7c39029f0d011f73b1f04bbf43018424678dad8137eaf6"
Sep 14 05:15:21.133289 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:15:21.133269    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85af6f68fff5ec45dc7c39029f0d011f73b1f04bbf43018424678dad8137eaf6\": container with ID starting with 85af6f68fff5ec45dc7c39029f0d011f73b1f04bbf43018424678dad8137eaf6 not found: ID does not exist" containerID="85af6f68fff5ec45dc7c39029f0d011f73b1f04bbf43018424678dad8137eaf6"
Sep 14 05:15:21.133356 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:21.133297    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85af6f68fff5ec45dc7c39029f0d011f73b1f04bbf43018424678dad8137eaf6"} err="failed to get container status \"85af6f68fff5ec45dc7c39029f0d011f73b1f04bbf43018424678dad8137eaf6\": rpc error: code = NotFound desc = could not find container \"85af6f68fff5ec45dc7c39029f0d011f73b1f04bbf43018424678dad8137eaf6\": container with ID starting with 85af6f68fff5ec45dc7c39029f0d011f73b1f04bbf43018424678dad8137eaf6 not found: ID does not exist"
Sep 14 05:15:21.133356 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:21.133312    2565 scope.go:117] "RemoveContainer" containerID="8f5085adadb13bcee960f3221962f4413f2e64cc38cb9a50e66a41736b3c0336"
Sep 14 05:15:21.141400 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:21.141262    2565 scope.go:117] "RemoveContainer" containerID="8f5085adadb13bcee960f3221962f4413f2e64cc38cb9a50e66a41736b3c0336"
Sep 14 05:15:21.141506 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:15:21.141487    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f5085adadb13bcee960f3221962f4413f2e64cc38cb9a50e66a41736b3c0336\": container with ID starting with 8f5085adadb13bcee960f3221962f4413f2e64cc38cb9a50e66a41736b3c0336 not found: ID does not exist" containerID="8f5085adadb13bcee960f3221962f4413f2e64cc38cb9a50e66a41736b3c0336"
Sep 14 05:15:21.141543 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:21.141512    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f5085adadb13bcee960f3221962f4413f2e64cc38cb9a50e66a41736b3c0336"} err="failed to get container status \"8f5085adadb13bcee960f3221962f4413f2e64cc38cb9a50e66a41736b3c0336\": rpc error: code = NotFound desc = could not find container \"8f5085adadb13bcee960f3221962f4413f2e64cc38cb9a50e66a41736b3c0336\": container with ID starting with 8f5085adadb13bcee960f3221962f4413f2e64cc38cb9a50e66a41736b3c0336 not found: ID does not exist"
Sep 14 05:15:21.147480 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:21.147455    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a8c46f-746b7b7d67-844g5"]
Sep 14 05:15:21.150820 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:21.150799    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a8c46f-746b7b7d67-844g5"]
Sep 14 05:15:21.159331 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:21.159303    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a8c46f-8585cb8f4-vjtwx"]
Sep 14 05:15:21.163346 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:21.163327    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a8c46f-8585cb8f4-vjtwx"]
Sep 14 05:15:22.335455 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:22.335418    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="993a22f2-b2c6-465b-b468-ce054475e45b" path="/var/lib/kubelet/pods/993a22f2-b2c6-465b-b468-ce054475e45b/volumes"
Sep 14 05:15:22.335990 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:22.335970    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc1256d8-060b-45b4-a613-5b57ea866bc9" path="/var/lib/kubelet/pods/fc1256d8-060b-45b4-a613-5b57ea866bc9/volumes"
Sep 14 05:15:24.532552 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:24.532517    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:15:34.137515 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:34.137483    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-27be50/sub-test-model-27be50-kserve-5bb56cd7fc-zdl65"]
Sep 14 05:15:34.138001 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:34.137801    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a8e3c54-b7db-486b-83c5-568572c1901f" containerName="revoke-keys"
Sep 14 05:15:34.138001 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:34.137813    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a8e3c54-b7db-486b-83c5-568572c1901f" containerName="revoke-keys"
Sep 14 05:15:34.138001 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:34.137828    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="993a22f2-b2c6-465b-b468-ce054475e45b" containerName="payload-processing"
Sep 14 05:15:34.138001 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:34.137834    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="993a22f2-b2c6-465b-b468-ce054475e45b" containerName="payload-processing"
Sep 14 05:15:34.138001 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:34.137843    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc1256d8-060b-45b4-a613-5b57ea866bc9" containerName="payload-pre-processing"
Sep 14 05:15:34.138001 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:34.137849    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc1256d8-060b-45b4-a613-5b57ea866bc9" containerName="payload-pre-processing"
Sep 14 05:15:34.138001 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:34.137901    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc1256d8-060b-45b4-a613-5b57ea866bc9" containerName="payload-pre-processing"
Sep 14 05:15:34.138001 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:34.137910    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a8e3c54-b7db-486b-83c5-568572c1901f" containerName="revoke-keys"
Sep 14 05:15:34.138001 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:34.137916    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="993a22f2-b2c6-465b-b468-ce054475e45b" containerName="payload-processing"
Sep 14 05:15:34.141698 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:34.141681    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-27be50/sub-test-model-27be50-kserve-5bb56cd7fc-zdl65"
Sep 14 05:15:34.144068 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:34.144048    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-27be50\"/\"openshift-service-ca.crt\""
Sep 14 05:15:34.144934 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:34.144917    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-27be50\"/\"default-dockercfg-m8bz9\""
Sep 14 05:15:34.145019 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:34.144949    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-27be50\"/\"sub-test-model-27be50-kserve-self-signed-certs\""
Sep 14 05:15:34.145019 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:34.144975    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-27be50\"/\"kube-root-ca.crt\""
Sep 14 05:15:34.149592 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:34.149572    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-27be50/sub-test-model-27be50-kserve-5bb56cd7fc-zdl65"]
Sep 14 05:15:34.254134 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:34.254083    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/545edddc-c8eb-4701-93ba-d8c74a2399e8-dshm\") pod \"sub-test-model-27be50-kserve-5bb56cd7fc-zdl65\" (UID: \"545edddc-c8eb-4701-93ba-d8c74a2399e8\") " pod="ai-tenant-e2e-shared-b-w3-27be50/sub-test-model-27be50-kserve-5bb56cd7fc-zdl65"
Sep 14 05:15:34.254134 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:34.254139    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/545edddc-c8eb-4701-93ba-d8c74a2399e8-home\") pod \"sub-test-model-27be50-kserve-5bb56cd7fc-zdl65\" (UID: \"545edddc-c8eb-4701-93ba-d8c74a2399e8\") " pod="ai-tenant-e2e-shared-b-w3-27be50/sub-test-model-27be50-kserve-5bb56cd7fc-zdl65"
Sep 14 05:15:34.254363 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:34.254197    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdgvr\" (UniqueName: \"kubernetes.io/projected/545edddc-c8eb-4701-93ba-d8c74a2399e8-kube-api-access-mdgvr\") pod \"sub-test-model-27be50-kserve-5bb56cd7fc-zdl65\" (UID: \"545edddc-c8eb-4701-93ba-d8c74a2399e8\") " pod="ai-tenant-e2e-shared-b-w3-27be50/sub-test-model-27be50-kserve-5bb56cd7fc-zdl65"
Sep 14 05:15:34.254363 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:34.254287    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/545edddc-c8eb-4701-93ba-d8c74a2399e8-tls-certs\") pod \"sub-test-model-27be50-kserve-5bb56cd7fc-zdl65\" (UID: \"545edddc-c8eb-4701-93ba-d8c74a2399e8\") " pod="ai-tenant-e2e-shared-b-w3-27be50/sub-test-model-27be50-kserve-5bb56cd7fc-zdl65"
Sep 14 05:15:34.254363 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:34.254352    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/545edddc-c8eb-4701-93ba-d8c74a2399e8-tmp-dir\") pod \"sub-test-model-27be50-kserve-5bb56cd7fc-zdl65\" (UID: \"545edddc-c8eb-4701-93ba-d8c74a2399e8\") " pod="ai-tenant-e2e-shared-b-w3-27be50/sub-test-model-27be50-kserve-5bb56cd7fc-zdl65"
Sep 14 05:15:34.254487 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:34.254380    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/545edddc-c8eb-4701-93ba-d8c74a2399e8-model-cache\") pod \"sub-test-model-27be50-kserve-5bb56cd7fc-zdl65\" (UID: \"545edddc-c8eb-4701-93ba-d8c74a2399e8\") " pod="ai-tenant-e2e-shared-b-w3-27be50/sub-test-model-27be50-kserve-5bb56cd7fc-zdl65"
Sep 14 05:15:34.355233 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:34.355200    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/545edddc-c8eb-4701-93ba-d8c74a2399e8-tmp-dir\") pod \"sub-test-model-27be50-kserve-5bb56cd7fc-zdl65\" (UID: \"545edddc-c8eb-4701-93ba-d8c74a2399e8\") " pod="ai-tenant-e2e-shared-b-w3-27be50/sub-test-model-27be50-kserve-5bb56cd7fc-zdl65"
Sep 14 05:15:34.355404 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:34.355241    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/545edddc-c8eb-4701-93ba-d8c74a2399e8-model-cache\") pod \"sub-test-model-27be50-kserve-5bb56cd7fc-zdl65\" (UID: \"545edddc-c8eb-4701-93ba-d8c74a2399e8\") " pod="ai-tenant-e2e-shared-b-w3-27be50/sub-test-model-27be50-kserve-5bb56cd7fc-zdl65"
Sep 14 05:15:34.355404 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:34.355270    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/545edddc-c8eb-4701-93ba-d8c74a2399e8-dshm\") pod \"sub-test-model-27be50-kserve-5bb56cd7fc-zdl65\" (UID: \"545edddc-c8eb-4701-93ba-d8c74a2399e8\") " pod="ai-tenant-e2e-shared-b-w3-27be50/sub-test-model-27be50-kserve-5bb56cd7fc-zdl65"
Sep 14 05:15:34.355404 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:34.355299    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/545edddc-c8eb-4701-93ba-d8c74a2399e8-home\") pod \"sub-test-model-27be50-kserve-5bb56cd7fc-zdl65\" (UID: \"545edddc-c8eb-4701-93ba-d8c74a2399e8\") " pod="ai-tenant-e2e-shared-b-w3-27be50/sub-test-model-27be50-kserve-5bb56cd7fc-zdl65"
Sep 14 05:15:34.355404 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:34.355334    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdgvr\" (UniqueName: \"kubernetes.io/projected/545edddc-c8eb-4701-93ba-d8c74a2399e8-kube-api-access-mdgvr\") pod \"sub-test-model-27be50-kserve-5bb56cd7fc-zdl65\" (UID: \"545edddc-c8eb-4701-93ba-d8c74a2399e8\") " pod="ai-tenant-e2e-shared-b-w3-27be50/sub-test-model-27be50-kserve-5bb56cd7fc-zdl65"
Sep 14 05:15:34.355404 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:34.355397    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/545edddc-c8eb-4701-93ba-d8c74a2399e8-tls-certs\") pod \"sub-test-model-27be50-kserve-5bb56cd7fc-zdl65\" (UID: \"545edddc-c8eb-4701-93ba-d8c74a2399e8\") " pod="ai-tenant-e2e-shared-b-w3-27be50/sub-test-model-27be50-kserve-5bb56cd7fc-zdl65"
Sep 14 05:15:34.355652 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:34.355566    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/545edddc-c8eb-4701-93ba-d8c74a2399e8-model-cache\") pod \"sub-test-model-27be50-kserve-5bb56cd7fc-zdl65\" (UID: \"545edddc-c8eb-4701-93ba-d8c74a2399e8\") " pod="ai-tenant-e2e-shared-b-w3-27be50/sub-test-model-27be50-kserve-5bb56cd7fc-zdl65"
Sep 14 05:15:34.355652 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:34.355637    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/545edddc-c8eb-4701-93ba-d8c74a2399e8-tmp-dir\") pod \"sub-test-model-27be50-kserve-5bb56cd7fc-zdl65\" (UID: \"545edddc-c8eb-4701-93ba-d8c74a2399e8\") " pod="ai-tenant-e2e-shared-b-w3-27be50/sub-test-model-27be50-kserve-5bb56cd7fc-zdl65"
Sep 14 05:15:34.355795 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:34.355745    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/545edddc-c8eb-4701-93ba-d8c74a2399e8-home\") pod \"sub-test-model-27be50-kserve-5bb56cd7fc-zdl65\" (UID: \"545edddc-c8eb-4701-93ba-d8c74a2399e8\") " pod="ai-tenant-e2e-shared-b-w3-27be50/sub-test-model-27be50-kserve-5bb56cd7fc-zdl65"
Sep 14 05:15:34.357500 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:34.357479    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/545edddc-c8eb-4701-93ba-d8c74a2399e8-dshm\") pod \"sub-test-model-27be50-kserve-5bb56cd7fc-zdl65\" (UID: \"545edddc-c8eb-4701-93ba-d8c74a2399e8\") " pod="ai-tenant-e2e-shared-b-w3-27be50/sub-test-model-27be50-kserve-5bb56cd7fc-zdl65"
Sep 14 05:15:34.357824 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:34.357809    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/545edddc-c8eb-4701-93ba-d8c74a2399e8-tls-certs\") pod \"sub-test-model-27be50-kserve-5bb56cd7fc-zdl65\" (UID: \"545edddc-c8eb-4701-93ba-d8c74a2399e8\") " pod="ai-tenant-e2e-shared-b-w3-27be50/sub-test-model-27be50-kserve-5bb56cd7fc-zdl65"
Sep 14 05:15:34.363914 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:34.363889    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdgvr\" (UniqueName: \"kubernetes.io/projected/545edddc-c8eb-4701-93ba-d8c74a2399e8-kube-api-access-mdgvr\") pod \"sub-test-model-27be50-kserve-5bb56cd7fc-zdl65\" (UID: \"545edddc-c8eb-4701-93ba-d8c74a2399e8\") " pod="ai-tenant-e2e-shared-b-w3-27be50/sub-test-model-27be50-kserve-5bb56cd7fc-zdl65"
Sep 14 05:15:34.452696 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:34.452601    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-27be50/sub-test-model-27be50-kserve-5bb56cd7fc-zdl65"
Sep 14 05:15:34.726014 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:34.725988    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-27be50/sub-test-model-27be50-kserve-5bb56cd7fc-zdl65"]
Sep 14 05:15:34.727566 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:15:34.727540    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod545edddc_c8eb_4701_93ba_d8c74a2399e8.slice/crio-f3bde3497dc078215e7736d2316f48aff949291506ab74b24ef091f43c8cbcdd WatchSource:0}: Error finding container f3bde3497dc078215e7736d2316f48aff949291506ab74b24ef091f43c8cbcdd: Status 404 returned error can't find the container with id f3bde3497dc078215e7736d2316f48aff949291506ab74b24ef091f43c8cbcdd
Sep 14 05:15:35.209274 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:35.209234    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-27be50/sub-test-model-27be50-kserve-5bb56cd7fc-zdl65" event={"ID":"545edddc-c8eb-4701-93ba-d8c74a2399e8","Type":"ContainerStarted","Data":"1eb2a4ebf77f846ff04cf44fc5235a7d7972297a3fc711c94a55217bc1b37d96"}
Sep 14 05:15:35.209274 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:35.209278    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-27be50/sub-test-model-27be50-kserve-5bb56cd7fc-zdl65" event={"ID":"545edddc-c8eb-4701-93ba-d8c74a2399e8","Type":"ContainerStarted","Data":"f3bde3497dc078215e7736d2316f48aff949291506ab74b24ef091f43c8cbcdd"}
Sep 14 05:15:35.229732 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:35.229678    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-27be50/sub-test-model-27be50-kserve-5bb56cd7fc-zdl65" podStartSLOduration=1.229663563 podStartE2EDuration="1.229663563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:15:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:15:35.227790554 +0000 UTC m=+1733.403562114" watchObservedRunningTime="2026-09-14 05:15:35.229663563 +0000 UTC m=+1733.405435121"
Sep 14 05:15:38.929217 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:38.929181    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:15:44.452729 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:44.452691    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-27be50/sub-test-model-27be50-kserve-5bb56cd7fc-zdl65"
Sep 14 05:15:44.453177 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:44.452743    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-27be50/sub-test-model-27be50-kserve-5bb56cd7fc-zdl65"
Sep 14 05:15:44.465604 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:44.465579    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-27be50/sub-test-model-27be50-kserve-5bb56cd7fc-zdl65"
Sep 14 05:15:45.279504 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:45.279475    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-27be50/sub-test-model-27be50-kserve-5bb56cd7fc-zdl65"
Sep 14 05:15:52.452953 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:15:52.452917    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:16:05.942483 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:05.942448    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:16:14.254026 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:14.253989    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2ff4a6/sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg"]
Sep 14 05:16:15.357655 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:15.357623    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-27be50/sub-test-model-27be50-kserve-5bb56cd7fc-zdl65"]
Sep 14 05:16:17.429740 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:17.429703    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:16:22.537330 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:22.537287    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-2ff4a6/rate-test-model-2ff4a6-kserve-594958d69f-lg8ql"]
Sep 14 05:16:22.541988 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:22.541958    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2ff4a6/rate-test-model-2ff4a6-kserve-594958d69f-lg8ql"
Sep 14 05:16:22.544949 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:22.544920    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2ff4a6\"/\"rate-test-model-2ff4a6-kserve-self-signed-certs\""
Sep 14 05:16:22.553866 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:22.553831    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2ff4a6/rate-test-model-2ff4a6-kserve-594958d69f-lg8ql"]
Sep 14 05:16:22.662619 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:22.662591    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/377c1063-c80a-41c5-91c9-5c9b265acebf-model-cache\") pod \"rate-test-model-2ff4a6-kserve-594958d69f-lg8ql\" (UID: \"377c1063-c80a-41c5-91c9-5c9b265acebf\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/rate-test-model-2ff4a6-kserve-594958d69f-lg8ql"
Sep 14 05:16:22.662778 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:22.662623    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/377c1063-c80a-41c5-91c9-5c9b265acebf-tls-certs\") pod \"rate-test-model-2ff4a6-kserve-594958d69f-lg8ql\" (UID: \"377c1063-c80a-41c5-91c9-5c9b265acebf\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/rate-test-model-2ff4a6-kserve-594958d69f-lg8ql"
Sep 14 05:16:22.662778 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:22.662660    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fkth\" (UniqueName: \"kubernetes.io/projected/377c1063-c80a-41c5-91c9-5c9b265acebf-kube-api-access-9fkth\") pod \"rate-test-model-2ff4a6-kserve-594958d69f-lg8ql\" (UID: \"377c1063-c80a-41c5-91c9-5c9b265acebf\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/rate-test-model-2ff4a6-kserve-594958d69f-lg8ql"
Sep 14 05:16:22.662778 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:22.662716    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/377c1063-c80a-41c5-91c9-5c9b265acebf-home\") pod \"rate-test-model-2ff4a6-kserve-594958d69f-lg8ql\" (UID: \"377c1063-c80a-41c5-91c9-5c9b265acebf\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/rate-test-model-2ff4a6-kserve-594958d69f-lg8ql"
Sep 14 05:16:22.662962 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:22.662787    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/377c1063-c80a-41c5-91c9-5c9b265acebf-tmp-dir\") pod \"rate-test-model-2ff4a6-kserve-594958d69f-lg8ql\" (UID: \"377c1063-c80a-41c5-91c9-5c9b265acebf\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/rate-test-model-2ff4a6-kserve-594958d69f-lg8ql"
Sep 14 05:16:22.662962 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:22.662830    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/377c1063-c80a-41c5-91c9-5c9b265acebf-dshm\") pod \"rate-test-model-2ff4a6-kserve-594958d69f-lg8ql\" (UID: \"377c1063-c80a-41c5-91c9-5c9b265acebf\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/rate-test-model-2ff4a6-kserve-594958d69f-lg8ql"
Sep 14 05:16:22.763304 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:22.763272    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fkth\" (UniqueName: \"kubernetes.io/projected/377c1063-c80a-41c5-91c9-5c9b265acebf-kube-api-access-9fkth\") pod \"rate-test-model-2ff4a6-kserve-594958d69f-lg8ql\" (UID: \"377c1063-c80a-41c5-91c9-5c9b265acebf\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/rate-test-model-2ff4a6-kserve-594958d69f-lg8ql"
Sep 14 05:16:22.763517 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:22.763322    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/377c1063-c80a-41c5-91c9-5c9b265acebf-home\") pod \"rate-test-model-2ff4a6-kserve-594958d69f-lg8ql\" (UID: \"377c1063-c80a-41c5-91c9-5c9b265acebf\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/rate-test-model-2ff4a6-kserve-594958d69f-lg8ql"
Sep 14 05:16:22.763517 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:22.763351    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/377c1063-c80a-41c5-91c9-5c9b265acebf-tmp-dir\") pod \"rate-test-model-2ff4a6-kserve-594958d69f-lg8ql\" (UID: \"377c1063-c80a-41c5-91c9-5c9b265acebf\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/rate-test-model-2ff4a6-kserve-594958d69f-lg8ql"
Sep 14 05:16:22.763517 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:22.763395    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/377c1063-c80a-41c5-91c9-5c9b265acebf-dshm\") pod \"rate-test-model-2ff4a6-kserve-594958d69f-lg8ql\" (UID: \"377c1063-c80a-41c5-91c9-5c9b265acebf\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/rate-test-model-2ff4a6-kserve-594958d69f-lg8ql"
Sep 14 05:16:22.763517 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:22.763431    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/377c1063-c80a-41c5-91c9-5c9b265acebf-model-cache\") pod \"rate-test-model-2ff4a6-kserve-594958d69f-lg8ql\" (UID: \"377c1063-c80a-41c5-91c9-5c9b265acebf\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/rate-test-model-2ff4a6-kserve-594958d69f-lg8ql"
Sep 14 05:16:22.763517 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:22.763454    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/377c1063-c80a-41c5-91c9-5c9b265acebf-tls-certs\") pod \"rate-test-model-2ff4a6-kserve-594958d69f-lg8ql\" (UID: \"377c1063-c80a-41c5-91c9-5c9b265acebf\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/rate-test-model-2ff4a6-kserve-594958d69f-lg8ql"
Sep 14 05:16:22.763827 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:22.763800    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/377c1063-c80a-41c5-91c9-5c9b265acebf-home\") pod \"rate-test-model-2ff4a6-kserve-594958d69f-lg8ql\" (UID: \"377c1063-c80a-41c5-91c9-5c9b265acebf\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/rate-test-model-2ff4a6-kserve-594958d69f-lg8ql"
Sep 14 05:16:22.763902 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:22.763845    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/377c1063-c80a-41c5-91c9-5c9b265acebf-tmp-dir\") pod \"rate-test-model-2ff4a6-kserve-594958d69f-lg8ql\" (UID: \"377c1063-c80a-41c5-91c9-5c9b265acebf\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/rate-test-model-2ff4a6-kserve-594958d69f-lg8ql"
Sep 14 05:16:22.763902 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:22.763875    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/377c1063-c80a-41c5-91c9-5c9b265acebf-model-cache\") pod \"rate-test-model-2ff4a6-kserve-594958d69f-lg8ql\" (UID: \"377c1063-c80a-41c5-91c9-5c9b265acebf\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/rate-test-model-2ff4a6-kserve-594958d69f-lg8ql"
Sep 14 05:16:22.766029 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:22.766002    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/377c1063-c80a-41c5-91c9-5c9b265acebf-dshm\") pod \"rate-test-model-2ff4a6-kserve-594958d69f-lg8ql\" (UID: \"377c1063-c80a-41c5-91c9-5c9b265acebf\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/rate-test-model-2ff4a6-kserve-594958d69f-lg8ql"
Sep 14 05:16:22.766235 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:22.766213    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/377c1063-c80a-41c5-91c9-5c9b265acebf-tls-certs\") pod \"rate-test-model-2ff4a6-kserve-594958d69f-lg8ql\" (UID: \"377c1063-c80a-41c5-91c9-5c9b265acebf\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/rate-test-model-2ff4a6-kserve-594958d69f-lg8ql"
Sep 14 05:16:22.774448 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:22.774429    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fkth\" (UniqueName: \"kubernetes.io/projected/377c1063-c80a-41c5-91c9-5c9b265acebf-kube-api-access-9fkth\") pod \"rate-test-model-2ff4a6-kserve-594958d69f-lg8ql\" (UID: \"377c1063-c80a-41c5-91c9-5c9b265acebf\") " pod="ai-tenant-e2e-shared-a-w3-2ff4a6/rate-test-model-2ff4a6-kserve-594958d69f-lg8ql"
Sep 14 05:16:22.851509 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:22.851484    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2ff4a6/rate-test-model-2ff4a6-kserve-594958d69f-lg8ql"
Sep 14 05:16:23.141813 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:23.141786    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2ff4a6/rate-test-model-2ff4a6-kserve-594958d69f-lg8ql"]
Sep 14 05:16:23.143291 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:16:23.143263    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod377c1063_c80a_41c5_91c9_5c9b265acebf.slice/crio-d9a3d92298bddb699137f3c1b3953e24a89699c57a1dc79d47bd6ece5794bc73 WatchSource:0}: Error finding container d9a3d92298bddb699137f3c1b3953e24a89699c57a1dc79d47bd6ece5794bc73: Status 404 returned error can't find the container with id d9a3d92298bddb699137f3c1b3953e24a89699c57a1dc79d47bd6ece5794bc73
Sep 14 05:16:23.474569 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:23.474481    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2ff4a6/rate-test-model-2ff4a6-kserve-594958d69f-lg8ql" event={"ID":"377c1063-c80a-41c5-91c9-5c9b265acebf","Type":"ContainerStarted","Data":"e23abe5e3f09de260f94578c4645599c02c1467fb7d4cdd12c0ce4271c43eaa7"}
Sep 14 05:16:23.474569 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:23.474531    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2ff4a6/rate-test-model-2ff4a6-kserve-594958d69f-lg8ql" event={"ID":"377c1063-c80a-41c5-91c9-5c9b265acebf","Type":"ContainerStarted","Data":"d9a3d92298bddb699137f3c1b3953e24a89699c57a1dc79d47bd6ece5794bc73"}
Sep 14 05:16:23.494489 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:23.494442    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-2ff4a6/rate-test-model-2ff4a6-kserve-594958d69f-lg8ql" podStartSLOduration=1.49442528 podStartE2EDuration="1.49442528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:16:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:16:23.49262056 +0000 UTC m=+1781.668392119" watchObservedRunningTime="2026-09-14 05:16:23.49442528 +0000 UTC m=+1781.670196840"
Sep 14 05:16:29.272460 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:29.272413    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-2ff4a6/sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg" podUID="f3929840-c8be-43d0-ad00-d669e75f8cdc" containerName="main" containerID="cri-o://8503981043f042244e8116cb8a6fd4f7e6a42ea58db774b1518e8a13d0693243" gracePeriod=45
Sep 14 05:16:29.516742 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:29.516711    2565 generic.go:356] "Generic (PLEG): container finished" podID="f3929840-c8be-43d0-ad00-d669e75f8cdc" containerID="8503981043f042244e8116cb8a6fd4f7e6a42ea58db774b1518e8a13d0693243" exitCode=0
Sep 14 05:16:29.516874 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:29.516789    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2ff4a6/sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg" event={"ID":"f3929840-c8be-43d0-ad00-d669e75f8cdc","Type":"ContainerDied","Data":"8503981043f042244e8116cb8a6fd4f7e6a42ea58db774b1518e8a13d0693243"}
Sep 14 05:16:29.516874 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:29.516823    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2ff4a6/sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg" event={"ID":"f3929840-c8be-43d0-ad00-d669e75f8cdc","Type":"ContainerDied","Data":"8ff7024e31d671f7318efa7bf5059cdd00d688a54314113de5dfdc9d927bdf3f"}
Sep 14 05:16:29.516874 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:29.516834    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ff7024e31d671f7318efa7bf5059cdd00d688a54314113de5dfdc9d927bdf3f"
Sep 14 05:16:29.521568 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:29.521548    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2ff4a6/sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg"
Sep 14 05:16:29.625699 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:29.625675    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3929840-c8be-43d0-ad00-d669e75f8cdc-dshm\") pod \"f3929840-c8be-43d0-ad00-d669e75f8cdc\" (UID: \"f3929840-c8be-43d0-ad00-d669e75f8cdc\") "
Sep 14 05:16:29.625852 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:29.625728    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3929840-c8be-43d0-ad00-d669e75f8cdc-tmp-dir\") pod \"f3929840-c8be-43d0-ad00-d669e75f8cdc\" (UID: \"f3929840-c8be-43d0-ad00-d669e75f8cdc\") "
Sep 14 05:16:29.625852 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:29.625804    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3929840-c8be-43d0-ad00-d669e75f8cdc-model-cache\") pod \"f3929840-c8be-43d0-ad00-d669e75f8cdc\" (UID: \"f3929840-c8be-43d0-ad00-d669e75f8cdc\") "
Sep 14 05:16:29.625852 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:29.625830    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6kxt\" (UniqueName: \"kubernetes.io/projected/f3929840-c8be-43d0-ad00-d669e75f8cdc-kube-api-access-k6kxt\") pod \"f3929840-c8be-43d0-ad00-d669e75f8cdc\" (UID: \"f3929840-c8be-43d0-ad00-d669e75f8cdc\") "
Sep 14 05:16:29.625852 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:29.625850    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3929840-c8be-43d0-ad00-d669e75f8cdc-tls-certs\") pod \"f3929840-c8be-43d0-ad00-d669e75f8cdc\" (UID: \"f3929840-c8be-43d0-ad00-d669e75f8cdc\") "
Sep 14 05:16:29.626075 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:29.625865    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3929840-c8be-43d0-ad00-d669e75f8cdc-home\") pod \"f3929840-c8be-43d0-ad00-d669e75f8cdc\" (UID: \"f3929840-c8be-43d0-ad00-d669e75f8cdc\") "
Sep 14 05:16:29.626138 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:29.626057    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3929840-c8be-43d0-ad00-d669e75f8cdc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f3929840-c8be-43d0-ad00-d669e75f8cdc" (UID: "f3929840-c8be-43d0-ad00-d669e75f8cdc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:16:29.626138 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:29.626097    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3929840-c8be-43d0-ad00-d669e75f8cdc-model-cache" (OuterVolumeSpecName: "model-cache") pod "f3929840-c8be-43d0-ad00-d669e75f8cdc" (UID: "f3929840-c8be-43d0-ad00-d669e75f8cdc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:16:29.626264 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:29.626226    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3929840-c8be-43d0-ad00-d669e75f8cdc-home" (OuterVolumeSpecName: "home") pod "f3929840-c8be-43d0-ad00-d669e75f8cdc" (UID: "f3929840-c8be-43d0-ad00-d669e75f8cdc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:16:29.627836 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:29.627811    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3929840-c8be-43d0-ad00-d669e75f8cdc-dshm" (OuterVolumeSpecName: "dshm") pod "f3929840-c8be-43d0-ad00-d669e75f8cdc" (UID: "f3929840-c8be-43d0-ad00-d669e75f8cdc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:16:29.627996 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:29.627973    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3929840-c8be-43d0-ad00-d669e75f8cdc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f3929840-c8be-43d0-ad00-d669e75f8cdc" (UID: "f3929840-c8be-43d0-ad00-d669e75f8cdc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 05:16:29.627996 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:29.627984    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3929840-c8be-43d0-ad00-d669e75f8cdc-kube-api-access-k6kxt" (OuterVolumeSpecName: "kube-api-access-k6kxt") pod "f3929840-c8be-43d0-ad00-d669e75f8cdc" (UID: "f3929840-c8be-43d0-ad00-d669e75f8cdc"). InnerVolumeSpecName "kube-api-access-k6kxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:16:29.726667 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:29.726635    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3929840-c8be-43d0-ad00-d669e75f8cdc-dshm\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:16:29.726811 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:29.726685    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3929840-c8be-43d0-ad00-d669e75f8cdc-tmp-dir\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:16:29.726811 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:29.726698    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3929840-c8be-43d0-ad00-d669e75f8cdc-model-cache\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:16:29.726811 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:29.726711    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6kxt\" (UniqueName: \"kubernetes.io/projected/f3929840-c8be-43d0-ad00-d669e75f8cdc-kube-api-access-k6kxt\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:16:29.726811 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:29.726723    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3929840-c8be-43d0-ad00-d669e75f8cdc-tls-certs\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:16:29.726811 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:29.726736    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3929840-c8be-43d0-ad00-d669e75f8cdc-home\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:16:30.376100 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:30.376050    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-27be50/sub-test-model-27be50-kserve-5bb56cd7fc-zdl65" podUID="545edddc-c8eb-4701-93ba-d8c74a2399e8" containerName="main" containerID="cri-o://1eb2a4ebf77f846ff04cf44fc5235a7d7972297a3fc711c94a55217bc1b37d96" gracePeriod=45
Sep 14 05:16:30.524883 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:30.524850    2565 generic.go:356] "Generic (PLEG): container finished" podID="545edddc-c8eb-4701-93ba-d8c74a2399e8" containerID="1eb2a4ebf77f846ff04cf44fc5235a7d7972297a3fc711c94a55217bc1b37d96" exitCode=0
Sep 14 05:16:30.525038 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:30.524918    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-27be50/sub-test-model-27be50-kserve-5bb56cd7fc-zdl65" event={"ID":"545edddc-c8eb-4701-93ba-d8c74a2399e8","Type":"ContainerDied","Data":"1eb2a4ebf77f846ff04cf44fc5235a7d7972297a3fc711c94a55217bc1b37d96"}
Sep 14 05:16:30.525038 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:30.524995    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2ff4a6/sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg"
Sep 14 05:16:30.547448 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:30.547420    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2ff4a6/sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg"]
Sep 14 05:16:30.555328 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:30.555301    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2ff4a6/sub-test-model-2ff4a6-kserve-5f68b5c469-msqbg"]
Sep 14 05:16:30.629636 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:30.629614    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-27be50/sub-test-model-27be50-kserve-5bb56cd7fc-zdl65"
Sep 14 05:16:30.733392 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:30.733356    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/545edddc-c8eb-4701-93ba-d8c74a2399e8-tls-certs\") pod \"545edddc-c8eb-4701-93ba-d8c74a2399e8\" (UID: \"545edddc-c8eb-4701-93ba-d8c74a2399e8\") "
Sep 14 05:16:30.733531 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:30.733405    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/545edddc-c8eb-4701-93ba-d8c74a2399e8-home\") pod \"545edddc-c8eb-4701-93ba-d8c74a2399e8\" (UID: \"545edddc-c8eb-4701-93ba-d8c74a2399e8\") "
Sep 14 05:16:30.733531 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:30.733434    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/545edddc-c8eb-4701-93ba-d8c74a2399e8-model-cache\") pod \"545edddc-c8eb-4701-93ba-d8c74a2399e8\" (UID: \"545edddc-c8eb-4701-93ba-d8c74a2399e8\") "
Sep 14 05:16:30.733531 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:30.733487    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/545edddc-c8eb-4701-93ba-d8c74a2399e8-dshm\") pod \"545edddc-c8eb-4701-93ba-d8c74a2399e8\" (UID: \"545edddc-c8eb-4701-93ba-d8c74a2399e8\") "
Sep 14 05:16:30.733663 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:30.733531    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/545edddc-c8eb-4701-93ba-d8c74a2399e8-tmp-dir\") pod \"545edddc-c8eb-4701-93ba-d8c74a2399e8\" (UID: \"545edddc-c8eb-4701-93ba-d8c74a2399e8\") "
Sep 14 05:16:30.733663 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:30.733590    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdgvr\" (UniqueName: \"kubernetes.io/projected/545edddc-c8eb-4701-93ba-d8c74a2399e8-kube-api-access-mdgvr\") pod \"545edddc-c8eb-4701-93ba-d8c74a2399e8\" (UID: \"545edddc-c8eb-4701-93ba-d8c74a2399e8\") "
Sep 14 05:16:30.733808 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:30.733686    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/545edddc-c8eb-4701-93ba-d8c74a2399e8-home" (OuterVolumeSpecName: "home") pod "545edddc-c8eb-4701-93ba-d8c74a2399e8" (UID: "545edddc-c8eb-4701-93ba-d8c74a2399e8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:16:30.733808 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:30.733740    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/545edddc-c8eb-4701-93ba-d8c74a2399e8-model-cache" (OuterVolumeSpecName: "model-cache") pod "545edddc-c8eb-4701-93ba-d8c74a2399e8" (UID: "545edddc-c8eb-4701-93ba-d8c74a2399e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:16:30.733926 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:30.733827    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/545edddc-c8eb-4701-93ba-d8c74a2399e8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "545edddc-c8eb-4701-93ba-d8c74a2399e8" (UID: "545edddc-c8eb-4701-93ba-d8c74a2399e8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:16:30.733987 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:30.733934    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/545edddc-c8eb-4701-93ba-d8c74a2399e8-tmp-dir\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:16:30.733987 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:30.733954    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/545edddc-c8eb-4701-93ba-d8c74a2399e8-home\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:16:30.733987 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:30.733968    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/545edddc-c8eb-4701-93ba-d8c74a2399e8-model-cache\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:16:30.735550 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:30.735530    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/545edddc-c8eb-4701-93ba-d8c74a2399e8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "545edddc-c8eb-4701-93ba-d8c74a2399e8" (UID: "545edddc-c8eb-4701-93ba-d8c74a2399e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 05:16:30.735695 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:30.735679    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/545edddc-c8eb-4701-93ba-d8c74a2399e8-dshm" (OuterVolumeSpecName: "dshm") pod "545edddc-c8eb-4701-93ba-d8c74a2399e8" (UID: "545edddc-c8eb-4701-93ba-d8c74a2399e8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:16:30.735747 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:30.735730    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/545edddc-c8eb-4701-93ba-d8c74a2399e8-kube-api-access-mdgvr" (OuterVolumeSpecName: "kube-api-access-mdgvr") pod "545edddc-c8eb-4701-93ba-d8c74a2399e8" (UID: "545edddc-c8eb-4701-93ba-d8c74a2399e8"). InnerVolumeSpecName "kube-api-access-mdgvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:16:30.834426 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:30.834402    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/545edddc-c8eb-4701-93ba-d8c74a2399e8-dshm\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:16:30.834426 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:30.834424    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdgvr\" (UniqueName: \"kubernetes.io/projected/545edddc-c8eb-4701-93ba-d8c74a2399e8-kube-api-access-mdgvr\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:16:30.834563 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:30.834433    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/545edddc-c8eb-4701-93ba-d8c74a2399e8-tls-certs\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:16:31.532411 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:31.532374    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-27be50/sub-test-model-27be50-kserve-5bb56cd7fc-zdl65" event={"ID":"545edddc-c8eb-4701-93ba-d8c74a2399e8","Type":"ContainerDied","Data":"f3bde3497dc078215e7736d2316f48aff949291506ab74b24ef091f43c8cbcdd"}
Sep 14 05:16:31.532899 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:31.532391    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-27be50/sub-test-model-27be50-kserve-5bb56cd7fc-zdl65"
Sep 14 05:16:31.532899 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:31.532422    2565 scope.go:117] "RemoveContainer" containerID="1eb2a4ebf77f846ff04cf44fc5235a7d7972297a3fc711c94a55217bc1b37d96"
Sep 14 05:16:31.560435 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:31.560410    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-27be50/sub-test-model-27be50-kserve-5bb56cd7fc-zdl65"]
Sep 14 05:16:31.564356 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:31.564335    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-27be50/sub-test-model-27be50-kserve-5bb56cd7fc-zdl65"]
Sep 14 05:16:32.334613 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:32.334577    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="545edddc-c8eb-4701-93ba-d8c74a2399e8" path="/var/lib/kubelet/pods/545edddc-c8eb-4701-93ba-d8c74a2399e8/volumes"
Sep 14 05:16:32.335008 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:32.334994    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3929840-c8be-43d0-ad00-d669e75f8cdc" path="/var/lib/kubelet/pods/f3929840-c8be-43d0-ad00-d669e75f8cdc/volumes"
Sep 14 05:16:32.477178 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:32.477146    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:16:32.852432 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:32.852401    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-2ff4a6/rate-test-model-2ff4a6-kserve-594958d69f-lg8ql"
Sep 14 05:16:32.852819 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:32.852477    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-2ff4a6/rate-test-model-2ff4a6-kserve-594958d69f-lg8ql"
Sep 14 05:16:32.864907 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:32.864886    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-2ff4a6/rate-test-model-2ff4a6-kserve-594958d69f-lg8ql"
Sep 14 05:16:33.555637 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:33.555602    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-2ff4a6/rate-test-model-2ff4a6-kserve-594958d69f-lg8ql"
Sep 14 05:16:42.430711 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:42.430685    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kndxq_e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae/ovn-acl-logging/0.log"
Sep 14 05:16:42.437292 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:42.437272    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kndxq_e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae/ovn-acl-logging/0.log"
Sep 14 05:16:42.662867 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:42.662838    2565 scope.go:117] "RemoveContainer" containerID="954240132f4e6c4e71c6313e2ac878c137fccaec0ee3b46be5349a26d4dbc9e8"
Sep 14 05:16:42.670958 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:42.670939    2565 scope.go:117] "RemoveContainer" containerID="86f5423f15a599665da405455b5467b700d9975ba8235b174f0f1dc896f04e0d"
Sep 14 05:16:47.935800 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:47.935746    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-27be50/rate-test-model-27be50-kserve-6c857f9f59-77w4m"]
Sep 14 05:16:47.936162 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:47.936090    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3929840-c8be-43d0-ad00-d669e75f8cdc" containerName="main"
Sep 14 05:16:47.936162 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:47.936109    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3929840-c8be-43d0-ad00-d669e75f8cdc" containerName="main"
Sep 14 05:16:47.936162 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:47.936121    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="545edddc-c8eb-4701-93ba-d8c74a2399e8" containerName="main"
Sep 14 05:16:47.936162 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:47.936126    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="545edddc-c8eb-4701-93ba-d8c74a2399e8" containerName="main"
Sep 14 05:16:47.936306 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:47.936170    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3929840-c8be-43d0-ad00-d669e75f8cdc" containerName="main"
Sep 14 05:16:47.936306 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:47.936177    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="545edddc-c8eb-4701-93ba-d8c74a2399e8" containerName="main"
Sep 14 05:16:47.938947 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:47.938928    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-27be50/rate-test-model-27be50-kserve-6c857f9f59-77w4m"
Sep 14 05:16:47.941473 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:47.941450    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-27be50\"/\"openshift-service-ca.crt\""
Sep 14 05:16:47.942868 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:47.942846    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-27be50\"/\"default-dockercfg-m8bz9\""
Sep 14 05:16:47.942981 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:47.942892    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-27be50\"/\"kube-root-ca.crt\""
Sep 14 05:16:47.942981 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:47.942907    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-27be50\"/\"rate-test-model-27be50-kserve-self-signed-certs\""
Sep 14 05:16:47.948222 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:47.948199    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-27be50/rate-test-model-27be50-kserve-6c857f9f59-77w4m"]
Sep 14 05:16:47.973894 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:47.973872    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/926109ad-1833-4b64-85d4-adc5e49319fb-home\") pod \"rate-test-model-27be50-kserve-6c857f9f59-77w4m\" (UID: \"926109ad-1833-4b64-85d4-adc5e49319fb\") " pod="ai-tenant-e2e-shared-b-w3-27be50/rate-test-model-27be50-kserve-6c857f9f59-77w4m"
Sep 14 05:16:47.973987 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:47.973920    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/926109ad-1833-4b64-85d4-adc5e49319fb-dshm\") pod \"rate-test-model-27be50-kserve-6c857f9f59-77w4m\" (UID: \"926109ad-1833-4b64-85d4-adc5e49319fb\") " pod="ai-tenant-e2e-shared-b-w3-27be50/rate-test-model-27be50-kserve-6c857f9f59-77w4m"
Sep 14 05:16:47.973987 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:47.973938    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/926109ad-1833-4b64-85d4-adc5e49319fb-tmp-dir\") pod \"rate-test-model-27be50-kserve-6c857f9f59-77w4m\" (UID: \"926109ad-1833-4b64-85d4-adc5e49319fb\") " pod="ai-tenant-e2e-shared-b-w3-27be50/rate-test-model-27be50-kserve-6c857f9f59-77w4m"
Sep 14 05:16:47.974068 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:47.974024    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/926109ad-1833-4b64-85d4-adc5e49319fb-tls-certs\") pod \"rate-test-model-27be50-kserve-6c857f9f59-77w4m\" (UID: \"926109ad-1833-4b64-85d4-adc5e49319fb\") " pod="ai-tenant-e2e-shared-b-w3-27be50/rate-test-model-27be50-kserve-6c857f9f59-77w4m"
Sep 14 05:16:47.974104 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:47.974089    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/926109ad-1833-4b64-85d4-adc5e49319fb-model-cache\") pod \"rate-test-model-27be50-kserve-6c857f9f59-77w4m\" (UID: \"926109ad-1833-4b64-85d4-adc5e49319fb\") " pod="ai-tenant-e2e-shared-b-w3-27be50/rate-test-model-27be50-kserve-6c857f9f59-77w4m"
Sep 14 05:16:47.974138 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:47.974114    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtct4\" (UniqueName: \"kubernetes.io/projected/926109ad-1833-4b64-85d4-adc5e49319fb-kube-api-access-mtct4\") pod \"rate-test-model-27be50-kserve-6c857f9f59-77w4m\" (UID: \"926109ad-1833-4b64-85d4-adc5e49319fb\") " pod="ai-tenant-e2e-shared-b-w3-27be50/rate-test-model-27be50-kserve-6c857f9f59-77w4m"
Sep 14 05:16:48.075115 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:48.075080    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/926109ad-1833-4b64-85d4-adc5e49319fb-tls-certs\") pod \"rate-test-model-27be50-kserve-6c857f9f59-77w4m\" (UID: \"926109ad-1833-4b64-85d4-adc5e49319fb\") " pod="ai-tenant-e2e-shared-b-w3-27be50/rate-test-model-27be50-kserve-6c857f9f59-77w4m"
Sep 14 05:16:48.075281 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:48.075137    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/926109ad-1833-4b64-85d4-adc5e49319fb-model-cache\") pod \"rate-test-model-27be50-kserve-6c857f9f59-77w4m\" (UID: \"926109ad-1833-4b64-85d4-adc5e49319fb\") " pod="ai-tenant-e2e-shared-b-w3-27be50/rate-test-model-27be50-kserve-6c857f9f59-77w4m"
Sep 14 05:16:48.075281 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:48.075163    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtct4\" (UniqueName: \"kubernetes.io/projected/926109ad-1833-4b64-85d4-adc5e49319fb-kube-api-access-mtct4\") pod \"rate-test-model-27be50-kserve-6c857f9f59-77w4m\" (UID: \"926109ad-1833-4b64-85d4-adc5e49319fb\") " pod="ai-tenant-e2e-shared-b-w3-27be50/rate-test-model-27be50-kserve-6c857f9f59-77w4m"
Sep 14 05:16:48.075281 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:48.075187    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/926109ad-1833-4b64-85d4-adc5e49319fb-home\") pod \"rate-test-model-27be50-kserve-6c857f9f59-77w4m\" (UID: \"926109ad-1833-4b64-85d4-adc5e49319fb\") " pod="ai-tenant-e2e-shared-b-w3-27be50/rate-test-model-27be50-kserve-6c857f9f59-77w4m"
Sep 14 05:16:48.075281 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:48.075221    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/926109ad-1833-4b64-85d4-adc5e49319fb-dshm\") pod \"rate-test-model-27be50-kserve-6c857f9f59-77w4m\" (UID: \"926109ad-1833-4b64-85d4-adc5e49319fb\") " pod="ai-tenant-e2e-shared-b-w3-27be50/rate-test-model-27be50-kserve-6c857f9f59-77w4m"
Sep 14 05:16:48.075493 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:48.075333    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/926109ad-1833-4b64-85d4-adc5e49319fb-tmp-dir\") pod \"rate-test-model-27be50-kserve-6c857f9f59-77w4m\" (UID: \"926109ad-1833-4b64-85d4-adc5e49319fb\") " pod="ai-tenant-e2e-shared-b-w3-27be50/rate-test-model-27be50-kserve-6c857f9f59-77w4m"
Sep 14 05:16:48.075635 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:48.075612    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/926109ad-1833-4b64-85d4-adc5e49319fb-model-cache\") pod \"rate-test-model-27be50-kserve-6c857f9f59-77w4m\" (UID: \"926109ad-1833-4b64-85d4-adc5e49319fb\") " pod="ai-tenant-e2e-shared-b-w3-27be50/rate-test-model-27be50-kserve-6c857f9f59-77w4m"
Sep 14 05:16:48.075695 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:48.075621    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/926109ad-1833-4b64-85d4-adc5e49319fb-home\") pod \"rate-test-model-27be50-kserve-6c857f9f59-77w4m\" (UID: \"926109ad-1833-4b64-85d4-adc5e49319fb\") " pod="ai-tenant-e2e-shared-b-w3-27be50/rate-test-model-27be50-kserve-6c857f9f59-77w4m"
Sep 14 05:16:48.075738 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:48.075695    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/926109ad-1833-4b64-85d4-adc5e49319fb-tmp-dir\") pod \"rate-test-model-27be50-kserve-6c857f9f59-77w4m\" (UID: \"926109ad-1833-4b64-85d4-adc5e49319fb\") " pod="ai-tenant-e2e-shared-b-w3-27be50/rate-test-model-27be50-kserve-6c857f9f59-77w4m"
Sep 14 05:16:48.077531 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:48.077503    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/926109ad-1833-4b64-85d4-adc5e49319fb-dshm\") pod \"rate-test-model-27be50-kserve-6c857f9f59-77w4m\" (UID: \"926109ad-1833-4b64-85d4-adc5e49319fb\") " pod="ai-tenant-e2e-shared-b-w3-27be50/rate-test-model-27be50-kserve-6c857f9f59-77w4m"
Sep 14 05:16:48.077641 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:48.077544    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/926109ad-1833-4b64-85d4-adc5e49319fb-tls-certs\") pod \"rate-test-model-27be50-kserve-6c857f9f59-77w4m\" (UID: \"926109ad-1833-4b64-85d4-adc5e49319fb\") " pod="ai-tenant-e2e-shared-b-w3-27be50/rate-test-model-27be50-kserve-6c857f9f59-77w4m"
Sep 14 05:16:48.083907 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:48.083890    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtct4\" (UniqueName: \"kubernetes.io/projected/926109ad-1833-4b64-85d4-adc5e49319fb-kube-api-access-mtct4\") pod \"rate-test-model-27be50-kserve-6c857f9f59-77w4m\" (UID: \"926109ad-1833-4b64-85d4-adc5e49319fb\") " pod="ai-tenant-e2e-shared-b-w3-27be50/rate-test-model-27be50-kserve-6c857f9f59-77w4m"
Sep 14 05:16:48.249060 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:48.248980    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-27be50/rate-test-model-27be50-kserve-6c857f9f59-77w4m"
Sep 14 05:16:48.518534 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:48.518508    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-27be50/rate-test-model-27be50-kserve-6c857f9f59-77w4m"]
Sep 14 05:16:48.520037 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:16:48.520012    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod926109ad_1833_4b64_85d4_adc5e49319fb.slice/crio-68501d18dcc282da4f7f142ce0f41f21c8f79b6636b40faa0bd1b7e52c532276 WatchSource:0}: Error finding container 68501d18dcc282da4f7f142ce0f41f21c8f79b6636b40faa0bd1b7e52c532276: Status 404 returned error can't find the container with id 68501d18dcc282da4f7f142ce0f41f21c8f79b6636b40faa0bd1b7e52c532276
Sep 14 05:16:48.629615 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:48.629569    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-27be50/rate-test-model-27be50-kserve-6c857f9f59-77w4m" event={"ID":"926109ad-1833-4b64-85d4-adc5e49319fb","Type":"ContainerStarted","Data":"82f920e360aa17d2d8e55406ee03d11d5de04f45c191693043bc447245624f0f"}
Sep 14 05:16:48.629806 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:48.629628    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-27be50/rate-test-model-27be50-kserve-6c857f9f59-77w4m" event={"ID":"926109ad-1833-4b64-85d4-adc5e49319fb","Type":"ContainerStarted","Data":"68501d18dcc282da4f7f142ce0f41f21c8f79b6636b40faa0bd1b7e52c532276"}
Sep 14 05:16:48.647856 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:48.647807    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-27be50/rate-test-model-27be50-kserve-6c857f9f59-77w4m" podStartSLOduration=1.647790539 podStartE2EDuration="1.647790539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:16:48.646302532 +0000 UTC m=+1806.822074092" watchObservedRunningTime="2026-09-14 05:16:48.647790539 +0000 UTC m=+1806.823562101"
Sep 14 05:16:51.840575 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:51.840533    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:16:58.249588 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:58.249547    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-27be50/rate-test-model-27be50-kserve-6c857f9f59-77w4m"
Sep 14 05:16:58.250012 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:58.249675    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-27be50/rate-test-model-27be50-kserve-6c857f9f59-77w4m"
Sep 14 05:16:58.262197 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:58.262172    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-27be50/rate-test-model-27be50-kserve-6c857f9f59-77w4m"
Sep 14 05:16:58.692974 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:16:58.692942    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-27be50/rate-test-model-27be50-kserve-6c857f9f59-77w4m"
Sep 14 05:17:07.044833 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:07.044797    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:17:16.535658 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:16.535610    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:17:22.347094 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:22.347061    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2ff4a6/rate-test-model-2ff4a6-kserve-594958d69f-lg8ql"]
Sep 14 05:17:23.407741 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:23.407709    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-27be50/rate-test-model-27be50-kserve-6c857f9f59-77w4m"]
Sep 14 05:17:27.242196 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:27.242164    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:17:29.557801 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:29.557750    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-524a38-64f67b69c5-4bpc4"]
Sep 14 05:17:29.561423 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:29.561406    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-524a38-64f67b69c5-4bpc4"
Sep 14 05:17:29.564214 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:29.564190    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-524a38-dockercfg-8qzn9\""
Sep 14 05:17:29.564335 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:29.564190    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-524a38\""
Sep 14 05:17:29.570438 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:29.570417    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-524a38-64f67b69c5-4bpc4"]
Sep 14 05:17:29.723646 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:29.723611    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6ljg\" (UniqueName: \"kubernetes.io/projected/efe291d5-bba9-494c-bf7a-e8bea65d52d6-kube-api-access-k6ljg\") pod \"payload-processing-e2e-ipp-a-524a38-64f67b69c5-4bpc4\" (UID: \"efe291d5-bba9-494c-bf7a-e8bea65d52d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-524a38-64f67b69c5-4bpc4"
Sep 14 05:17:29.723832 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:29.723671    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efe291d5-bba9-494c-bf7a-e8bea65d52d6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-524a38-64f67b69c5-4bpc4\" (UID: \"efe291d5-bba9-494c-bf7a-e8bea65d52d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-524a38-64f67b69c5-4bpc4"
Sep 14 05:17:29.825060 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:29.824971    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efe291d5-bba9-494c-bf7a-e8bea65d52d6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-524a38-64f67b69c5-4bpc4\" (UID: \"efe291d5-bba9-494c-bf7a-e8bea65d52d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-524a38-64f67b69c5-4bpc4"
Sep 14 05:17:29.825213 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:29.825076    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6ljg\" (UniqueName: \"kubernetes.io/projected/efe291d5-bba9-494c-bf7a-e8bea65d52d6-kube-api-access-k6ljg\") pod \"payload-processing-e2e-ipp-a-524a38-64f67b69c5-4bpc4\" (UID: \"efe291d5-bba9-494c-bf7a-e8bea65d52d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-524a38-64f67b69c5-4bpc4"
Sep 14 05:17:29.825705 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:29.825681    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efe291d5-bba9-494c-bf7a-e8bea65d52d6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-524a38-64f67b69c5-4bpc4\" (UID: \"efe291d5-bba9-494c-bf7a-e8bea65d52d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-524a38-64f67b69c5-4bpc4"
Sep 14 05:17:29.832395 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:29.832374    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6ljg\" (UniqueName: \"kubernetes.io/projected/efe291d5-bba9-494c-bf7a-e8bea65d52d6-kube-api-access-k6ljg\") pod \"payload-processing-e2e-ipp-a-524a38-64f67b69c5-4bpc4\" (UID: \"efe291d5-bba9-494c-bf7a-e8bea65d52d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-524a38-64f67b69c5-4bpc4"
Sep 14 05:17:29.872748 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:29.872719    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-524a38-64f67b69c5-4bpc4"
Sep 14 05:17:30.148891 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:30.148860    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-524a38-64f67b69c5-4bpc4"]
Sep 14 05:17:30.150256 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:17:30.150231    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefe291d5_bba9_494c_bf7a_e8bea65d52d6.slice/crio-71be9ab618e1353548bcfe09829f274bada3b92ec8a4455e4343006c265e43c3 WatchSource:0}: Error finding container 71be9ab618e1353548bcfe09829f274bada3b92ec8a4455e4343006c265e43c3: Status 404 returned error can't find the container with id 71be9ab618e1353548bcfe09829f274bada3b92ec8a4455e4343006c265e43c3
Sep 14 05:17:30.854464 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:30.854426    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-524a38-64f67b69c5-4bpc4" event={"ID":"efe291d5-bba9-494c-bf7a-e8bea65d52d6","Type":"ContainerStarted","Data":"842f1d999a0b38333d1694d260e732172c24068b7ed57a966d8ed1bf698049fa"}
Sep 14 05:17:30.854953 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:30.854487    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-524a38-64f67b69c5-4bpc4" event={"ID":"efe291d5-bba9-494c-bf7a-e8bea65d52d6","Type":"ContainerStarted","Data":"71be9ab618e1353548bcfe09829f274bada3b92ec8a4455e4343006c265e43c3"}
Sep 14 05:17:30.854953 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:30.854539    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-524a38-64f67b69c5-4bpc4"
Sep 14 05:17:30.873053 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:30.873002    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-524a38-64f67b69c5-4bpc4" podStartSLOduration=1.872983199 podStartE2EDuration="1.872983199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:17:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:17:30.870814435 +0000 UTC m=+1849.046586002" watchObservedRunningTime="2026-09-14 05:17:30.872983199 +0000 UTC m=+1849.048754758"
Sep 14 05:17:37.365715 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:37.365676    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-2ff4a6/rate-test-model-2ff4a6-kserve-594958d69f-lg8ql" podUID="377c1063-c80a-41c5-91c9-5c9b265acebf" containerName="main" containerID="cri-o://e23abe5e3f09de260f94578c4645599c02c1467fb7d4cdd12c0ce4271c43eaa7" gracePeriod=45
Sep 14 05:17:37.615273 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:37.615248    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2ff4a6/rate-test-model-2ff4a6-kserve-594958d69f-lg8ql"
Sep 14 05:17:37.795955 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:37.795853    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/377c1063-c80a-41c5-91c9-5c9b265acebf-tls-certs\") pod \"377c1063-c80a-41c5-91c9-5c9b265acebf\" (UID: \"377c1063-c80a-41c5-91c9-5c9b265acebf\") "
Sep 14 05:17:37.795955 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:37.795914    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/377c1063-c80a-41c5-91c9-5c9b265acebf-tmp-dir\") pod \"377c1063-c80a-41c5-91c9-5c9b265acebf\" (UID: \"377c1063-c80a-41c5-91c9-5c9b265acebf\") "
Sep 14 05:17:37.795955 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:37.795946    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/377c1063-c80a-41c5-91c9-5c9b265acebf-home\") pod \"377c1063-c80a-41c5-91c9-5c9b265acebf\" (UID: \"377c1063-c80a-41c5-91c9-5c9b265acebf\") "
Sep 14 05:17:37.796245 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:37.795972    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/377c1063-c80a-41c5-91c9-5c9b265acebf-model-cache\") pod \"377c1063-c80a-41c5-91c9-5c9b265acebf\" (UID: \"377c1063-c80a-41c5-91c9-5c9b265acebf\") "
Sep 14 05:17:37.796245 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:37.796010    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fkth\" (UniqueName: \"kubernetes.io/projected/377c1063-c80a-41c5-91c9-5c9b265acebf-kube-api-access-9fkth\") pod \"377c1063-c80a-41c5-91c9-5c9b265acebf\" (UID: \"377c1063-c80a-41c5-91c9-5c9b265acebf\") "
Sep 14 05:17:37.796245 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:37.796041    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/377c1063-c80a-41c5-91c9-5c9b265acebf-dshm\") pod \"377c1063-c80a-41c5-91c9-5c9b265acebf\" (UID: \"377c1063-c80a-41c5-91c9-5c9b265acebf\") "
Sep 14 05:17:37.796245 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:37.796195    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377c1063-c80a-41c5-91c9-5c9b265acebf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "377c1063-c80a-41c5-91c9-5c9b265acebf" (UID: "377c1063-c80a-41c5-91c9-5c9b265acebf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:17:37.796245 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:37.796202    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377c1063-c80a-41c5-91c9-5c9b265acebf-home" (OuterVolumeSpecName: "home") pod "377c1063-c80a-41c5-91c9-5c9b265acebf" (UID: "377c1063-c80a-41c5-91c9-5c9b265acebf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:17:37.796497 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:37.796247    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377c1063-c80a-41c5-91c9-5c9b265acebf-model-cache" (OuterVolumeSpecName: "model-cache") pod "377c1063-c80a-41c5-91c9-5c9b265acebf" (UID: "377c1063-c80a-41c5-91c9-5c9b265acebf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:17:37.798132 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:37.798109    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/377c1063-c80a-41c5-91c9-5c9b265acebf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "377c1063-c80a-41c5-91c9-5c9b265acebf" (UID: "377c1063-c80a-41c5-91c9-5c9b265acebf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 05:17:37.798232 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:37.798205    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377c1063-c80a-41c5-91c9-5c9b265acebf-dshm" (OuterVolumeSpecName: "dshm") pod "377c1063-c80a-41c5-91c9-5c9b265acebf" (UID: "377c1063-c80a-41c5-91c9-5c9b265acebf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:17:37.798293 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:37.798276    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/377c1063-c80a-41c5-91c9-5c9b265acebf-kube-api-access-9fkth" (OuterVolumeSpecName: "kube-api-access-9fkth") pod "377c1063-c80a-41c5-91c9-5c9b265acebf" (UID: "377c1063-c80a-41c5-91c9-5c9b265acebf"). InnerVolumeSpecName "kube-api-access-9fkth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:17:37.896657 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:37.896623    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/377c1063-c80a-41c5-91c9-5c9b265acebf-tls-certs\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:17:37.896657 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:37.896655    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/377c1063-c80a-41c5-91c9-5c9b265acebf-tmp-dir\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:17:37.896657 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:37.896665    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/377c1063-c80a-41c5-91c9-5c9b265acebf-home\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:17:37.896948 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:37.896673    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/377c1063-c80a-41c5-91c9-5c9b265acebf-model-cache\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:17:37.896948 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:37.896682    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fkth\" (UniqueName: \"kubernetes.io/projected/377c1063-c80a-41c5-91c9-5c9b265acebf-kube-api-access-9fkth\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:17:37.896948 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:37.896690    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/377c1063-c80a-41c5-91c9-5c9b265acebf-dshm\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:17:37.897258 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:37.897237    2565 generic.go:356] "Generic (PLEG): container finished" podID="377c1063-c80a-41c5-91c9-5c9b265acebf" containerID="e23abe5e3f09de260f94578c4645599c02c1467fb7d4cdd12c0ce4271c43eaa7" exitCode=0
Sep 14 05:17:37.897318 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:37.897281    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2ff4a6/rate-test-model-2ff4a6-kserve-594958d69f-lg8ql"
Sep 14 05:17:37.897351 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:37.897325    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2ff4a6/rate-test-model-2ff4a6-kserve-594958d69f-lg8ql" event={"ID":"377c1063-c80a-41c5-91c9-5c9b265acebf","Type":"ContainerDied","Data":"e23abe5e3f09de260f94578c4645599c02c1467fb7d4cdd12c0ce4271c43eaa7"}
Sep 14 05:17:37.897383 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:37.897372    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2ff4a6/rate-test-model-2ff4a6-kserve-594958d69f-lg8ql" event={"ID":"377c1063-c80a-41c5-91c9-5c9b265acebf","Type":"ContainerDied","Data":"d9a3d92298bddb699137f3c1b3953e24a89699c57a1dc79d47bd6ece5794bc73"}
Sep 14 05:17:37.897419 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:37.897393    2565 scope.go:117] "RemoveContainer" containerID="e23abe5e3f09de260f94578c4645599c02c1467fb7d4cdd12c0ce4271c43eaa7"
Sep 14 05:17:37.907293 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:37.907275    2565 scope.go:117] "RemoveContainer" containerID="e23abe5e3f09de260f94578c4645599c02c1467fb7d4cdd12c0ce4271c43eaa7"
Sep 14 05:17:37.907600 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:17:37.907571    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e23abe5e3f09de260f94578c4645599c02c1467fb7d4cdd12c0ce4271c43eaa7\": container with ID starting with e23abe5e3f09de260f94578c4645599c02c1467fb7d4cdd12c0ce4271c43eaa7 not found: ID does not exist" containerID="e23abe5e3f09de260f94578c4645599c02c1467fb7d4cdd12c0ce4271c43eaa7"
Sep 14 05:17:37.907674 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:37.907609    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e23abe5e3f09de260f94578c4645599c02c1467fb7d4cdd12c0ce4271c43eaa7"} err="failed to get container status \"e23abe5e3f09de260f94578c4645599c02c1467fb7d4cdd12c0ce4271c43eaa7\": rpc error: code = NotFound desc = could not find container \"e23abe5e3f09de260f94578c4645599c02c1467fb7d4cdd12c0ce4271c43eaa7\": container with ID starting with e23abe5e3f09de260f94578c4645599c02c1467fb7d4cdd12c0ce4271c43eaa7 not found: ID does not exist"
Sep 14 05:17:37.917965 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:37.917940    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2ff4a6/rate-test-model-2ff4a6-kserve-594958d69f-lg8ql"]
Sep 14 05:17:37.920362 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:37.920336    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2ff4a6/rate-test-model-2ff4a6-kserve-594958d69f-lg8ql"]
Sep 14 05:17:37.925256 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:37.925235    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:17:38.334252 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:38.334207    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="377c1063-c80a-41c5-91c9-5c9b265acebf" path="/var/lib/kubelet/pods/377c1063-c80a-41c5-91c9-5c9b265acebf/volumes"
Sep 14 05:17:38.426399 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:38.426335    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-27be50/rate-test-model-27be50-kserve-6c857f9f59-77w4m" podUID="926109ad-1833-4b64-85d4-adc5e49319fb" containerName="main" containerID="cri-o://82f920e360aa17d2d8e55406ee03d11d5de04f45c191693043bc447245624f0f" gracePeriod=45
Sep 14 05:17:38.677091 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:38.677068    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-27be50/rate-test-model-27be50-kserve-6c857f9f59-77w4m"
Sep 14 05:17:38.803072 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:38.803037    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/926109ad-1833-4b64-85d4-adc5e49319fb-tls-certs\") pod \"926109ad-1833-4b64-85d4-adc5e49319fb\" (UID: \"926109ad-1833-4b64-85d4-adc5e49319fb\") "
Sep 14 05:17:38.803218 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:38.803094    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/926109ad-1833-4b64-85d4-adc5e49319fb-model-cache\") pod \"926109ad-1833-4b64-85d4-adc5e49319fb\" (UID: \"926109ad-1833-4b64-85d4-adc5e49319fb\") "
Sep 14 05:17:38.803218 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:38.803123    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtct4\" (UniqueName: \"kubernetes.io/projected/926109ad-1833-4b64-85d4-adc5e49319fb-kube-api-access-mtct4\") pod \"926109ad-1833-4b64-85d4-adc5e49319fb\" (UID: \"926109ad-1833-4b64-85d4-adc5e49319fb\") "
Sep 14 05:17:38.803218 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:38.803154    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/926109ad-1833-4b64-85d4-adc5e49319fb-home\") pod \"926109ad-1833-4b64-85d4-adc5e49319fb\" (UID: \"926109ad-1833-4b64-85d4-adc5e49319fb\") "
Sep 14 05:17:38.803218 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:38.803170    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/926109ad-1833-4b64-85d4-adc5e49319fb-tmp-dir\") pod \"926109ad-1833-4b64-85d4-adc5e49319fb\" (UID: \"926109ad-1833-4b64-85d4-adc5e49319fb\") "
Sep 14 05:17:38.803459 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:38.803225    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/926109ad-1833-4b64-85d4-adc5e49319fb-dshm\") pod \"926109ad-1833-4b64-85d4-adc5e49319fb\" (UID: \"926109ad-1833-4b64-85d4-adc5e49319fb\") "
Sep 14 05:17:38.803459 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:38.803424    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/926109ad-1833-4b64-85d4-adc5e49319fb-model-cache" (OuterVolumeSpecName: "model-cache") pod "926109ad-1833-4b64-85d4-adc5e49319fb" (UID: "926109ad-1833-4b64-85d4-adc5e49319fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:17:38.803553 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:38.803445    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/926109ad-1833-4b64-85d4-adc5e49319fb-home" (OuterVolumeSpecName: "home") pod "926109ad-1833-4b64-85d4-adc5e49319fb" (UID: "926109ad-1833-4b64-85d4-adc5e49319fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:17:38.805233 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:38.803833    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/926109ad-1833-4b64-85d4-adc5e49319fb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "926109ad-1833-4b64-85d4-adc5e49319fb" (UID: "926109ad-1833-4b64-85d4-adc5e49319fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:17:38.805894 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:38.805867    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/926109ad-1833-4b64-85d4-adc5e49319fb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "926109ad-1833-4b64-85d4-adc5e49319fb" (UID: "926109ad-1833-4b64-85d4-adc5e49319fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 05:17:38.807799 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:38.806349    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/926109ad-1833-4b64-85d4-adc5e49319fb-kube-api-access-mtct4" (OuterVolumeSpecName: "kube-api-access-mtct4") pod "926109ad-1833-4b64-85d4-adc5e49319fb" (UID: "926109ad-1833-4b64-85d4-adc5e49319fb"). InnerVolumeSpecName "kube-api-access-mtct4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:17:38.811166 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:38.811134    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/926109ad-1833-4b64-85d4-adc5e49319fb-dshm" (OuterVolumeSpecName: "dshm") pod "926109ad-1833-4b64-85d4-adc5e49319fb" (UID: "926109ad-1833-4b64-85d4-adc5e49319fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:17:38.904342 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:38.904259    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/926109ad-1833-4b64-85d4-adc5e49319fb-dshm\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:17:38.904342 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:38.904296    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/926109ad-1833-4b64-85d4-adc5e49319fb-tls-certs\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:17:38.904342 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:38.904290    2565 generic.go:356] "Generic (PLEG): container finished" podID="926109ad-1833-4b64-85d4-adc5e49319fb" containerID="82f920e360aa17d2d8e55406ee03d11d5de04f45c191693043bc447245624f0f" exitCode=0
Sep 14 05:17:38.904342 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:38.904310    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/926109ad-1833-4b64-85d4-adc5e49319fb-model-cache\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:17:38.904342 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:38.904322    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtct4\" (UniqueName: \"kubernetes.io/projected/926109ad-1833-4b64-85d4-adc5e49319fb-kube-api-access-mtct4\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:17:38.904342 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:38.904334    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/926109ad-1833-4b64-85d4-adc5e49319fb-home\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:17:38.904705 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:38.904346    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-27be50/rate-test-model-27be50-kserve-6c857f9f59-77w4m"
Sep 14 05:17:38.904705 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:38.904368    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/926109ad-1833-4b64-85d4-adc5e49319fb-tmp-dir\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:17:38.904705 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:38.904384    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-27be50/rate-test-model-27be50-kserve-6c857f9f59-77w4m" event={"ID":"926109ad-1833-4b64-85d4-adc5e49319fb","Type":"ContainerDied","Data":"82f920e360aa17d2d8e55406ee03d11d5de04f45c191693043bc447245624f0f"}
Sep 14 05:17:38.904705 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:38.904431    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-27be50/rate-test-model-27be50-kserve-6c857f9f59-77w4m" event={"ID":"926109ad-1833-4b64-85d4-adc5e49319fb","Type":"ContainerDied","Data":"68501d18dcc282da4f7f142ce0f41f21c8f79b6636b40faa0bd1b7e52c532276"}
Sep 14 05:17:38.904705 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:38.904447    2565 scope.go:117] "RemoveContainer" containerID="82f920e360aa17d2d8e55406ee03d11d5de04f45c191693043bc447245624f0f"
Sep 14 05:17:38.914233 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:38.914215    2565 scope.go:117] "RemoveContainer" containerID="82f920e360aa17d2d8e55406ee03d11d5de04f45c191693043bc447245624f0f"
Sep 14 05:17:38.914500 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:17:38.914479    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82f920e360aa17d2d8e55406ee03d11d5de04f45c191693043bc447245624f0f\": container with ID starting with 82f920e360aa17d2d8e55406ee03d11d5de04f45c191693043bc447245624f0f not found: ID does not exist" containerID="82f920e360aa17d2d8e55406ee03d11d5de04f45c191693043bc447245624f0f"
Sep 14 05:17:38.914562 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:38.914507    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82f920e360aa17d2d8e55406ee03d11d5de04f45c191693043bc447245624f0f"} err="failed to get container status \"82f920e360aa17d2d8e55406ee03d11d5de04f45c191693043bc447245624f0f\": rpc error: code = NotFound desc = could not find container \"82f920e360aa17d2d8e55406ee03d11d5de04f45c191693043bc447245624f0f\": container with ID starting with 82f920e360aa17d2d8e55406ee03d11d5de04f45c191693043bc447245624f0f not found: ID does not exist"
Sep 14 05:17:38.926139 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:38.926115    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-27be50/rate-test-model-27be50-kserve-6c857f9f59-77w4m"]
Sep 14 05:17:38.929737 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:38.929716    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-27be50/rate-test-model-27be50-kserve-6c857f9f59-77w4m"]
Sep 14 05:17:40.333977 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:40.333943    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="926109ad-1833-4b64-85d4-adc5e49319fb" path="/var/lib/kubelet/pods/926109ad-1833-4b64-85d4-adc5e49319fb/volumes"
Sep 14 05:17:41.861937 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:41.861906    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-524a38-64f67b69c5-4bpc4"
Sep 14 05:17:53.329637 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:53.329602    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"]
Sep 14 05:17:53.330221 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:53.330085    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="926109ad-1833-4b64-85d4-adc5e49319fb" containerName="main"
Sep 14 05:17:53.330221 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:53.330101    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="926109ad-1833-4b64-85d4-adc5e49319fb" containerName="main"
Sep 14 05:17:53.330221 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:53.330120    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="377c1063-c80a-41c5-91c9-5c9b265acebf" containerName="main"
Sep 14 05:17:53.330221 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:53.330129    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="377c1063-c80a-41c5-91c9-5c9b265acebf" containerName="main"
Sep 14 05:17:53.330221 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:53.330204    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="926109ad-1833-4b64-85d4-adc5e49319fb" containerName="main"
Sep 14 05:17:53.330221 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:53.330215    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="377c1063-c80a-41c5-91c9-5c9b265acebf" containerName="main"
Sep 14 05:17:53.333534 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:53.333511    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"
Sep 14 05:17:53.335619 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:53.335597    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-7bbea9-openshift-default-dockercfg-g62gc\""
Sep 14 05:17:53.344085 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:53.344063    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"]
Sep 14 05:17:53.420570 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:53.420533    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-istiod-ca-cert\") pod \"e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5\" (UID: \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\") " pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"
Sep 14 05:17:53.420785 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:53.420590    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-istio-token\") pod \"e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5\" (UID: \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\") " pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"
Sep 14 05:17:53.420785 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:53.420625    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-workload-certs\") pod \"e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5\" (UID: \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\") " pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"
Sep 14 05:17:53.420785 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:53.420662    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khjrx\" (UniqueName: \"kubernetes.io/projected/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-kube-api-access-khjrx\") pod \"e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5\" (UID: \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\") " pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"
Sep 14 05:17:53.420785 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:53.420696    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-istio-envoy\") pod \"e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5\" (UID: \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\") " pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"
Sep 14 05:17:53.421058 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:53.420789    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-credential-socket\") pod \"e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5\" (UID: \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\") " pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"
Sep 14 05:17:53.421058 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:53.420813    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-istio-data\") pod \"e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5\" (UID: \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\") " pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"
Sep 14 05:17:53.421058 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:53.420836    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-istio-podinfo\") pod \"e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5\" (UID: \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\") " pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"
Sep 14 05:17:53.421058 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:53.420877    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-workload-socket\") pod \"e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5\" (UID: \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\") " pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"
Sep 14 05:17:53.521881 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:53.521844    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-workload-certs\") pod \"e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5\" (UID: \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\") " pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"
Sep 14 05:17:53.522052 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:53.521898    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khjrx\" (UniqueName: \"kubernetes.io/projected/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-kube-api-access-khjrx\") pod \"e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5\" (UID: \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\") " pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"
Sep 14 05:17:53.522052 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:53.521931    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-istio-envoy\") pod \"e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5\" (UID: \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\") " pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"
Sep 14 05:17:53.522052 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:53.521994    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-credential-socket\") pod \"e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5\" (UID: \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\") " pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"
Sep 14 05:17:53.522052 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:53.522018    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-istio-data\") pod \"e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5\" (UID: \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\") " pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"
Sep 14 05:17:53.522052 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:53.522043    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-istio-podinfo\") pod \"e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5\" (UID: \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\") " pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"
Sep 14 05:17:53.522329 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:53.522150    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-workload-socket\") pod \"e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5\" (UID: \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\") " pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"
Sep 14 05:17:53.522329 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:53.522262    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-workload-certs\") pod \"e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5\" (UID: \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\") " pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"
Sep 14 05:17:53.522441 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:53.522387    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-credential-socket\") pod \"e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5\" (UID: \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\") " pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"
Sep 14 05:17:53.522441 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:53.522414    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-istio-data\") pod \"e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5\" (UID: \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\") " pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"
Sep 14 05:17:53.522441 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:53.522393    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-istiod-ca-cert\") pod \"e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5\" (UID: \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\") " pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"
Sep 14 05:17:53.522600 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:53.522487    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-istio-token\") pod \"e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5\" (UID: \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\") " pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"
Sep 14 05:17:53.522600 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:53.522498    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-workload-socket\") pod \"e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5\" (UID: \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\") " pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"
Sep 14 05:17:53.523084 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:53.523057    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-istiod-ca-cert\") pod \"e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5\" (UID: \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\") " pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"
Sep 14 05:17:53.524566 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:53.524545    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-istio-envoy\") pod \"e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5\" (UID: \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\") " pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"
Sep 14 05:17:53.524566 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:53.524561    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-istio-podinfo\") pod \"e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5\" (UID: \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\") " pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"
Sep 14 05:17:53.529119 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:53.529095    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-istio-token\") pod \"e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5\" (UID: \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\") " pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"
Sep 14 05:17:53.529507 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:53.529485    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khjrx\" (UniqueName: \"kubernetes.io/projected/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-kube-api-access-khjrx\") pod \"e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5\" (UID: \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\") " pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"
Sep 14 05:17:53.645089 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:53.645056    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"
Sep 14 05:17:53.922214 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:53.922182    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"]
Sep 14 05:17:53.923508 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:17:53.923481    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0b92efa_2cbb_4a99_87ea_b8b0d0e9eda9.slice/crio-f7584a55069fa9c9fd4b01bb22ed17e605ab25d158978608ae3826558fe379c8 WatchSource:0}: Error finding container f7584a55069fa9c9fd4b01bb22ed17e605ab25d158978608ae3826558fe379c8: Status 404 returned error can't find the container with id f7584a55069fa9c9fd4b01bb22ed17e605ab25d158978608ae3826558fe379c8
Sep 14 05:17:53.925909 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:53.925867    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:17:53.925980 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:53.925951    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:17:53.926024 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:53.925994    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:17:53.989578 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:53.989548    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5" event={"ID":"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9","Type":"ContainerStarted","Data":"f7584a55069fa9c9fd4b01bb22ed17e605ab25d158978608ae3826558fe379c8"}
Sep 14 05:17:54.996108 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:54.996065    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5" event={"ID":"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9","Type":"ContainerStarted","Data":"db2e063f1d35024837eae0bc80ba877219cf663db5d0c3c90068948f2f747a88"}
Sep 14 05:17:55.016539 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:55.016490    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5" podStartSLOduration=2.016472442 podStartE2EDuration="2.016472442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:17:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:17:55.01466874 +0000 UTC m=+1873.190440311" watchObservedRunningTime="2026-09-14 05:17:55.016472442 +0000 UTC m=+1873.192244065"
Sep 14 05:17:55.645920 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:55.645888    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"
Sep 14 05:17:55.651069 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:55.651042    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"
Sep 14 05:17:56.001582 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:56.001498    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"
Sep 14 05:17:56.002370 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:56.002352    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"
Sep 14 05:17:59.866207 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:59.866174    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7bbea9-6c44bcf854-8mbns"]
Sep 14 05:17:59.869522 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:59.869505    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7bbea9-6c44bcf854-8mbns"
Sep 14 05:17:59.871539 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:59.871516    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-7bbea9-dockercfg-k9skg\""
Sep 14 05:17:59.871686 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:59.871547    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-7bbea9\""
Sep 14 05:17:59.878236 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:59.878212    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7bbea9-6c44bcf854-8mbns"]
Sep 14 05:17:59.977897 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:59.977863    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xlt4\" (UniqueName: \"kubernetes.io/projected/5edad9b0-9829-4009-ab8a-8fed1c7ff844-kube-api-access-6xlt4\") pod \"payload-pre-processing-e2e-ipp-b-7bbea9-6c44bcf854-8mbns\" (UID: \"5edad9b0-9829-4009-ab8a-8fed1c7ff844\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7bbea9-6c44bcf854-8mbns"
Sep 14 05:17:59.978067 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:17:59.977905    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5edad9b0-9829-4009-ab8a-8fed1c7ff844-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7bbea9-6c44bcf854-8mbns\" (UID: \"5edad9b0-9829-4009-ab8a-8fed1c7ff844\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7bbea9-6c44bcf854-8mbns"
Sep 14 05:18:00.079181 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:00.079135    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xlt4\" (UniqueName: \"kubernetes.io/projected/5edad9b0-9829-4009-ab8a-8fed1c7ff844-kube-api-access-6xlt4\") pod \"payload-pre-processing-e2e-ipp-b-7bbea9-6c44bcf854-8mbns\" (UID: \"5edad9b0-9829-4009-ab8a-8fed1c7ff844\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7bbea9-6c44bcf854-8mbns"
Sep 14 05:18:00.079391 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:00.079190    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5edad9b0-9829-4009-ab8a-8fed1c7ff844-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7bbea9-6c44bcf854-8mbns\" (UID: \"5edad9b0-9829-4009-ab8a-8fed1c7ff844\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7bbea9-6c44bcf854-8mbns"
Sep 14 05:18:00.079938 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:00.079913    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5edad9b0-9829-4009-ab8a-8fed1c7ff844-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7bbea9-6c44bcf854-8mbns\" (UID: \"5edad9b0-9829-4009-ab8a-8fed1c7ff844\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7bbea9-6c44bcf854-8mbns"
Sep 14 05:18:00.086802 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:00.086743    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xlt4\" (UniqueName: \"kubernetes.io/projected/5edad9b0-9829-4009-ab8a-8fed1c7ff844-kube-api-access-6xlt4\") pod \"payload-pre-processing-e2e-ipp-b-7bbea9-6c44bcf854-8mbns\" (UID: \"5edad9b0-9829-4009-ab8a-8fed1c7ff844\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7bbea9-6c44bcf854-8mbns"
Sep 14 05:18:00.180314 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:00.180209    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7bbea9-6c44bcf854-8mbns"
Sep 14 05:18:00.450289 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:00.450264    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7bbea9-6c44bcf854-8mbns"]
Sep 14 05:18:00.451785 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:18:00.451737    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5edad9b0_9829_4009_ab8a_8fed1c7ff844.slice/crio-b8885691911a925922a1bddec85584eac86e5413bf3a83929c49bf6e3c545405 WatchSource:0}: Error finding container b8885691911a925922a1bddec85584eac86e5413bf3a83929c49bf6e3c545405: Status 404 returned error can't find the container with id b8885691911a925922a1bddec85584eac86e5413bf3a83929c49bf6e3c545405
Sep 14 05:18:01.030111 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:01.030069    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7bbea9-6c44bcf854-8mbns" event={"ID":"5edad9b0-9829-4009-ab8a-8fed1c7ff844","Type":"ContainerStarted","Data":"970e13fd6cc879b95305f3781a424ee4988dabd24cac2e4246e39261dfa20aa6"}
Sep 14 05:18:01.030500 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:01.030139    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7bbea9-6c44bcf854-8mbns" event={"ID":"5edad9b0-9829-4009-ab8a-8fed1c7ff844","Type":"ContainerStarted","Data":"b8885691911a925922a1bddec85584eac86e5413bf3a83929c49bf6e3c545405"}
Sep 14 05:18:01.030500 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:01.030262    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7bbea9-6c44bcf854-8mbns"
Sep 14 05:18:01.046936 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:01.046870    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7bbea9-6c44bcf854-8mbns" podStartSLOduration=2.046852597 podStartE2EDuration="2.046852597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:17:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:18:01.046840341 +0000 UTC m=+1879.222611902" watchObservedRunningTime="2026-09-14 05:18:01.046852597 +0000 UTC m=+1879.222624156"
Sep 14 05:18:04.229601 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:04.229566    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:18:12.038600 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:12.038571    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7bbea9-6c44bcf854-8mbns"
Sep 14 05:18:12.042915 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:12.042882    2565 patch_prober.go:28] interesting pod/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 14 05:18:12.042985 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:12.042947    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5" podUID="f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 14 05:18:14.107460 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:14.107427    2565 generic.go:356] "Generic (PLEG): container finished" podID="f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9" containerID="db2e063f1d35024837eae0bc80ba877219cf663db5d0c3c90068948f2f747a88" exitCode=0
Sep 14 05:18:14.107911 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:14.107480    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5" event={"ID":"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9","Type":"ContainerDied","Data":"db2e063f1d35024837eae0bc80ba877219cf663db5d0c3c90068948f2f747a88"}
Sep 14 05:18:14.107911 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:14.107850    2565 scope.go:117] "RemoveContainer" containerID="db2e063f1d35024837eae0bc80ba877219cf663db5d0c3c90068948f2f747a88"
Sep 14 05:18:14.108951 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:14.108915    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:18:14.109017 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:14.108992    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:18:14.109056 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:14.109024    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 14 05:18:15.119918 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:15.119880    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5" event={"ID":"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9","Type":"ContainerStarted","Data":"fcf23f7fbf413ea3e03b883955dfe89a59a9c7de3974cf8f20b7652d5ffb796d"}
Sep 14 05:18:15.645840 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:15.645796    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"
Sep 14 05:18:15.646056 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:15.645936    2565 patch_prober.go:28] interesting pod/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Sep 14 05:18:15.646056 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:15.645997    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5" podUID="f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Sep 14 05:18:16.646268 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:16.646220    2565 patch_prober.go:28] interesting pod/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Sep 14 05:18:16.646800 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:16.646303    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5" podUID="f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Sep 14 05:18:17.645543 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:17.645497    2565 patch_prober.go:28] interesting pod/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Sep 14 05:18:17.645767 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:17.645565    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5" podUID="f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Sep 14 05:18:18.646284 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:18.646244    2565 patch_prober.go:28] interesting pod/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Sep 14 05:18:18.646919 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:18.646888    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5" podUID="f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Sep 14 05:18:19.646834 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:19.645572    2565 patch_prober.go:28] interesting pod/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Sep 14 05:18:19.646834 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:19.645644    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5" podUID="f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Sep 14 05:18:20.725159 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:20.725128    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"
Sep 14 05:18:20.725601 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:20.725427    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"
Sep 14 05:18:20.726221 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:20.726204    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"
Sep 14 05:18:43.825667 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:43.825634    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-524a38/ipp-route-524a38-kserve-cd947985f-jchh8"]
Sep 14 05:18:43.830440 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:43.830414    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-524a38/ipp-route-524a38-kserve-cd947985f-jchh8"
Sep 14 05:18:43.833866 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:43.833841    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-524a38\"/\"default-dockercfg-46kgr\""
Sep 14 05:18:43.833866 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:43.833841    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-524a38\"/\"ipp-route-524a38-kserve-self-signed-certs\""
Sep 14 05:18:43.833866 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:43.833913    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-524a38\"/\"kube-root-ca.crt\""
Sep 14 05:18:43.833866 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:43.833958    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-524a38\"/\"openshift-service-ca.crt\""
Sep 14 05:18:43.837671 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:43.837342    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-524a38/ipp-route-524a38-kserve-cd947985f-jchh8"]
Sep 14 05:18:43.893859 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:43.893829    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58e8ba9c-7f9f-484a-adf7-2853282da751-home\") pod \"ipp-route-524a38-kserve-cd947985f-jchh8\" (UID: \"58e8ba9c-7f9f-484a-adf7-2853282da751\") " pod="ai-tenant-e2e-ipp-a-524a38/ipp-route-524a38-kserve-cd947985f-jchh8"
Sep 14 05:18:43.894013 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:43.893866    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58e8ba9c-7f9f-484a-adf7-2853282da751-tls-certs\") pod \"ipp-route-524a38-kserve-cd947985f-jchh8\" (UID: \"58e8ba9c-7f9f-484a-adf7-2853282da751\") " pod="ai-tenant-e2e-ipp-a-524a38/ipp-route-524a38-kserve-cd947985f-jchh8"
Sep 14 05:18:43.894013 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:43.893895    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58e8ba9c-7f9f-484a-adf7-2853282da751-tmp-dir\") pod \"ipp-route-524a38-kserve-cd947985f-jchh8\" (UID: \"58e8ba9c-7f9f-484a-adf7-2853282da751\") " pod="ai-tenant-e2e-ipp-a-524a38/ipp-route-524a38-kserve-cd947985f-jchh8"
Sep 14 05:18:43.894013 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:43.893963    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jgrm\" (UniqueName: \"kubernetes.io/projected/58e8ba9c-7f9f-484a-adf7-2853282da751-kube-api-access-6jgrm\") pod \"ipp-route-524a38-kserve-cd947985f-jchh8\" (UID: \"58e8ba9c-7f9f-484a-adf7-2853282da751\") " pod="ai-tenant-e2e-ipp-a-524a38/ipp-route-524a38-kserve-cd947985f-jchh8"
Sep 14 05:18:43.894013 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:43.893987    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58e8ba9c-7f9f-484a-adf7-2853282da751-dshm\") pod \"ipp-route-524a38-kserve-cd947985f-jchh8\" (UID: \"58e8ba9c-7f9f-484a-adf7-2853282da751\") " pod="ai-tenant-e2e-ipp-a-524a38/ipp-route-524a38-kserve-cd947985f-jchh8"
Sep 14 05:18:43.894013 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:43.894009    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58e8ba9c-7f9f-484a-adf7-2853282da751-model-cache\") pod \"ipp-route-524a38-kserve-cd947985f-jchh8\" (UID: \"58e8ba9c-7f9f-484a-adf7-2853282da751\") " pod="ai-tenant-e2e-ipp-a-524a38/ipp-route-524a38-kserve-cd947985f-jchh8"
Sep 14 05:18:43.994898 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:43.994866    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jgrm\" (UniqueName: \"kubernetes.io/projected/58e8ba9c-7f9f-484a-adf7-2853282da751-kube-api-access-6jgrm\") pod \"ipp-route-524a38-kserve-cd947985f-jchh8\" (UID: \"58e8ba9c-7f9f-484a-adf7-2853282da751\") " pod="ai-tenant-e2e-ipp-a-524a38/ipp-route-524a38-kserve-cd947985f-jchh8"
Sep 14 05:18:43.994898 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:43.994900    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58e8ba9c-7f9f-484a-adf7-2853282da751-dshm\") pod \"ipp-route-524a38-kserve-cd947985f-jchh8\" (UID: \"58e8ba9c-7f9f-484a-adf7-2853282da751\") " pod="ai-tenant-e2e-ipp-a-524a38/ipp-route-524a38-kserve-cd947985f-jchh8"
Sep 14 05:18:43.995154 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:43.994920    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58e8ba9c-7f9f-484a-adf7-2853282da751-model-cache\") pod \"ipp-route-524a38-kserve-cd947985f-jchh8\" (UID: \"58e8ba9c-7f9f-484a-adf7-2853282da751\") " pod="ai-tenant-e2e-ipp-a-524a38/ipp-route-524a38-kserve-cd947985f-jchh8"
Sep 14 05:18:43.995154 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:43.994971    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58e8ba9c-7f9f-484a-adf7-2853282da751-home\") pod \"ipp-route-524a38-kserve-cd947985f-jchh8\" (UID: \"58e8ba9c-7f9f-484a-adf7-2853282da751\") " pod="ai-tenant-e2e-ipp-a-524a38/ipp-route-524a38-kserve-cd947985f-jchh8"
Sep 14 05:18:43.995154 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:43.994994    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58e8ba9c-7f9f-484a-adf7-2853282da751-tls-certs\") pod \"ipp-route-524a38-kserve-cd947985f-jchh8\" (UID: \"58e8ba9c-7f9f-484a-adf7-2853282da751\") " pod="ai-tenant-e2e-ipp-a-524a38/ipp-route-524a38-kserve-cd947985f-jchh8"
Sep 14 05:18:43.995154 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:43.995020    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58e8ba9c-7f9f-484a-adf7-2853282da751-tmp-dir\") pod \"ipp-route-524a38-kserve-cd947985f-jchh8\" (UID: \"58e8ba9c-7f9f-484a-adf7-2853282da751\") " pod="ai-tenant-e2e-ipp-a-524a38/ipp-route-524a38-kserve-cd947985f-jchh8"
Sep 14 05:18:43.995629 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:43.995576    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58e8ba9c-7f9f-484a-adf7-2853282da751-home\") pod \"ipp-route-524a38-kserve-cd947985f-jchh8\" (UID: \"58e8ba9c-7f9f-484a-adf7-2853282da751\") " pod="ai-tenant-e2e-ipp-a-524a38/ipp-route-524a38-kserve-cd947985f-jchh8"
Sep 14 05:18:43.995629 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:43.995599    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58e8ba9c-7f9f-484a-adf7-2853282da751-tmp-dir\") pod \"ipp-route-524a38-kserve-cd947985f-jchh8\" (UID: \"58e8ba9c-7f9f-484a-adf7-2853282da751\") " pod="ai-tenant-e2e-ipp-a-524a38/ipp-route-524a38-kserve-cd947985f-jchh8"
Sep 14 05:18:43.996011 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:43.995661    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58e8ba9c-7f9f-484a-adf7-2853282da751-model-cache\") pod \"ipp-route-524a38-kserve-cd947985f-jchh8\" (UID: \"58e8ba9c-7f9f-484a-adf7-2853282da751\") " pod="ai-tenant-e2e-ipp-a-524a38/ipp-route-524a38-kserve-cd947985f-jchh8"
Sep 14 05:18:43.997583 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:43.997532    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58e8ba9c-7f9f-484a-adf7-2853282da751-dshm\") pod \"ipp-route-524a38-kserve-cd947985f-jchh8\" (UID: \"58e8ba9c-7f9f-484a-adf7-2853282da751\") " pod="ai-tenant-e2e-ipp-a-524a38/ipp-route-524a38-kserve-cd947985f-jchh8"
Sep 14 05:18:43.998013 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:43.997994    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58e8ba9c-7f9f-484a-adf7-2853282da751-tls-certs\") pod \"ipp-route-524a38-kserve-cd947985f-jchh8\" (UID: \"58e8ba9c-7f9f-484a-adf7-2853282da751\") " pod="ai-tenant-e2e-ipp-a-524a38/ipp-route-524a38-kserve-cd947985f-jchh8"
Sep 14 05:18:44.008624 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:44.008602    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jgrm\" (UniqueName: \"kubernetes.io/projected/58e8ba9c-7f9f-484a-adf7-2853282da751-kube-api-access-6jgrm\") pod \"ipp-route-524a38-kserve-cd947985f-jchh8\" (UID: \"58e8ba9c-7f9f-484a-adf7-2853282da751\") " pod="ai-tenant-e2e-ipp-a-524a38/ipp-route-524a38-kserve-cd947985f-jchh8"
Sep 14 05:18:44.140431 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:44.140402    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-524a38/ipp-route-524a38-kserve-cd947985f-jchh8"
Sep 14 05:18:44.413809 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:44.413784    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-524a38/ipp-route-524a38-kserve-cd947985f-jchh8"]
Sep 14 05:18:44.415939 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:18:44.415908    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58e8ba9c_7f9f_484a_adf7_2853282da751.slice/crio-a3ab586f7406179a0b4ef36dba826b6da416874ecf7766f6d3706f3d180fee92 WatchSource:0}: Error finding container a3ab586f7406179a0b4ef36dba826b6da416874ecf7766f6d3706f3d180fee92: Status 404 returned error can't find the container with id a3ab586f7406179a0b4ef36dba826b6da416874ecf7766f6d3706f3d180fee92
Sep 14 05:18:45.285438 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:45.285402    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-524a38/ipp-route-524a38-kserve-cd947985f-jchh8" event={"ID":"58e8ba9c-7f9f-484a-adf7-2853282da751","Type":"ContainerStarted","Data":"6519939dd380f4bad2dec9e593f476b3baddce0746458738613f0fb677845add"}
Sep 14 05:18:45.285438 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:45.285439    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-524a38/ipp-route-524a38-kserve-cd947985f-jchh8" event={"ID":"58e8ba9c-7f9f-484a-adf7-2853282da751","Type":"ContainerStarted","Data":"a3ab586f7406179a0b4ef36dba826b6da416874ecf7766f6d3706f3d180fee92"}
Sep 14 05:18:45.303808 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:45.303743    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-524a38/ipp-route-524a38-kserve-cd947985f-jchh8" podStartSLOduration=2.303727646 podStartE2EDuration="2.303727646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:18:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:18:45.301812756 +0000 UTC m=+1923.477584315" watchObservedRunningTime="2026-09-14 05:18:45.303727646 +0000 UTC m=+1923.479499205"
Sep 14 05:18:48.521096 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:48.521063    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:18:54.141522 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:54.141483    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-524a38/ipp-route-524a38-kserve-cd947985f-jchh8"
Sep 14 05:18:54.142038 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:54.141541    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-524a38/ipp-route-524a38-kserve-cd947985f-jchh8"
Sep 14 05:18:54.153827 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:54.153800    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-524a38/ipp-route-524a38-kserve-cd947985f-jchh8"
Sep 14 05:18:54.351305 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:18:54.351273    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-524a38/ipp-route-524a38-kserve-cd947985f-jchh8"
Sep 14 05:19:02.728496 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:02.728460    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:19:08.780268 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:08.780239    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-524a38-64f67b69c5-4bpc4_efe291d5-bba9-494c-bf7a-e8bea65d52d6/payload-processing/0.log"
Sep 14 05:19:16.857937 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:16.857895    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2339fb-9f8c9fb5d-lwgwm"]
Sep 14 05:19:16.861596 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:16.861572    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2339fb-9f8c9fb5d-lwgwm"
Sep 14 05:19:16.863969 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:16.863941    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-2339fb-dockercfg-g7cjf\""
Sep 14 05:19:16.864092 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:16.863941    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-2339fb\""
Sep 14 05:19:16.869788 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:16.869747    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2339fb-9f8c9fb5d-lwgwm"]
Sep 14 05:19:16.875993 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:16.875972    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgw72\" (UniqueName: \"kubernetes.io/projected/a28b0b06-c4e3-4510-a369-31bb55592080-kube-api-access-xgw72\") pod \"payload-pre-processing-e2e-ipp-cleanup-2339fb-9f8c9fb5d-lwgwm\" (UID: \"a28b0b06-c4e3-4510-a369-31bb55592080\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2339fb-9f8c9fb5d-lwgwm"
Sep 14 05:19:16.876086 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:16.876005    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a28b0b06-c4e3-4510-a369-31bb55592080-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-2339fb-9f8c9fb5d-lwgwm\" (UID: \"a28b0b06-c4e3-4510-a369-31bb55592080\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2339fb-9f8c9fb5d-lwgwm"
Sep 14 05:19:16.884200 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:16.884177    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2339fb-9967895f7-db2s9"]
Sep 14 05:19:16.887564 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:16.887545    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2339fb-9967895f7-db2s9"
Sep 14 05:19:16.898927 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:16.898907    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2339fb-9967895f7-db2s9"]
Sep 14 05:19:16.977117 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:16.977078    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6009e86-5fa0-43ce-befd-58dd5ef39d82-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-2339fb-9967895f7-db2s9\" (UID: \"f6009e86-5fa0-43ce-befd-58dd5ef39d82\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2339fb-9967895f7-db2s9"
Sep 14 05:19:16.977307 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:16.977150    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7l8t\" (UniqueName: \"kubernetes.io/projected/f6009e86-5fa0-43ce-befd-58dd5ef39d82-kube-api-access-g7l8t\") pod \"payload-processing-e2e-ipp-cleanup-2339fb-9967895f7-db2s9\" (UID: \"f6009e86-5fa0-43ce-befd-58dd5ef39d82\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2339fb-9967895f7-db2s9"
Sep 14 05:19:16.977307 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:16.977215    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgw72\" (UniqueName: \"kubernetes.io/projected/a28b0b06-c4e3-4510-a369-31bb55592080-kube-api-access-xgw72\") pod \"payload-pre-processing-e2e-ipp-cleanup-2339fb-9f8c9fb5d-lwgwm\" (UID: \"a28b0b06-c4e3-4510-a369-31bb55592080\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2339fb-9f8c9fb5d-lwgwm"
Sep 14 05:19:16.977307 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:16.977251    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a28b0b06-c4e3-4510-a369-31bb55592080-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-2339fb-9f8c9fb5d-lwgwm\" (UID: \"a28b0b06-c4e3-4510-a369-31bb55592080\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2339fb-9f8c9fb5d-lwgwm"
Sep 14 05:19:16.978026 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:16.978001    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a28b0b06-c4e3-4510-a369-31bb55592080-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-2339fb-9f8c9fb5d-lwgwm\" (UID: \"a28b0b06-c4e3-4510-a369-31bb55592080\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2339fb-9f8c9fb5d-lwgwm"
Sep 14 05:19:16.985571 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:16.985548    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgw72\" (UniqueName: \"kubernetes.io/projected/a28b0b06-c4e3-4510-a369-31bb55592080-kube-api-access-xgw72\") pod \"payload-pre-processing-e2e-ipp-cleanup-2339fb-9f8c9fb5d-lwgwm\" (UID: \"a28b0b06-c4e3-4510-a369-31bb55592080\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2339fb-9f8c9fb5d-lwgwm"
Sep 14 05:19:17.078388 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:17.078352    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6009e86-5fa0-43ce-befd-58dd5ef39d82-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-2339fb-9967895f7-db2s9\" (UID: \"f6009e86-5fa0-43ce-befd-58dd5ef39d82\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2339fb-9967895f7-db2s9"
Sep 14 05:19:17.078550 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:17.078407    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7l8t\" (UniqueName: \"kubernetes.io/projected/f6009e86-5fa0-43ce-befd-58dd5ef39d82-kube-api-access-g7l8t\") pod \"payload-processing-e2e-ipp-cleanup-2339fb-9967895f7-db2s9\" (UID: \"f6009e86-5fa0-43ce-befd-58dd5ef39d82\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2339fb-9967895f7-db2s9"
Sep 14 05:19:17.079055 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:17.079036    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6009e86-5fa0-43ce-befd-58dd5ef39d82-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-2339fb-9967895f7-db2s9\" (UID: \"f6009e86-5fa0-43ce-befd-58dd5ef39d82\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2339fb-9967895f7-db2s9"
Sep 14 05:19:17.086950 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:17.086924    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7l8t\" (UniqueName: \"kubernetes.io/projected/f6009e86-5fa0-43ce-befd-58dd5ef39d82-kube-api-access-g7l8t\") pod \"payload-processing-e2e-ipp-cleanup-2339fb-9967895f7-db2s9\" (UID: \"f6009e86-5fa0-43ce-befd-58dd5ef39d82\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2339fb-9967895f7-db2s9"
Sep 14 05:19:17.173544 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:17.173477    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2339fb-9f8c9fb5d-lwgwm"
Sep 14 05:19:17.197975 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:17.197949    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2339fb-9967895f7-db2s9"
Sep 14 05:19:17.448315 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:17.448288    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2339fb-9f8c9fb5d-lwgwm"]
Sep 14 05:19:17.449508 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:19:17.449479    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda28b0b06_c4e3_4510_a369_31bb55592080.slice/crio-a4e8baae37dfe648e94939060d49967240d6657582bb6197b9aba595282a1d22 WatchSource:0}: Error finding container a4e8baae37dfe648e94939060d49967240d6657582bb6197b9aba595282a1d22: Status 404 returned error can't find the container with id a4e8baae37dfe648e94939060d49967240d6657582bb6197b9aba595282a1d22
Sep 14 05:19:17.476644 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:17.476619    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2339fb-9f8c9fb5d-lwgwm" event={"ID":"a28b0b06-c4e3-4510-a369-31bb55592080","Type":"ContainerStarted","Data":"a4e8baae37dfe648e94939060d49967240d6657582bb6197b9aba595282a1d22"}
Sep 14 05:19:17.489922 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:17.489905    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2339fb-9967895f7-db2s9"]
Sep 14 05:19:17.491191 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:19:17.491168    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6009e86_5fa0_43ce_befd_58dd5ef39d82.slice/crio-8afe5f413693473b604f48c79682dcb9b6d10d29b0dea9eef2a7d8f68c09c882 WatchSource:0}: Error finding container 8afe5f413693473b604f48c79682dcb9b6d10d29b0dea9eef2a7d8f68c09c882: Status 404 returned error can't find the container with id 8afe5f413693473b604f48c79682dcb9b6d10d29b0dea9eef2a7d8f68c09c882
Sep 14 05:19:18.483526 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:18.483488    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2339fb-9967895f7-db2s9" event={"ID":"f6009e86-5fa0-43ce-befd-58dd5ef39d82","Type":"ContainerStarted","Data":"f76a1e15bc51b6e295e2de7ec3d9b5d4940426ec0df8fcc803e6cac9dc07750b"}
Sep 14 05:19:18.484012 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:18.483553    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2339fb-9967895f7-db2s9"
Sep 14 05:19:18.484012 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:18.483578    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2339fb-9967895f7-db2s9" event={"ID":"f6009e86-5fa0-43ce-befd-58dd5ef39d82","Type":"ContainerStarted","Data":"8afe5f413693473b604f48c79682dcb9b6d10d29b0dea9eef2a7d8f68c09c882"}
Sep 14 05:19:18.485141 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:18.485117    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2339fb-9f8c9fb5d-lwgwm" event={"ID":"a28b0b06-c4e3-4510-a369-31bb55592080","Type":"ContainerStarted","Data":"2c4d7ee252b4adf587db48e7b1917ec6e1af72e28a962d5d9ec36a5af32c89d9"}
Sep 14 05:19:18.485239 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:18.485161    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2339fb-9f8c9fb5d-lwgwm"
Sep 14 05:19:18.500087 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:18.500043    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2339fb-9967895f7-db2s9" podStartSLOduration=2.500029621 podStartE2EDuration="2.500029621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:19:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:19:18.498396995 +0000 UTC m=+1956.674168554" watchObservedRunningTime="2026-09-14 05:19:18.500029621 +0000 UTC m=+1956.675801176"
Sep 14 05:19:18.515001 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:18.514963    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2339fb-9f8c9fb5d-lwgwm" podStartSLOduration=2.51495031 podStartE2EDuration="2.51495031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:19:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:19:18.512892565 +0000 UTC m=+1956.688664124" watchObservedRunningTime="2026-09-14 05:19:18.51495031 +0000 UTC m=+1956.690721869"
Sep 14 05:19:22.935621 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:22.935557    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:19:29.492530 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:29.492496    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2339fb-9f8c9fb5d-lwgwm"
Sep 14 05:19:29.492903 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:29.492671    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2339fb-9967895f7-db2s9"
Sep 14 05:19:54.688236 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:54.688151    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2339fb-9967895f7-db2s9"]
Sep 14 05:19:54.688824 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:54.688415    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2339fb-9967895f7-db2s9" podUID="f6009e86-5fa0-43ce-befd-58dd5ef39d82" containerName="payload-processing" containerID="cri-o://f76a1e15bc51b6e295e2de7ec3d9b5d4940426ec0df8fcc803e6cac9dc07750b" gracePeriod=30
Sep 14 05:19:54.693202 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:54.693172    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2339fb-9f8c9fb5d-lwgwm"]
Sep 14 05:19:54.693616 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:54.693585    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2339fb-9f8c9fb5d-lwgwm" podUID="a28b0b06-c4e3-4510-a369-31bb55592080" containerName="payload-pre-processing" containerID="cri-o://2c4d7ee252b4adf587db48e7b1917ec6e1af72e28a962d5d9ec36a5af32c89d9" gracePeriod=30
Sep 14 05:19:54.937267 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:54.937241    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2339fb-9f8c9fb5d-lwgwm"
Sep 14 05:19:54.967487 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:54.967462    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2339fb-9967895f7-db2s9"
Sep 14 05:19:55.118034 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:55.117996    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a28b0b06-c4e3-4510-a369-31bb55592080-plugins-config-volume\") pod \"a28b0b06-c4e3-4510-a369-31bb55592080\" (UID: \"a28b0b06-c4e3-4510-a369-31bb55592080\") "
Sep 14 05:19:55.118221 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:55.118055    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7l8t\" (UniqueName: \"kubernetes.io/projected/f6009e86-5fa0-43ce-befd-58dd5ef39d82-kube-api-access-g7l8t\") pod \"f6009e86-5fa0-43ce-befd-58dd5ef39d82\" (UID: \"f6009e86-5fa0-43ce-befd-58dd5ef39d82\") "
Sep 14 05:19:55.118221 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:55.118094    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgw72\" (UniqueName: \"kubernetes.io/projected/a28b0b06-c4e3-4510-a369-31bb55592080-kube-api-access-xgw72\") pod \"a28b0b06-c4e3-4510-a369-31bb55592080\" (UID: \"a28b0b06-c4e3-4510-a369-31bb55592080\") "
Sep 14 05:19:55.118221 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:55.118148    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6009e86-5fa0-43ce-befd-58dd5ef39d82-plugins-config-volume\") pod \"f6009e86-5fa0-43ce-befd-58dd5ef39d82\" (UID: \"f6009e86-5fa0-43ce-befd-58dd5ef39d82\") "
Sep 14 05:19:55.118493 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:55.118458    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a28b0b06-c4e3-4510-a369-31bb55592080-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a28b0b06-c4e3-4510-a369-31bb55592080" (UID: "a28b0b06-c4e3-4510-a369-31bb55592080"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 05:19:55.118637 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:55.118596    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6009e86-5fa0-43ce-befd-58dd5ef39d82-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f6009e86-5fa0-43ce-befd-58dd5ef39d82" (UID: "f6009e86-5fa0-43ce-befd-58dd5ef39d82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 05:19:55.120306 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:55.120285    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6009e86-5fa0-43ce-befd-58dd5ef39d82-kube-api-access-g7l8t" (OuterVolumeSpecName: "kube-api-access-g7l8t") pod "f6009e86-5fa0-43ce-befd-58dd5ef39d82" (UID: "f6009e86-5fa0-43ce-befd-58dd5ef39d82"). InnerVolumeSpecName "kube-api-access-g7l8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:19:55.120367 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:55.120303    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a28b0b06-c4e3-4510-a369-31bb55592080-kube-api-access-xgw72" (OuterVolumeSpecName: "kube-api-access-xgw72") pod "a28b0b06-c4e3-4510-a369-31bb55592080" (UID: "a28b0b06-c4e3-4510-a369-31bb55592080"). InnerVolumeSpecName "kube-api-access-xgw72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:19:55.219462 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:55.219350    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a28b0b06-c4e3-4510-a369-31bb55592080-plugins-config-volume\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:19:55.219462 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:55.219399    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7l8t\" (UniqueName: \"kubernetes.io/projected/f6009e86-5fa0-43ce-befd-58dd5ef39d82-kube-api-access-g7l8t\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:19:55.219462 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:55.219412    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgw72\" (UniqueName: \"kubernetes.io/projected/a28b0b06-c4e3-4510-a369-31bb55592080-kube-api-access-xgw72\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:19:55.219462 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:55.219424    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6009e86-5fa0-43ce-befd-58dd5ef39d82-plugins-config-volume\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:19:55.695606 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:55.695566    2565 generic.go:356] "Generic (PLEG): container finished" podID="f6009e86-5fa0-43ce-befd-58dd5ef39d82" containerID="f76a1e15bc51b6e295e2de7ec3d9b5d4940426ec0df8fcc803e6cac9dc07750b" exitCode=0
Sep 14 05:19:55.695606 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:55.695611    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2339fb-9967895f7-db2s9"
Sep 14 05:19:55.696155 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:55.695657    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2339fb-9967895f7-db2s9" event={"ID":"f6009e86-5fa0-43ce-befd-58dd5ef39d82","Type":"ContainerDied","Data":"f76a1e15bc51b6e295e2de7ec3d9b5d4940426ec0df8fcc803e6cac9dc07750b"}
Sep 14 05:19:55.696155 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:55.695723    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2339fb-9967895f7-db2s9" event={"ID":"f6009e86-5fa0-43ce-befd-58dd5ef39d82","Type":"ContainerDied","Data":"8afe5f413693473b604f48c79682dcb9b6d10d29b0dea9eef2a7d8f68c09c882"}
Sep 14 05:19:55.696155 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:55.695778    2565 scope.go:117] "RemoveContainer" containerID="f76a1e15bc51b6e295e2de7ec3d9b5d4940426ec0df8fcc803e6cac9dc07750b"
Sep 14 05:19:55.697146 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:55.697079    2565 generic.go:356] "Generic (PLEG): container finished" podID="a28b0b06-c4e3-4510-a369-31bb55592080" containerID="2c4d7ee252b4adf587db48e7b1917ec6e1af72e28a962d5d9ec36a5af32c89d9" exitCode=0
Sep 14 05:19:55.697146 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:55.697111    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2339fb-9f8c9fb5d-lwgwm" event={"ID":"a28b0b06-c4e3-4510-a369-31bb55592080","Type":"ContainerDied","Data":"2c4d7ee252b4adf587db48e7b1917ec6e1af72e28a962d5d9ec36a5af32c89d9"}
Sep 14 05:19:55.697146 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:55.697134    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2339fb-9f8c9fb5d-lwgwm"
Sep 14 05:19:55.697317 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:55.697162    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2339fb-9f8c9fb5d-lwgwm" event={"ID":"a28b0b06-c4e3-4510-a369-31bb55592080","Type":"ContainerDied","Data":"a4e8baae37dfe648e94939060d49967240d6657582bb6197b9aba595282a1d22"}
Sep 14 05:19:55.713749 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:55.713733    2565 scope.go:117] "RemoveContainer" containerID="f76a1e15bc51b6e295e2de7ec3d9b5d4940426ec0df8fcc803e6cac9dc07750b"
Sep 14 05:19:55.714034 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:19:55.714016    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f76a1e15bc51b6e295e2de7ec3d9b5d4940426ec0df8fcc803e6cac9dc07750b\": container with ID starting with f76a1e15bc51b6e295e2de7ec3d9b5d4940426ec0df8fcc803e6cac9dc07750b not found: ID does not exist" containerID="f76a1e15bc51b6e295e2de7ec3d9b5d4940426ec0df8fcc803e6cac9dc07750b"
Sep 14 05:19:55.714082 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:55.714043    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f76a1e15bc51b6e295e2de7ec3d9b5d4940426ec0df8fcc803e6cac9dc07750b"} err="failed to get container status \"f76a1e15bc51b6e295e2de7ec3d9b5d4940426ec0df8fcc803e6cac9dc07750b\": rpc error: code = NotFound desc = could not find container \"f76a1e15bc51b6e295e2de7ec3d9b5d4940426ec0df8fcc803e6cac9dc07750b\": container with ID starting with f76a1e15bc51b6e295e2de7ec3d9b5d4940426ec0df8fcc803e6cac9dc07750b not found: ID does not exist"
Sep 14 05:19:55.714082 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:55.714061    2565 scope.go:117] "RemoveContainer" containerID="2c4d7ee252b4adf587db48e7b1917ec6e1af72e28a962d5d9ec36a5af32c89d9"
Sep 14 05:19:55.721904 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:55.721886    2565 scope.go:117] "RemoveContainer" containerID="2c4d7ee252b4adf587db48e7b1917ec6e1af72e28a962d5d9ec36a5af32c89d9"
Sep 14 05:19:55.722184 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:19:55.722165    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c4d7ee252b4adf587db48e7b1917ec6e1af72e28a962d5d9ec36a5af32c89d9\": container with ID starting with 2c4d7ee252b4adf587db48e7b1917ec6e1af72e28a962d5d9ec36a5af32c89d9 not found: ID does not exist" containerID="2c4d7ee252b4adf587db48e7b1917ec6e1af72e28a962d5d9ec36a5af32c89d9"
Sep 14 05:19:55.722241 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:55.722191    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c4d7ee252b4adf587db48e7b1917ec6e1af72e28a962d5d9ec36a5af32c89d9"} err="failed to get container status \"2c4d7ee252b4adf587db48e7b1917ec6e1af72e28a962d5d9ec36a5af32c89d9\": rpc error: code = NotFound desc = could not find container \"2c4d7ee252b4adf587db48e7b1917ec6e1af72e28a962d5d9ec36a5af32c89d9\": container with ID starting with 2c4d7ee252b4adf587db48e7b1917ec6e1af72e28a962d5d9ec36a5af32c89d9 not found: ID does not exist"
Sep 14 05:19:55.725119 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:55.725097    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2339fb-9f8c9fb5d-lwgwm"]
Sep 14 05:19:55.729180 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:55.729157    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2339fb-9f8c9fb5d-lwgwm"]
Sep 14 05:19:55.740271 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:55.740248    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2339fb-9967895f7-db2s9"]
Sep 14 05:19:55.746314 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:55.746293    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2339fb-9967895f7-db2s9"]
Sep 14 05:19:56.334302 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:56.334270    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a28b0b06-c4e3-4510-a369-31bb55592080" path="/var/lib/kubelet/pods/a28b0b06-c4e3-4510-a369-31bb55592080/volumes"
Sep 14 05:19:56.334625 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:56.334612    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6009e86-5fa0-43ce-befd-58dd5ef39d82" path="/var/lib/kubelet/pods/f6009e86-5fa0-43ce-befd-58dd5ef39d82/volumes"
Sep 14 05:19:59.231297 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:19:59.231259    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:20:05.276551 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:05.276509    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-524a38-rfnv5"]
Sep 14 05:20:05.277048 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:05.277032    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a28b0b06-c4e3-4510-a369-31bb55592080" containerName="payload-pre-processing"
Sep 14 05:20:05.277122 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:05.277052    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a28b0b06-c4e3-4510-a369-31bb55592080" containerName="payload-pre-processing"
Sep 14 05:20:05.277122 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:05.277070    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6009e86-5fa0-43ce-befd-58dd5ef39d82" containerName="payload-processing"
Sep 14 05:20:05.277122 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:05.277079    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6009e86-5fa0-43ce-befd-58dd5ef39d82" containerName="payload-processing"
Sep 14 05:20:05.277271 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:05.277168    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a28b0b06-c4e3-4510-a369-31bb55592080" containerName="payload-pre-processing"
Sep 14 05:20:05.277271 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:05.277182    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6009e86-5fa0-43ce-befd-58dd5ef39d82" containerName="payload-processing"
Sep 14 05:20:05.281607 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:05.281586    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-524a38-rfnv5"
Sep 14 05:20:05.284148 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:05.284125    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 05:20:05.284248 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:05.284210    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k6hpm\""
Sep 14 05:20:05.288463 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:05.288438    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-524a38-rfnv5"]
Sep 14 05:20:05.307706 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:05.307674    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb5f8b77-4321-40a3-acff-38e7b5e5c45b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-524a38-rfnv5\" (UID: \"cb5f8b77-4321-40a3-acff-38e7b5e5c45b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-524a38-rfnv5"
Sep 14 05:20:05.408596 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:05.408559    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb5f8b77-4321-40a3-acff-38e7b5e5c45b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-524a38-rfnv5\" (UID: \"cb5f8b77-4321-40a3-acff-38e7b5e5c45b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-524a38-rfnv5"
Sep 14 05:20:05.409215 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:05.409196    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb5f8b77-4321-40a3-acff-38e7b5e5c45b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-524a38-rfnv5\" (UID: \"cb5f8b77-4321-40a3-acff-38e7b5e5c45b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-524a38-rfnv5"
Sep 14 05:20:05.592858 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:05.592825    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-524a38-rfnv5"
Sep 14 05:20:05.863999 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:05.863974    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-524a38-rfnv5"]
Sep 14 05:20:05.865956 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:20:05.865922    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb5f8b77_4321_40a3_acff_38e7b5e5c45b.slice/crio-6232b5da0358600b4f85fdf1cff2e047a11176e20d2e29645137d63be689b2e6 WatchSource:0}: Error finding container 6232b5da0358600b4f85fdf1cff2e047a11176e20d2e29645137d63be689b2e6: Status 404 returned error can't find the container with id 6232b5da0358600b4f85fdf1cff2e047a11176e20d2e29645137d63be689b2e6
Sep 14 05:20:06.766377 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:06.766336    2565 generic.go:356] "Generic (PLEG): container finished" podID="cb5f8b77-4321-40a3-acff-38e7b5e5c45b" containerID="d0a88e7d8eaf180fa07c8c517885fdb152a0f15294531bf13ad4cdd9db3f81b0" exitCode=0
Sep 14 05:20:06.766834 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:06.766421    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-524a38-rfnv5" event={"ID":"cb5f8b77-4321-40a3-acff-38e7b5e5c45b","Type":"ContainerDied","Data":"d0a88e7d8eaf180fa07c8c517885fdb152a0f15294531bf13ad4cdd9db3f81b0"}
Sep 14 05:20:06.766834 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:06.766468    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-524a38-rfnv5" event={"ID":"cb5f8b77-4321-40a3-acff-38e7b5e5c45b","Type":"ContainerStarted","Data":"6232b5da0358600b4f85fdf1cff2e047a11176e20d2e29645137d63be689b2e6"}
Sep 14 05:20:07.904282 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:07.904244    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-524a38-rfnv5"
Sep 14 05:20:07.925604 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:07.925576    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb5f8b77-4321-40a3-acff-38e7b5e5c45b-maas-api-service-ca\") pod \"cb5f8b77-4321-40a3-acff-38e7b5e5c45b\" (UID: \"cb5f8b77-4321-40a3-acff-38e7b5e5c45b\") "
Sep 14 05:20:07.925958 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:07.925932    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb5f8b77-4321-40a3-acff-38e7b5e5c45b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cb5f8b77-4321-40a3-acff-38e7b5e5c45b" (UID: "cb5f8b77-4321-40a3-acff-38e7b5e5c45b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 05:20:08.026384 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:08.026290    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb5f8b77-4321-40a3-acff-38e7b5e5c45b-maas-api-service-ca\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:20:08.779358 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:08.779277    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-524a38-rfnv5"
Sep 14 05:20:08.779358 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:08.779305    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-524a38-rfnv5" event={"ID":"cb5f8b77-4321-40a3-acff-38e7b5e5c45b","Type":"ContainerDied","Data":"6232b5da0358600b4f85fdf1cff2e047a11176e20d2e29645137d63be689b2e6"}
Sep 14 05:20:08.779358 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:08.779348    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6232b5da0358600b4f85fdf1cff2e047a11176e20d2e29645137d63be689b2e6"
Sep 14 05:20:18.729260 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:18.729215    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:20:21.085134 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:21.085098    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-524a38-64f67b69c5-4bpc4"]
Sep 14 05:20:21.085587 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:21.085394    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-524a38-64f67b69c5-4bpc4" podUID="efe291d5-bba9-494c-bf7a-e8bea65d52d6" containerName="payload-processing" containerID="cri-o://842f1d999a0b38333d1694d260e732172c24068b7ed57a966d8ed1bf698049fa" gracePeriod=30
Sep 14 05:20:21.271038 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:21.271007    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-524a38-rfnv5"]
Sep 14 05:20:21.276046 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:21.276017    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-524a38-rfnv5"]
Sep 14 05:20:21.339315 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:21.339257    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-524a38-64f67b69c5-4bpc4"
Sep 14 05:20:21.445488 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:21.445438    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efe291d5-bba9-494c-bf7a-e8bea65d52d6-plugins-config-volume\") pod \"efe291d5-bba9-494c-bf7a-e8bea65d52d6\" (UID: \"efe291d5-bba9-494c-bf7a-e8bea65d52d6\") "
Sep 14 05:20:21.445771 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:21.445721    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6ljg\" (UniqueName: \"kubernetes.io/projected/efe291d5-bba9-494c-bf7a-e8bea65d52d6-kube-api-access-k6ljg\") pod \"efe291d5-bba9-494c-bf7a-e8bea65d52d6\" (UID: \"efe291d5-bba9-494c-bf7a-e8bea65d52d6\") "
Sep 14 05:20:21.445997 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:21.445956    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efe291d5-bba9-494c-bf7a-e8bea65d52d6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "efe291d5-bba9-494c-bf7a-e8bea65d52d6" (UID: "efe291d5-bba9-494c-bf7a-e8bea65d52d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 05:20:21.446350 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:21.446327    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efe291d5-bba9-494c-bf7a-e8bea65d52d6-plugins-config-volume\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:20:21.448020 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:21.447995    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efe291d5-bba9-494c-bf7a-e8bea65d52d6-kube-api-access-k6ljg" (OuterVolumeSpecName: "kube-api-access-k6ljg") pod "efe291d5-bba9-494c-bf7a-e8bea65d52d6" (UID: "efe291d5-bba9-494c-bf7a-e8bea65d52d6"). InnerVolumeSpecName "kube-api-access-k6ljg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:20:21.547671 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:21.547643    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6ljg\" (UniqueName: \"kubernetes.io/projected/efe291d5-bba9-494c-bf7a-e8bea65d52d6-kube-api-access-k6ljg\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:20:21.851899 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:21.851865    2565 generic.go:356] "Generic (PLEG): container finished" podID="efe291d5-bba9-494c-bf7a-e8bea65d52d6" containerID="842f1d999a0b38333d1694d260e732172c24068b7ed57a966d8ed1bf698049fa" exitCode=0
Sep 14 05:20:21.852098 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:21.851910    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-524a38-64f67b69c5-4bpc4"
Sep 14 05:20:21.852098 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:21.851943    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-524a38-64f67b69c5-4bpc4" event={"ID":"efe291d5-bba9-494c-bf7a-e8bea65d52d6","Type":"ContainerDied","Data":"842f1d999a0b38333d1694d260e732172c24068b7ed57a966d8ed1bf698049fa"}
Sep 14 05:20:21.852098 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:21.851993    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-524a38-64f67b69c5-4bpc4" event={"ID":"efe291d5-bba9-494c-bf7a-e8bea65d52d6","Type":"ContainerDied","Data":"71be9ab618e1353548bcfe09829f274bada3b92ec8a4455e4343006c265e43c3"}
Sep 14 05:20:21.852098 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:21.852022    2565 scope.go:117] "RemoveContainer" containerID="842f1d999a0b38333d1694d260e732172c24068b7ed57a966d8ed1bf698049fa"
Sep 14 05:20:21.861207 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:21.861189    2565 scope.go:117] "RemoveContainer" containerID="842f1d999a0b38333d1694d260e732172c24068b7ed57a966d8ed1bf698049fa"
Sep 14 05:20:21.861486 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:20:21.861466    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"842f1d999a0b38333d1694d260e732172c24068b7ed57a966d8ed1bf698049fa\": container with ID starting with 842f1d999a0b38333d1694d260e732172c24068b7ed57a966d8ed1bf698049fa not found: ID does not exist" containerID="842f1d999a0b38333d1694d260e732172c24068b7ed57a966d8ed1bf698049fa"
Sep 14 05:20:21.861541 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:21.861495    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"842f1d999a0b38333d1694d260e732172c24068b7ed57a966d8ed1bf698049fa"} err="failed to get container status \"842f1d999a0b38333d1694d260e732172c24068b7ed57a966d8ed1bf698049fa\": rpc error: code = NotFound desc = could not find container \"842f1d999a0b38333d1694d260e732172c24068b7ed57a966d8ed1bf698049fa\": container with ID starting with 842f1d999a0b38333d1694d260e732172c24068b7ed57a966d8ed1bf698049fa not found: ID does not exist"
Sep 14 05:20:21.874412 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:21.874388    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-524a38-64f67b69c5-4bpc4"]
Sep 14 05:20:21.878416 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:21.878396    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-524a38-64f67b69c5-4bpc4"]
Sep 14 05:20:22.334440 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:22.334413    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb5f8b77-4321-40a3-acff-38e7b5e5c45b" path="/var/lib/kubelet/pods/cb5f8b77-4321-40a3-acff-38e7b5e5c45b/volumes"
Sep 14 05:20:22.334802 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:22.334776    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efe291d5-bba9-494c-bf7a-e8bea65d52d6" path="/var/lib/kubelet/pods/efe291d5-bba9-494c-bf7a-e8bea65d52d6/volumes"
Sep 14 05:20:26.637796 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:26.637745    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-524a38/ipp-route-524a38-kserve-cd947985f-jchh8"]
Sep 14 05:20:32.932736 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:32.932704    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:20:41.656368 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:41.656326    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-524a38/ipp-route-524a38-kserve-cd947985f-jchh8" podUID="58e8ba9c-7f9f-484a-adf7-2853282da751" containerName="main" containerID="cri-o://6519939dd380f4bad2dec9e593f476b3baddce0746458738613f0fb677845add" gracePeriod=45
Sep 14 05:20:41.902025 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:41.902001    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-524a38/ipp-route-524a38-kserve-cd947985f-jchh8"
Sep 14 05:20:41.971237 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:41.971152    2565 generic.go:356] "Generic (PLEG): container finished" podID="58e8ba9c-7f9f-484a-adf7-2853282da751" containerID="6519939dd380f4bad2dec9e593f476b3baddce0746458738613f0fb677845add" exitCode=0
Sep 14 05:20:41.971237 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:41.971194    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-524a38/ipp-route-524a38-kserve-cd947985f-jchh8"
Sep 14 05:20:41.971444 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:41.971242    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-524a38/ipp-route-524a38-kserve-cd947985f-jchh8" event={"ID":"58e8ba9c-7f9f-484a-adf7-2853282da751","Type":"ContainerDied","Data":"6519939dd380f4bad2dec9e593f476b3baddce0746458738613f0fb677845add"}
Sep 14 05:20:41.971444 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:41.971295    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-524a38/ipp-route-524a38-kserve-cd947985f-jchh8" event={"ID":"58e8ba9c-7f9f-484a-adf7-2853282da751","Type":"ContainerDied","Data":"a3ab586f7406179a0b4ef36dba826b6da416874ecf7766f6d3706f3d180fee92"}
Sep 14 05:20:41.971444 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:41.971317    2565 scope.go:117] "RemoveContainer" containerID="6519939dd380f4bad2dec9e593f476b3baddce0746458738613f0fb677845add"
Sep 14 05:20:41.980536 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:41.980513    2565 scope.go:117] "RemoveContainer" containerID="6519939dd380f4bad2dec9e593f476b3baddce0746458738613f0fb677845add"
Sep 14 05:20:41.980883 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:20:41.980862    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6519939dd380f4bad2dec9e593f476b3baddce0746458738613f0fb677845add\": container with ID starting with 6519939dd380f4bad2dec9e593f476b3baddce0746458738613f0fb677845add not found: ID does not exist" containerID="6519939dd380f4bad2dec9e593f476b3baddce0746458738613f0fb677845add"
Sep 14 05:20:41.980939 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:41.980893    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6519939dd380f4bad2dec9e593f476b3baddce0746458738613f0fb677845add"} err="failed to get container status \"6519939dd380f4bad2dec9e593f476b3baddce0746458738613f0fb677845add\": rpc error: code = NotFound desc = could not find container \"6519939dd380f4bad2dec9e593f476b3baddce0746458738613f0fb677845add\": container with ID starting with 6519939dd380f4bad2dec9e593f476b3baddce0746458738613f0fb677845add not found: ID does not exist"
Sep 14 05:20:42.040703 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:42.040672    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58e8ba9c-7f9f-484a-adf7-2853282da751-home\") pod \"58e8ba9c-7f9f-484a-adf7-2853282da751\" (UID: \"58e8ba9c-7f9f-484a-adf7-2853282da751\") "
Sep 14 05:20:42.040893 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:42.040738    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58e8ba9c-7f9f-484a-adf7-2853282da751-tmp-dir\") pod \"58e8ba9c-7f9f-484a-adf7-2853282da751\" (UID: \"58e8ba9c-7f9f-484a-adf7-2853282da751\") "
Sep 14 05:20:42.040893 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:42.040816    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58e8ba9c-7f9f-484a-adf7-2853282da751-dshm\") pod \"58e8ba9c-7f9f-484a-adf7-2853282da751\" (UID: \"58e8ba9c-7f9f-484a-adf7-2853282da751\") "
Sep 14 05:20:42.040893 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:42.040847    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jgrm\" (UniqueName: \"kubernetes.io/projected/58e8ba9c-7f9f-484a-adf7-2853282da751-kube-api-access-6jgrm\") pod \"58e8ba9c-7f9f-484a-adf7-2853282da751\" (UID: \"58e8ba9c-7f9f-484a-adf7-2853282da751\") "
Sep 14 05:20:42.040893 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:42.040891    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58e8ba9c-7f9f-484a-adf7-2853282da751-model-cache\") pod \"58e8ba9c-7f9f-484a-adf7-2853282da751\" (UID: \"58e8ba9c-7f9f-484a-adf7-2853282da751\") "
Sep 14 05:20:42.041099 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:42.040917    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58e8ba9c-7f9f-484a-adf7-2853282da751-tls-certs\") pod \"58e8ba9c-7f9f-484a-adf7-2853282da751\" (UID: \"58e8ba9c-7f9f-484a-adf7-2853282da751\") "
Sep 14 05:20:42.041099 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:42.040970    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e8ba9c-7f9f-484a-adf7-2853282da751-home" (OuterVolumeSpecName: "home") pod "58e8ba9c-7f9f-484a-adf7-2853282da751" (UID: "58e8ba9c-7f9f-484a-adf7-2853282da751"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:20:42.041099 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:42.040987    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e8ba9c-7f9f-484a-adf7-2853282da751-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "58e8ba9c-7f9f-484a-adf7-2853282da751" (UID: "58e8ba9c-7f9f-484a-adf7-2853282da751"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:20:42.041257 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:42.041187    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58e8ba9c-7f9f-484a-adf7-2853282da751-home\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:20:42.041257 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:42.041202    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58e8ba9c-7f9f-484a-adf7-2853282da751-tmp-dir\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:20:42.041257 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:42.041204    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e8ba9c-7f9f-484a-adf7-2853282da751-model-cache" (OuterVolumeSpecName: "model-cache") pod "58e8ba9c-7f9f-484a-adf7-2853282da751" (UID: "58e8ba9c-7f9f-484a-adf7-2853282da751"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:20:42.043118 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:42.043096    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e8ba9c-7f9f-484a-adf7-2853282da751-dshm" (OuterVolumeSpecName: "dshm") pod "58e8ba9c-7f9f-484a-adf7-2853282da751" (UID: "58e8ba9c-7f9f-484a-adf7-2853282da751"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:20:42.043118 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:42.043107    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58e8ba9c-7f9f-484a-adf7-2853282da751-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "58e8ba9c-7f9f-484a-adf7-2853282da751" (UID: "58e8ba9c-7f9f-484a-adf7-2853282da751"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 05:20:42.043222 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:42.043199    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58e8ba9c-7f9f-484a-adf7-2853282da751-kube-api-access-6jgrm" (OuterVolumeSpecName: "kube-api-access-6jgrm") pod "58e8ba9c-7f9f-484a-adf7-2853282da751" (UID: "58e8ba9c-7f9f-484a-adf7-2853282da751"). InnerVolumeSpecName "kube-api-access-6jgrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:20:42.142390 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:42.142349    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58e8ba9c-7f9f-484a-adf7-2853282da751-dshm\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:20:42.142390 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:42.142394    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jgrm\" (UniqueName: \"kubernetes.io/projected/58e8ba9c-7f9f-484a-adf7-2853282da751-kube-api-access-6jgrm\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:20:42.142595 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:42.142405    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58e8ba9c-7f9f-484a-adf7-2853282da751-model-cache\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:20:42.142595 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:42.142414    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58e8ba9c-7f9f-484a-adf7-2853282da751-tls-certs\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:20:42.297604 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:42.297576    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-524a38/ipp-route-524a38-kserve-cd947985f-jchh8"]
Sep 14 05:20:42.306385 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:42.306359    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-524a38/ipp-route-524a38-kserve-cd947985f-jchh8"]
Sep 14 05:20:42.334856 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:42.334828    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58e8ba9c-7f9f-484a-adf7-2853282da751" path="/var/lib/kubelet/pods/58e8ba9c-7f9f-484a-adf7-2853282da751/volumes"
Sep 14 05:20:42.737785 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:42.735870    2565 scope.go:117] "RemoveContainer" containerID="281af8a0142143ff5c2d99a6ea23cf72aafcd8014a22f782ff63fd88722bca87"
Sep 14 05:20:42.747243 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:20:42.747217    2565 scope.go:117] "RemoveContainer" containerID="41b4bb50bbde6e62b2d29093db8f2f2b618a3efa6e9573b10b06333a6ec3886e"
Sep 14 05:21:11.377876 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:11.377793    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7bbea9-qp6b8"]
Sep 14 05:21:11.378340 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:11.378128    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58e8ba9c-7f9f-484a-adf7-2853282da751" containerName="main"
Sep 14 05:21:11.378340 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:11.378139    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="58e8ba9c-7f9f-484a-adf7-2853282da751" containerName="main"
Sep 14 05:21:11.378340 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:11.378154    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efe291d5-bba9-494c-bf7a-e8bea65d52d6" containerName="payload-processing"
Sep 14 05:21:11.378340 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:11.378159    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="efe291d5-bba9-494c-bf7a-e8bea65d52d6" containerName="payload-processing"
Sep 14 05:21:11.378340 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:11.378170    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb5f8b77-4321-40a3-acff-38e7b5e5c45b" containerName="revoke-keys"
Sep 14 05:21:11.378340 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:11.378175    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb5f8b77-4321-40a3-acff-38e7b5e5c45b" containerName="revoke-keys"
Sep 14 05:21:11.378340 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:11.378224    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="58e8ba9c-7f9f-484a-adf7-2853282da751" containerName="main"
Sep 14 05:21:11.378340 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:11.378235    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="efe291d5-bba9-494c-bf7a-e8bea65d52d6" containerName="payload-processing"
Sep 14 05:21:11.378340 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:11.378243    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb5f8b77-4321-40a3-acff-38e7b5e5c45b" containerName="revoke-keys"
Sep 14 05:21:11.381411 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:11.381393    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7bbea9-qp6b8"
Sep 14 05:21:11.385929 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:11.385905    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k6hpm\""
Sep 14 05:21:11.386153 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:11.386139    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 05:21:11.391541 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:11.391516    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7bbea9-qp6b8"]
Sep 14 05:21:11.498420 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:11.498387    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1587987-eb18-4054-9792-6be94c731efd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7bbea9-qp6b8\" (UID: \"b1587987-eb18-4054-9792-6be94c731efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7bbea9-qp6b8"
Sep 14 05:21:11.599189 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:11.599154    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1587987-eb18-4054-9792-6be94c731efd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7bbea9-qp6b8\" (UID: \"b1587987-eb18-4054-9792-6be94c731efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7bbea9-qp6b8"
Sep 14 05:21:11.599724 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:11.599706    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1587987-eb18-4054-9792-6be94c731efd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7bbea9-qp6b8\" (UID: \"b1587987-eb18-4054-9792-6be94c731efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7bbea9-qp6b8"
Sep 14 05:21:11.692714 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:11.692639    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7bbea9-qp6b8"
Sep 14 05:21:11.974937 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:11.974910    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7bbea9-qp6b8"]
Sep 14 05:21:11.976811 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:21:11.976780    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1587987_eb18_4054_9792_6be94c731efd.slice/crio-a155ba3be6c574d366d37c506235043a38941e80f9060c769e087be69ebb5bc3 WatchSource:0}: Error finding container a155ba3be6c574d366d37c506235043a38941e80f9060c769e087be69ebb5bc3: Status 404 returned error can't find the container with id a155ba3be6c574d366d37c506235043a38941e80f9060c769e087be69ebb5bc3
Sep 14 05:21:12.126368 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:12.126333    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7bbea9-qp6b8" event={"ID":"b1587987-eb18-4054-9792-6be94c731efd","Type":"ContainerStarted","Data":"a155ba3be6c574d366d37c506235043a38941e80f9060c769e087be69ebb5bc3"}
Sep 14 05:21:13.133304 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:13.133266    2565 generic.go:356] "Generic (PLEG): container finished" podID="b1587987-eb18-4054-9792-6be94c731efd" containerID="a820d0277a87b38c63fa57a8342850509e500f720f0b78c94424ff5525356f5f" exitCode=0
Sep 14 05:21:13.133713 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:13.133352    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7bbea9-qp6b8" event={"ID":"b1587987-eb18-4054-9792-6be94c731efd","Type":"ContainerDied","Data":"a820d0277a87b38c63fa57a8342850509e500f720f0b78c94424ff5525356f5f"}
Sep 14 05:21:14.268788 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:14.268744    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7bbea9-qp6b8"
Sep 14 05:21:14.325424 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:14.325395    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1587987-eb18-4054-9792-6be94c731efd-maas-api-service-ca\") pod \"b1587987-eb18-4054-9792-6be94c731efd\" (UID: \"b1587987-eb18-4054-9792-6be94c731efd\") "
Sep 14 05:21:14.325735 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:14.325714    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1587987-eb18-4054-9792-6be94c731efd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b1587987-eb18-4054-9792-6be94c731efd" (UID: "b1587987-eb18-4054-9792-6be94c731efd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 05:21:14.427117 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:14.427025    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1587987-eb18-4054-9792-6be94c731efd-maas-api-service-ca\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:21:15.146632 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:15.146602    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7bbea9-qp6b8"
Sep 14 05:21:15.146826 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:15.146637    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7bbea9-qp6b8" event={"ID":"b1587987-eb18-4054-9792-6be94c731efd","Type":"ContainerDied","Data":"a155ba3be6c574d366d37c506235043a38941e80f9060c769e087be69ebb5bc3"}
Sep 14 05:21:15.146826 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:15.146681    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a155ba3be6c574d366d37c506235043a38941e80f9060c769e087be69ebb5bc3"
Sep 14 05:21:22.162704 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:22.162672    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7bbea9-6c44bcf854-8mbns"]
Sep 14 05:21:22.163143 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:22.162905    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7bbea9-6c44bcf854-8mbns" podUID="5edad9b0-9829-4009-ab8a-8fed1c7ff844" containerName="payload-pre-processing" containerID="cri-o://970e13fd6cc879b95305f3781a424ee4988dabd24cac2e4246e39261dfa20aa6" gracePeriod=30
Sep 14 05:21:22.321624 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:22.321591    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7bbea9-qp6b8"]
Sep 14 05:21:22.324782 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:22.324740    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7bbea9-qp6b8"]
Sep 14 05:21:22.334428 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:22.334402    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1587987-eb18-4054-9792-6be94c731efd" path="/var/lib/kubelet/pods/b1587987-eb18-4054-9792-6be94c731efd/volumes"
Sep 14 05:21:22.427830 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:22.427438    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7bbea9-6c44bcf854-8mbns"
Sep 14 05:21:22.498567 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:22.498534    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5edad9b0-9829-4009-ab8a-8fed1c7ff844-plugins-config-volume\") pod \"5edad9b0-9829-4009-ab8a-8fed1c7ff844\" (UID: \"5edad9b0-9829-4009-ab8a-8fed1c7ff844\") "
Sep 14 05:21:22.498709 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:22.498609    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xlt4\" (UniqueName: \"kubernetes.io/projected/5edad9b0-9829-4009-ab8a-8fed1c7ff844-kube-api-access-6xlt4\") pod \"5edad9b0-9829-4009-ab8a-8fed1c7ff844\" (UID: \"5edad9b0-9829-4009-ab8a-8fed1c7ff844\") "
Sep 14 05:21:22.498930 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:22.498904    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5edad9b0-9829-4009-ab8a-8fed1c7ff844-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5edad9b0-9829-4009-ab8a-8fed1c7ff844" (UID: "5edad9b0-9829-4009-ab8a-8fed1c7ff844"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 05:21:22.500828 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:22.500792    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5edad9b0-9829-4009-ab8a-8fed1c7ff844-kube-api-access-6xlt4" (OuterVolumeSpecName: "kube-api-access-6xlt4") pod "5edad9b0-9829-4009-ab8a-8fed1c7ff844" (UID: "5edad9b0-9829-4009-ab8a-8fed1c7ff844"). InnerVolumeSpecName "kube-api-access-6xlt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:21:22.599898 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:22.599872    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5edad9b0-9829-4009-ab8a-8fed1c7ff844-plugins-config-volume\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:21:22.599898 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:22.599896    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xlt4\" (UniqueName: \"kubernetes.io/projected/5edad9b0-9829-4009-ab8a-8fed1c7ff844-kube-api-access-6xlt4\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:21:23.192949 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:23.192912    2565 generic.go:356] "Generic (PLEG): container finished" podID="5edad9b0-9829-4009-ab8a-8fed1c7ff844" containerID="970e13fd6cc879b95305f3781a424ee4988dabd24cac2e4246e39261dfa20aa6" exitCode=0
Sep 14 05:21:23.193382 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:23.192970    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7bbea9-6c44bcf854-8mbns"
Sep 14 05:21:23.193382 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:23.193003    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7bbea9-6c44bcf854-8mbns" event={"ID":"5edad9b0-9829-4009-ab8a-8fed1c7ff844","Type":"ContainerDied","Data":"970e13fd6cc879b95305f3781a424ee4988dabd24cac2e4246e39261dfa20aa6"}
Sep 14 05:21:23.193382 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:23.193085    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7bbea9-6c44bcf854-8mbns" event={"ID":"5edad9b0-9829-4009-ab8a-8fed1c7ff844","Type":"ContainerDied","Data":"b8885691911a925922a1bddec85584eac86e5413bf3a83929c49bf6e3c545405"}
Sep 14 05:21:23.193382 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:23.193108    2565 scope.go:117] "RemoveContainer" containerID="970e13fd6cc879b95305f3781a424ee4988dabd24cac2e4246e39261dfa20aa6"
Sep 14 05:21:23.202091 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:23.202075    2565 scope.go:117] "RemoveContainer" containerID="970e13fd6cc879b95305f3781a424ee4988dabd24cac2e4246e39261dfa20aa6"
Sep 14 05:21:23.202395 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:21:23.202366    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"970e13fd6cc879b95305f3781a424ee4988dabd24cac2e4246e39261dfa20aa6\": container with ID starting with 970e13fd6cc879b95305f3781a424ee4988dabd24cac2e4246e39261dfa20aa6 not found: ID does not exist" containerID="970e13fd6cc879b95305f3781a424ee4988dabd24cac2e4246e39261dfa20aa6"
Sep 14 05:21:23.202517 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:23.202404    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"970e13fd6cc879b95305f3781a424ee4988dabd24cac2e4246e39261dfa20aa6"} err="failed to get container status \"970e13fd6cc879b95305f3781a424ee4988dabd24cac2e4246e39261dfa20aa6\": rpc error: code = NotFound desc = could not find container \"970e13fd6cc879b95305f3781a424ee4988dabd24cac2e4246e39261dfa20aa6\": container with ID starting with 970e13fd6cc879b95305f3781a424ee4988dabd24cac2e4246e39261dfa20aa6 not found: ID does not exist"
Sep 14 05:21:23.216786 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:23.216745    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7bbea9-6c44bcf854-8mbns"]
Sep 14 05:21:23.224798 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:23.224779    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7bbea9-6c44bcf854-8mbns"]
Sep 14 05:21:24.334377 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:24.334346    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5edad9b0-9829-4009-ab8a-8fed1c7ff844" path="/var/lib/kubelet/pods/5edad9b0-9829-4009-ab8a-8fed1c7ff844/volumes"
Sep 14 05:21:25.731521 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:25.731484    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:21:29.128806 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:29.128772    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"]
Sep 14 05:21:29.129258 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:29.129004    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5" podUID="f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9" containerName="istio-proxy" containerID="cri-o://fcf23f7fbf413ea3e03b883955dfe89a59a9c7de3974cf8f20b7652d5ffb796d" gracePeriod=30
Sep 14 05:21:34.256717 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:34.256686    2565 generic.go:356] "Generic (PLEG): container finished" podID="f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9" containerID="fcf23f7fbf413ea3e03b883955dfe89a59a9c7de3974cf8f20b7652d5ffb796d" exitCode=0
Sep 14 05:21:34.257080 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:34.256734    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5" event={"ID":"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9","Type":"ContainerDied","Data":"fcf23f7fbf413ea3e03b883955dfe89a59a9c7de3974cf8f20b7652d5ffb796d"}
Sep 14 05:21:34.257080 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:34.256802    2565 scope.go:117] "RemoveContainer" containerID="db2e063f1d35024837eae0bc80ba877219cf663db5d0c3c90068948f2f747a88"
Sep 14 05:21:34.379750 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:34.379725    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"
Sep 14 05:21:34.508218 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:34.508129    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-istio-podinfo\") pod \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\" (UID: \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\") "
Sep 14 05:21:34.508218 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:34.508190    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-workload-certs\") pod \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\" (UID: \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\") "
Sep 14 05:21:34.508218 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:34.508224    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khjrx\" (UniqueName: \"kubernetes.io/projected/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-kube-api-access-khjrx\") pod \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\" (UID: \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\") "
Sep 14 05:21:34.508507 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:34.508250    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-istio-envoy\") pod \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\" (UID: \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\") "
Sep 14 05:21:34.508507 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:34.508275    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-credential-socket\") pod \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\" (UID: \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\") "
Sep 14 05:21:34.508507 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:34.508294    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-workload-socket\") pod \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\" (UID: \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\") "
Sep 14 05:21:34.508507 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:34.508312    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-istiod-ca-cert\") pod \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\" (UID: \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\") "
Sep 14 05:21:34.508507 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:34.508380    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-istio-token\") pod \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\" (UID: \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\") "
Sep 14 05:21:34.508507 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:34.508420    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-istio-data\") pod \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\" (UID: \"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9\") "
Sep 14 05:21:34.508855 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:34.508575    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9" (UID: "f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:21:34.508855 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:34.508591    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9" (UID: "f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:21:34.508855 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:34.508796    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-credential-socket\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:21:34.508855 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:34.508816    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-workload-certs\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:21:34.508855 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:34.508832    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-istio-data" (OuterVolumeSpecName: "istio-data") pod "f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9" (UID: "f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:21:34.508855 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:34.508846    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9" (UID: "f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:21:34.509182 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:34.508975    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9" (UID: "f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 05:21:34.510560 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:34.510537    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-istio-token" (OuterVolumeSpecName: "istio-token") pod "f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9" (UID: "f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:21:34.510660 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:34.510641    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-kube-api-access-khjrx" (OuterVolumeSpecName: "kube-api-access-khjrx") pod "f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9" (UID: "f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9"). InnerVolumeSpecName "kube-api-access-khjrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:21:34.510864 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:34.510845    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9" (UID: "f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 05:21:34.510864 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:34.510851    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9" (UID: "f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:21:34.609443 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:34.609409    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khjrx\" (UniqueName: \"kubernetes.io/projected/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-kube-api-access-khjrx\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:21:34.609443 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:34.609443    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-istio-envoy\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:21:34.609443 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:34.609454    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-workload-socket\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:21:34.609622 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:34.609464    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-istiod-ca-cert\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:21:34.609622 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:34.609473    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-istio-token\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:21:34.609622 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:34.609481    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-istio-data\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:21:34.609622 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:34.609488    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9-istio-podinfo\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:21:35.262887 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:35.262850    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5" event={"ID":"f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9","Type":"ContainerDied","Data":"f7584a55069fa9c9fd4b01bb22ed17e605ab25d158978608ae3826558fe379c8"}
Sep 14 05:21:35.262887 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:35.262881    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"
Sep 14 05:21:35.263370 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:35.262907    2565 scope.go:117] "RemoveContainer" containerID="fcf23f7fbf413ea3e03b883955dfe89a59a9c7de3974cf8f20b7652d5ffb796d"
Sep 14 05:21:35.285257 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:35.285231    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"]
Sep 14 05:21:35.292716 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:35.292683    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-7bbea9-openshift-default-56cd4b86d4-l28j5"]
Sep 14 05:21:36.333685 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:36.333646    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9" path="/var/lib/kubelet/pods/f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9/volumes"
Sep 14 05:21:42.465313 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:42.465284    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kndxq_e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae/ovn-acl-logging/0.log"
Sep 14 05:21:42.473403 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:42.473384    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kndxq_e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae/ovn-acl-logging/0.log"
Sep 14 05:21:42.785191 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:42.785096    2565 scope.go:117] "RemoveContainer" containerID="715da2ea1683fd50c7b655b0ddcd5763a995e1ddc65f2ad5022bd5cf07fc3e78"
Sep 14 05:21:42.793503 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:42.793480    2565 scope.go:117] "RemoveContainer" containerID="7dcedd3c00577d42c6f997a25bd60c4e00b21825b078736e399372c0ca7038f4"
Sep 14 05:21:42.800968 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:42.800951    2565 scope.go:117] "RemoveContainer" containerID="8503981043f042244e8116cb8a6fd4f7e6a42ea58db774b1518e8a13d0693243"
Sep 14 05:21:43.332225 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:43.332193    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:21:47.173713 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:47.173677    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"]
Sep 14 05:21:47.174193 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:47.174031    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5" podUID="6a5a2e92-8a45-4f45-a89a-afd3d81f0699" containerName="istio-proxy" containerID="cri-o://732d8ac1fb86b3447c8eeb98092f735d22bb94e0a684f731b5d9df085ad103a3" gracePeriod=30
Sep 14 05:21:52.351386 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:52.351339    2565 generic.go:356] "Generic (PLEG): container finished" podID="6a5a2e92-8a45-4f45-a89a-afd3d81f0699" containerID="732d8ac1fb86b3447c8eeb98092f735d22bb94e0a684f731b5d9df085ad103a3" exitCode=0
Sep 14 05:21:52.351813 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:52.351517    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5" event={"ID":"6a5a2e92-8a45-4f45-a89a-afd3d81f0699","Type":"ContainerDied","Data":"732d8ac1fb86b3447c8eeb98092f735d22bb94e0a684f731b5d9df085ad103a3"}
Sep 14 05:21:52.351813 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:52.351724    2565 scope.go:117] "RemoveContainer" containerID="69fa9bfb9ec720356935fcaae89bf89fbd08dff90b0ba33ab7aa670e25fe898d"
Sep 14 05:21:52.413739 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:52.413717    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"
Sep 14 05:21:52.457170 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:52.457115    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-credential-socket\") pod \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\" (UID: \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\") "
Sep 14 05:21:52.457170 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:52.457146    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-istio-envoy\") pod \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\" (UID: \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\") "
Sep 14 05:21:52.457170 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:52.457163    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-istio-token\") pod \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\" (UID: \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\") "
Sep 14 05:21:52.457412 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:52.457189    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwkk7\" (UniqueName: \"kubernetes.io/projected/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-kube-api-access-qwkk7\") pod \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\" (UID: \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\") "
Sep 14 05:21:52.457412 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:52.457206    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-istio-data\") pod \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\" (UID: \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\") "
Sep 14 05:21:52.457412 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:52.457234    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-workload-socket\") pod \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\" (UID: \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\") "
Sep 14 05:21:52.457412 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:52.457279    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-istio-podinfo\") pod \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\" (UID: \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\") "
Sep 14 05:21:52.457412 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:52.457305    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-workload-certs\") pod \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\" (UID: \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\") "
Sep 14 05:21:52.457412 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:52.457338    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-istiod-ca-cert\") pod \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\" (UID: \"6a5a2e92-8a45-4f45-a89a-afd3d81f0699\") "
Sep 14 05:21:52.457412 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:52.457398    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6a5a2e92-8a45-4f45-a89a-afd3d81f0699" (UID: "6a5a2e92-8a45-4f45-a89a-afd3d81f0699"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:21:52.457834 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:52.457555    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-istio-data" (OuterVolumeSpecName: "istio-data") pod "6a5a2e92-8a45-4f45-a89a-afd3d81f0699" (UID: "6a5a2e92-8a45-4f45-a89a-afd3d81f0699"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:21:52.457834 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:52.457602    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-credential-socket\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:21:52.457834 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:52.457612    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6a5a2e92-8a45-4f45-a89a-afd3d81f0699" (UID: "6a5a2e92-8a45-4f45-a89a-afd3d81f0699"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:21:52.457834 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:52.457732    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6a5a2e92-8a45-4f45-a89a-afd3d81f0699" (UID: "6a5a2e92-8a45-4f45-a89a-afd3d81f0699"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:21:52.458057 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:52.457933    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6a5a2e92-8a45-4f45-a89a-afd3d81f0699" (UID: "6a5a2e92-8a45-4f45-a89a-afd3d81f0699"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 05:21:52.459651 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:52.459626    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6a5a2e92-8a45-4f45-a89a-afd3d81f0699" (UID: "6a5a2e92-8a45-4f45-a89a-afd3d81f0699"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:21:52.459740 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:52.459688    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-istio-token" (OuterVolumeSpecName: "istio-token") pod "6a5a2e92-8a45-4f45-a89a-afd3d81f0699" (UID: "6a5a2e92-8a45-4f45-a89a-afd3d81f0699"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:21:52.459882 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:52.459856    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-kube-api-access-qwkk7" (OuterVolumeSpecName: "kube-api-access-qwkk7") pod "6a5a2e92-8a45-4f45-a89a-afd3d81f0699" (UID: "6a5a2e92-8a45-4f45-a89a-afd3d81f0699"). InnerVolumeSpecName "kube-api-access-qwkk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:21:52.459973 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:52.459879    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6a5a2e92-8a45-4f45-a89a-afd3d81f0699" (UID: "6a5a2e92-8a45-4f45-a89a-afd3d81f0699"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 05:21:52.558322 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:52.558289    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-istio-envoy\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:21:52.558322 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:52.558317    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-istio-token\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:21:52.558322 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:52.558326    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwkk7\" (UniqueName: \"kubernetes.io/projected/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-kube-api-access-qwkk7\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:21:52.558548 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:52.558336    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-istio-data\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:21:52.558548 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:52.558346    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-workload-socket\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:21:52.558548 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:52.558355    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-istio-podinfo\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:21:52.558548 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:52.558363    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-workload-certs\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:21:52.558548 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:52.558370    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a5a2e92-8a45-4f45-a89a-afd3d81f0699-istiod-ca-cert\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:21:53.357978 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:53.357950    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"
Sep 14 05:21:53.358444 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:53.357993    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5" event={"ID":"6a5a2e92-8a45-4f45-a89a-afd3d81f0699","Type":"ContainerDied","Data":"e9acbefbf5130751fffd73e254777ac364b12f65facb329597cdbfd276bd6a74"}
Sep 14 05:21:53.358444 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:53.358058    2565 scope.go:117] "RemoveContainer" containerID="732d8ac1fb86b3447c8eeb98092f735d22bb94e0a684f731b5d9df085ad103a3"
Sep 14 05:21:53.380332 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:53.380301    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"]
Sep 14 05:21:53.385925 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:53.385900    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2ff4a6-openshift-default-6cdbbdbf5d-glrj5"]
Sep 14 05:21:54.333881 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:54.333846    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a5a2e92-8a45-4f45-a89a-afd3d81f0699" path="/var/lib/kubelet/pods/6a5a2e92-8a45-4f45-a89a-afd3d81f0699/volumes"
Sep 14 05:21:58.108549 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:58.108510    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-27be50-29822715-2hm9r"]
Sep 14 05:21:58.110770 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:58.110733    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-27be50-29822715-2hm9r"]
Sep 14 05:21:58.334163 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:21:58.334128    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2244a16f-ad71-4fd5-a783-3bf80befdde4" path="/var/lib/kubelet/pods/2244a16f-ad71-4fd5-a783-3bf80befdde4/volumes"
Sep 14 05:22:00.929800 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:00.929749    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:22:05.111183 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:05.111148    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"]
Sep 14 05:22:05.111600 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:05.111386    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4" podUID="09dff87c-b6f1-45da-8d31-c6a906b19a6a" containerName="istio-proxy" containerID="cri-o://2872ff980962ce39aa52d51b989f7b8f36ee80fce9ea4ec05d77c2dd085eba66" gracePeriod=30
Sep 14 05:22:10.355164 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:10.355136    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"
Sep 14 05:22:10.405890 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:10.405825    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09dff87c-b6f1-45da-8d31-c6a906b19a6a-credential-socket\") pod \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\" (UID: \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\") "
Sep 14 05:22:10.405890 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:10.405856    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09dff87c-b6f1-45da-8d31-c6a906b19a6a-istio-envoy\") pod \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\" (UID: \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\") "
Sep 14 05:22:10.405890 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:10.405881    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09dff87c-b6f1-45da-8d31-c6a906b19a6a-workload-certs\") pod \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\" (UID: \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\") "
Sep 14 05:22:10.406098 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:10.405907    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09dff87c-b6f1-45da-8d31-c6a906b19a6a-workload-socket\") pod \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\" (UID: \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\") "
Sep 14 05:22:10.406098 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:10.405932    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09dff87c-b6f1-45da-8d31-c6a906b19a6a-istio-data\") pod \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\" (UID: \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\") "
Sep 14 05:22:10.406098 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:10.405979    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09dff87c-b6f1-45da-8d31-c6a906b19a6a-istiod-ca-cert\") pod \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\" (UID: \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\") "
Sep 14 05:22:10.406098 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:10.405997    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09dff87c-b6f1-45da-8d31-c6a906b19a6a-istio-token\") pod \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\" (UID: \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\") "
Sep 14 05:22:10.406098 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:10.406029    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09dff87c-b6f1-45da-8d31-c6a906b19a6a-istio-podinfo\") pod \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\" (UID: \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\") "
Sep 14 05:22:10.406098 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:10.406067    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xmpj\" (UniqueName: \"kubernetes.io/projected/09dff87c-b6f1-45da-8d31-c6a906b19a6a-kube-api-access-6xmpj\") pod \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\" (UID: \"09dff87c-b6f1-45da-8d31-c6a906b19a6a\") "
Sep 14 05:22:10.406409 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:10.406138    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09dff87c-b6f1-45da-8d31-c6a906b19a6a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "09dff87c-b6f1-45da-8d31-c6a906b19a6a" (UID: "09dff87c-b6f1-45da-8d31-c6a906b19a6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:22:10.406409 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:10.406175    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09dff87c-b6f1-45da-8d31-c6a906b19a6a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "09dff87c-b6f1-45da-8d31-c6a906b19a6a" (UID: "09dff87c-b6f1-45da-8d31-c6a906b19a6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:22:10.406409 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:10.406251    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09dff87c-b6f1-45da-8d31-c6a906b19a6a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "09dff87c-b6f1-45da-8d31-c6a906b19a6a" (UID: "09dff87c-b6f1-45da-8d31-c6a906b19a6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:22:10.406409 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:10.406292    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09dff87c-b6f1-45da-8d31-c6a906b19a6a-istio-data" (OuterVolumeSpecName: "istio-data") pod "09dff87c-b6f1-45da-8d31-c6a906b19a6a" (UID: "09dff87c-b6f1-45da-8d31-c6a906b19a6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:22:10.406409 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:10.406365    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09dff87c-b6f1-45da-8d31-c6a906b19a6a-workload-socket\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:22:10.406409 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:10.406383    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09dff87c-b6f1-45da-8d31-c6a906b19a6a-istio-data\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:22:10.406409 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:10.406393    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09dff87c-b6f1-45da-8d31-c6a906b19a6a-credential-socket\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:22:10.406409 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:10.406408    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09dff87c-b6f1-45da-8d31-c6a906b19a6a-workload-certs\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:22:10.406780 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:10.406676    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09dff87c-b6f1-45da-8d31-c6a906b19a6a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "09dff87c-b6f1-45da-8d31-c6a906b19a6a" (UID: "09dff87c-b6f1-45da-8d31-c6a906b19a6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 05:22:10.408014 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:10.407994    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09dff87c-b6f1-45da-8d31-c6a906b19a6a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "09dff87c-b6f1-45da-8d31-c6a906b19a6a" (UID: "09dff87c-b6f1-45da-8d31-c6a906b19a6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:22:10.408284 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:10.408267    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09dff87c-b6f1-45da-8d31-c6a906b19a6a-kube-api-access-6xmpj" (OuterVolumeSpecName: "kube-api-access-6xmpj") pod "09dff87c-b6f1-45da-8d31-c6a906b19a6a" (UID: "09dff87c-b6f1-45da-8d31-c6a906b19a6a"). InnerVolumeSpecName "kube-api-access-6xmpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:22:10.408553 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:10.408531    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09dff87c-b6f1-45da-8d31-c6a906b19a6a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "09dff87c-b6f1-45da-8d31-c6a906b19a6a" (UID: "09dff87c-b6f1-45da-8d31-c6a906b19a6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 05:22:10.408684 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:10.408547    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09dff87c-b6f1-45da-8d31-c6a906b19a6a-istio-token" (OuterVolumeSpecName: "istio-token") pod "09dff87c-b6f1-45da-8d31-c6a906b19a6a" (UID: "09dff87c-b6f1-45da-8d31-c6a906b19a6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:22:10.442828 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:10.442799    2565 generic.go:356] "Generic (PLEG): container finished" podID="09dff87c-b6f1-45da-8d31-c6a906b19a6a" containerID="2872ff980962ce39aa52d51b989f7b8f36ee80fce9ea4ec05d77c2dd085eba66" exitCode=0
Sep 14 05:22:10.442932 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:10.442843    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4" event={"ID":"09dff87c-b6f1-45da-8d31-c6a906b19a6a","Type":"ContainerDied","Data":"2872ff980962ce39aa52d51b989f7b8f36ee80fce9ea4ec05d77c2dd085eba66"}
Sep 14 05:22:10.442932 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:10.442885    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4" event={"ID":"09dff87c-b6f1-45da-8d31-c6a906b19a6a","Type":"ContainerDied","Data":"322d64e5efcdd6f23b5c7579d63841d98911c5949ce023730bee1bf46c27237a"}
Sep 14 05:22:10.442932 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:10.442908    2565 scope.go:117] "RemoveContainer" containerID="2872ff980962ce39aa52d51b989f7b8f36ee80fce9ea4ec05d77c2dd085eba66"
Sep 14 05:22:10.442932 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:10.442847    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"
Sep 14 05:22:10.451978 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:10.451961    2565 scope.go:117] "RemoveContainer" containerID="f31d9520cd40097950f3a6a0278b85a54a05fb845fa9679cf8aed98b4b6380c1"
Sep 14 05:22:10.462104 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:10.462084    2565 scope.go:117] "RemoveContainer" containerID="2872ff980962ce39aa52d51b989f7b8f36ee80fce9ea4ec05d77c2dd085eba66"
Sep 14 05:22:10.462696 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:22:10.462671    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2872ff980962ce39aa52d51b989f7b8f36ee80fce9ea4ec05d77c2dd085eba66\": container with ID starting with 2872ff980962ce39aa52d51b989f7b8f36ee80fce9ea4ec05d77c2dd085eba66 not found: ID does not exist" containerID="2872ff980962ce39aa52d51b989f7b8f36ee80fce9ea4ec05d77c2dd085eba66"
Sep 14 05:22:10.462819 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:10.462707    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2872ff980962ce39aa52d51b989f7b8f36ee80fce9ea4ec05d77c2dd085eba66"} err="failed to get container status \"2872ff980962ce39aa52d51b989f7b8f36ee80fce9ea4ec05d77c2dd085eba66\": rpc error: code = NotFound desc = could not find container \"2872ff980962ce39aa52d51b989f7b8f36ee80fce9ea4ec05d77c2dd085eba66\": container with ID starting with 2872ff980962ce39aa52d51b989f7b8f36ee80fce9ea4ec05d77c2dd085eba66 not found: ID does not exist"
Sep 14 05:22:10.462819 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:10.462730    2565 scope.go:117] "RemoveContainer" containerID="f31d9520cd40097950f3a6a0278b85a54a05fb845fa9679cf8aed98b4b6380c1"
Sep 14 05:22:10.463101 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:22:10.463078    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f31d9520cd40097950f3a6a0278b85a54a05fb845fa9679cf8aed98b4b6380c1\": container with ID starting with f31d9520cd40097950f3a6a0278b85a54a05fb845fa9679cf8aed98b4b6380c1 not found: ID does not exist" containerID="f31d9520cd40097950f3a6a0278b85a54a05fb845fa9679cf8aed98b4b6380c1"
Sep 14 05:22:10.463170 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:10.463108    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f31d9520cd40097950f3a6a0278b85a54a05fb845fa9679cf8aed98b4b6380c1"} err="failed to get container status \"f31d9520cd40097950f3a6a0278b85a54a05fb845fa9679cf8aed98b4b6380c1\": rpc error: code = NotFound desc = could not find container \"f31d9520cd40097950f3a6a0278b85a54a05fb845fa9679cf8aed98b4b6380c1\": container with ID starting with f31d9520cd40097950f3a6a0278b85a54a05fb845fa9679cf8aed98b4b6380c1 not found: ID does not exist"
Sep 14 05:22:10.464929 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:10.464910    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"]
Sep 14 05:22:10.470031 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:10.470009    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-27be50-openshift-default-6d48cf8d84-sklh4"]
Sep 14 05:22:10.507481 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:10.507460    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09dff87c-b6f1-45da-8d31-c6a906b19a6a-istiod-ca-cert\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:22:10.507481 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:10.507478    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09dff87c-b6f1-45da-8d31-c6a906b19a6a-istio-token\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:22:10.507605 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:10.507487    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09dff87c-b6f1-45da-8d31-c6a906b19a6a-istio-podinfo\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:22:10.507605 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:10.507496    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xmpj\" (UniqueName: \"kubernetes.io/projected/09dff87c-b6f1-45da-8d31-c6a906b19a6a-kube-api-access-6xmpj\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:22:10.507605 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:10.507506    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09dff87c-b6f1-45da-8d31-c6a906b19a6a-istio-envoy\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:22:11.023691 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:11.023653    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:22:12.334834 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:12.334802    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09dff87c-b6f1-45da-8d31-c6a906b19a6a" path="/var/lib/kubelet/pods/09dff87c-b6f1-45da-8d31-c6a906b19a6a/volumes"
Sep 14 05:22:42.835226 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:22:42.835148    2565 scope.go:117] "RemoveContainer" containerID="8600ebb76fa1b7ccfc4e003e8f10a923a38c857bd1ac5f9d7323e501da449efa"
Sep 14 05:23:33.941512 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:23:33.941468    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:23:58.930586 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:23:58.930547    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:24:08.426862 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:24:08.426765    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:24:14.730323 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:24:14.730286    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:24:22.726246 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:24:22.726211    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:24:37.632615 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:24:37.632580    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:24:43.830781 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:24:43.830731    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:26:16.129645 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:16.129607    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:26:21.934767 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:21.934720    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:26:33.829522 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:33.829492    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j"]
Sep 14 05:26:33.829925 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:33.829846    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a5a2e92-8a45-4f45-a89a-afd3d81f0699" containerName="istio-proxy"
Sep 14 05:26:33.829925 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:33.829862    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a5a2e92-8a45-4f45-a89a-afd3d81f0699" containerName="istio-proxy"
Sep 14 05:26:33.829925 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:33.829871    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1587987-eb18-4054-9792-6be94c731efd" containerName="revoke-keys"
Sep 14 05:26:33.829925 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:33.829877    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1587987-eb18-4054-9792-6be94c731efd" containerName="revoke-keys"
Sep 14 05:26:33.829925 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:33.829883    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a5a2e92-8a45-4f45-a89a-afd3d81f0699" containerName="istio-proxy"
Sep 14 05:26:33.829925 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:33.829889    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a5a2e92-8a45-4f45-a89a-afd3d81f0699" containerName="istio-proxy"
Sep 14 05:26:33.829925 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:33.829895    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09dff87c-b6f1-45da-8d31-c6a906b19a6a" containerName="istio-proxy"
Sep 14 05:26:33.829925 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:33.829901    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="09dff87c-b6f1-45da-8d31-c6a906b19a6a" containerName="istio-proxy"
Sep 14 05:26:33.829925 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:33.829906    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5edad9b0-9829-4009-ab8a-8fed1c7ff844" containerName="payload-pre-processing"
Sep 14 05:26:33.829925 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:33.829911    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5edad9b0-9829-4009-ab8a-8fed1c7ff844" containerName="payload-pre-processing"
Sep 14 05:26:33.829925 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:33.829920    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09dff87c-b6f1-45da-8d31-c6a906b19a6a" containerName="istio-proxy"
Sep 14 05:26:33.829925 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:33.829925    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="09dff87c-b6f1-45da-8d31-c6a906b19a6a" containerName="istio-proxy"
Sep 14 05:26:33.829925 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:33.829931    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9" containerName="istio-proxy"
Sep 14 05:26:33.830362 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:33.829936    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9" containerName="istio-proxy"
Sep 14 05:26:33.830362 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:33.829944    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9" containerName="istio-proxy"
Sep 14 05:26:33.830362 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:33.829949    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9" containerName="istio-proxy"
Sep 14 05:26:33.830362 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:33.830009    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a5a2e92-8a45-4f45-a89a-afd3d81f0699" containerName="istio-proxy"
Sep 14 05:26:33.830362 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:33.830016    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a5a2e92-8a45-4f45-a89a-afd3d81f0699" containerName="istio-proxy"
Sep 14 05:26:33.830362 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:33.830022    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9" containerName="istio-proxy"
Sep 14 05:26:33.830362 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:33.830028    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5edad9b0-9829-4009-ab8a-8fed1c7ff844" containerName="payload-pre-processing"
Sep 14 05:26:33.830362 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:33.830033    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1587987-eb18-4054-9792-6be94c731efd" containerName="revoke-keys"
Sep 14 05:26:33.830362 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:33.830038    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0b92efa-2cbb-4a99-87ea-b8b0d0e9eda9" containerName="istio-proxy"
Sep 14 05:26:33.830362 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:33.830045    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="09dff87c-b6f1-45da-8d31-c6a906b19a6a" containerName="istio-proxy"
Sep 14 05:26:33.830362 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:33.830049    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="09dff87c-b6f1-45da-8d31-c6a906b19a6a" containerName="istio-proxy"
Sep 14 05:26:33.833004 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:33.832985    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j"
Sep 14 05:26:33.835380 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:33.835356    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-e8fb9296-kserve-self-signed-certs\""
Sep 14 05:26:33.843673 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:33.843646    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j"]
Sep 14 05:26:33.918015 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:33.917965    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82a0ab60-542e-42fa-9142-0665cd63073f-model-cache\") pod \"e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j\" (UID: \"82a0ab60-542e-42fa-9142-0665cd63073f\") " pod="llm/e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j"
Sep 14 05:26:33.918015 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:33.918021    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9xdr\" (UniqueName: \"kubernetes.io/projected/82a0ab60-542e-42fa-9142-0665cd63073f-kube-api-access-s9xdr\") pod \"e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j\" (UID: \"82a0ab60-542e-42fa-9142-0665cd63073f\") " pod="llm/e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j"
Sep 14 05:26:33.918256 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:33.918082    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82a0ab60-542e-42fa-9142-0665cd63073f-tmp-dir\") pod \"e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j\" (UID: \"82a0ab60-542e-42fa-9142-0665cd63073f\") " pod="llm/e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j"
Sep 14 05:26:33.918256 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:33.918103    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82a0ab60-542e-42fa-9142-0665cd63073f-home\") pod \"e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j\" (UID: \"82a0ab60-542e-42fa-9142-0665cd63073f\") " pod="llm/e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j"
Sep 14 05:26:33.918256 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:33.918142    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82a0ab60-542e-42fa-9142-0665cd63073f-dshm\") pod \"e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j\" (UID: \"82a0ab60-542e-42fa-9142-0665cd63073f\") " pod="llm/e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j"
Sep 14 05:26:33.918256 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:33.918243    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82a0ab60-542e-42fa-9142-0665cd63073f-tls-certs\") pod \"e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j\" (UID: \"82a0ab60-542e-42fa-9142-0665cd63073f\") " pod="llm/e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j"
Sep 14 05:26:34.018695 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:34.018648    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82a0ab60-542e-42fa-9142-0665cd63073f-tls-certs\") pod \"e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j\" (UID: \"82a0ab60-542e-42fa-9142-0665cd63073f\") " pod="llm/e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j"
Sep 14 05:26:34.018695 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:34.018699    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82a0ab60-542e-42fa-9142-0665cd63073f-model-cache\") pod \"e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j\" (UID: \"82a0ab60-542e-42fa-9142-0665cd63073f\") " pod="llm/e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j"
Sep 14 05:26:34.018975 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:34.018719    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9xdr\" (UniqueName: \"kubernetes.io/projected/82a0ab60-542e-42fa-9142-0665cd63073f-kube-api-access-s9xdr\") pod \"e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j\" (UID: \"82a0ab60-542e-42fa-9142-0665cd63073f\") " pod="llm/e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j"
Sep 14 05:26:34.018975 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:34.018845    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82a0ab60-542e-42fa-9142-0665cd63073f-tmp-dir\") pod \"e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j\" (UID: \"82a0ab60-542e-42fa-9142-0665cd63073f\") " pod="llm/e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j"
Sep 14 05:26:34.018975 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:34.018885    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82a0ab60-542e-42fa-9142-0665cd63073f-home\") pod \"e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j\" (UID: \"82a0ab60-542e-42fa-9142-0665cd63073f\") " pod="llm/e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j"
Sep 14 05:26:34.018975 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:34.018931    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82a0ab60-542e-42fa-9142-0665cd63073f-dshm\") pod \"e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j\" (UID: \"82a0ab60-542e-42fa-9142-0665cd63073f\") " pod="llm/e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j"
Sep 14 05:26:34.019193 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:34.019144    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82a0ab60-542e-42fa-9142-0665cd63073f-model-cache\") pod \"e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j\" (UID: \"82a0ab60-542e-42fa-9142-0665cd63073f\") " pod="llm/e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j"
Sep 14 05:26:34.019193 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:34.019172    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82a0ab60-542e-42fa-9142-0665cd63073f-tmp-dir\") pod \"e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j\" (UID: \"82a0ab60-542e-42fa-9142-0665cd63073f\") " pod="llm/e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j"
Sep 14 05:26:34.019306 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:34.019288    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82a0ab60-542e-42fa-9142-0665cd63073f-home\") pod \"e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j\" (UID: \"82a0ab60-542e-42fa-9142-0665cd63073f\") " pod="llm/e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j"
Sep 14 05:26:34.021094 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:34.021069    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82a0ab60-542e-42fa-9142-0665cd63073f-dshm\") pod \"e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j\" (UID: \"82a0ab60-542e-42fa-9142-0665cd63073f\") " pod="llm/e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j"
Sep 14 05:26:34.021383 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:34.021364    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82a0ab60-542e-42fa-9142-0665cd63073f-tls-certs\") pod \"e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j\" (UID: \"82a0ab60-542e-42fa-9142-0665cd63073f\") " pod="llm/e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j"
Sep 14 05:26:34.026162 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:34.026136    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9xdr\" (UniqueName: \"kubernetes.io/projected/82a0ab60-542e-42fa-9142-0665cd63073f-kube-api-access-s9xdr\") pod \"e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j\" (UID: \"82a0ab60-542e-42fa-9142-0665cd63073f\") " pod="llm/e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j"
Sep 14 05:26:34.142419 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:34.142344    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j"
Sep 14 05:26:34.411728 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:34.411700    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j"]
Sep 14 05:26:34.413092 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:26:34.413065    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82a0ab60_542e_42fa_9142_0665cd63073f.slice/crio-378ff620edf07f8bec46e01d12a3af8504c0a157eb559211f3a697fda9524def WatchSource:0}: Error finding container 378ff620edf07f8bec46e01d12a3af8504c0a157eb559211f3a697fda9524def: Status 404 returned error can't find the container with id 378ff620edf07f8bec46e01d12a3af8504c0a157eb559211f3a697fda9524def
Sep 14 05:26:34.724361 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:34.724263    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j" event={"ID":"82a0ab60-542e-42fa-9142-0665cd63073f","Type":"ContainerStarted","Data":"e1b5dc5d3186f91ed0704859d32bb0da55e85388d4070468e07b21ce2f9a3e4d"}
Sep 14 05:26:34.724361 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:34.724311    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j" event={"ID":"82a0ab60-542e-42fa-9142-0665cd63073f","Type":"ContainerStarted","Data":"378ff620edf07f8bec46e01d12a3af8504c0a157eb559211f3a697fda9524def"}
Sep 14 05:26:34.744583 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:34.744535    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j" podStartSLOduration=1.744520214 podStartE2EDuration="1.744520214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:26:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:26:34.74307754 +0000 UTC m=+2392.918849100" watchObservedRunningTime="2026-09-14 05:26:34.744520214 +0000 UTC m=+2392.920291770"
Sep 14 05:26:36.727007 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:36.726971    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:26:42.499713 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:42.499687    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kndxq_e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae/ovn-acl-logging/0.log"
Sep 14 05:26:42.508956 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:42.508935    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kndxq_e46993d2-9dee-49c0-aa4f-57ecc2f3f9ae/ovn-acl-logging/0.log"
Sep 14 05:26:42.882533 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:42.882505    2565 scope.go:117] "RemoveContainer" containerID="d0a88e7d8eaf180fa07c8c517885fdb152a0f15294531bf13ad4cdd9db3f81b0"
Sep 14 05:26:44.142912 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:44.142876    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j"
Sep 14 05:26:44.143345 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:44.142925    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j"
Sep 14 05:26:44.155521 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:44.155495    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j"
Sep 14 05:26:44.787664 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:44.787636    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j"
Sep 14 05:26:53.624429 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:53.624391    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w"]
Sep 14 05:26:53.627859 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:53.627838    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w"
Sep 14 05:26:53.630349 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:53.630326    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-e8fb9296-kserve-self-signed-certs\""
Sep 14 05:26:53.637441 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:53.637418    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w"]
Sep 14 05:26:53.786092 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:53.786055    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b0282c7-c6a4-4a03-942a-9904bf6be4f3-tmp-dir\") pod \"e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w\" (UID: \"6b0282c7-c6a4-4a03-942a-9904bf6be4f3\") " pod="llm/e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w"
Sep 14 05:26:53.786302 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:53.786108    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b0282c7-c6a4-4a03-942a-9904bf6be4f3-dshm\") pod \"e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w\" (UID: \"6b0282c7-c6a4-4a03-942a-9904bf6be4f3\") " pod="llm/e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w"
Sep 14 05:26:53.786302 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:53.786123    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b0282c7-c6a4-4a03-942a-9904bf6be4f3-tls-certs\") pod \"e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w\" (UID: \"6b0282c7-c6a4-4a03-942a-9904bf6be4f3\") " pod="llm/e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w"
Sep 14 05:26:53.786302 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:53.786205    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b0282c7-c6a4-4a03-942a-9904bf6be4f3-model-cache\") pod \"e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w\" (UID: \"6b0282c7-c6a4-4a03-942a-9904bf6be4f3\") " pod="llm/e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w"
Sep 14 05:26:53.786302 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:53.786237    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khf4t\" (UniqueName: \"kubernetes.io/projected/6b0282c7-c6a4-4a03-942a-9904bf6be4f3-kube-api-access-khf4t\") pod \"e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w\" (UID: \"6b0282c7-c6a4-4a03-942a-9904bf6be4f3\") " pod="llm/e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w"
Sep 14 05:26:53.786302 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:53.786298    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b0282c7-c6a4-4a03-942a-9904bf6be4f3-home\") pod \"e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w\" (UID: \"6b0282c7-c6a4-4a03-942a-9904bf6be4f3\") " pod="llm/e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w"
Sep 14 05:26:53.887069 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:53.886970    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b0282c7-c6a4-4a03-942a-9904bf6be4f3-tmp-dir\") pod \"e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w\" (UID: \"6b0282c7-c6a4-4a03-942a-9904bf6be4f3\") " pod="llm/e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w"
Sep 14 05:26:53.887069 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:53.887031    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b0282c7-c6a4-4a03-942a-9904bf6be4f3-dshm\") pod \"e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w\" (UID: \"6b0282c7-c6a4-4a03-942a-9904bf6be4f3\") " pod="llm/e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w"
Sep 14 05:26:53.887069 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:53.887056    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b0282c7-c6a4-4a03-942a-9904bf6be4f3-tls-certs\") pod \"e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w\" (UID: \"6b0282c7-c6a4-4a03-942a-9904bf6be4f3\") " pod="llm/e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w"
Sep 14 05:26:53.887379 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:53.887090    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b0282c7-c6a4-4a03-942a-9904bf6be4f3-model-cache\") pod \"e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w\" (UID: \"6b0282c7-c6a4-4a03-942a-9904bf6be4f3\") " pod="llm/e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w"
Sep 14 05:26:53.887379 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:53.887113    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khf4t\" (UniqueName: \"kubernetes.io/projected/6b0282c7-c6a4-4a03-942a-9904bf6be4f3-kube-api-access-khf4t\") pod \"e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w\" (UID: \"6b0282c7-c6a4-4a03-942a-9904bf6be4f3\") " pod="llm/e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w"
Sep 14 05:26:53.887379 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:53.887150    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b0282c7-c6a4-4a03-942a-9904bf6be4f3-home\") pod \"e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w\" (UID: \"6b0282c7-c6a4-4a03-942a-9904bf6be4f3\") " pod="llm/e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w"
Sep 14 05:26:53.887608 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:53.887496    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b0282c7-c6a4-4a03-942a-9904bf6be4f3-tmp-dir\") pod \"e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w\" (UID: \"6b0282c7-c6a4-4a03-942a-9904bf6be4f3\") " pod="llm/e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w"
Sep 14 05:26:53.887608 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:53.887545    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b0282c7-c6a4-4a03-942a-9904bf6be4f3-model-cache\") pod \"e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w\" (UID: \"6b0282c7-c6a4-4a03-942a-9904bf6be4f3\") " pod="llm/e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w"
Sep 14 05:26:53.887608 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:53.887561    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b0282c7-c6a4-4a03-942a-9904bf6be4f3-home\") pod \"e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w\" (UID: \"6b0282c7-c6a4-4a03-942a-9904bf6be4f3\") " pod="llm/e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w"
Sep 14 05:26:53.889385 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:53.889362    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b0282c7-c6a4-4a03-942a-9904bf6be4f3-dshm\") pod \"e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w\" (UID: \"6b0282c7-c6a4-4a03-942a-9904bf6be4f3\") " pod="llm/e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w"
Sep 14 05:26:53.889781 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:53.889750    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b0282c7-c6a4-4a03-942a-9904bf6be4f3-tls-certs\") pod \"e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w\" (UID: \"6b0282c7-c6a4-4a03-942a-9904bf6be4f3\") " pod="llm/e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w"
Sep 14 05:26:53.894955 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:53.894937    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khf4t\" (UniqueName: \"kubernetes.io/projected/6b0282c7-c6a4-4a03-942a-9904bf6be4f3-kube-api-access-khf4t\") pod \"e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w\" (UID: \"6b0282c7-c6a4-4a03-942a-9904bf6be4f3\") " pod="llm/e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w"
Sep 14 05:26:53.938934 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:53.938906    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w"
Sep 14 05:26:54.209873 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:54.209843    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w"]
Sep 14 05:26:54.211118 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:26:54.211094    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b0282c7_c6a4_4a03_942a_9904bf6be4f3.slice/crio-daed40492dc3ed2ac1d23db27f7d16158d6eb134572d5eb3a31d9d6ae7f27780 WatchSource:0}: Error finding container daed40492dc3ed2ac1d23db27f7d16158d6eb134572d5eb3a31d9d6ae7f27780: Status 404 returned error can't find the container with id daed40492dc3ed2ac1d23db27f7d16158d6eb134572d5eb3a31d9d6ae7f27780
Sep 14 05:26:54.830616 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:54.830576    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w" event={"ID":"6b0282c7-c6a4-4a03-942a-9904bf6be4f3","Type":"ContainerStarted","Data":"042453de40035b929a15ba79f9e5e465ba537eb70b41bca50bdcffeaa8ca8db4"}
Sep 14 05:26:54.830616 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:54.830619    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w" event={"ID":"6b0282c7-c6a4-4a03-942a-9904bf6be4f3","Type":"ContainerStarted","Data":"daed40492dc3ed2ac1d23db27f7d16158d6eb134572d5eb3a31d9d6ae7f27780"}
Sep 14 05:26:54.848826 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:54.848769    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w" podStartSLOduration=1.848739683 podStartE2EDuration="1.848739683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:26:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:26:54.846931258 +0000 UTC m=+2413.022702816" watchObservedRunningTime="2026-09-14 05:26:54.848739683 +0000 UTC m=+2413.024511242"
Sep 14 05:26:56.531680 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:26:56.531642    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:27:03.939502 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:03.939462    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w"
Sep 14 05:27:03.939909 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:03.939516    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w"
Sep 14 05:27:03.952079 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:03.952051    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w"
Sep 14 05:27:04.901024 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:04.900943    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w"
Sep 14 05:27:21.854920 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:21.854888    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j"]
Sep 14 05:27:22.607496 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:22.607455    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w"]
Sep 14 05:27:25.030435 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:25.030400    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:27:34.628216 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:34.628178    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:27:36.873551 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:36.873509    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j" podUID="82a0ab60-542e-42fa-9142-0665cd63073f" containerName="main" containerID="cri-o://e1b5dc5d3186f91ed0704859d32bb0da55e85388d4070468e07b21ce2f9a3e4d" gracePeriod=45
Sep 14 05:27:37.059967 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:37.059937    2565 generic.go:356] "Generic (PLEG): container finished" podID="82a0ab60-542e-42fa-9142-0665cd63073f" containerID="e1b5dc5d3186f91ed0704859d32bb0da55e85388d4070468e07b21ce2f9a3e4d" exitCode=0
Sep 14 05:27:37.060110 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:37.059986    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j" event={"ID":"82a0ab60-542e-42fa-9142-0665cd63073f","Type":"ContainerDied","Data":"e1b5dc5d3186f91ed0704859d32bb0da55e85388d4070468e07b21ce2f9a3e4d"}
Sep 14 05:27:37.109014 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:37.108988    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j"
Sep 14 05:27:37.223239 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:37.223159    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82a0ab60-542e-42fa-9142-0665cd63073f-model-cache\") pod \"82a0ab60-542e-42fa-9142-0665cd63073f\" (UID: \"82a0ab60-542e-42fa-9142-0665cd63073f\") "
Sep 14 05:27:37.223239 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:37.223195    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82a0ab60-542e-42fa-9142-0665cd63073f-tmp-dir\") pod \"82a0ab60-542e-42fa-9142-0665cd63073f\" (UID: \"82a0ab60-542e-42fa-9142-0665cd63073f\") "
Sep 14 05:27:37.223239 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:37.223236    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82a0ab60-542e-42fa-9142-0665cd63073f-tls-certs\") pod \"82a0ab60-542e-42fa-9142-0665cd63073f\" (UID: \"82a0ab60-542e-42fa-9142-0665cd63073f\") "
Sep 14 05:27:37.223561 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:37.223255    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9xdr\" (UniqueName: \"kubernetes.io/projected/82a0ab60-542e-42fa-9142-0665cd63073f-kube-api-access-s9xdr\") pod \"82a0ab60-542e-42fa-9142-0665cd63073f\" (UID: \"82a0ab60-542e-42fa-9142-0665cd63073f\") "
Sep 14 05:27:37.223561 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:37.223277    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82a0ab60-542e-42fa-9142-0665cd63073f-home\") pod \"82a0ab60-542e-42fa-9142-0665cd63073f\" (UID: \"82a0ab60-542e-42fa-9142-0665cd63073f\") "
Sep 14 05:27:37.223561 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:37.223309    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82a0ab60-542e-42fa-9142-0665cd63073f-dshm\") pod \"82a0ab60-542e-42fa-9142-0665cd63073f\" (UID: \"82a0ab60-542e-42fa-9142-0665cd63073f\") "
Sep 14 05:27:37.223561 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:37.223479    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a0ab60-542e-42fa-9142-0665cd63073f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "82a0ab60-542e-42fa-9142-0665cd63073f" (UID: "82a0ab60-542e-42fa-9142-0665cd63073f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:27:37.223561 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:37.223492    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a0ab60-542e-42fa-9142-0665cd63073f-model-cache" (OuterVolumeSpecName: "model-cache") pod "82a0ab60-542e-42fa-9142-0665cd63073f" (UID: "82a0ab60-542e-42fa-9142-0665cd63073f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:27:37.223779 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:37.223576    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a0ab60-542e-42fa-9142-0665cd63073f-home" (OuterVolumeSpecName: "home") pod "82a0ab60-542e-42fa-9142-0665cd63073f" (UID: "82a0ab60-542e-42fa-9142-0665cd63073f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:27:37.223779 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:37.223597    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82a0ab60-542e-42fa-9142-0665cd63073f-tmp-dir\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:27:37.225527 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:37.225495    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82a0ab60-542e-42fa-9142-0665cd63073f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "82a0ab60-542e-42fa-9142-0665cd63073f" (UID: "82a0ab60-542e-42fa-9142-0665cd63073f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 05:27:37.225633 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:37.225545    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a0ab60-542e-42fa-9142-0665cd63073f-dshm" (OuterVolumeSpecName: "dshm") pod "82a0ab60-542e-42fa-9142-0665cd63073f" (UID: "82a0ab60-542e-42fa-9142-0665cd63073f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:27:37.225697 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:37.225632    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82a0ab60-542e-42fa-9142-0665cd63073f-kube-api-access-s9xdr" (OuterVolumeSpecName: "kube-api-access-s9xdr") pod "82a0ab60-542e-42fa-9142-0665cd63073f" (UID: "82a0ab60-542e-42fa-9142-0665cd63073f"). InnerVolumeSpecName "kube-api-access-s9xdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:27:37.324857 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:37.324820    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82a0ab60-542e-42fa-9142-0665cd63073f-model-cache\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:27:37.324857 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:37.324852    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82a0ab60-542e-42fa-9142-0665cd63073f-tls-certs\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:27:37.325038 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:37.324865    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9xdr\" (UniqueName: \"kubernetes.io/projected/82a0ab60-542e-42fa-9142-0665cd63073f-kube-api-access-s9xdr\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:27:37.325038 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:37.324877    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82a0ab60-542e-42fa-9142-0665cd63073f-home\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:27:37.325038 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:37.324889    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82a0ab60-542e-42fa-9142-0665cd63073f-dshm\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:27:37.626364 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:37.626304    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w" podUID="6b0282c7-c6a4-4a03-942a-9904bf6be4f3" containerName="main" containerID="cri-o://042453de40035b929a15ba79f9e5e465ba537eb70b41bca50bdcffeaa8ca8db4" gracePeriod=45
Sep 14 05:27:37.855302 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:37.855278    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w"
Sep 14 05:27:37.929991 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:37.929965    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b0282c7-c6a4-4a03-942a-9904bf6be4f3-home\") pod \"6b0282c7-c6a4-4a03-942a-9904bf6be4f3\" (UID: \"6b0282c7-c6a4-4a03-942a-9904bf6be4f3\") "
Sep 14 05:27:37.930328 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:37.930052    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b0282c7-c6a4-4a03-942a-9904bf6be4f3-tls-certs\") pod \"6b0282c7-c6a4-4a03-942a-9904bf6be4f3\" (UID: \"6b0282c7-c6a4-4a03-942a-9904bf6be4f3\") "
Sep 14 05:27:37.930328 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:37.930070    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khf4t\" (UniqueName: \"kubernetes.io/projected/6b0282c7-c6a4-4a03-942a-9904bf6be4f3-kube-api-access-khf4t\") pod \"6b0282c7-c6a4-4a03-942a-9904bf6be4f3\" (UID: \"6b0282c7-c6a4-4a03-942a-9904bf6be4f3\") "
Sep 14 05:27:37.930328 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:37.930190    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b0282c7-c6a4-4a03-942a-9904bf6be4f3-model-cache\") pod \"6b0282c7-c6a4-4a03-942a-9904bf6be4f3\" (UID: \"6b0282c7-c6a4-4a03-942a-9904bf6be4f3\") "
Sep 14 05:27:37.930328 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:37.930238    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b0282c7-c6a4-4a03-942a-9904bf6be4f3-tmp-dir\") pod \"6b0282c7-c6a4-4a03-942a-9904bf6be4f3\" (UID: \"6b0282c7-c6a4-4a03-942a-9904bf6be4f3\") "
Sep 14 05:27:37.930328 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:37.930261    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b0282c7-c6a4-4a03-942a-9904bf6be4f3-home" (OuterVolumeSpecName: "home") pod "6b0282c7-c6a4-4a03-942a-9904bf6be4f3" (UID: "6b0282c7-c6a4-4a03-942a-9904bf6be4f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:27:37.930328 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:37.930270    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b0282c7-c6a4-4a03-942a-9904bf6be4f3-dshm\") pod \"6b0282c7-c6a4-4a03-942a-9904bf6be4f3\" (UID: \"6b0282c7-c6a4-4a03-942a-9904bf6be4f3\") "
Sep 14 05:27:37.930652 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:37.930485    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b0282c7-c6a4-4a03-942a-9904bf6be4f3-model-cache" (OuterVolumeSpecName: "model-cache") pod "6b0282c7-c6a4-4a03-942a-9904bf6be4f3" (UID: "6b0282c7-c6a4-4a03-942a-9904bf6be4f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:27:37.930652 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:37.930514    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b0282c7-c6a4-4a03-942a-9904bf6be4f3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6b0282c7-c6a4-4a03-942a-9904bf6be4f3" (UID: "6b0282c7-c6a4-4a03-942a-9904bf6be4f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:27:37.930652 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:37.930603    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b0282c7-c6a4-4a03-942a-9904bf6be4f3-model-cache\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:27:37.930652 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:37.930623    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b0282c7-c6a4-4a03-942a-9904bf6be4f3-tmp-dir\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:27:37.930652 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:37.930636    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b0282c7-c6a4-4a03-942a-9904bf6be4f3-home\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:27:37.932161 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:37.932134    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b0282c7-c6a4-4a03-942a-9904bf6be4f3-dshm" (OuterVolumeSpecName: "dshm") pod "6b0282c7-c6a4-4a03-942a-9904bf6be4f3" (UID: "6b0282c7-c6a4-4a03-942a-9904bf6be4f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 05:27:37.932251 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:37.932171    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b0282c7-c6a4-4a03-942a-9904bf6be4f3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6b0282c7-c6a4-4a03-942a-9904bf6be4f3" (UID: "6b0282c7-c6a4-4a03-942a-9904bf6be4f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 05:27:37.932251 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:37.932204    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b0282c7-c6a4-4a03-942a-9904bf6be4f3-kube-api-access-khf4t" (OuterVolumeSpecName: "kube-api-access-khf4t") pod "6b0282c7-c6a4-4a03-942a-9904bf6be4f3" (UID: "6b0282c7-c6a4-4a03-942a-9904bf6be4f3"). InnerVolumeSpecName "kube-api-access-khf4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:27:38.031306 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:38.031280    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b0282c7-c6a4-4a03-942a-9904bf6be4f3-tls-certs\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:27:38.031306 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:38.031304    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khf4t\" (UniqueName: \"kubernetes.io/projected/6b0282c7-c6a4-4a03-942a-9904bf6be4f3-kube-api-access-khf4t\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:27:38.031469 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:38.031313    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b0282c7-c6a4-4a03-942a-9904bf6be4f3-dshm\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:27:38.066096 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:38.066063    2565 generic.go:356] "Generic (PLEG): container finished" podID="6b0282c7-c6a4-4a03-942a-9904bf6be4f3" containerID="042453de40035b929a15ba79f9e5e465ba537eb70b41bca50bdcffeaa8ca8db4" exitCode=0
Sep 14 05:27:38.066230 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:38.066104    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w"
Sep 14 05:27:38.066230 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:38.066146    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w" event={"ID":"6b0282c7-c6a4-4a03-942a-9904bf6be4f3","Type":"ContainerDied","Data":"042453de40035b929a15ba79f9e5e465ba537eb70b41bca50bdcffeaa8ca8db4"}
Sep 14 05:27:38.066230 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:38.066203    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w" event={"ID":"6b0282c7-c6a4-4a03-942a-9904bf6be4f3","Type":"ContainerDied","Data":"daed40492dc3ed2ac1d23db27f7d16158d6eb134572d5eb3a31d9d6ae7f27780"}
Sep 14 05:27:38.066382 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:38.066231    2565 scope.go:117] "RemoveContainer" containerID="042453de40035b929a15ba79f9e5e465ba537eb70b41bca50bdcffeaa8ca8db4"
Sep 14 05:27:38.067575 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:38.067552    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j"
Sep 14 05:27:38.067647 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:38.067589    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j" event={"ID":"82a0ab60-542e-42fa-9142-0665cd63073f","Type":"ContainerDied","Data":"378ff620edf07f8bec46e01d12a3af8504c0a157eb559211f3a697fda9524def"}
Sep 14 05:27:38.075509 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:38.075494    2565 scope.go:117] "RemoveContainer" containerID="042453de40035b929a15ba79f9e5e465ba537eb70b41bca50bdcffeaa8ca8db4"
Sep 14 05:27:38.075804 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:27:38.075784    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"042453de40035b929a15ba79f9e5e465ba537eb70b41bca50bdcffeaa8ca8db4\": container with ID starting with 042453de40035b929a15ba79f9e5e465ba537eb70b41bca50bdcffeaa8ca8db4 not found: ID does not exist" containerID="042453de40035b929a15ba79f9e5e465ba537eb70b41bca50bdcffeaa8ca8db4"
Sep 14 05:27:38.075855 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:38.075810    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"042453de40035b929a15ba79f9e5e465ba537eb70b41bca50bdcffeaa8ca8db4"} err="failed to get container status \"042453de40035b929a15ba79f9e5e465ba537eb70b41bca50bdcffeaa8ca8db4\": rpc error: code = NotFound desc = could not find container \"042453de40035b929a15ba79f9e5e465ba537eb70b41bca50bdcffeaa8ca8db4\": container with ID starting with 042453de40035b929a15ba79f9e5e465ba537eb70b41bca50bdcffeaa8ca8db4 not found: ID does not exist"
Sep 14 05:27:38.075855 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:38.075825    2565 scope.go:117] "RemoveContainer" containerID="e1b5dc5d3186f91ed0704859d32bb0da55e85388d4070468e07b21ce2f9a3e4d"
Sep 14 05:27:38.090414 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:38.090388    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w"]
Sep 14 05:27:38.093651 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:38.093630    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-e8fb9296-kserve-5c8bf77b47-cff9w"]
Sep 14 05:27:38.104935 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:38.104916    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j"]
Sep 14 05:27:38.110267 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:38.110244    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-e8fb9296-kserve-6f9c488979-2227j"]
Sep 14 05:27:38.334471 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:38.334444    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b0282c7-c6a4-4a03-942a-9904bf6be4f3" path="/var/lib/kubelet/pods/6b0282c7-c6a4-4a03-942a-9904bf6be4f3/volumes"
Sep 14 05:27:38.334885 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:38.334870    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82a0ab60-542e-42fa-9142-0665cd63073f" path="/var/lib/kubelet/pods/82a0ab60-542e-42fa-9142-0665cd63073f/volumes"
Sep 14 05:27:42.426174 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:42.426144    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:27:42.723137 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:42.723061    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5f795d4696-spjjx"]
Sep 14 05:27:42.723341 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:42.723329    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b0282c7-c6a4-4a03-942a-9904bf6be4f3" containerName="main"
Sep 14 05:27:42.723391 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:42.723342    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b0282c7-c6a4-4a03-942a-9904bf6be4f3" containerName="main"
Sep 14 05:27:42.723391 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:42.723354    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82a0ab60-542e-42fa-9142-0665cd63073f" containerName="main"
Sep 14 05:27:42.723391 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:42.723360    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="82a0ab60-542e-42fa-9142-0665cd63073f" containerName="main"
Sep 14 05:27:42.723484 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:42.723422    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="82a0ab60-542e-42fa-9142-0665cd63073f" containerName="main"
Sep 14 05:27:42.723484 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:42.723429    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b0282c7-c6a4-4a03-942a-9904bf6be4f3" containerName="main"
Sep 14 05:27:42.727791 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:42.727773    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5f795d4696-spjjx"
Sep 14 05:27:42.730870 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:42.730845    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 14 05:27:42.731052 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:42.730945    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 05:27:42.731052 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:42.730972    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 14 05:27:42.731213 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:42.731197    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-m9nv8\""
Sep 14 05:27:42.731284 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:42.731206    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 14 05:27:42.735816 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:42.735797    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5f795d4696-spjjx"]
Sep 14 05:27:42.872829 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:42.872788    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkf2q\" (UniqueName: \"kubernetes.io/projected/a37b8103-697c-4a63-a911-939039e58279-kube-api-access-bkf2q\") pod \"maas-api-5f795d4696-spjjx\" (UID: \"a37b8103-697c-4a63-a911-939039e58279\") " pod="odh-ai-gateway-infra/maas-api-5f795d4696-spjjx"
Sep 14 05:27:42.872829 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:42.872829    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a37b8103-697c-4a63-a911-939039e58279-maas-api-tls\") pod \"maas-api-5f795d4696-spjjx\" (UID: \"a37b8103-697c-4a63-a911-939039e58279\") " pod="odh-ai-gateway-infra/maas-api-5f795d4696-spjjx"
Sep 14 05:27:42.873038 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:42.872907    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a37b8103-697c-4a63-a911-939039e58279-metrics-certs\") pod \"maas-api-5f795d4696-spjjx\" (UID: \"a37b8103-697c-4a63-a911-939039e58279\") " pod="odh-ai-gateway-infra/maas-api-5f795d4696-spjjx"
Sep 14 05:27:42.873038 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:42.872950    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a37b8103-697c-4a63-a911-939039e58279-openshift-service-ca\") pod \"maas-api-5f795d4696-spjjx\" (UID: \"a37b8103-697c-4a63-a911-939039e58279\") " pod="odh-ai-gateway-infra/maas-api-5f795d4696-spjjx"
Sep 14 05:27:42.902068 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:42.902039    2565 scope.go:117] "RemoveContainer" containerID="a820d0277a87b38c63fa57a8342850509e500f720f0b78c94424ff5525356f5f"
Sep 14 05:27:42.974244 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:42.974172    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkf2q\" (UniqueName: \"kubernetes.io/projected/a37b8103-697c-4a63-a911-939039e58279-kube-api-access-bkf2q\") pod \"maas-api-5f795d4696-spjjx\" (UID: \"a37b8103-697c-4a63-a911-939039e58279\") " pod="odh-ai-gateway-infra/maas-api-5f795d4696-spjjx"
Sep 14 05:27:42.974244 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:42.974205    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a37b8103-697c-4a63-a911-939039e58279-maas-api-tls\") pod \"maas-api-5f795d4696-spjjx\" (UID: \"a37b8103-697c-4a63-a911-939039e58279\") " pod="odh-ai-gateway-infra/maas-api-5f795d4696-spjjx"
Sep 14 05:27:42.974425 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:42.974284    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a37b8103-697c-4a63-a911-939039e58279-metrics-certs\") pod \"maas-api-5f795d4696-spjjx\" (UID: \"a37b8103-697c-4a63-a911-939039e58279\") " pod="odh-ai-gateway-infra/maas-api-5f795d4696-spjjx"
Sep 14 05:27:42.974425 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:42.974340    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a37b8103-697c-4a63-a911-939039e58279-openshift-service-ca\") pod \"maas-api-5f795d4696-spjjx\" (UID: \"a37b8103-697c-4a63-a911-939039e58279\") " pod="odh-ai-gateway-infra/maas-api-5f795d4696-spjjx"
Sep 14 05:27:42.974980 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:42.974957    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a37b8103-697c-4a63-a911-939039e58279-openshift-service-ca\") pod \"maas-api-5f795d4696-spjjx\" (UID: \"a37b8103-697c-4a63-a911-939039e58279\") " pod="odh-ai-gateway-infra/maas-api-5f795d4696-spjjx"
Sep 14 05:27:42.976617 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:42.976590    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a37b8103-697c-4a63-a911-939039e58279-maas-api-tls\") pod \"maas-api-5f795d4696-spjjx\" (UID: \"a37b8103-697c-4a63-a911-939039e58279\") " pod="odh-ai-gateway-infra/maas-api-5f795d4696-spjjx"
Sep 14 05:27:42.976705 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:42.976628    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a37b8103-697c-4a63-a911-939039e58279-metrics-certs\") pod \"maas-api-5f795d4696-spjjx\" (UID: \"a37b8103-697c-4a63-a911-939039e58279\") " pod="odh-ai-gateway-infra/maas-api-5f795d4696-spjjx"
Sep 14 05:27:42.982039 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:42.982017    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkf2q\" (UniqueName: \"kubernetes.io/projected/a37b8103-697c-4a63-a911-939039e58279-kube-api-access-bkf2q\") pod \"maas-api-5f795d4696-spjjx\" (UID: \"a37b8103-697c-4a63-a911-939039e58279\") " pod="odh-ai-gateway-infra/maas-api-5f795d4696-spjjx"
Sep 14 05:27:43.038961 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:43.038937    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5f795d4696-spjjx"
Sep 14 05:27:43.309559 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:43.309531    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5f795d4696-spjjx"]
Sep 14 05:27:43.311465 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:27:43.311435    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda37b8103_697c_4a63_a911_939039e58279.slice/crio-76ef69eb19a7268f6b61d9577f22184c6c335da10398884702c082b43234bdd5 WatchSource:0}: Error finding container 76ef69eb19a7268f6b61d9577f22184c6c335da10398884702c082b43234bdd5: Status 404 returned error can't find the container with id 76ef69eb19a7268f6b61d9577f22184c6c335da10398884702c082b43234bdd5
Sep 14 05:27:43.312795 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:43.312774    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 05:27:44.102766 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:44.102679    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5f795d4696-spjjx" event={"ID":"a37b8103-697c-4a63-a911-939039e58279","Type":"ContainerStarted","Data":"76ef69eb19a7268f6b61d9577f22184c6c335da10398884702c082b43234bdd5"}
Sep 14 05:27:46.115782 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:46.115718    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5f795d4696-spjjx" event={"ID":"a37b8103-697c-4a63-a911-939039e58279","Type":"ContainerStarted","Data":"38f12009a498f9d179e2b230448671821338381fe77be21ebabfff4852c9e483"}
Sep 14 05:27:46.116229 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:46.115814    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5f795d4696-spjjx"
Sep 14 05:27:46.134065 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:46.134021    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5f795d4696-spjjx" podStartSLOduration=1.814654354 podStartE2EDuration="4.134007688s" totalImagesPullingTime="2.319353334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:27:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:27:46.131900449 +0000 UTC m=+2464.307672008" watchObservedRunningTime="2026-09-14 05:27:46.134007688 +0000 UTC m=+2464.309779246"
Sep 14 05:27:50.135814 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:50.135779    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:27:50.412389 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:50.412308    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5f795d4696-spjjx"]
Sep 14 05:27:50.412539 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:50.412494    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5f795d4696-spjjx" podUID="a37b8103-697c-4a63-a911-939039e58279" containerName="maas-api" containerID="cri-o://38f12009a498f9d179e2b230448671821338381fe77be21ebabfff4852c9e483" gracePeriod=30
Sep 14 05:27:50.418514 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:50.418492    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5f795d4696-spjjx"
Sep 14 05:27:50.655051 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:50.655025    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5f795d4696-spjjx"
Sep 14 05:27:50.744295 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:50.744217    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkf2q\" (UniqueName: \"kubernetes.io/projected/a37b8103-697c-4a63-a911-939039e58279-kube-api-access-bkf2q\") pod \"a37b8103-697c-4a63-a911-939039e58279\" (UID: \"a37b8103-697c-4a63-a911-939039e58279\") "
Sep 14 05:27:50.744295 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:50.744269    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a37b8103-697c-4a63-a911-939039e58279-metrics-certs\") pod \"a37b8103-697c-4a63-a911-939039e58279\" (UID: \"a37b8103-697c-4a63-a911-939039e58279\") "
Sep 14 05:27:50.744463 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:50.744305    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a37b8103-697c-4a63-a911-939039e58279-maas-api-tls\") pod \"a37b8103-697c-4a63-a911-939039e58279\" (UID: \"a37b8103-697c-4a63-a911-939039e58279\") "
Sep 14 05:27:50.744463 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:50.744336    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a37b8103-697c-4a63-a911-939039e58279-openshift-service-ca\") pod \"a37b8103-697c-4a63-a911-939039e58279\" (UID: \"a37b8103-697c-4a63-a911-939039e58279\") "
Sep 14 05:27:50.744749 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:50.744720    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a37b8103-697c-4a63-a911-939039e58279-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a37b8103-697c-4a63-a911-939039e58279" (UID: "a37b8103-697c-4a63-a911-939039e58279"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 05:27:50.746511 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:50.746479    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a37b8103-697c-4a63-a911-939039e58279-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a37b8103-697c-4a63-a911-939039e58279" (UID: "a37b8103-697c-4a63-a911-939039e58279"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 05:27:50.746627 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:50.746532    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a37b8103-697c-4a63-a911-939039e58279-kube-api-access-bkf2q" (OuterVolumeSpecName: "kube-api-access-bkf2q") pod "a37b8103-697c-4a63-a911-939039e58279" (UID: "a37b8103-697c-4a63-a911-939039e58279"). InnerVolumeSpecName "kube-api-access-bkf2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 05:27:50.746627 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:50.746590    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a37b8103-697c-4a63-a911-939039e58279-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a37b8103-697c-4a63-a911-939039e58279" (UID: "a37b8103-697c-4a63-a911-939039e58279"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 05:27:50.845762 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:50.845735    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkf2q\" (UniqueName: \"kubernetes.io/projected/a37b8103-697c-4a63-a911-939039e58279-kube-api-access-bkf2q\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:27:50.845875 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:50.845781    2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a37b8103-697c-4a63-a911-939039e58279-metrics-certs\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:27:50.845875 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:50.845791    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a37b8103-697c-4a63-a911-939039e58279-maas-api-tls\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:27:50.845875 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:50.845799    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a37b8103-697c-4a63-a911-939039e58279-openshift-service-ca\") on node \"ip-10-0-133-99.ec2.internal\" DevicePath \"\""
Sep 14 05:27:51.141454 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:51.141420    2565 generic.go:356] "Generic (PLEG): container finished" podID="a37b8103-697c-4a63-a911-939039e58279" containerID="38f12009a498f9d179e2b230448671821338381fe77be21ebabfff4852c9e483" exitCode=0
Sep 14 05:27:51.141873 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:51.141466    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5f795d4696-spjjx"
Sep 14 05:27:51.141873 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:51.141474    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5f795d4696-spjjx" event={"ID":"a37b8103-697c-4a63-a911-939039e58279","Type":"ContainerDied","Data":"38f12009a498f9d179e2b230448671821338381fe77be21ebabfff4852c9e483"}
Sep 14 05:27:51.141873 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:51.141516    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5f795d4696-spjjx" event={"ID":"a37b8103-697c-4a63-a911-939039e58279","Type":"ContainerDied","Data":"76ef69eb19a7268f6b61d9577f22184c6c335da10398884702c082b43234bdd5"}
Sep 14 05:27:51.141873 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:51.141538    2565 scope.go:117] "RemoveContainer" containerID="38f12009a498f9d179e2b230448671821338381fe77be21ebabfff4852c9e483"
Sep 14 05:27:51.151522 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:51.151506    2565 scope.go:117] "RemoveContainer" containerID="38f12009a498f9d179e2b230448671821338381fe77be21ebabfff4852c9e483"
Sep 14 05:27:51.151792 ip-10-0-133-99 kubenswrapper[2565]: E0914 05:27:51.151765    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38f12009a498f9d179e2b230448671821338381fe77be21ebabfff4852c9e483\": container with ID starting with 38f12009a498f9d179e2b230448671821338381fe77be21ebabfff4852c9e483 not found: ID does not exist" containerID="38f12009a498f9d179e2b230448671821338381fe77be21ebabfff4852c9e483"
Sep 14 05:27:51.151916 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:51.151799    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38f12009a498f9d179e2b230448671821338381fe77be21ebabfff4852c9e483"} err="failed to get container status \"38f12009a498f9d179e2b230448671821338381fe77be21ebabfff4852c9e483\": rpc error: code = NotFound desc = could not find container \"38f12009a498f9d179e2b230448671821338381fe77be21ebabfff4852c9e483\": container with ID starting with 38f12009a498f9d179e2b230448671821338381fe77be21ebabfff4852c9e483 not found: ID does not exist"
Sep 14 05:27:51.163786 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:51.163749    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5f795d4696-spjjx"]
Sep 14 05:27:51.167521 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:51.167502    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5f795d4696-spjjx"]
Sep 14 05:27:52.335207 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:52.335171    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a37b8103-697c-4a63-a911-939039e58279" path="/var/lib/kubelet/pods/a37b8103-697c-4a63-a911-939039e58279/volumes"
Sep 14 05:27:57.026648 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:27:57.026615    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:28:03.125370 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:03.125330    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:28:10.135152 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:10.135117    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dbf8d"]
Sep 14 05:28:12.796498 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:12.796467    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-78cfd7c89-xbsm5_b16d10bd-7c66-48bb-b9a4-011e1f9f9e19/authorino/0.log"
Sep 14 05:28:19.064955 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:19.064924    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-78b6456dc5-v2m8w_03e983db-f240-4e1f-af0a-726228526dab/copy-manifests/0.log"
Sep 14 05:28:19.071563 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:19.071535    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-78b6456dc5-v2m8w_03e983db-f240-4e1f-af0a-726228526dab/manager/0.log"
Sep 14 05:28:19.791663 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:19.791583    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29822700-zxqrq_5c78c401-caf2-41eb-8729-1113340b8f2e/cleanup/0.log"
Sep 14 05:28:19.904306 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:19.904276    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29822715-rxqzw_7d362a61-8b9d-4ea8-95b6-cd6ba9537e8d/cleanup/0.log"
Sep 14 05:28:20.858388 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:20.858352    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-78cfd7c89-xbsm5_b16d10bd-7c66-48bb-b9a4-011e1f9f9e19/authorino/0.log"
Sep 14 05:28:21.485093 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:21.485064    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-dbf8d_98d95388-675b-41a4-a488-c76222746864/limitador/0.log"
Sep 14 05:28:22.334917 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:22.334885    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-96745c74f-xnltl_2a5dca15-e3df-4cab-898d-6716ee6f4615/kube-auth-proxy/0.log"
Sep 14 05:28:22.795543 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:22.795464    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c9d6db5f8-lqphc_93a9bb73-aa22-4294-b119-20aa06f4bfde/router/0.log"
Sep 14 05:28:23.580710 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:23.580681    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5976_114d8063-89e7-4b4c-8ae2-20e344640805/main/0.log"
Sep 14 05:28:23.816310 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:23.816280    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-8wpxd_8e7b27aa-4d5a-4784-b6ee-ed3471d5fecf/main/0.log"
Sep 14 05:28:29.440080 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:29.440046    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rvbqd/must-gather-7dr2b"]
Sep 14 05:28:29.440573 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:29.440369    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a37b8103-697c-4a63-a911-939039e58279" containerName="maas-api"
Sep 14 05:28:29.440573 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:29.440381    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a37b8103-697c-4a63-a911-939039e58279" containerName="maas-api"
Sep 14 05:28:29.440573 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:29.440444    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a37b8103-697c-4a63-a911-939039e58279" containerName="maas-api"
Sep 14 05:28:29.443451 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:29.443431    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rvbqd/must-gather-7dr2b"
Sep 14 05:28:29.445851 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:29.445827    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rvbqd\"/\"kube-root-ca.crt\""
Sep 14 05:28:29.446000 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:29.445979    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rvbqd\"/\"default-dockercfg-s74lc\""
Sep 14 05:28:29.446000 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:29.445993    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rvbqd\"/\"openshift-service-ca.crt\""
Sep 14 05:28:29.453897 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:29.453878    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rvbqd/must-gather-7dr2b"]
Sep 14 05:28:29.546013 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:29.545986    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7de67573-a8f9-4e7d-af11-77b2e63eb250-must-gather-output\") pod \"must-gather-7dr2b\" (UID: \"7de67573-a8f9-4e7d-af11-77b2e63eb250\") " pod="openshift-must-gather-rvbqd/must-gather-7dr2b"
Sep 14 05:28:29.546190 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:29.546023    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m87st\" (UniqueName: \"kubernetes.io/projected/7de67573-a8f9-4e7d-af11-77b2e63eb250-kube-api-access-m87st\") pod \"must-gather-7dr2b\" (UID: \"7de67573-a8f9-4e7d-af11-77b2e63eb250\") " pod="openshift-must-gather-rvbqd/must-gather-7dr2b"
Sep 14 05:28:29.646801 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:29.646763    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7de67573-a8f9-4e7d-af11-77b2e63eb250-must-gather-output\") pod \"must-gather-7dr2b\" (UID: \"7de67573-a8f9-4e7d-af11-77b2e63eb250\") " pod="openshift-must-gather-rvbqd/must-gather-7dr2b"
Sep 14 05:28:29.646948 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:29.646818    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m87st\" (UniqueName: \"kubernetes.io/projected/7de67573-a8f9-4e7d-af11-77b2e63eb250-kube-api-access-m87st\") pod \"must-gather-7dr2b\" (UID: \"7de67573-a8f9-4e7d-af11-77b2e63eb250\") " pod="openshift-must-gather-rvbqd/must-gather-7dr2b"
Sep 14 05:28:29.647114 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:29.647089    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7de67573-a8f9-4e7d-af11-77b2e63eb250-must-gather-output\") pod \"must-gather-7dr2b\" (UID: \"7de67573-a8f9-4e7d-af11-77b2e63eb250\") " pod="openshift-must-gather-rvbqd/must-gather-7dr2b"
Sep 14 05:28:29.657260 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:29.657240    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m87st\" (UniqueName: \"kubernetes.io/projected/7de67573-a8f9-4e7d-af11-77b2e63eb250-kube-api-access-m87st\") pod \"must-gather-7dr2b\" (UID: \"7de67573-a8f9-4e7d-af11-77b2e63eb250\") " pod="openshift-must-gather-rvbqd/must-gather-7dr2b"
Sep 14 05:28:29.753981 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:29.753921    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rvbqd/must-gather-7dr2b"
Sep 14 05:28:30.023941 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:30.023917    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rvbqd/must-gather-7dr2b"]
Sep 14 05:28:30.025772 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:28:30.025730    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7de67573_a8f9_4e7d_af11_77b2e63eb250.slice/crio-553bf4c72b9b267e19eb686c8eb3fc13797e68082b09ff91177aced4b4fd2d59 WatchSource:0}: Error finding container 553bf4c72b9b267e19eb686c8eb3fc13797e68082b09ff91177aced4b4fd2d59: Status 404 returned error can't find the container with id 553bf4c72b9b267e19eb686c8eb3fc13797e68082b09ff91177aced4b4fd2d59
Sep 14 05:28:30.334210 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:30.334178    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rvbqd/must-gather-7dr2b" event={"ID":"7de67573-a8f9-4e7d-af11-77b2e63eb250","Type":"ContainerStarted","Data":"553bf4c72b9b267e19eb686c8eb3fc13797e68082b09ff91177aced4b4fd2d59"}
Sep 14 05:28:31.343887 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:31.343844    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rvbqd/must-gather-7dr2b" event={"ID":"7de67573-a8f9-4e7d-af11-77b2e63eb250","Type":"ContainerStarted","Data":"a4f004b96f5b49cb8d3e69690ce1b804cd7092e3660086c7e554299f9800ad12"}
Sep 14 05:28:31.344365 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:31.343904    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rvbqd/must-gather-7dr2b" event={"ID":"7de67573-a8f9-4e7d-af11-77b2e63eb250","Type":"ContainerStarted","Data":"0be2732dc8a3f6ddb4b99040fd790527db696f0d19b614b6a4b8e28f8631c257"}
Sep 14 05:28:31.362601 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:31.362553    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rvbqd/must-gather-7dr2b" podStartSLOduration=1.597849907 podStartE2EDuration="2.362535014s" totalImagesPullingTime="764.685107ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 05:28:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:28:31.360474854 +0000 UTC m=+2509.536246414" watchObservedRunningTime="2026-09-14 05:28:31.362535014 +0000 UTC m=+2509.538306573"
Sep 14 05:28:32.483450 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:32.483417    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-whcnp_ca784040-efa0-4d69-8c25-2b284f3bbe65/global-pull-secret-syncer/0.log"
Sep 14 05:28:32.605674 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:32.605639    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zv7p8_9f1e07bf-2d62-4d64-a473-150a0830ba42/konnectivity-agent/0.log"
Sep 14 05:28:32.654935 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:32.654903    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-99.ec2.internal_d5acb9fe33c38810a5100cf8c524f161/haproxy/0.log"
Sep 14 05:28:37.110218 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:37.110180    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-78cfd7c89-xbsm5_b16d10bd-7c66-48bb-b9a4-011e1f9f9e19/authorino/0.log"
Sep 14 05:28:37.292600 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:37.292571    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-dbf8d_98d95388-675b-41a4-a488-c76222746864/limitador/0.log"
Sep 14 05:28:39.063246 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:39.063209    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-vhtjr_ce13cb9e-09d6-4a9a-b4d4-7db45242c8bc/cluster-monitoring-operator/0.log"
Sep 14 05:28:39.427918 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:39.427860    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xrqvz_ad643666-9ad7-4c4f-9d4a-f18bdc21740d/node-exporter/0.log"
Sep 14 05:28:39.454447 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:39.454397    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xrqvz_ad643666-9ad7-4c4f-9d4a-f18bdc21740d/kube-rbac-proxy/0.log"
Sep 14 05:28:39.484366 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:39.484336    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xrqvz_ad643666-9ad7-4c4f-9d4a-f18bdc21740d/init-textfile/0.log"
Sep 14 05:28:42.492812 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:42.492778    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rvbqd/perf-node-gather-daemonset-z95dw"]
Sep 14 05:28:42.503851 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:42.503823    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rvbqd/perf-node-gather-daemonset-z95dw"
Sep 14 05:28:42.504576 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:42.504054    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rvbqd/perf-node-gather-daemonset-z95dw"]
Sep 14 05:28:42.567783 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:42.567714    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6dac4c5a-f772-496d-b3f2-4de668238b99-sys\") pod \"perf-node-gather-daemonset-z95dw\" (UID: \"6dac4c5a-f772-496d-b3f2-4de668238b99\") " pod="openshift-must-gather-rvbqd/perf-node-gather-daemonset-z95dw"
Sep 14 05:28:42.567783 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:42.567773    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gpcn\" (UniqueName: \"kubernetes.io/projected/6dac4c5a-f772-496d-b3f2-4de668238b99-kube-api-access-2gpcn\") pod \"perf-node-gather-daemonset-z95dw\" (UID: \"6dac4c5a-f772-496d-b3f2-4de668238b99\") " pod="openshift-must-gather-rvbqd/perf-node-gather-daemonset-z95dw"
Sep 14 05:28:42.568048 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:42.567869    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6dac4c5a-f772-496d-b3f2-4de668238b99-proc\") pod \"perf-node-gather-daemonset-z95dw\" (UID: \"6dac4c5a-f772-496d-b3f2-4de668238b99\") " pod="openshift-must-gather-rvbqd/perf-node-gather-daemonset-z95dw"
Sep 14 05:28:42.568048 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:42.567948    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6dac4c5a-f772-496d-b3f2-4de668238b99-lib-modules\") pod \"perf-node-gather-daemonset-z95dw\" (UID: \"6dac4c5a-f772-496d-b3f2-4de668238b99\") " pod="openshift-must-gather-rvbqd/perf-node-gather-daemonset-z95dw"
Sep 14 05:28:42.568048 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:42.567979    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6dac4c5a-f772-496d-b3f2-4de668238b99-podres\") pod \"perf-node-gather-daemonset-z95dw\" (UID: \"6dac4c5a-f772-496d-b3f2-4de668238b99\") " pod="openshift-must-gather-rvbqd/perf-node-gather-daemonset-z95dw"
Sep 14 05:28:42.669134 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:42.669076    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6dac4c5a-f772-496d-b3f2-4de668238b99-sys\") pod \"perf-node-gather-daemonset-z95dw\" (UID: \"6dac4c5a-f772-496d-b3f2-4de668238b99\") " pod="openshift-must-gather-rvbqd/perf-node-gather-daemonset-z95dw"
Sep 14 05:28:42.669325 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:42.669144    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gpcn\" (UniqueName: \"kubernetes.io/projected/6dac4c5a-f772-496d-b3f2-4de668238b99-kube-api-access-2gpcn\") pod \"perf-node-gather-daemonset-z95dw\" (UID: \"6dac4c5a-f772-496d-b3f2-4de668238b99\") " pod="openshift-must-gather-rvbqd/perf-node-gather-daemonset-z95dw"
Sep 14 05:28:42.669325 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:42.669199    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6dac4c5a-f772-496d-b3f2-4de668238b99-proc\") pod \"perf-node-gather-daemonset-z95dw\" (UID: \"6dac4c5a-f772-496d-b3f2-4de668238b99\") " pod="openshift-must-gather-rvbqd/perf-node-gather-daemonset-z95dw"
Sep 14 05:28:42.669325 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:42.669229    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6dac4c5a-f772-496d-b3f2-4de668238b99-sys\") pod \"perf-node-gather-daemonset-z95dw\" (UID: \"6dac4c5a-f772-496d-b3f2-4de668238b99\") " pod="openshift-must-gather-rvbqd/perf-node-gather-daemonset-z95dw"
Sep 14 05:28:42.669325 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:42.669292    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6dac4c5a-f772-496d-b3f2-4de668238b99-lib-modules\") pod \"perf-node-gather-daemonset-z95dw\" (UID: \"6dac4c5a-f772-496d-b3f2-4de668238b99\") " pod="openshift-must-gather-rvbqd/perf-node-gather-daemonset-z95dw"
Sep 14 05:28:42.669542 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:42.669320    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6dac4c5a-f772-496d-b3f2-4de668238b99-podres\") pod \"perf-node-gather-daemonset-z95dw\" (UID: \"6dac4c5a-f772-496d-b3f2-4de668238b99\") " pod="openshift-must-gather-rvbqd/perf-node-gather-daemonset-z95dw"
Sep 14 05:28:42.669542 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:42.669366    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6dac4c5a-f772-496d-b3f2-4de668238b99-proc\") pod \"perf-node-gather-daemonset-z95dw\" (UID: \"6dac4c5a-f772-496d-b3f2-4de668238b99\") " pod="openshift-must-gather-rvbqd/perf-node-gather-daemonset-z95dw"
Sep 14 05:28:42.669542 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:42.669527    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6dac4c5a-f772-496d-b3f2-4de668238b99-podres\") pod \"perf-node-gather-daemonset-z95dw\" (UID: \"6dac4c5a-f772-496d-b3f2-4de668238b99\") " pod="openshift-must-gather-rvbqd/perf-node-gather-daemonset-z95dw"
Sep 14 05:28:42.669718 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:42.669545    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6dac4c5a-f772-496d-b3f2-4de668238b99-lib-modules\") pod \"perf-node-gather-daemonset-z95dw\" (UID: \"6dac4c5a-f772-496d-b3f2-4de668238b99\") " pod="openshift-must-gather-rvbqd/perf-node-gather-daemonset-z95dw"
Sep 14 05:28:42.678207 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:42.678181    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gpcn\" (UniqueName: \"kubernetes.io/projected/6dac4c5a-f772-496d-b3f2-4de668238b99-kube-api-access-2gpcn\") pod \"perf-node-gather-daemonset-z95dw\" (UID: \"6dac4c5a-f772-496d-b3f2-4de668238b99\") " pod="openshift-must-gather-rvbqd/perf-node-gather-daemonset-z95dw"
Sep 14 05:28:42.819271 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:42.819184    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rvbqd/perf-node-gather-daemonset-z95dw"
Sep 14 05:28:43.111328 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:43.111287    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rvbqd/perf-node-gather-daemonset-z95dw"]
Sep 14 05:28:43.111889 ip-10-0-133-99 kubenswrapper[2565]: W0914 05:28:43.111858    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6dac4c5a_f772_496d_b3f2_4de668238b99.slice/crio-0db3969d1eb981f4810d6bee2dbc9edee445cc493a68063368a64a1b99e9226d WatchSource:0}: Error finding container 0db3969d1eb981f4810d6bee2dbc9edee445cc493a68063368a64a1b99e9226d: Status 404 returned error can't find the container with id 0db3969d1eb981f4810d6bee2dbc9edee445cc493a68063368a64a1b99e9226d
Sep 14 05:28:43.437441 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:43.437353    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rvbqd/perf-node-gather-daemonset-z95dw" event={"ID":"6dac4c5a-f772-496d-b3f2-4de668238b99","Type":"ContainerStarted","Data":"fb70bf88d373a92e58933b316eec07160054a7ed004092677c179d149b6cd9b6"}
Sep 14 05:28:43.437607 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:43.437527    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rvbqd/perf-node-gather-daemonset-z95dw"
Sep 14 05:28:43.437607 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:43.437569    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rvbqd/perf-node-gather-daemonset-z95dw" event={"ID":"6dac4c5a-f772-496d-b3f2-4de668238b99","Type":"ContainerStarted","Data":"0db3969d1eb981f4810d6bee2dbc9edee445cc493a68063368a64a1b99e9226d"}
Sep 14 05:28:43.454814 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:43.454766    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rvbqd/perf-node-gather-daemonset-z95dw" podStartSLOduration=1.4547323460000001 podStartE2EDuration="1.454732346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 05:28:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 05:28:43.452207687 +0000 UTC m=+2521.627979247" watchObservedRunningTime="2026-09-14 05:28:43.454732346 +0000 UTC m=+2521.630503905"
Sep 14 05:28:44.560660 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:44.560631    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2ms2f_bb2c56d0-20d7-49ba-857c-febc67911fcd/dns/0.log"
Sep 14 05:28:44.582420 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:44.582395    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2ms2f_bb2c56d0-20d7-49ba-857c-febc67911fcd/kube-rbac-proxy/0.log"
Sep 14 05:28:44.761418 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:44.761375    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rqw8q_0cfe37ed-e18d-4e25-84d6-cba59e2b2d81/dns-node-resolver/0.log"
Sep 14 05:28:45.319932 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:45.319894    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-654b486d49-g55wq_4d4725e3-157e-4b82-acc4-44492baf85a7/registry/0.log"
Sep 14 05:28:45.393312 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:45.393286    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pfl42_7f7e9f24-ddd1-4c3b-acdf-371ea4c648e8/node-ca/0.log"
Sep 14 05:28:46.739174 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:46.739145    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-96745c74f-xnltl_2a5dca15-e3df-4cab-898d-6716ee6f4615/kube-auth-proxy/0.log"
Sep 14 05:28:46.861878 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:46.861848    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c9d6db5f8-lqphc_93a9bb73-aa22-4294-b119-20aa06f4bfde/router/0.log"
Sep 14 05:28:47.839301 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:47.839271    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zxrd4_9f897b0d-d9b9-4ae8-b545-ce1f5dfa63c5/serve-healthcheck-canary/0.log"
Sep 14 05:28:48.363555 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:48.363528    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2bkxf_3bbc1798-b60c-4a19-bab3-c463a8bdd51a/kube-rbac-proxy/0.log"
Sep 14 05:28:48.387046 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:48.387018    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2bkxf_3bbc1798-b60c-4a19-bab3-c463a8bdd51a/exporter/0.log"
Sep 14 05:28:48.409490 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:48.409459    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2bkxf_3bbc1798-b60c-4a19-bab3-c463a8bdd51a/extractor/0.log"
Sep 14 05:28:49.460516 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:49.460490    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rvbqd/perf-node-gather-daemonset-z95dw"
Sep 14 05:28:51.129817 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:51.129787    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-78b6456dc5-v2m8w_03e983db-f240-4e1f-af0a-726228526dab/manager/0.log"
Sep 14 05:28:51.150255 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:51.150225    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-78b6456dc5-v2m8w_03e983db-f240-4e1f-af0a-726228526dab/copy-manifests/0.log"
Sep 14 05:28:52.696689 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:52.696653    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5854c46596-shm4c_989234dc-b2f0-4b28-95f6-df049e7fbe4b/manager/0.log"
Sep 14 05:28:53.238315 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:53.238284    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-96745c74f-xnltl_2a5dca15-e3df-4cab-898d-6716ee6f4615/kube-auth-proxy/0.log"
Sep 14 05:28:53.314278 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:53.314241    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c9d6db5f8-lqphc_93a9bb73-aa22-4294-b119-20aa06f4bfde/router/0.log"
Sep 14 05:28:55.259479 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:55.259448    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-96745c74f-xnltl_2a5dca15-e3df-4cab-898d-6716ee6f4615/kube-auth-proxy/0.log"
Sep 14 05:28:55.359213 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:55.359173    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c9d6db5f8-lqphc_93a9bb73-aa22-4294-b119-20aa06f4bfde/router/0.log"
Sep 14 05:28:57.851454 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:57.851421    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-7cxw6_3684d0a2-72f8-48fa-92aa-45f04832a96d/migrator/0.log"
Sep 14 05:28:57.877740 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:57.877707    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-7cxw6_3684d0a2-72f8-48fa-92aa-45f04832a96d/graceful-termination/0.log"
Sep 14 05:28:59.287423 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:59.287389    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-6g897_2e7a5d9f-3918-4bba-995c-1dc18f9611ac/kube-multus/0.log"
Sep 14 05:28:59.631913 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:59.631871    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z9dck_f6ac6666-f468-4757-8748-38b7e4fa90a9/kube-multus-additional-cni-plugins/0.log"
Sep 14 05:28:59.655818 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:59.655731    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z9dck_f6ac6666-f468-4757-8748-38b7e4fa90a9/egress-router-binary-copy/0.log"
Sep 14 05:28:59.678696 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:59.678665    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z9dck_f6ac6666-f468-4757-8748-38b7e4fa90a9/cni-plugins/0.log"
Sep 14 05:28:59.701830 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:59.701802    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z9dck_f6ac6666-f468-4757-8748-38b7e4fa90a9/bond-cni-plugin/0.log"
Sep 14 05:28:59.729561 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:59.729517    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z9dck_f6ac6666-f468-4757-8748-38b7e4fa90a9/routeoverride-cni/0.log"
Sep 14 05:28:59.757051 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:59.757023    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z9dck_f6ac6666-f468-4757-8748-38b7e4fa90a9/whereabouts-cni-bincopy/0.log"
Sep 14 05:28:59.781951 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:59.781925    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z9dck_f6ac6666-f468-4757-8748-38b7e4fa90a9/whereabouts-cni/0.log"
Sep 14 05:28:59.901334 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:59.901213    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2gwsl_ee8a9555-8b90-4ad8-a399-22d414487421/network-metrics-daemon/0.log"
Sep 14 05:28:59.919379 ip-10-0-133-99 kubenswrapper[2565]: I0914 05:28:59.919339    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2gwsl_ee8a9555-8b90-4ad8-a399-22d414487421/kube-rbac-proxy/0.log"